CINECA · Bologna, IT
Tier-1 supercomputer · 4× NVIDIA V100 per node · ~32 PFLOPS
Marconi100 — an NVIDIA V100 node drifting toward 2× its power.
A real degradation event on CINECA's Marconi100, replayed across 30 months of continuous telemetry. OrionLinks flagged the failing node months before it reached its worst operating point — while a traditional rack-average dashboard showed nothing wrong.
54 dayslead time · first signal → peak power
2.07×baseline power the node climbed to
4 / 847nodes flagged across the fleet
933 dayscontinuous telemetry analysed
The model walked Node 978 from STABLE → WATCH → DRIFTING → DEGRADING while its 16 rack-mates — sharing the same cooling, air, and workload pool — stayed flat. That divergence is the signal that separates a hardware fault from ordinary fleet-wide drift. Built on 698,461 daily power records.
GWDG · Göttingen, DE
A100 GPU fleet · DCGM telemetry
GWDG — every NVIDIA A100 GPU failure with telemetry, caught early.
The same model, the same thresholds, pointed at a different cluster — GWDG's A100 GPU fleet — with no re-tuning. Scored forward-only against a year of operator-logged incidents as ground truth.
16 / 16telemetered GPU failures caught
4 daysmedian lead over the operator's report
15 / 16flagged before the human incident report
69incidents in the ground-truth catalog
Three forward-only layers — self-normalised behaviour, slow-drift slope, and DCGM hardware faults — score each node on telemetry alone. The detector ported straight from Marconi100 generalised to a new vendor and workload with zero domain tuning, confirming the approach isn't cluster-specific.