Most fault detection projects for building systems do not fail on a metrics dashboard. They fail in a control room six weeks after go-live, when an operator silences the alert feed because it cries wolf four times a day. After that, the model can be 99% accurate and it does not matter. Nobody is reading it.
This is the part that does not show up in the model card. The constraint on an HVAC fault detection system is not how many faults you can catch. It is how many false alarms an operator will tolerate before they tune you out for good. That number is low, and it is the number you should be optimising for.
A slightly less sensitive model that operators trust catches more real faults than a sensitive one they ignore.
Why false-positive rate is the metric that matters
Recall feels like the goal. You want to catch every failing chiller, every fouled coil, every stuck damper. So teams push the threshold to catch more, recall climbs, and the false-positive rate climbs with it. Each false alarm is a maintenance crew dispatched to a healthy unit, a work order that turns out to be nothing, a small withdrawal from the operator's trust. Trust does not come back at the rate it leaves.
Precision in alerting is what keeps the system alive in production. An alert that is right most of the time gets acted on. An alert that is wrong a third of the time gets a filter rule in Outlook. We tune for a low false-positive rate even when it costs us some recall, because a slightly less sensitive model that operators trust catches more real faults in practice than a sensitive one they ignore.
The 93 / 7 operating point
On our HVAC building management work we landed on roughly 93% true positive rate at about 7% false positive rate. We did not pick that point to win a benchmark. We picked it by walking the ROC curve with the operations team and asking, at each candidate threshold, how many false dispatches per week this produces and whether the crew would still trust the feed at that volume. Seven percent was the point where the alerts stayed credible and the missed faults were the marginal, slow-developing ones that the maintenance schedule would catch anyway.
Detection is only the floor. On top of the classifier we run remaining-useful-life estimation, so a flagged component comes with a runway rather than a binary alarm, and load-forecasting energy optimisation that adjusts setpoints inside comfort constraints instead of overriding them. The constraint matters: an optimizer that saves energy by letting a zone drift out of comfort range gets switched off as fast as a noisy alarm does.
How it is built
- Telemetry streams off the BMS over Kafka, which decouples ingestion from inference and lets us replay history when we retrain or re-tune a threshold.
- Models are scikit-learn, versioned and tracked in MLflow so every deployed threshold is tied to the exact run, dataset, and operating point it came from.
- Alert suppression sits between the model and the operator: debounce, hold-down windows, and grouping so one root fault does not fan out into twenty alerts. This logic is what protects the false-positive budget in the real world, separate from the model itself.
- Serving is FastAPI, which keeps inference a plain, observable HTTP boundary the rest of the platform can call.
The suppression layer is worth dwelling on. Your offline false-positive rate is measured per prediction. The operator experiences false alarms per shift. A correlated burst of true-but-redundant alerts reads as noise even when every one is technically correct, so the debounce and grouping logic does as much for trust as the threshold does.
The operating point is seasonal
A threshold tuned in the dry season lies to you in the wet one. HVAC behaviour shifts with weather, occupancy patterns, and equipment ageing, so a false-positive rate that held at 7% in commissioning month drifts upward quietly as conditions move away from the training window. We treat the operating point as a maintained artefact: input distributions are monitored for drift, the threshold is reviewed against the last quarter's confirmed outcomes each season, and retraining is triggered by measured drift rather than by calendar habit. The review is deliberately joint, the same walk along the curve with the operations team that set the original point, because the tolerance for false dispatches is theirs to spend, not ours. A model that was trusted at handover and silently degrades is how the alert feed ends up filtered after all, one season late.
What it returns
Run this way, the suite delivered 15 to 20% energy savings across the sites, and shifted maintenance from reactive to predictive: crews work off a ranked list of components with time-to-failure estimates instead of chasing breakdowns. Expect a shadow period before any of that: the first weeks run the model silently against operator logs, so the threshold conversation happens over real dispatch records rather than a slideware ROC curve, and the feed goes live already calibrated to the building. Because the value is measurable at the meter, the commercial model can be too. We are comfortable structuring these engagements around outcomes, saved energy and avoided downtime, rather than billing for a dashboard nobody opens. The model being accurate was never the hard part. The model still being believed in month six is the whole job.
