The Model That Goes Quiet: MLOps on the Plant Floor

A model rarely dies with an alarm. It goes quiet.

Picture the anomaly detector somebody bolted onto a boiler-feed pump last commissioning season. For a few months it earned its place. It flagged a rising bearing defect a good two weeks before the vibration route would have caught it, and the reliability crew started trusting the little amber dial on the mimic. Then the flags stopped. Not because the pumps got healthier. A fitter re-seated the accelerometer a centimetre off its old pad during a mechanical seal change, the resonance it watches shifted a few hundred hertz, and the features the model leaned on slid out from under it. The score still refreshes every minute. It reads green. It's also blind.

Nothing in the control system announces this. No PLC faults, no red on the SCADA, no line stop. The pump keeps running, the model keeps scoring, and the one thing that changed, the quiet erosion of the model's grip on reality, lives in a layer nobody is paid to watch. A tripped breaker gets a work order in minutes. A model that has drifted into uselessness can sit there for a year, dressed as a healthy green indicator, until the bearing it was meant to catch lets go on a night shift.

That gap is the subject here. In industrial ML, the day you deploy a model is day one, not the finish line. A pump, a controller tuning, a feedstock, a sensor: every input the model depends on is bolted to a living process that refuses to hold still, and the model itself is a snapshot of a moment that's already passing. Keeping it useful is a maintenance job, closer to lubrication routes and calibration schedules than to the data-science sprint that produced it. The field borrowed a word from software for the tools and habits that do this work: MLOps. On the plant floor it looks different from the version the web companies wrote about, and the differences are exactly where models go to die.

Web MLOps grew up around a very different problem. Fast, dense labels; cheap experiments; a rollback that's a config flag and a shrug. A plant offers none of that. Labels are rare and expensive, an experiment runs on equipment that can hurt someone, and a bad model doesn't serve a worse ad, it sends a technician to the wrong pump at two in the morning or masks a fault until it becomes a fire. So the borrowed toolkit stays and the priorities invert. Less about shipping fast, more about never going quietly wrong. Less about the model, more about the machinery around it that keeps the model honest.

Two different things get called drift, and they fail you in different ways. The first is data drift: the inputs themselves move. Ambient air comes off a cold Baltic winter and lands in July, cooling-water temperature climbs, and a model that learned a pump's vibration signature at ten degrees now meets it at thirty. The physics of the pump hasn't changed. The distribution the model was fed has. The second is concept drift, the nastier one, where the relationship between inputs and the thing you're predicting changes underneath a stable-looking signal. A heat exchanger fouls, so the same inlet temperature now means something different downstream. Retune a control loop and the healthy operating point moves. Swap a centrifugal pump for a different maker's casing and the whole vibration-to-health mapping the model built is quietly void.

Once you start looking, the triggers are everywhere on a running site. A transmitter gets recalibrated and its bias steps a fraction of a percent. A 4-20 mA loop gets re-ranged after an instrument swap. An edge gateway takes a firmware update that changes an anti-alias filter or nudges the sampling rate, and every frequency feature downstream lands in a slightly different bin. In waste-to-energy the refuse itself is the moving part: the calorific value of what comes through the tipping hall swings with the season and the collection round, so a combustion model trained on autumn waste meets a wetter, denser feed by spring. Food and beverage changes SKU on a schedule, and every recipe and clean-in-place cycle rewrites what normal looks like on the same line. Metals melt whatever scrap the market delivers that week. A duty machine quietly becomes a standby (a compressor swap nobody logged, say), and its whole load profile inverts. None of these are faults. All of them move the ground the model stands on.

What makes this genuinely hard, harder than the same problem in a recommender or an ad model, is that you often can't tell whether the model is still right. A web team retrains against click-through it measures by the hour. On a boiler-feed pump, ground truth is a bearing failure, and those arrive months or years apart, if you're unlucky enough to see one at all. So the label that would tell you the model has gone wrong is the exact event the model exists to prevent. You can't wait for it. By the time a failure confirms the model was blind, you've already had the outage the whole system was bought to avoid.

It's part of why the datasets the field grew up on can mislead. Aircraft-engine run-to-failure sets and lab bearing rigs hand you dense, clean, labelled failures; a real pump hands you one ambiguous trip in three years. Benchmarks made of easy labels quietly teach the wrong lesson about how much you'll know in production.

Watching the model, not just the machine

Because you can't lean on failure labels, you watch the model's vital signs instead, and most of them live upstream of any prediction. Start with the inputs. Is every tag that feeds the model actually arriving, on time, and moving? A frozen transmitter that holds its last good value looks perfectly healthy to a naive model, and a flatline is one of the commonest ways a feature quietly dies. NAMUR NE 107 status, the diagnostic flag a decent field device already emits over OPC-UA, tells you the sensor thinks it's degraded before the number does anything visibly wrong. Carry that flag through the pipeline. A model that swallows a "maintenance required" signal as though it were a clean reading is being lied to by its own instruments.

Above the plumbing sits distribution monitoring. You keep a statistical fingerprint of what each input looked like during training, then watch the live data against it. When the running mean of a feature walks off, or its spread collapses, or a value the model never saw in training starts showing up, that's your early warning, and it fires long before any prediction has visibly failed. The cheap version is a set of range and rate-of-change limits per tag. A better version tracks the joint behaviour, because inputs on a process move together, and a pair of readings that each look fine alone can sit in a combination the model has never encountered. Either way the aim is the same: catch the world moving before you catch the model being wrong.

Detecting input drift and proving a worse prediction are two different jobs, and it pays to keep them apart in your head. Watching a feature's distribution wander is cheap and early, but a shifted input doesn't always mean a degraded output; sometimes the model generalises across it fine. A confirmed drop in prediction quality is what you actually care about, and it's usually the thing you can't measure directly, because the labels are missing. So you triangulate. Input drift as the smoke alarm, physics residuals and operator feedback as the closest thing to ground truth, and a standing rule that a model tripped by two independent monitors gets a human review before anyone trusts its next call. One alarm is noise. Two that don't share a cause are a reason to open the hood.

Then watch what the model emits. A sudden change in how often it raises a flag, a confidence score that piles up at the extremes, an output distribution that no longer resembles last quarter's: each is a symptom worth an engineer's attention even when you can't yet prove the prediction wrong. The strongest check we lean on pairs the learned model with a dumb physical one. A first-principles estimate of pump head from flow and speed, an energy balance across an exchanger, a mass balance that has to close: when the model and the physics disagree and the gap keeps growing, something has moved, and the residual between them is a label you didn't have to wait years for. Physics doesn't drift. That's what makes it such a useful adversary for a model that does.

None of this survives contact with a control room if the operators stop believing it. A model that cries wolf twice a shift gets muted, then ignored, then quietly disabled by someone who's had enough, and no amount of AUC on your test set buys that trust back. So the false-alarm rate isn't a vanity metric. It decides whether the model is still in service in the human sense, whatever the config file says. Watch it the way you watch the process. When it climbs, the model has already begun dying in the only place that counts, which is an operator's willingness to walk out and check.

Where you can manufacture labels, do it. Every time the model raises a flag and a technician goes to look, the outcome of that trip (false alarm or genuine defect) is a label, if somebody bothers to write it back. A thumbs-up, thumbs-down on the alarm screen, fed into the record, turns the operators into a slow labelling pipeline and gives you a live read on the false-alarm rate, the number that decides whether they keep trusting the system at all. Making the model's health visible in the same place as the process it serves, its inputs, its residuals, its alarm rate on one screen an engineer already watches, is most of what an edge telemetry and analytics platform is for. The model isn't a black box hanging off the side of the plant. It's an instrument, and instruments get calibrated and trended like everything else in the cabinet.

The trap in retraining

The obvious fix for a drifted model is to retrain it, and the obvious way to retrain is the one that will burn you. Point a pipeline at the last few months of plant data, refit, redeploy, repeat on a schedule. It feels disciplined. But think about what that data holds when an asset is slowly degrading: the bearing that's three months into a nine-month failure is teaching the model that rising vibration is the new normal. Retrain naively on a window that includes the fault you're trying to catch, and you train the alarm right out of the model. It's the boiled-frog problem rendered in gradient descent. Each retrain nudges the definition of healthy a little nearer to broken, and every version passes its own validation because the yardstick moved along with it.

So retraining needs a gate on what's allowed to teach the model. Curate the training window the way you'd curate calibration data: drop known upsets, drop periods when an input was flagged bad, drop the run-up to a failure once hindsight has told you it was one. Keep a held-out set that genuinely reflects current healthy operation, refreshed as the plant ages, because a validation set frozen at commissioning will happily bless a model that has drifted right along with the process. And resist retraining on a fixed calendar just because a sprint board says so. Retrain when the monitoring says the world moved, not when a cron job fires.

One failure mode is pure MLOps and catches good teams anyway: training-serving skew. You train on historian data, resampled and averaged the way the historian stored it, then you serve on raw signal at the edge, aggregated by slightly different code. The model meets a subtly different feature at inference than the one it learned, and it underperforms without anything having drifted at all. The fix is boring and non-negotiable. The exact transformation that built a training feature has to be the one that builds it live, ideally the same code path, not a reimplementation somebody wrote for the gateway in a hurry. A feature that means one thing offline and another thing online is a bug the model can't see and the monitoring rarely catches.

Then there's the cold-start question every multi-asset site hits. You've got twelve near-identical pumps and a model that took months of data to earn its keep on the first one. Do you wait months again for each of the rest? A fleet model trained across similar assets gets you a running start, and you specialise it per machine as data accumulates, but similar isn't identical: two pumps of the same part number sit on different foundations, see different suction conditions, and wear at different rates. A fleet baseline blind to that will flag the quirky-but-healthy unit as sick and miss the genuinely failing one whose baseline always ran a little high. Transfer buys you speed. It doesn't buy you the right to skip per-asset validation.

Before a new model touches a decision, run it where it can't do harm. Shadow deployment means the challenger scores live data in parallel with the champion that's actually in service, and you compare them for weeks before promoting anything. Does the new one catch the events the old one caught? Does it throw up a wall of fresh false alarms? Shadow mode answers both without betting the plant on the answer. The promotion, when it comes, should be a boring, logged, reversible act, and the minute a freshly promoted model starts behaving oddly you want to fall back to the previous one in minutes, not spend a night shift arguing about whether to trust it.

That reversibility has to be built, and it's where plant MLOps borrows hardest from ordinary operations discipline. Every model in service needs a version, a record of exactly which data window and features and code produced it, and a note of which line and which asset it's running on. When an estimate looks wrong at three in the morning, the shift engineer's first question is which model is this and what changed, and if the answer lives in a departed contractor's notebook, you have no answer. A registry that pins model version to data lineage to deployment target, with a last-known-good you can roll back to, is the difference between a ten-minute recovery and a forensic dig. None of this is exotic. It's configuration management applied to a model instead of a PLC program.

All of it assumes you kept the data, which plenty of sites haven't. You can't retrain on a period you threw away, and you can't diagnose a drifted model without the raw signal from when it still worked, so a retention policy that holds enough high-resolution history to rebuild and re-validate is part of the model's life support, not a storage line to trim. The same goes for the model's own provenance. A model you can't rebuild because the library versions moved and nobody pinned them is one you can only replace, never repair, and the day it misbehaves you'll wish you'd treated reproducibility as a maintenance requirement rather than a nicety.

The edge makes all of this physically harder. A model that watches a pump might run on a DIN-rail gateway the size of a paperback, bolted in a cabinet, one of forty scattered across a site on intermittent connectivity. Push an update and you've got version skew: line three runs last month's model, line seven got the new one, and a fault that shows on one but not the other might be the asset or might be the software. So the same over-the-air discipline you'd demand for firmware has to apply to models. Staged rollout, a way to know exactly what's running where, and a rollback that doesn't need a technician standing at the cabinet with a laptop. A model you can't update safely at the edge is a model you'll be scared to fix, and a fix you're scared to ship is a fix that never happens.

The box you run on also shapes the model you can keep alive. A gateway with a few hundred megabytes of RAM and no accelerator won't run the heavyweight architecture that won the offline bake-off, and a model the edge can't serve is one that never leaves the notebook. Picking something the hardware can run, and the on-site engineers can reason about, usually beats the last two points of accuracy from a model only its author understands. A slightly worse model that runs, gets watched, and gets maintained will outlast a brilliant one that's gone stale inside a year, every time.

And a retrain is a change to the plant, so treat it like one. The instinct on the software side is to ship continuously, lots of small updates, move fast. On the OT side a model that shapes maintenance or control decisions belongs under the same management-of-change reflex as a loop-tuning change: someone reviews what the new model does differently, someone signs that it was validated against current conditions, and the old version stays parked and recoverable. That sounds heavy until the first time an unreviewed "improvement" quietly lifts the missed-detection rate and nobody notices for a month. The cadence that works sits between the two cultures, quicker than a plant's annual habits, slower and more accountable than continuous deployment.

It helps to give the model a home inside structures the plant already trusts. ISA-95 tells you which level the thing lives at and what it's allowed to touch, so a health estimate doesn't quietly end up writing setpoints nobody signed off on. And the condition-monitoring world mapped this pipeline out decades before anyone said MLOps. ISO 13374, first published in 2003, breaks machine condition monitoring into ordered blocks: data acquisition, data manipulation, state detection, health assessment, prognostics, then advisory generation. Read it now and it's a description of exactly the pipeline a learning model sits inside, from cleaning the signal to detecting the state, judging the health, projecting it forward, and telling someone. The learning part is one block. The other five are the plumbing that decides whether the clever block ever gets believed.

Security rides along with all of this, and it's easy to forget that a model is an attack surface. A thing that ingests sensor data and shapes maintenance or control decisions is worth tampering with, and the integrity question, who is allowed to push a model to that edge box and how you'd know if the wrong one landed, sits squarely in the IEC 62443 conversation about zones, conduits, and change control. A poisoned or simply careless model update can do damage a bad setpoint never could, because it arrives looking authorized. Treat model artefacts like any other controlled change into the OT network, signed, logged, and access-controlled, and you close a door most plants currently leave wide open.

All of it comes back to a question that has nothing to do with algorithms: who owns the model after the launch party? The pump has a custodian. The control loop has an owner and a change procedure. Too often the model has neither, because it arrived as a project with an end date instead of an asset with a lifecycle, and the person who understood it was a contractor whose statement of work closed at go-live. The deployments that last hand the model to the people who'll be on site at three in the morning: the controls and reliability engineers who live with it, not the data scientists who built it. Much of the industrial AI deployment work we do is exactly this unglamorous handover, wiring a model's health into the tools an operator already watches and writing down how to retrain it, validate it, and roll it back, so the thing outlives whoever built it.

So try this test before the next model goes into service, or the next time somebody tells you the one you've got is working fine. Walk up to the screen. It says green. Ask the only question that matters: how would you know if it were wrong? If the answer is a monitored input distribution, a physics residual that hasn't moved, a false-alarm rate the operators still believe, and a version you could roll back tonight, then the green means something. If the answer is that it hasn't alarmed lately, you aren't looking at a healthy model. You're looking at a quiet one, and by now you know those aren't the same thing.

Notes

Where this applies: continuous and batch process plants running their own condition-monitoring or soft-sensing models, at the edge or near it. The mechanisms here, drift, monitoring, gated retraining, versioned rollback, hold whether you built the model or bought it; the thresholds don't transfer. What counts as a healthy input distribution or an acceptable false-alarm rate is specific to your assets, your feedstock, and your operators' tolerance, and it takes a season or two of running to learn. Two limits deserve stating plainly. A physics residual only helps where a first-principles model exists, and some assets don't give you one. And no amount of monitoring rescues a model whose failure mode is an event you'll see once a decade, where the honest answer is a wider safety margin, not a cleverer algorithm. The one outside reference below is ISO 13374 for the condition-monitoring block model; everything else is general engineering practice.