The Fleet Started Checking Itself

The Fleet Started Checking Itself

Day 13 put Humi Assistant on top of a cold-chain fleet.

That was the visible layer: Slack, email, MySQL, reports, cron, a dedicated machine, about 1,040 sensor units, and one agent profile that could answer questions without a dashboard in front of it.

The next capability was less visible. More important.

I started checking whether the intelligence layer was still trustworthy.

Monitoring the monitors

A sensor fleet does not fail in one clean way.

A freezer warms. A compressor cycles badly. A probe stops reporting. A location mapping drifts out of sync. A forecast model gets stale. A daily batch completes but leaves gaps. An alert path looks alive until the one message that matters does not arrive.

The first deployed version of Humi Assistant could read the data and explain it. The next version watches the machinery around the data.

A forecast health check now runs every morning. It checks whether the latest analysis and forecast batches completed, whether forecast coverage matches analysis coverage, how confidence is distributed across the fleet, and whether critical sensor signals are showing up in the forecast layer.

That is not a temperature alert. It is a data-quality alert.

The difference matters. If a walk-in cooler is warm, operations needs to know. If the forecast system silently stopped producing rows for part of the fleet, operations needs to know that first.

Calibration became scheduled work

Forecasting is not a one-time setup.

The analytics engine writes SARIMA forecasts into MySQL. Humi reads them, interprets them, and turns them into usable language for operators and customer-facing reports. That only works if the models stay calibrated against actual behavior.

A scheduled calibration job now runs three times a day. It tests candidate model orders, records every attempt, and only promotes a model when the candidate beats the current baseline. Failed and rejected attempts stay visible. They do not mutate the active model.

This is the correct shape for fleet forecasting: bounded, observable, and boring.

The agent does not wake up and decide to rebuild the whole model layer. It watches calibration progress, reads the tables, summarizes health, and points attention where the forecast surface is getting weak.

Alert health is its own signal

There is also a five-minute alert health monitor.

That sounds small. It is not.

An alerting system without a health signal is a superstition. You believe it works because yesterday it worked. The monitor changes that boundary. The alert path is no longer assumed. It is checked.

This is the operational pattern I keep finding useful: do not just automate the work. Automate the evidence that the work is still happening.

Inventory drift is different from temperature drift

The fleet has another failure mode: registration drift.

One system knows which sensors exist. Another system knows where those sensors are mapped. If those two views diverge, the analytics can still run and the reports can still render, but the conclusions start pointing at the wrong operational reality.

A daily registration audit now compares the source sensor inventory against the application mapping layer. It looks for missing or mismatched sensor IDs before those gaps become reporting errors.

That is not glamorous. Good. Glamour is expensive in operations.

Risk factors became explainable

The analytics engine also gained a thermal event signal.

A raw score is useful. A reason is better.

The engine now detects warm excursions, recent sustained events, and escalation patterns. Those signals feed into the risk factors exposed through the Humi agent view. Humi can now say why a sensor looks risky: not just that the probability rose, but that a recent warm excursion lasted long enough to matter, or that excursions are escalating in duration and peak temperature.

That changes the report language. The agent no longer has to infer a story from a score alone. It can read the factors the engine produced and turn them into operational language.

The boundary moved

Day 13 was about deployment.

This continuation is about supervision.

Humi Assistant does not replace the analytics engine. It sits above it with read-only access, scheduled jobs, narrow skills, and a small amount of memory. The engine computes. The database stores. The agent watches, explains, reports, and checks the checks.

That is the useful boundary.

The fleet is no longer just being monitored.

The monitoring system is being monitored too.