Trex Pi
An exception gets logged, an alert fires, someone fixes it. That loop works - for failures loud enough to announce themselves. The failures that cost our clients real money are the quiet ones: the vendor silently rotated an SFTP key, the nightly export produced a valid file with zero rows, the queue consumer is "running" but processing nothing.
Nothing errored. Everything stopped.
Flow metrics over health checks
A health check answers "is the process alive?" The question that matters is "did the work happen?" So the primary signal on every integration we run is a flow metric - records per interval, compared against what that interval normally carries:
- Orders synced per hour, with weekday/weekend baselines - a quiet Sunday is normal; a quiet Tuesday at 10:00 is an incident.
- File arrivals as deadlines, not events: "partner X's price file is due by 06:30" - the alert fires when it doesn't arrive.
- End-to-end lag, measured from source timestamp to destination commit, not from queue entry.
The zero-row problem
Special mention for the most common silent failure we see: a scheduled job that succeeds with empty output. Upstream changed a filter, a credential lost read access to one table, a date-window bug excluded everything - and the job exits 0, green across the board.
Every extract we ship carries a plausibility floor: the smallest output size that has ever been legitimate for that job and calendar context. Below the floor, the run is held and a human is asked, "this produced 0 rows; the last 90 Tuesdays averaged 4,200 - proceed?" That single guard has caught more real incidents than all our exception alerting combined.
Silence is a signal. Instrument for it deliberately, because it will never announce itself.