The problem, precisely
An MSSP watches log feeds for many clients at once. The failure mode nobody budgets for is not a noisy alert — it is silence. When a log source dies, the dashboard simply stops mentioning it, and everything looks calm. In practice that meant dead feeds were discovered during an investigation, days later, when someone went looking for evidence that was never collected.
Why it is harder than it sounds
“Is this input alive?” has no single answer. An input can be running but receiving nothing. It can be receiving a trickle when it should be receiving thousands. It can be perfectly healthy but belong to a client whose office is closed for the weekend. A naive zero-message check produces so many false positives that people mute it — which is worse than no monitoring at all.
How I built it
A Python service runs hourly against each client's SIEM over its REST API. For every input it pulls the cluster-wide state, counts messages in the last 60 minutes and the 60 minutes before that, and fetches the timestamp of the most recent message across a 7-day lookback. Those three signals together separate dead from merely quiet far better than any one of them alone.
Classification
CRITICAL covers an unreachable instance, a stopped input, a zero-message window, or an expected input that has vanished from the configuration entirely. WARNING covers a volume drop of more than 80% against the previous window — but only when the baseline was meaningful, which kills the false positives that come from low-traffic sources. Every threshold is tunable per client, with ignore lists and expected-input assertions.
Access model
This is the part most monitoring tools get wrong. Rather than a shared admin credential, each client gets a dedicated service account with a custom role scoped to read-only input monitoring — nothing else. Authentication is by API token, and instances that are not directly reachable are reached through Cloudflare Access service tokens rather than a VPN. The monitoring system cannot read log contents; it can only ask whether logs arrived.
What happened in the pilot
On the first client, within the first day, it caught a Domain Controller feed that fell from 26,548 messages per hour to zero in the middle of a working day. It also surfaced two Check Point inputs that had been silently dead long enough that nobody remembered them being alive. Detection time for failed log feeds went from days to under an hour.
Where it is now
Rolling out across the remaining clients. Reports are severity-sorted HTML and Excel, emailed to the IT team, with monitoring-friendly exit codes and an only-on-issues mode so it stays quiet when everything is healthy. The package ships with a mock SIEM server for demos and regression testing.