An exit is not a verdict

The page came in a little after dawn, and it was wrong. A periodic sync job, one of a class that runs every twenty minutes, does a small amount of work, and exits, had tripped its failure hook and sent up an alarm. The notification said the process had been killed, signal fifteen, a plain termination. Nothing was actually broken. Overnight the machine had run its unattended upgrade, and somewhere in that upgrade cycle a stop signal went out to whatever units were running as it landed; the sync had started thirty seconds earlier and caught it mid-run. Its next tick, twenty minutes later, ran clean, as did every one after. The job had never been unhealthy for a second. And the notification was not empty: it carried the whole story, the signal number, the fact that the process was killed rather than that it returned an error. The monitor had all of it, and still called a deliberate shutdown a failure, because for this kind of unit the rule it applied read a termination signal as a fault.

This is the failure mode hiding inside the cheapest thing a monitor can do, which is to notice that something is gone. Absence is easy to detect. It is also almost always the wrong thing to page on, because an absence is not a cause, it is the shadow of one, and the causes fall into two piles that look alike from where the monitor stands. In one pile the work broke: a bug, a bad input, a dependency that vanished. In the other the work was ended from outside, usually by something allowed to end it: a deploy rolling the fleet, a reboot, an operator running a stop by hand, an idle sweep reclaiming what it took for a stale process, the upgrade that caught my sync. A monitor that only asks whether the process is still there pages for both piles at once, because that question has no way to tell them apart. A monitor wired instead to the unit's own failure verdict is only as good as that verdict, and the dawn page is what a miscalibrated verdict looks like. Either way, what a page should turn on is not the fact of the absence but its provenance: what ended the process, and whether it was supposed to.

The reflex, when a pager gets noisy, is to reach for time. Do not page on a single sample; wait to see whether it sticks. This instinct is sound and it is built into the tools: a Prometheus alerting rule with a for clause holds a firing condition in a pending state and only promotes it to a real alert once the condition has stayed true continuously for the whole duration, where a rule with no for goes off on the first evaluation that trips it. And time is the right axis for one shape of non-failure. A periodic poll of an external API times out once, because a network is a network, and self-heals on its very next attempt; paging a human for that single blip is how you teach the human to swipe the pager away without reading it, so you wait for a streak instead, an hour of the thing being unreachable, before you decide it is real. That is time doing exactly the job it is good at, filtering a transient down to a trend.

But time is a partial fix at best for the dawn page, and it is worth being precise about why, because it looks like it should be the whole answer. A for clause sorts absences by how long they last, and duration is only a proxy for legitimacy, never a reading of it. Some of the proxy holds: a deploy that rolls a long-running service is gone for seconds, a reboot for a minute or two, and a for of a few minutes waits both of those out, which is genuinely useful and covers a good share of routine stops. But it buys that only by delaying every real page by the same window, and it goes blind the moment a legitimate stop is not brief. A unit taken down for maintenance and left down is a clean, deliberate absence that lasts for hours. A periodic job stopped mid-run stays gone until its next tick, which for my twenty-minute sync is a long, clean silence a ten-minute for sails straight through and pages on anyway. Waiting longer separates the transient from the sustained. It cannot separate the failed from the intended, because a legitimate stop can be every bit as sustained as a crash, and often more so.

The distinction the pager actually needs is not longer, it is deeper: it has to know why the process is gone. The surprising part is that the operating system already carries part of that answer, and then, for one kind of unit, throws it away. systemd, supervising an ordinary long-running service, treats a termination by SIGTERM as a success by default. Its exit-status rules count the normal exit code of zero as clean, and, for any service that is not a one-shot, they count the signals SIGHUP, SIGINT, SIGTERM and SIGPIPE as clean too. That default is the supervisor recording provenance the process cannot forge: not what the program said about itself, but which signal arrived, and that a signal rather than a bad return value is what ended it. A stop signal sent to a service you asked to stop is not a fault, and the contract says as much. But that default is written with an exception, for the one-shot type, and a job that starts, does a bounded piece of work, and exits is exactly a one-shot. For those units the stop signal is dropped back out of the clean set, so the very SIGTERM that means graceful shutdown for a daemon means failure for the periodic job, and the failure hook fires. The repair is to restore the distinction by hand, declaring through SuccessExitStatus=SIGTERM that a termination signal on this unit counts as a clean end, at which point an external stop stops tripping the alarm while a genuine fault, a non-zero code the job returned, an out-of-memory kill by the harder SIGKILL, still pages. This does not silence the alarm. It hands the alarm a rule that stands in for the one distinction it could not draw on its own, failed against ended, and how good a stand-in that rule is turns out to be the question the rest of this rests on.

It would be easy to file this next to a problem it resembles, so it is worth drawing the line. There is a nearby argument, the case that a presence check is the wrong way to watch a long-running agent because it cannot tell an agent thinking hard from an agent wedged forever, and the repair there is to stop asking whether the process exists and read progress off the durable trail the work leaves behind. That is a real fix, and it works while the work is still running and still leaving a trail to read. A page is the opposite situation. The thing is gone. There is no trail still being written, because the process that would write it is not there, and no progress to consult, because nothing is progressing. All that remains is the manner of the leaving: the exit code, the signal, whatever the supervisor recorded about who sent it. For an absence, provenance is not one signal among several you might choose to read. It is the only evidence there is, and the part of it worth trusting is precisely the part the departed process did not author. So the two decisions come apart and stay apart: the health check asks whether the live thing is moving, and reads a trail to find out; the pager asks whether the ending was a failure or a choice, and has nothing to read but the shape of the ending itself.

Pull back from systemd and the principle is plain, and a little uncomfortable. An exit code is a one-byte summary of a story the monitor never watched, and worse, a summary the subject wrote about itself. This is the old, hard-won caution from operating distributed things: you do not ask a component whether it is healthy and simply believe the answer, because the component that is broken and does not know it is the one whose word is worth least. Provenance is trustworthy exactly to the degree something other than the process recorded it. SuccessExitStatus can do its work only because the supervisor kept the out-of-band half of the story, which signal arrived and that the kernel, not the program, delivered it. The half the program chooses, the return value it hands back on its way out, inherits every reason not to trust a thing's account of itself. So the move that pays, where you can manage it, is to page on a fact recorded by something other than the failing process, and to hold the bare absence, the one with no such fact attached to it, as a question rather than a verdict. A unit whose supervisor logged an out-of-memory kill has failed in a way you can stand behind. A unit that is merely gone, with nothing recorded about why, has raised a question, and a question is not yet a page. This is the alerting world's oldest discipline turned onto absence: wake a person for what they must act on, not for every event that crosses a threshold.

But do not let that harden into permission to stay silent when a job simply never runs, which is the most dangerous absence of all, because it makes no signal to classify. The complement to never paging on bare absence is to page on the absence of a success. A last-good-completion timestamp that has gone stale is a positive, attributable fact, the work has not succeeded since a time you can name, and it makes the job that quietly stopped being scheduled, or that failed and exited zero, loud again without waiting for the thing itself to file a complaint it may never file.

That reclassification is where the honesty has to live, because encoding provenance by the shape of a signal is not the same as observing it. Widen the clean set to include SIGTERM and you are betting that nothing broken ever leaves through that particular door. Mostly nothing does: a real fault tends to return a non-zero code, or to run the machine out of memory and take the harder kill your rule does not cover. But the bet has edges. A fault that happens to exit by way of SIGTERM, a handler that swallows the signal and returns cleanly, a stop issued for a reason that was actually wrong, an idle sweep reclaiming a process that was only quiet, now inherits the innocence the rule granted the signal. A hang leaves no signal at all, so this rule never sees it; a one-shot with no start timeout set can sit wedged forever without ever failing, which is the health check's problem next door, not the pager's. And the out-of-memory kill you kept in the paging pile is itself a judgment call, because a process killed for a neighbor's appetite did not so much break as lose an argument about capacity. None of this is a reason to skip the trade. You have swapped a large and certain source of false pages, every deploy and every reboot for the life of the unit, for a small and rare source of false silence, and for a job that self-heals on its next tick that is almost always the right swap. What it is not is a reading. When you can do better than a bet, the way to do it is to stop inferring provenance and start recording it at the source: have the deploy, the sweep, the maintenance window write down that it stopped this unit, at this time, for this reason, and the absence stops being a question at all.

The pager is the one instrument in the whole stack whose entire purpose is to spend a human's attention, and it is the one most often built to fire on the cheapest signal it can get, which is bare absence. An alarm that goes off every time something is stopped on purpose is not a safety device. It is a machine for teaching its operator, patiently and effectively, that the alarm means nothing, and it does its best teaching at dawn on the morning after a routine upgrade. What it should answer instead is the question the person actually has the moment the pager wakes them, which is not is it gone, a thing they can see for themselves, but did it fail or did something end it, and to admit, in the one place the two genuinely cannot be told apart, that the answer is a bet and not a reading. An exit is not a verdict. It is a piece of evidence, and the whole discipline is in refusing to page until you know what it is evidence of.