A model is a dependency that won't hold still

One of my scheduled jobs started coming back worse, and the first thing I did was open my own config.

That is the right instinct almost every time. When something you built degrades, the bug is usually on your side of the line: a prompt you edited, a default you forgot you changed. So I went looking, and I found one thing. I had restored an old formatting rule that week. It did not obviously explain what I was seeing, and the model underneath had also rolled to a new version around the same time. Two changes, one symptom, and the honest position was that I could not yet say which one was doing it.

What I wanted to say was that the model had gotten worse. That is the feeling every developer gets when a vendor ships a new version and the output shifts. But reaching for the model as the culprit, with a change of my own sitting right there in the same week, is the move that has cost me a week before. This is about the instrument I built so I would not have to guess.

Because a model does something no other dependency does. A foundation is the part you build on and then stop thinking about, since it does not move. A model moves. It moves under its own name, on someone else's schedule, and it takes away the one property that makes depending on code you do not own survivable: the ability to tell, when the output changes, whether the dependency changed or you did.

A pinned library holds still. A model doesn't

Everything about relying on code you did not write rests on one inference: if the output changed and I did not change my side, the dependency changed, and I can find out exactly how, because a pinned library is an artifact I can inspect. I can read its bytes, walk its dependency graph, and run it again in a controlled environment. Lock the version with a lockfile and a hash and the thing you depend on stops being a moving target. It is not that the library is magically deterministic, since it can still lean on a clock or a random seed or a network call. It is that what it is can be pinned down and checked, so a surprising output has a short list of places to hide.

A hosted model gives you almost none of that. Start with the name. Point at a mutable alias and the vendor can remap it to a new snapshot without your version string changing at all, and even around a fixed snapshot the routing, the serving stack, and the safety filters can move between one run and the next. Then there is the output itself. It is a distribution, not a value, so setting the temperature to zero does not rescue you either: on hosted inference, batch composition and kernel nondeterminism leak through, and "it got worse" and "I drew an unlucky sample" are the same observation until you look harder. Put those two together and a degraded job hands you a change with no cause you can point at. The version string, the one control you would normally grab, may not even be holding still.

So the pin is not for stability. It's for attribution

So I pin the model version everywhere my scheduled work runs, and I am precise about what that buys. It does not guarantee the model stays put, because a pinned snapshot still sits on serving infrastructure the vendor moves. What it removes is a suspect. When something drifts and I never touched the version, the "I took an upgrade" explanation is off the table. And a pin does one thing that reading the served version back out of each response cannot: it lets me choose the moment the version changes, so the before and after fall on either side of a line I drew, with everything else held. That clean baseline is most of the game.

The pin alone is still not enough, because the model's own noise keeps small regressions hidden inside the run-to-run scatter, and because a single number cannot pull two concurrent changes apart. Both my restored rule and a model shift would move "the output changed." So the other half is a small panel of cheap metrics I fix before I touch anything: not one number, but two or three whose sensitivities differ, so that which of them moves tells me what moved. That job from the opening, the batch of short status summaries that came back worse, is the one I want. I was watching two numbers I could compute on each summary: how long it ran, and how many bullet points it broke the status into. The formatting rule I had restored requires a bullet per item, so it drives the bullet count; how long the run is stays the model's to move. When I rolled the version back to the pinned one, the length dropped to its old band while the bullet count kept climbing. Length had tracked the model, the bullets had tracked my rule. Two suspects, two numbers, and they came apart because I had chosen numbers that would.

None of this is proof, and I hold it loosely. A handful of runs against a distribution does not buy certainty. But it pointed the arrow: the length bloat was the model's, the extra structure was mine, and I knew where to look first instead of rewriting a prompt against the wrong cause. That is the entire value. Without a number fixed in advance, "the model got worse" can be neither shown nor ruled out, and you will do the human thing and reopen your own prompt to fight a confound you cannot see. The first time I hit this I did exactly that, and my tidy story that the new model was simply worse did not survive a skeptical reader who showed me I had blamed the model for a change of my own. The panel is what I built so the next confound would not need a reviewer to untangle it.

Capacity is not stable either, and it fails the same way

You might grant all of that about behavior and still assume the amount of model you get is fixed, because that part is written into your plan. It is not fixed. Community reports I could not check against my own meter describe a promoted capacity change landing as a cut: the announcement said more and the throughput people saw said less. Set aside whether the vendor moved the number. The attribution question underneath is the same one as before, one layer down. When your throughput drops, did their supply shrink, or did your own workload grow? Those are two live causes for one symptom, and nothing but a record of your own load, kept over time, tells them apart. Nobody in those threads could say for certain, and that is the tell: without your own meter, the vendor's description is the only account you have, and the description is marketing copy, not a guarantee.

The upgrade decision is the same problem wearing a friendlier face. A new generation ships, and the migration note tells you a cheaper setting now gives results similar to or better than the last version. Maybe it does. But "similar to or better" is a claim about an aggregate benchmark, not about the one narrow job you run while you are asleep, and the only way to learn which it is for your job is to have kept the panel and run it across the switch. Diagnosing a regression and deciding whether to accept an upgrade turn out to be the same measurement, asked in two directions.

What the dependency framing actually buys you

Once you stop calling the model a foundation and start treating it as a dependency you cannot fully control, the rest of the design falls out the way it does for any such dependency. Isolate it behind an interface and swapping the engine becomes a configuration change instead of a rebuild. Run a portfolio rather than a single model, matched to the job, and you meet the case that looks upside down until you have lived it, where the disposition that makes a model a poor unattended worker is the same disposition that makes it a sharp reviewer. For the mechanical bulk that never needed the expensive model, a cheaper one stays wired in. And because a supply that can shrink without warning is not something to build a hard dependency on, you design the degraded mode before the cut lands rather than scrambling after it.

All of that is real, and all of it comes after. The portfolio, the routing, the fallback: none of them help if you cannot say which member of the portfolio changed, or whether it changed at all. The measurement comes first, because it is the foundation the model itself refuses to be.

A foundation you get to forget about. That is the whole point of the word. This dependency you do not get to forget, because it will move under its own name on a schedule you do not set, and the grown-up response is to treat it as exactly that: the most important and least reliable thing in the stack. Pin it so you own the moment it changes. Fix your metrics before it does, so the change stays legible when it comes. And keep the record yourself, because the version number is the vendor's promise about identity, and it is a promise they do not entirely keep.