Use Models to Reason. Use Software to Operate.

Where to put the model in an AI platform, and where to keep it out.

The reflex, when you are building AI infrastructure, is to reach for a model. If a decision is hard, ask the model. If a system is complex, wrap it in an agent. For a large class of problems that instinct is correct. It is also how you end up with a platform that is slow, expensive, nondeterministic, and impossible to reason about at exactly the moment you need to.

Use models to reason about work. Use software to operate infrastructure.

That rule has held up every time we have built against it. Here is why, and where the line actually falls.

Two kinds of decisions

Most decisions in an AI platform fall into one of two buckets, and the buckets want opposite tools.

The first is judgment under ambiguity. Is this code correct. Is this task complex enough to need the expensive model. Did this failed attempt fail for a reason worth escalating. These questions have no closed-form answer. They reward reading context, weighing tradeoffs, and producing a considered call. That is what models are good at, and nothing else does it as well.

The second is deterministic control. Is the pod healthy. Are we over budget for the month. Is the request queue full. Should this deployment have three replicas or four right now. These are not matters of judgment. They are functions of observable state, with correct answers you can compute, test, and replay. A model is the wrong tool here, not because it would fail often, but because it would fail unpredictably, and infrastructure that fails unpredictably is infrastructure you cannot operate.

The whole discipline is knowing which bucket you are in and refusing to cross the streams.

Why software wins the operating layer

Put a model in the control path and you inherit four problems at once.

It is nondeterministic. The same inputs can produce different actions, which means you cannot reason about what the system will do, only about what it usually does. For a reconcile loop that decides whether to destroy a running GPU, "usually" is not a word you want anywhere near the logic.

It is slow and expensive. Every control decision now costs a model call. A loop that ticks every ten seconds across hundreds of deployments becomes a bill and a latency source, to decide things a comparison operator could have decided for free.

It is hard to test. You cannot enumerate the decision space of a model. You can enumerate the decision space of a pure function, and assert on every branch of it.

And it is hard to audit. When spend drifts or a pod gets torn down, you want to point at the exact rule that fired and the exact state that triggered it. "The model decided to" is not an answer a regulated customer, or a tired on-call engineer, will accept.

Deterministic software has the opposite properties on all four counts. That is not a knock on models. It is a statement about what operating infrastructure actually requires.

What this looks like in practice

OpenLease is built on this line, and the control plane is deliberately, almost boringly deterministic.

The heart of it is a reconcile loop over a desired state and an observed state. Its decision core is a pure function: it takes a deployment and what the world currently looks like, and returns the one action to take next. It reads no clock, makes no network call, and asks no model. Everything that needs the wall-clock or a metric is resolved before the pure function is called, and the resolved answer is passed in. The result is that the entire decision space of the system is a function you can exhaustively unit-test and replay, and every state transition has exactly one rule behind it.

The capacity controls that Operate Your AI Capacity describes sit on the same foundation. An operating schedule resolves from the wall-clock against a set of windows: a comparison, not a consultation. A spend ceiling is computed from the cost records that already exist, checked against a limit, and enforced through the ordinary teardown path. A concurrency limit is a bounded semaphore in the proxy. Autoscaling reads the served request rate and adjusts the replica count between a floor and a ceiling. None of these asks a model whether it should act. They are software doing what software is good at: the same thing, correctly, every time, in a way you can test and point at afterward.

The hard parts were software problems, not model problems

It is worth noticing that the genuinely tricky decisions in this system were not the ones a model would have helped with.

When a proxy holds a concurrency slot for a streaming response, the slot has to be released exactly once, even if the client disconnects halfway through the stream. Get it wrong and slots leak until the deployment wedges. That is a lifecycle problem with a precise correct answer, and the way you get it right is careful software and a test that pulls the client mid-stream, not a model's best guess.

When an exceeded budget says stop but a schedule says the deployment should be running, something has to win. We made the budget outrank the schedule, because a ceiling that a schedule can override is not a ceiling. That is a precedence rule: one deterministic line that a person can read, argue with, and verify. An LLM adjudicating it case by case would be slower, unauditable, and occasionally wrong, in a place where occasionally wrong means a runaway bill.

The pattern holds in general. The hard operational decisions are hard because they demand precision and repeatability. Those are software's strengths and a model's weaknesses.

Where the model absolutely belongs

None of this argues for a dumb platform. It argues for putting the intelligence where it earns its cost, which is the reasoning layer that sits above the control plane, not inside it.

Deciding whether an incoming task is simple enough for a local model or hard enough to warrant a frontier one. Reading a failed run and recommending whether to escalate, retry, or give up. Summarizing why spend spiked this week and what to change. Proposing which model to route a class of work to, given its accuracy and cost profile. These are judgment calls under ambiguity, and they are exactly where a model is the best tool available.

The mistake is never using models. The mistake is using a model to do a semaphore's job.

The takeaway

An operating system for AI workloads should be boringly deterministic where it counts, and it should save its intelligence for the decisions that actually need intelligence. Software operates the infrastructure: health, budgets, schedules, concurrency, placement, retries, cleanup. Models reason about the work: what is hard, what failed, what to run where, what to do next.

Draw that line clearly and hold it, and you get a platform you can test, audit, afford, and trust. Blur it, and you get an agent wrapped around a comparison operator, wondering why the bill is so high and the logs make no sense.