LLM cost optimization: what a token bill actually is

LLM cost optimization, as the guides tell it, is an engineering discipline — model tiers, token counting, prompt compression. For the COO who opened the invoice, it is a simpler question: what are we actually paying for? What a token bill is, the four levers that move it, and what we saw when a fifteen-person firm asked what its token budget should be.

The question arrived the way it usually does — after the bill. A lower-middle-market private equity sponsor, fifteen to twenty people at the firm and nine portfolio companies, had rolled out frontier-model subscriptions a couple of months earlier. Adoption was real. Then the usage meter became real too. One partner’s verdict: “I’ve been burning through tokens.” Another’s confession: “I alone somehow hit $400 and I don’t know how that’s possible.” So the COO asked us the question this article exists to answer: what should our token budget be?

Search “LLM cost optimization” and you will find good engineering guides — model tiering, prompt compression, token counting — written for the people building the system. This is the version for the person who got surprised by the invoice: what the bill actually is, why it grows faster than headcount, and the levers that move it. None of them require your team to use AI less.

What a token bill actually is

A token is roughly three-quarters of a word, and the meter runs on everything the model reads and everything it writes. That much the guides will tell you. The part that surprises operators is the reading side. The model has no memory: every question re-sends the context it needs — the conversation so far, the documents attached, the instructions — and you pay for that context again with every single call. A ten-minute working session with a document is not one charge; it is the document, re-read, dozens of times.

That is why the bill scales with ambition instead of seats. The sponsor saw it directly: token usage scales up with the more analytical, deeper questions — the ones where the assistant chews through deal documents and call transcripts — while the shallow “rewrite this email” usage barely registers. And the heavy usage is the valuable usage. The goal of AI cost optimization is not a smaller number; it is the same capability with the waste engineered out.

LLM routing: the lever with the most slack

The single largest source of waste we find is not extravagant questions — it is every question going to the most expensive model. A status lookup and a forty-page document analysis get the same frontier treatment, because the default is the default. LLM routing fixes that with one upstream decision: a cheap step reads each request and sends it to the smallest model that can actually handle it. Lookups, extraction, and drafting go to fast, inexpensive models; the frontier model only sees the analytical work that justifies it. The user notices nothing. The invoice notices immediately.

The same logic applies behind the scenes. The sponsor was building an ingestion layer — reading the firm’s documents so the assistant can retrieve them later — and running it on the frontier model, which gets quite expensive at bulk. Bulk reading is exactly the work a lower-cost model does well. Their own hope, phrased better than most vendor copy: reduce the token costs by shifting that work over to lower cost. That is routing, applied to the plumbing as well as the questions.

Caching, right-sizing, and batching

Three smaller levers compound the routing gain. Caching: when the same large context goes out with every call — your firm’s standing instructions, a document being worked over an afternoon — the provider can hold it instead of re-billing it in full each time; the re-read problem shrinks to a delta. Right-sizing: model choice is per task, not per company, and most firms’ task mix is far more ordinary than their model bill suggests. Batching: work that does not need an answer this second — the nightly ingestion run, the weekly summary — goes out as a batch job at a discount instead of as impatient one-off calls.

Underneath all four levers sits a fifth that is really a design rule: retrieve before you read. An assistant that answers from the three relevant pages costs a fraction of one that re-reads the whole folder to find them. Most “LLM cost” problems we are brought in to look at turn out to be retrieval problems wearing a billing costume.

A token bill is not an AI problem. It is an architecture problem with an invoice attached.

What should our token budget be?

The honest answer the COO got: nobody can hand you a benchmark that survives contact with your usage, because the number is an output, not an input. One partner alone hit $400 in a month with no architecture at all — every question to the top model, full context re-sent every time, ingestion running on the frontier tier. Fifteen more people adopting the same habits does not produce a budget; it produces a surprise. The budget that means anything is set per workflow: this question class goes to this model tier, this context gets cached, this job runs in batch. When the routing table is written down, the monthly number becomes a consequence of design decisions you can point at — and the answer to “what should it be” stops being a guess.

The question is never how much tokens should cost. It is which questions deserve the expensive model.

There is a volume at which the meter itself becomes the problem: if you have routed, cached, right-sized, and batched, and the frontier bill still dominates, the next conversation is about running open-weight models in your own cloud tenant at a fraction of per-token pricing — the private-deployment math is on the on-premise AI page (zaigo.ai/services/on-premise-ai), including the thread where open-source models run 90–95% cheaper than frontier tokens. The build this firm’s bill came from — retrieval over the firm’s own documents, notes, and deal history — is on the enterprise AI search page (zaigo.ai/services/enterprise-ai-search).

If the invoice landed before the architecture did, the next step is a 30-minute working call: zaigo.ai/book-a-call. Bring a month of the bill. We will tell you on the call which of the levers your usage pattern needs first — and what the number looks like once the expensive model only sees the questions that deserve it.