Inference budgets: route before you prompt

The model is not the architecture

Most LLM integrations start by picking a model and end by discovering the cost on the invoice. Reverse the order: every endpoint in your system declares a budget (say, 800 ms p95 and €0.002 per request) and the budget decides which model serves each case. The model is an execution detail; the budget is design.

Measure before you optimize

Without inference telemetry you are flying blind. Log per provider, model and endpoint: input and output tokens, p50 and p99 latency, error rate and attributed cost. Two weeks of real data will tell you which share of your requests truly needs the big model. In practice it is usually under 20%.

Tiered routing

The pattern we run in production (SynthHires orchestrates ten providers on the Vercel AI SDK, with OpenRouter as the safety net):

1. Base tier: the small cheap model serves everything by default.
2. Signal-based escalation: step up to the big model only with cause (low confidence, long input, task flagged critical, retry after failure).
3. Chained fallback: when a provider drops or returns 429, the request jumps to the next one without the user noticing. Without this, your availability equals your weakest provider.

Cache and streaming

Repeated requests (greetings, templates, FAQs) answer from exact cache without touching any model. For the nearly repeated, a semantic cache over embeddings retrieves past answers when similarity clears your threshold. And anything slower than a second answers in streaming: perceived latency drops even when real latency does not move.

Exercise

Take one real endpoint of yours and write its budget in one line: target p95, max cost per request, and what happens on breach (degrade, queue, or fail loudly). If you cannot write it, you have no design yet: you have a prompt with infrastructure around it.

Next lesson: where API keys live when every user brings their own.