Cascading Model Routing: How We Cut LLM Inference Costs 85% Without Losing Quality

When you're running multi-agent systems that process thousands of LLM calls per day, the cost difference between sending everything to Opus versus routing intelligently across Haiku, Sonnet, and Opus is the difference between a viable product and burning cash. Here's how we built a cascading model router at PlanCodeShip that achieves 85%+ blended margin on inference spend.

The Problem: One Model Doesn't Fit All

Most AI products start by hardcoding a single model — usually the most capable one available. For us that meant Claude Sonnet for everything: greeting messages, FAQ lookups, multi-step reasoning chains, tool-call planning. The quality was great. The bill was not.

A quick audit of our production traffic revealed a pattern that's probably universal: roughly 60% of LLM calls are simple — classification, entity extraction, template filling, or single-hop retrieval. Another 25% need moderate reasoning. Only 15% genuinely require frontier-level capability.

The Architecture: Three-Tier Cascade

The router sits between the application layer and the LLM providers. Every request passes through a lightweight complexity classifier before hitting any model. The classifier itself runs on Haiku — costing fractions of a cent and adding under 200ms of latency.

Complexity Classification

The classifier is a structured output call to Haiku that returns a complexity score (1-10) and a category tag. We trained the prompt on 2,000 labeled examples from our production traffic. The features it considers:

• ▸ Token count — longer prompts with more context tend to need more capable models
• ▸ Tool-call count — requests requiring 3+ tool calls in sequence need planning capability
• ▸ Domain signal — medical, legal, and financial queries get bumped up by default
• ▸ Conversational depth — turn 8 of a complex negotiation is different from turn 1 of a greeting

Quality Gates and Automatic Escalation

Routing alone isn't enough. You need a quality feedback loop. After each Haiku or Sonnet response, a lightweight validator checks for signs of degradation: hallucination markers (claims without grounding), truncated reasoning, or schema violations in structured output.

If the validator flags an issue, the request automatically escalates to the next tier with the original context plus the failed attempt. This retry-with-escalation pattern means we can be aggressive with Haiku routing — the worst case is a slightly slower response, not a bad one.

Outcome-Based Metering

The final piece is tying cost to outcome. We don't just track tokens spent — we track cost per successful task completion. Every agent action gets an HMAC-signed billing event that records: model used, tokens consumed, latency, and whether the task was resolved or escalated to a human.

This lets us compute blended margins per customer, per agent type, and per task category. Our current production numbers:

At these ratios, our blended cost per 1M tokens is roughly $1.80 — compared to $15 if everything went to Opus. That's the 85% margin.

Lessons Learned

Start with logging, not routing. Before building any router, instrument your existing single-model setup to log prompt complexity signals. You need real data to know your traffic distribution.

The classifier pays for itself in one hour. A Haiku classification call costs ~$0.0001. If it prevents even one unnecessary Opus call per thousand requests, it's already profitable.

Quality gates matter more than initial routing. Being wrong about initial tier selection is cheap if your escalation path is fast. Being wrong without a fallback is expensive.

Track cost per outcome, not cost per token. A task that takes 3 Haiku calls and succeeds is cheaper than 1 Opus call that fails and requires human intervention.