AI & Routing
How requests get scored, ranked, cached, and moved to a backup provider when one fails.
18 providers, one client
5 native adapters cover OpenAI, Anthropic, Gemini, OpenRouter, and vLLM. The other 13 — Groq, Together, Fireworks, and more — are OpenAI-compatible, so they're served by a single shared client rather than a bespoke adapter each. Fewer adapters to maintain means fewer places for provider drift to break something quietly.
What the router actually returns
Given a policy, a request classification, and current provider health, the engine returns a full decision: the chosen provider, the strategy used, a complete ranking of candidates (not just the winner), any rejected candidates and why, a confidence score, and estimated cost/latency/savings.
| Mode | Optimizes for |
|---|---|
| latency | Fastest expected response |
| quality | Highest-capability model available |
| balanced | A blend of cost, latency, and quality |
| coding | Models suited to code generation |
| reasoning | Models suited to multi-step reasoning |
| vision | Multimodal / image-capable models |
| long_context | Largest context windows |
| embeddings | Embedding-optimized models |
| json | Structured / JSON-mode output |
You don't have to trust it blindly
POST /v1/routing/explain replays the same decision without dispatching the request. If a routing choice looks wrong, this is where you check first.
Failover, precisely
Failover walks the ranking from the decision above. A retryable, pre-commit failure moves to the next candidate automatically — your app doesn't see it. Once a streaming response has committed to the client, though, it stays put; switching providers mid-stream would mean handing back a corrupted response, which is worse than a clean failure. A per-provider circuit breaker also removes consistently unhealthy providers from consideration until they recover.
Caching
Two layers: exact-match for identical prompts, and a pgvector-backed semantic cache that catches near-duplicates by embedding similarity. Policy is configurable per org, so you can be aggressive in dev and conservative in a compliance-sensitive prod environment.
Shadow mode
Mirror live traffic to a candidate provider or routing config without it touching what users actually see. It's the way to validate a new model or a policy change against real traffic before you flip it live.
About those savings numbers
Estimated, not billed
"Estimated savings" compares against the most expensive eligible candidate for that specific request. It explains routing behavior — it is not a billing figure. See Data Model for where it's stored and Business & Pricing for how plans are actually metered.
Related