Overview
What Inferect Labs actually does, and how the pieces fit together.
What is Inferect Labs?
Inferect Labs sits between your application and the model providers you already use. You keep your own OpenAI, Anthropic, Groq, or other provider keys — we call this BYOK — and Inferect routes each request to the right model, fails over when a provider hiccups, caches what's worth caching, and gives you one bill and one dashboard instead of eighteen.
It's not a wrapper SDK. It's a gateway you point your existing OpenAI client at. If your code already calls chat.completions.create, switching base_url is most of the migration.
Data plane
The OpenAI-compatible surface — chat completions, model listing, routing explain. Authenticated by API key, and it forwards using your provider credentials, not ours.
Control plane
Everything that isn't inference: organizations, members, API keys, credentials, provider config, policies, billing.
Dashboard
A Next.js app that talks to the control plane through a same-origin proxy, so your browser session never holds a raw provider token.
What you get out of the box
| Capability | In practice |
|---|---|
| OpenAI-compatible completions | Existing SDKs work unchanged, streaming included. |
| 18 providers behind one client | 5 native adapters (OpenAI, Anthropic, Gemini, OpenRouter, vLLM) plus 13 OpenAI-compatible ones — Groq, Together, Fireworks, and more. |
| Routing you can actually inspect | /v1/routing/explain shows the ranked candidates and why one won, before you spend a token on it. |
| Failover that doesn't break streams | A retryable failure before commit moves to the next candidate. Once a stream has started, it doesn't jump providers mid-response — that would corrupt output. |
| Two-layer response cache | Exact-match, plus a pgvector semantic cache for near-duplicate prompts. |
| Shadow mode | Mirror production traffic to a candidate provider or policy with zero effect on what users see. |
| Envelope-encrypted BYOK | AES-256-GCM, scoped per org, per provider, per purpose. |
| Postgres RLS, not app-layer checks | Tenant isolation is enforced by the database, so an application bug can't leak another org's rows. |
| RBAC and an audit trail | Permissions and roles scoped to the org, with a record of who changed what. |
How a single request flows
Your app calls the gateway
A normal POST /v1/chat/completions, authenticated with an Inferect API key.
The router scores candidates
Using the request's classification and current provider health, not a static default.
Cache is checked
Exact match first, then semantic similarity — a hit skips dispatch entirely.
The request is dispatched
Using your BYOK credential for the chosen provider, never a shared platform key.
The response comes back
Streamed or not, in the same shape the OpenAI API would return. If dispatch fails before commit, the executor retries the next-ranked candidate automatically.
What it's built on
- Backend
- Go, standard-library
net/httprouting, a deliberately small dependency list - Database
- PostgreSQL + pgvector, schema tracked with goose migrations
- Frontend
- Next.js App Router, React Query, TypeScript, Tailwind
- Deploy
- One Docker image, one Postgres instance — no cluster to operate
- CI
- GitHub Actions, tests run against a real Postgres service — never a mock
Why the stack is this boring
A single binary and a single database are easier to reason about, audit, and operate than a sprawl of services. Architectural changes go through an ADR (Architecture Decision Record) — the v1.0 architecture is frozen, and nothing gets bolted on informally.