System Architecture
The data plane, the control plane, and everything a request touches in between.
Three surfaces, one database
Inferect Labs isn't three separate products glued together — it's one Postgres database shared by three cooperating surfaces, each with a different job and a different auth model.
| Layer | Job | Auth |
|---|---|---|
| Data plane | Chat completions, model discovery, routing explain | API key |
| Control plane | Orgs, members, keys, credentials, providers, policies, dashboards, billing | Session or API key, via the BFF |
| Web app | The operator dashboard — Next.js App Router | Browser session |
How the backend is organized
The Go backend is split into domain packages with hexagonal boundaries — nothing reaches into another package's internals. Grouped by concern:
| Group | Packages | Role |
|---|---|---|
| Data plane | gateway, dispatch, routing, cache, classification, ingestion | Request handling, provider abstraction, routing, caching |
| Discovery / catalog | discovery, catalog | Model sync and the per-tenant model catalogue |
| Tenancy / identity | tenancy, account, organization, user, auth, apikey, authz, role, permission | Multi-tenant foundation, auth, RBAC |
| Credentials | credential, secrets | BYOK envelope encryption |
| Billing | billing, entitlement, quota, usage, ratelimit | Plans, subscriptions, entitlements, metering, limits |
| Control / experiments | control, experiment | Dashboard aggregation, shadow mode, recommendations |
| Cross-cutting | config, lifecycle, logging, metrics, observability, health, requestid, version | Runtime foundation |
What happens on a request
Authenticate
The API key is validated and resolved to an organization.
Classify
The request is classified — model family, capability needs — to feed the router.
Route
The engine scores candidates and produces a full ranking, not just a single pick.
Check cache
Exact match, then semantic. A hit returns immediately, no provider call made.
Dispatch
The top-ranked provider is called using your BYOK credential.
Stream or return
The response is returned in the OpenAI-compatible shape.
Fail over if needed
A retryable, pre-commit failure walks to the next-ranked candidate. A committed stream never fails over — that would corrupt what the client is already reading.
Record usage
Latency, tokens, retries, and outcome are written for the dashboard and billing.
Note
A per-provider circuit breaker tracks health and pulls unhealthy providers out of the ranking until they recover — so a struggling provider doesn't keep eating retries.
Multi-tenancy is a database property, not a code convention
Every tenant table carries a Postgres Row-Level Security policy, and the application connects as a non-owner role. A query bug in application code can't leak another organization's rows — the database refuses it at the policy level, not because someone remembered to add a WHERE org_id = ? clause.
Deployment topology
One Go binary, one managed Postgres with pgvector enabled. No Kubernetes cluster, no message queue, no fleet of microservices to keep in sync. A multi-stage Docker build produces the service binary, a migration runner, and a small utility that provisions the RLS application role; the entrypoint runs migrations to completion, then execs the service.
How architectural change works here
Architecture decisions are recorded as ADRs — dated, binding documents covering the stack, provider abstraction, credential handling, observability, auth, routing, caching, shadow mode, and billing. v1.0 is frozen: a change to it requires a new ADR, not a quiet patch.
Related