How It Works

Roles

Customers are identified by a workspace_id string you assign (user ID, org ID, team ID). You bring your own auth system.

Request Flow

Every AI request follows this path:

1. Customer sends an OpenAI-compatible request to the gateway (POST /v1/chat/completions)
2. Gateway authenticates the API key (SHA-256 lookup) and checks rate limits
3. Route resolution: match the requested model to a provider using model routes and routing strategy
4. Pre-billing check: validate the workspace has sufficient balance (cached for 5 minutes)
5. Forward the request to the AI provider, with automatic retry/fallback on 429/5xx errors
6. Calculate cost from token usage using the model pricing database
7. Deduct cost from workspace balance (with markup) and operator wallet (raw cost)
8. Return the response with cost and usage metadata in headers

Model Routing

The router resolves which provider and model handles each request.

Route Actions

Routes match by exact model name or wildcard patterns.

Routing Strategies

Fallback Chains

When fallbacks are configured on a route, the gateway retries automatically on retryable errors (429, 500, 502, 503, 504).

Primary: OpenAI (gpt-4o)
  ↓ fails with 503
Fallback 1: Azure OpenAI (gpt-4o)
  ↓ fails with 429
Fallback 2: Anthropic (claude-sonnet-4-20250514)
  ✓ success → return response

Non-retryable errors (400, 401, 402, 403, 404) are returned immediately. Billing is calculated against the actual provider that produced the response.

Two-Level Billing

Operator Wallet

Platform-managed balance between the operator and Assistant Billing. When customers make AI requests, the raw provider cost is deducted from this wallet. Operators top up by paying Assistant Billing directly through the Dashboard.

Customer Balances

Per-workspace balances between customers and the operator. Customers top up via Stripe checkout (paying the operator). The cost with markup is deducted per-request.

Fee Breakdown

base_cost = (input_tokens * input_price) + (output_tokens * output_price)
markup    = base_cost * markup_percentage
total     = base_cost + markup

For top-ups, the platform fee is added:

Example: $10 top-up, 10% markup, 5% platform fee
  Base amount:    $10.00  (credited to balance)
  Markup:         $1.00   (operator margin)
  Platform fee:   $0.50   (goes to Assistant Billing)
  Charged:        $11.50

BYOK (Bring Your Own Key)

Customers can supply their own provider API keys. The gateway checks for keys in order:

1. Customer's own key (workspace-level)  → use if set
2. Operator's key (project-level)        → fallback
3. No key available                      → error

When a customer uses their own key, no balance deduction occurs. Usage is tracked per API key with optional monthly budget limits.

Response Headers

The gateway returns metadata in response headers: