Gateway API

The gateway speaks both the OpenAI API and the Anthropic Messages API, and routes either one to any provider you have configured. Point an OpenAI or Anthropic SDK at your gateway URL and it works out of the box.

https://gateway.assistant-billing.dev/v1

To use a custom domain (e.g., ai.yourapp.com), see Custom Domains.

All requests require a gateway API key (sk_ab_* or temp_ab_*) via the Authorization: Bearer header.

Endpoints

Chat Completions

POST /v1/chat/completions

OpenAI-compatible chat completions with routing, billing, caching, and fallback.

Request Body

All standard OpenAI fields are supported (model, messages, temperature, tools, stream, etc.) plus gateway-specific fields:

import { GatewayClient } from "@assistant-billing/sdk"
 
const gateway = new GatewayClient({
  apiKey: "sk_ab_live_xyz789",
  baseURL: "https://gateway.assistant-billing.dev",
})
 
const response = await gateway.chat.completions.create({
  model: "gpt-4o",
  messages: [
    { role: "system", content: "You are a helpful assistant." },
    { role: "user", content: "Explain quantum computing in one paragraph." },
  ],
  temperature: 0.7,
  max_tokens: 500,
})
 
console.log(response.choices[0].message.content)

Streaming

When stream: true, the gateway:

1. Injects stream_options: { include_usage: true } to capture token usage
2. Streams SSE chunks to the client in real-time
3. Extracts usage from the final chunk for billing
4. Calculates cost and deducts after stream completes
5. Returns metadata in response headers (sent before the first chunk)

const stream = await gateway.chat.completions.create({
  model: "gpt-4o",
  messages: [{ role: "user", content: "Write a haiku about TypeScript." }],
  stream: true,
})
 
for await (const chunk of stream) {
  const content = chunk.choices[0]?.delta?.content
  if (content) process.stdout.write(content)
}

Provider Preferences

Control routing per-request via the provider field in the request body:

{
  "model": "gpt-4o",
  "messages": [{"role": "user", "content": "Hello!"}],
  "provider": {
    "order": ["openai", "azure"],
    "only": ["openai", "azure"],
    "ignore": ["groq"],
    "allow_fallbacks": true
  }
}

Model Fallback

Try multiple models in order until one resolves:

const response = await gateway.chat.completions.create({
  model: "gpt-4o",
  models: ["gpt-4o", "gpt-4o-mini", "claude-sonnet-4-20250514"],
  route: "fallback",
  messages: [{ role: "user", content: "Hello!" }],
})

The gateway tries each model's route in order. The first successful resolution is used. This is client-side model selection -- different from provider fallback chains configured in the dashboard, which retry a single model across multiple providers.

Transforms

Pre-process the request body before sending to the provider.

middle-out -- Compresses long conversations by keeping the first 2 and last 6 non-system messages (plus all system messages). Only applies when the total message count exceeds 10 and there are more than 8 non-system messages.

{
  "model": "gpt-4o",
  "messages": [...],
  "transforms": ["middle-out"]
}

Messages

POST /v1/messages

The Anthropic Messages API, with the same routing, billing, caching, and fallback as Chat Completions. Point the Anthropic SDK at your gateway URL and it works unchanged.

This endpoint forwards the Anthropic request shape to the provider's own messages endpoint. It does not convert between provider protocols, so the model you name has to be served by a provider that speaks Messages (Anthropic, Bedrock, Vertex). To reach an OpenAI model, use /v1/chat/completions or /v1/responses instead.

Request Body

All standard Anthropic fields are supported (model, messages, system, max_tokens, tools, thinking, stream, and so on), plus the same gateway fields Chat Completions accepts:

curl https://gateway.assistant-billing.dev/v1/messages \
  -H "Authorization: Bearer sk_ab_live_xyz789" \
  -H "Content-Type: application/json" \
  -H "anthropic-version: 2023-06-01" \
  -d '{
    "model": "claude-opus-5",
    "max_tokens": 1024,
    "system": "You are a concise assistant.",
    "messages": [{ "role": "user", "content": "Explain prompt caching in one paragraph." }]
  }'

Reasoning effort

effort and OpenAI's reasoning_effort are the same control spelled two ways, and the gateway translates whichever one you send into the form the routed provider accepts. Two rules are applied on the way through:

• reasoning_effort: "none" has no Anthropic equivalent, so it becomes effort: "low" with thinking: { "type": "disabled" }.
• Claude Opus 5 rejects thinking: { "type": "disabled" } above high. If you disable thinking explicitly and also ask for xhigh or max, the effort is clamped to high rather than letting the request fail.

Prompt caching

cache_control blocks pass through untouched, and the response's cache_read_input_tokens and cache_creation_input_tokens are billed at Anthropic's published multipliers: 0.1x for reads, 1.25x for 5-minute writes, and 2x for 1-hour writes. When usage.cache_creation splits writes by TTL, each bucket is priced at its own rate.

Requests sharing a cacheable prefix are routed to the same upstream account where the project has several, so a warm cache survives across turns. X-AB-Cache reports whether the gateway's own response cache served the request.

Streaming

Set stream: true for standard Anthropic SSE. Usage arrives split across message_start and message_delta; the gateway merges both before billing, and reads output_tokens_details.thinking_tokens from the final message_delta when present.

A response that ends in stop_reason: "refusal" is not billed, matching Anthropic's own behaviour for a request declined before any output.

Token counting

POST /v1/messages/count_tokens

Counts tokens for a prompt without running it. Takes the same body as /v1/messages. Useful now that Claude Fable 5 and Opus 4.7 and later use a tokenizer that produces roughly 30% more tokens for the same text than earlier models, which makes local estimates unreliable.

curl https://gateway.assistant-billing.dev/v1/messages/count_tokens \
  -H "Authorization: Bearer sk_ab_live_xyz789" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "claude-opus-5",
    "messages": [{ "role": "user", "content": "How many tokens is this?" }]
  }'

Batches

POST /v1/messages/batches submits an Anthropic message batch at the discounted tier; GET /v1/messages/batches/{id} reports status and /results streams the output. POST /v1/messages/batches/{id}/cancel stops one.

A batch belongs to the account that created it, so the gateway records that account and routes every later call back to it. Server tool calls inside a batch are priced the same as they are outside one.

Requests Without Provider Keys

A model whose provider you have not configured is still served, through an aggregator account that pays the provider on your behalf. X-AB-Route-Type: virtual marks these requests, and X-AB-Carrier names what delivered them.

Requests carried by the platform's accounts are charged to your wallet at the catalog rate. Connect your own Vercel AI Gateway or OpenRouter account under Providers and keyless requests use it instead, with nothing charged to the wallet for them. A provider key of your own always takes precedence over either.

Where both platform carriers can serve a request, Cloudflare is tried first because only its responses are cached, and Vercel takes over when Cloudflare is unavailable. To require Vercel AI Gateway for a single request, spell the model the way Vercel does behind a vercel/ prefix, such as vercel/anthropic/claude-sonnet-4.5.

Caching

The gateway supports response caching via Cloudflare AI Gateway. It applies to every request that passes through Cloudflare AI Gateway: credentialed routes (using your own provider API keys), which reach the provider through your project's Cloudflare AI Gateway, and virtual routes that Cloudflare carries (X-AB-Carrier: cloudflare). Virtual routes carried by Vercel AI Gateway are not cached.

• Cache hits return X-AB-Cache: HIT header
• Cached responses cost $0 -- no balance deduction
• Saved costs tracked in X-AB-Cost-Usd: 0 and logged as savedCostUsd
• Cache behavior is managed by Cloudflare AI Gateway -- configure it via your Cloudflare AI Gateway dashboard
• Cache is keyed on the full request body (model + messages + params)

Providers also bill a prompt prefix they have already processed at a discount. For OpenAI models the gateway sends one prompt_cache_key per project and workspace (per API key for a key without a workspace) with chat completions and responses requests, so every request of a workspace can reuse a prefix another one wrote; a prompt_cache_key in your request is sent as it is. Cache reads and writes are billed at the provider's rates, and the dashboard reports the share of input read from the provider's cache and what it saved.

Embeddings

POST /v1/embeddings

const result = await gateway.embeddings.create({
  model: "text-embedding-3-small",
  input: "The quick brown fox jumps over the lazy dog",
  dimensions: 256,
})
 
console.log(result.data[0].embedding) // [0.0023, -0.0091, ...]

Does not support streaming or model fallback.

Images

POST /v1/images/generations

curl https://gateway.assistant-billing.dev/v1/images/generations \
  -H "Authorization: Bearer sk_ab_live_xyz789" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "dall-e-3",
    "prompt": "A sunset over mountains, oil painting style",
    "size": "1024x1024",
    "n": 1
  }'

POST /v1/images/edits and POST /v1/images/variations default to dall-e-2 and use multipart form data.

Audio

Speech defaults to mp3. The OpenAI-compatible response_format selects mp3, opus, aac, flac, wav, or pcm when the selected provider supports it. stream_format: "sse" requests speech events instead of a binary audio response. The gpt-4o-mini-tts family reports token usage only in that SSE stream; a request on the platform's account without SSE is refused because it cannot be billed from the binary response. Vendor adapters for ElevenLabs, Deepgram, and Cartesia do not support SSE speech.

Transcription and translation default to JSON. response_format can request json, verbose_json, text, srt, or vtt; diarized_json is available when the model supplies diarization data. For a nonstreaming request, the gateway asks the upstream for a JSON format that reports usage, then renders the format requested by the client. Set multipart stream=true for streaming transcription with the gpt-4o-transcribe family. Translation does not stream.

For a nonstreaming audio response, X-AB-Cost-Usd reports the calculated request cost when a usable rate and usage are available. Streaming usage is recorded after the stream ends, so its cost is not in the initial response headers. Speech without a usable billing rate is refused on the platform's account rather than served without a charge.

Use the catalog model ID in model. For ElevenLabs and Cartesia speech, voice is that vendor's voice ID. For Deepgram speech, voice is its voice name, such as thalia-en with aura-2, which the gateway combines with the model into Deepgram's own model name. Vercel AI Gateway can serve listed audio models without a direct provider key, either through the platform account when the billing unit is supported or through your project's own Vercel account. A project's OpenRouter account can also serve its listed audio models. Streaming speech and transcription are unavailable through Vercel AI Gateway. Translation is available only through OpenAI and Groq provider keys.

# Text-to-speech
curl https://gateway.assistant-billing.dev/v1/audio/speech \
  -H "Authorization: Bearer sk_ab_live_xyz789" \
  -H "Content-Type: application/json" \
  -d '{"model": "tts-1", "input": "Hello, welcome to Assistant Billing!", "voice": "alloy"}' \
  --output speech.mp3
 
# Transcription (multipart)
curl https://gateway.assistant-billing.dev/v1/audio/transcriptions \
  -H "Authorization: Bearer sk_ab_live_xyz789" \
  -F file=@audio.mp3 \
  -F model=whisper-1

Moderations

POST /v1/moderations

Default model: omni-moderation-latest. Model field is optional.

curl https://gateway.assistant-billing.dev/v1/moderations \
  -H "Authorization: Bearer sk_ab_live_xyz789" \
  -H "Content-Type: application/json" \
  -d '{"input": "Some text to check for policy violations"}'

Feedback

POST /v1/feedback

Attach scores to a completed request for evaluation tracking. The request_id must reference an existing request log (prefixed with rl_).

// Single numeric score
await gateway.feedback("rl_a1b2c3d4e5f6", 5, { session: "abc" })
 
// Multiple named scores
await gateway.scores("rl_a1b2c3d4e5f6", {
  quality: 5,
  relevance: 3,
  thumbs_up: true,
})

You can pass either score (single numeric value with optional name) or scores (object with multiple named values). Boolean values are stored as 1/0.

Models

GET /v1/models

Returns the list of models available to the current API key, based on enabled providers and model permissions.

const models = await gateway.models.list()
// [{ id: "gpt-4o", object: "model", owned_by: "openai", context_window: 128000, ... }]

The response includes extended fields beyond the OpenAI spec: context_window, max_output, capabilities, and pricing. Capabilities include audio_input and audio_output alongside reasoning, tools, images, and structured output. pricing uses input, output, audio_input, audio_cached_input, and audio_output for USD per million tokens; characters for USD per million input characters; and audio_minute for USD per minute of audio. Each pricing key appears only when that model has a rate for the unit, and pricing is absent when no rate is available.

Stateful resources

Background responses, uploaded files, and batches live on one upstream account. The gateway records which account created each one, so a later retrieve, cancel, or delete reaches it even when the project has several credentials for the same provider. The same mapping backs previous_response_id: a turn continuing a stored response is pinned to the account holding it.

Response Headers

Every proxy response includes metadata headers:

virtual means the request was served without local provider credentials, by an aggregator account (see Requests Without Provider Keys). credentialed means the request was routed through a provider with your own API key. passthrough means the request was forwarded as-is via pass-through proxy mode.

For streaming responses, headers are sent before the first chunk. X-AB-Cost-Usd, X-AB-Tokens-In, and X-AB-Tokens-Out are only available in non-streaming responses (usage is extracted from the final stream chunk and logged asynchronously).

Vercel AI SDK

The gateway is OpenAI-compatible, so it works as a drop-in with the Vercel AI SDK:

import { createOpenAI } from "@ai-sdk/openai"
import { generateText, streamText } from "ai"
 
const provider = createOpenAI({
  baseURL: "https://gateway.assistant-billing.dev/v1",
  apiKey: "sk_ab_live_xyz789",
})
 
// Non-streaming
const { text } = await generateText({
  model: provider("gpt-4o"),
  prompt: "Hello!",
})
 
// Streaming
const result = streamText({
  model: provider("gpt-4o"),
  prompt: "Hello!",
})
 
for await (const chunk of result.textStream) {
  process.stdout.write(chunk)
}

No special adapter needed -- just point baseURL at your gateway.

OpenAI SDK

Any OpenAI-compatible SDK works. Just override the base_url and api_key:

import OpenAI from "openai"
 
const client = new OpenAI({
  baseURL: "https://gateway.assistant-billing.dev/v1",
  apiKey: "sk_ab_live_xyz789",
})
 
const completion = await client.chat.completions.create({
  model: "gpt-4o",
  messages: [{ role: "user", content: "Hello!" }],
})