Gateway

Every model call goes through the Forgebench gateway, which routes, caches, budgets, meters and audits it. No client reaches a provider directly. This page covers the model catalog, reliability routing, fallbacks and the response cache.

The page is read-only. Your platform administrator manages the policy.

Call the gateway

The gateway has an OpenAI-compatible endpoint. Point any OpenAI client at it and pass an sk_... key. Your code never holds a provider credential.

from forgebench import Forgebench

client = Forgebench(api_key="sk_...", base_url="https://api.forgebench.ai")
resp = client.chat.completions.create(
  model="mock-gpt",
  messages=[{"role": "user", "content": "hello"}],
)

import { Forgebench } from "@seedlinglabs/forgebench-sdk";

const forgebench = new Forgebench({ baseUrl: "https://api.forgebench.ai", apiKey: "sk_..." });
const res = await forgebench.chat.create({
model: "mock-gpt",
messages: [{ role: "user", content: "hello" }],
});

import ai.forgebench.ChatCompletionRequest;
import ai.forgebench.ChatMessage;
import ai.forgebench.Forgebench;
import ai.forgebench.types.ChatCompletion;

Forgebench client = Forgebench.builder().apiKey("sk_...").baseUrl("https://api.forgebench.ai").build();
ChatCompletion resp = client.chat().completions().create(ChatCompletionRequest.builder()
  .model("mock-gpt")
  .addMessage(ChatMessage.user("hello"))
  .build());

curl -X POST https://api.forgebench.ai/v1/chat/completions \
-H "Authorization: Bearer sk_..." \
-H "Content-Type: application/json" \
-d '{
  "model": "mock-gpt",
  "messages": [{"role": "user", "content": "hello"}]
}'

The response is OpenAI-shaped, plus a trace_id. See Quickstart for the full body.

To route coding tools such as Claude Code and Codex instead, run forgebench gateway install.

The model catalog

A searchable list of every model a call can use: label, provider, fallback chain, availability, and price. Local models show "free" and unpriced ones are blank. A model whose key you haven't configured shows as unavailable, with a link to add one. See Model providers for each provider's credentials.

Above the list are the total model count and how many have a fallback chain.

Prices are in each model's billing unit:

• Chat: per 1K tokens.
• Image: per image.
• Transcription: per minute of audio processed.
• Speech Synthesis: per 1K characters spoken.

All are charged at provider cost and draw down the same dollar budget as chat. See Govern your voice agent.

Sort by name, price, context window or tier, and filter by vendor. Available models can be used in Playground and by agents, within the calling key's allowlist.

Reliability routing and fallbacks

Every call is load-balanced across providers, retried on failure, and escalated through a fallback chain. You don't write retry logic.

A model marked protected in the catalog has a fallback chain. The call moves to the next model in the chain on either trigger:

• On failure: the primary model errored or is unavailable.
• Over context limit: the request is too large for the primary model's context window.

Response cache

A cache hit skips the provider call. The request is still budgeted, metered and audited as if it had gone through.

Cache entries are never shared across tenants. Entries expire after a fixed TTL set by Forgebench.

This page shows whether caching is on and how long entries live. Hit rate and requests saved are in Observability.

Thinking models and small max_tokens

Thinking models (for example vertex-gemini-2.5-pro) spend max_tokens on hidden reasoning before they write the answer. With a small budget you can get a few words back and finish_reason: "length".

The gateway never changes your max_tokens. When a thinking model stops on length with almost no output, the response carries a header:

X-Forgebench-Warning: thinking_budget_exhausted: thinking model stopped on max_tokens ...

Raise max_tokens to fix it.

• The call's usage record is flagged warning = thinking_budget_exhausted, so you can find these calls later. Streams send headers before the model finishes, so they get the flag only.
• Browser clients can read the header (it's exposed for CORS).
• GET /v1/models marks thinking models with reasoning: true.
• Routing's live-test probes and LLM judge get enough tokens to reason (a floor, capped at the model's maximum output), so a thinking model is never scored on a truncated answer.

Gateway vs Routing

• Gateway handles reliability routing: which provider serves a model, with retries and failover the caller never sees. Configure fallback chains here.
• Routing handles capability routing: which model a request should use, based on what it needs. Configure it in Routing.

Next