TypeScript SDK

A typed client for the Forgebench API. Built on fetch, it runs on Node 18+, Deno, browsers and edge runtimes with no extra dependencies.

Your code holds a Forgebench key (sk_...) instead of a provider key, and Forgebench forwards each request to the model you asked for. Every call is budget-checked before money is spent and recorded in the audit log. See Core concepts for terms such as Agent, Budget and Policy.

Install

The SDK isn't on the npm registry. Install the prebuilt package from the latest stable release with npm, pnpm or yarn:

npm install https://github.com/seedlinglabs/forgebench-sdk/releases/latest/download/forgebench-sdk-ts.tgz

To pin a version, use releases/download/ts-vX.Y.Z/forgebench-sdk-ts.tgz. The releases page lists every version; previews end in -preview.

Connect to the gateway

You never send a provider credential; Forgebench strips one if a client sends it.

import { Forgebench } from "@seedlinglabs/forgebench-sdk";

const forgebench = new Forgebench({
  baseUrl: "https://api.forgebench.ai",
  apiKey: process.env.FORGEBENCH_API_KEY!,
});

Both are sent as Authorization: Bearer <credential>. With neither, new Forgebench({}) throws immediately ("Forgebench: one of \apiKey` or `token` is required."`).

Set baseUrl to https://api.forgebench.ai for the hosted service, or your own domain if self-hosted. The SDK doesn't read FORGEBENCH_BASE_URL; the example below reads it itself:

const forgebench = new Forgebench({
  baseUrl: process.env.FORGEBENCH_BASE_URL ?? "http://localhost:8000",
  apiKey: process.env.FORGEBENCH_API_KEY!,
});

whoami

Confirms the credential is valid and shows the identity it resolves to. Call it at startup to catch a bad key early:

const me = await forgebench.whoami();
console.log(me.tenant_id, me.roles, me.auth_method);

await forgebench.health(); // unauthenticated liveness

Chat completions

const res = await forgebench.chat.create({
  model: "mock-gpt",
  messages: [{ role: "user", content: "Hello, Forgebench." }],
});

console.log(res.choices[0]?.message.content);
console.log("tokens:", res.usage.total_tokens);
console.log("trace:", res.trace_id, "call:", res.call_id);

Omit model to use the server default. trace_id links the response to its audit entry, metering record and trace. call_id identifies this call; see Call lineage. See Make your first call for the full response shape.

Streaming

for await (const chunk of forgebench.chat.stream({
  messages: [{ role: "user", content: "Stream me a reply." }],
})) {
  process.stdout.write(chunk.choices[0]?.delta.content ?? "");
}

Or collect it into a string:

const text = await forgebench.chat.streamToText(
  { messages: [{ role: "user", content: "hi" }] },
  { onToken: (t) => process.stdout.write(t) },
);

The client parses the SSE stream, including the final [DONE]. The budget check runs before the stream opens, so a 402 throws immediately, never mid-stream. A stream is retried only on a 429 or a failure to connect, before it opens.

Call lineage

Pass a response's call_id as parentCallId on the calls it causes, such as a sub-agent's turn or the next step of a tool loop. Cost rolls up to the root call, and the console shows the run as a tree. It doesn't affect whether a call is allowed or what it costs.

const plan = await forgebench.chat.create({ messages: [...] });

const step = await forgebench.chat.create(
  { messages: [...] },
  { parentCallId: plan.call_id },
);

Tools

Which allowlist authorizes a tool call depends on the tool's name:

• Names starting with mcp_ (such as mcp_time_get_current_time) are checked against the MCP allowlist. Your agent's MCP client calls the server directly.
• Every other name (such as search or send_email) is checked against the Tool Registry. Your own code runs the tool.

Register each tool in the right place, or neither allowlist will authorize it.

Sending a tool schema

tools and tool_choice are typed fields of ChatCompletionRequest (SDK 1.2.0+), as are tool_calls and tool_call_id on messages.

Below, search is a Tool Registry tool and mcp_time_get_current_time is an MCP tool. If the model calls a tool this agent isn't bound to, the whole response is refused with 403 tool_not_authorized (Tool Registry) or 403 mcp_tool_not_authorized (MCP); see Errors.

import { Forgebench, PermissionDeniedError } from "@seedlinglabs/forgebench-sdk";

try {
  const res = await forgebench.chat.create({
    model: "gpt-4o",
    messages,
    tools: [
      {
        type: "function",
        function: {
          name: "search",
          description: "Search the KB",
          parameters: { type: "object", properties: { q: { type: "string" } } },
        },
      },
      {
        type: "function",
        function: {
          name: "mcp_time_get_current_time",
          description: "Current time in a timezone",
          parameters: { type: "object", properties: { tz: { type: "string" } } },
        },
      },
    ],
    tool_choice: "auto",
  });

  for (const call of res.choices[0]?.message.tool_calls ?? []) {
    if (call.function.name.startsWith("mcp_")) {
      const result = await yourMcpClient.call(call.function.name, call.function.arguments);
    } else {
      const result = await yourToolDispatch(call.function.name, call.function.arguments);
    }
  }
} catch (e) {
  if (e instanceof PermissionDeniedError) {
    console.error("tool not authorized", e.code, e.body);
  } else {
    throw e;
  }
}

tool_calls on the response is OpenAI-shaped ({ id, type, function: { name, arguments } }). arguments is a JSON string, not parsed. Bind tools on the Tool Registry or MCP page first.

Other fields only reachable through a widened type

Forgebench accepts a few fields that ChatCompletionRequest doesn't declare. Send them with an as ChatCompletionRequest & Record<string, unknown> cast:

Any other undeclared field is silently dropped.

The agentTools resource (agent credentials only)

With an agent key, forgebench.agentTools gives you the agent's current tool allowlist and a dispatch loop. A binding revoked in the console drops out on the next turn, with no redeploy.

• openaiSchema() builds the model's tool list from the allowlist (GET /v1/agent-tools).
• dispatch runs each allowed tool call, reports the outcome, and returns the role: "tool" messages to append before the next turn.

const tools = await forgebench.agentTools.openaiSchema();

const turn = await forgebench.chat.create({
  model: "gpt-4o",
  messages,
  tools,
} as ChatCompletionRequest & Record<string, unknown>);

messages.push(
  ...(await forgebench.agentTools.dispatch(
    turn.choices[0].message.tool_calls,
    (name, args) => yourMcpClient.call(name, args),
    { callId: turn.call_id! },
  )),
);

dispatch doesn't stop on a failing tool. A throwing executor is reported as that tool's error and returned to the model as { error }. Forgebench stores only a tool's name and description, so pass argument schemas with openaiSchema({ parameters: { search: {...json schema...} } }).

agentTools.report(...) is what dispatch calls internally. Use it directly only if you run tools outside an executor callback. Reports (success, error, latency) go to the audit log and traces; the model never sees them.

Agents, runs, deployments

const agent = await forgebench.agents.create({
  name: "support-bot",
  model: "mock-gpt",
  system_prompt: "You are concise.",
});

const run = await forgebench.runs.create({ agent_id: agent.id, input: { topic: "refunds" } });
const finished = await forgebench.runs.waitForCompletion(run.id, { intervalMs: 1000 });
console.log(finished.status, finished.output);

const deployment = await forgebench.deployments.create({ agent_id: agent.id, name: "prod" });

Agent → agent (A2A tasks)

An agent bound to another agent can open a task on it. The callee's governed calls are recorded under the task:

const task = await forgebench.agents.call("qp-research", {
  text: "10 MCQs on photosynthesis",
  data: { grade: 9 },
  parentCallId: plan.call_id,
});
const notes = artifactText(task);

To receive tasks without an inbound port, long-poll for this agent's tasks:

await forgebench.agents.serve(async (task, reply) => {
  const turn = await forgebench.chat.create(
    { model: "mock-gpt", messages: [{ role: "user", content: messageText(task.input.message) }] },
    { parentCallId: task.call_id },
  );
  return reply.done({ text: turn.choices[0].message.content ?? "" });
});

The handler may return a TaskReply (reply.done(...), reply.ask(...), reply.fail(...)), or a string, object or array of artifacts, which becomes a completed reply. A thrown error fails the task without stopping the loop.

Workflows

A workflow is a named group of agents (steps) that one app runs in order, one agent and one key per step. Register it once, then drive it with forgebench.workflow:

const me = await forgebench.whoami();
const created = await forgebench.workflows.register({
  name: "Question paper",
  owner_identity_id: me.identity_id,
  steps: [{ name: "grading", model: "mock-gpt" }, { name: "retry", model: "mock-gpt" }],
  edges: [{ from: "grading", to: "retry", kind: "sequence" }],
});
const keys = Object.fromEntries(created.credentials.map((c) => [c.step, c.credential!.api_key])); // shown once

const run = forgebench.workflow("question-paper", keys).run();
await run.step("grading").chat({ model: "mock-gpt", messages });  // uses grading's key
await run.step("retry").chat({ model: "mock-gpt", messages });    // parent = grading's call
console.log(run.rootCallId);

await forgebench.workflows.checks("question-paper");              // wired? threaded? seeing traffic?

Each step call uses that step's own key and sends the previous call of the run as its parent (X-Parent-Call); the first call is a root. A streamed step call (step.stream) becomes the next call's parent once its callId is known, so read it to the end first. forgebench.workflows also has list, get, update, archive, addStep, removeStep, addEdge and removeEdge.

Voice

Transcription, Speech Synthesis and streaming agent runs are governed like chat. See Govern your voice agent.

try {
  const heard = await forgebench.audio.transcriptions.create({
    file: new Blob([bytes], { type: "audio/wav" }),
    model: "gpt-4o-transcribe",
    session: "call-8841",
  }); // heard.text, heard.raw, heard.traceId
  const spoken = await forgebench.audio.speech.create({
    model: "gpt-4o-mini-tts", input: "Hello", voice: "alloy", session: "call-8841",
  }); // spoken.audio (ArrayBuffer), spoken.contentType, spoken.traceId
  for await (const event of forgebench.agents.run(agentId, { message: "Hi", stream: true })) {
    console.log(event); // chat.completion.chunk objects, then { type: "agent.run.completed", answer, trace_id, cost_usd, citations }
  }
} catch (e) {
  if (e instanceof BudgetExceededError) console.log("over budget");
  else throw e;
}

file accepts a Blob/File, bytes, or { data, filename, contentType }. session sets X-Forgebench-Session. Without stream: true, agents.run resolves to the cited answer. Admin helpers: forgebench.voice.policy, forgebench.voice.assets and forgebench.voice.consents. See Voice governance.

A streaming agents.run throws ForgebenchStreamError (e.code, e.message) if the server sends an event: error frame mid-stream, or if the stream ends without agent.run.completed / [DONE] (code: "stream_truncated"). To clear a Voice Asset field pass null, for example assets.update(id, { expires_at: null }). voiceHeaders(...) builds the session, turn and traceparent headers for your own pipeline's calls; AUDIO_TRACE_ID_HEADER is the /v1/audio/* trace-id response header (chat uses TRACE_ID_HEADER).

Budget enforcement (HTTP 402)

Over-budget calls are refused before any provider call, so they cost nothing:

import { BudgetExceededError } from "@seedlinglabs/forgebench-sdk";

try {
  await forgebench.chat.create({ messages: [{ role: "user", content: "again" }] });
} catch (err) {
  if (err instanceof BudgetExceededError) {
    console.error("Tenant is over budget:", err.code, err.requestId);
  } else {
    throw err;
  }
}

Tracing

Mark your agent's steps and each run becomes one nested trace: the agent, its functions and tools, and each model call with its tokens and cost. Steps are sent in the background with the same API key. Read traces in the console. See Tracing your agent.

Wrap a function

observe wraps a function (sync or async) and records each call: arguments as input (a single argument as itself), result as output, duration, and any error (re-thrown unchanged). Model calls inside it nest under it.

import { Forgebench, observe, updateActiveTrace } from "@seedlinglabs/forgebench-sdk";

const forgebench = new Forgebench({ apiKey: process.env.FORGEBENCH_API_KEY! });

const lookupOrder = observe(async (orderId: string) => orders.get(orderId), {
  name: "lookup_order",
  asType: "tool",
});

const run = observe(
  async (question: string) => {
    updateActiveTrace({ userId: "customer-42", sessionId: "chat-7", tags: ["support"] });
    const order = await lookupOrder("A-1001");
    return forgebench.chat.create({ messages: [{ role: "user", content: question }] }); // nested
  },
  { name: "support-agent", asType: "agent" },
);

name defaults to the function's name; asType is one of span (default), agent, tool, chain, retriever, embedding, evaluator, guardrail, event. Set captureInput: false or captureOutput: false for values that are large or must not leave the process. Use forgebench.observe(...) to bind the wrapper to one client when your app uses several API keys.

Mark a block of code

const steps = await forgebench.startActiveObservation(
  "plan",
  async (plan) => {
    const steps = await makePlan(question);
    plan.update({ output: { steps }, metadata: { planner: "v2" } });
    return steps;
  },
  { asType: "chain", input: { q: question } },
);

startActiveObservation returns what your function returns; steps and model calls made inside nest under it, and an error is recorded and re-thrown.

Steps that start and end in different places

forgebench.startObservation(...) creates a step that isn't the active one, for a callback or background job. step.startObservation(...) adds a child. Call end() when done; a step that never ends is never sent.

const job = forgebench.startObservation("nightly-sync", { asType: "chain" });
const page = job.startObservation("fetch-page", { asType: "retriever" });
page.update({ output: { rows: 120 } });
page.end();
job.end();

Parallel work

Steps started with Promise.all or any other async code keep their own parents on Node 20.16 or later, Node 22 and Bun (through AsyncLocalStorage). Other runtimes nest code that runs one step at a time correctly.

Configuration

const forgebench = new Forgebench({
  apiKey: process.env.FORGEBENCH_API_KEY!,
  tracing: {
    sampleRate: 0.2,            // keep 20% of traces (whole traces, never half)
    environment: "production",
    release: "2.3.0",
    mask: myMask,               // runs on every value and error message before it leaves
  },
});

Unset fields come from FORGEBENCH_TRACING_ENABLED, FORGEBENCH_SAMPLE_RATE, FORGEBENCH_ENVIRONMENT and FORGEBENCH_RELEASE; a value that does not parse is ignored with a warning. enabled: false turns tracing into a no-op without touching your code.

Sending before exit

Steps are sent about every second. In a short script or serverless function, await forgebench.flush() before returning; forgebench.shutdown() sends what's left and stops. Node also flushes when its event loop empties, but not on process.exit(). A failed send is logged and dropped, never thrown.

The module-level functions (observe, startActiveObservation, …) use the client you created. With several clients they run untraced and warn once, so call the methods on a client instead (forgebench.observe(...)).

Errors

All non-2xx responses map to a specific subclass of ForgebenchAPIError, which carries status, code, body and requestId:

Network and abort failures raise ForgebenchConnectionError; timeouts raise its subclass ForgebenchTimeoutError. By the time you catch a 429, or a 5xx or network failure on a read, the SDK has already retried it with backoff up to maxRetries times (default 2).

A write (chat call, A2A task, tool report) is never replayed after a 5xx or a dropped connection, because the server may already have run and billed it. An InternalServerError from a chat call means it may or may not have run; check before repeating it.

Other errors will fail again if retried unchanged. Three unauthorized attempts at the same tool within ten minutes escalate to a 429 (tool_repeatedly_unauthorized).

PermissionDeniedError reports its cause in different places. Tool refusals set .code (for example "tool_not_authorized"). For model_not_allowed, .code is null and the identifier is in .body.detail.error:

import {
  BudgetExceededError,
  PermissionDeniedError,
  AuthenticationError,
  NotFoundError,
  ConflictError,
  UnprocessableEntityError,
  RateLimitError,
  InternalServerError,
  ForgebenchConnectionError,
  ForgebenchTimeoutError,
} from "@seedlinglabs/forgebench-sdk";

try {
  const res = await forgebench.chat.create({ model: "gpt-4o", messages, tools } as ChatCompletionRequest & Record<string, unknown>);
} catch (e) {
  if (e instanceof BudgetExceededError) {
    console.warn("budget gate refused the call", e.body);
  } else if (e instanceof PermissionDeniedError) {
    console.error("not authorized", e.code, e.body);
  } else if (e instanceof AuthenticationError) {
    console.error("auth failed", e.body);
  } else if (e instanceof UnprocessableEntityError) {
    console.error("request rejected", e.body);
  } else if (e instanceof NotFoundError) {
    console.error("not found", e.body);
  } else if (e instanceof ConflictError) {
    console.error("conflicting state", e.body);
  } else if (e instanceof RateLimitError) {
    console.warn("rate limited after retries", e.body);
  } else if (e instanceof InternalServerError) {
    console.error("control-plane error after retries", e.body);
  } else if (e instanceof ForgebenchTimeoutError) {
    console.error("request timed out");
  } else if (e instanceof ForgebenchConnectionError) {
    console.error("could not reach the control plane", e);
  } else {
    throw e;
  }
}

Every 402, 403, 400 and 429 refusal is written to the Refusals log with its cause. See Errors and refusals for each body shape.

Retries

The Python, TypeScript and Java SDKs share one retry rule: a call the server may already have run and billed is never sent twice.

• A stream that is already delivering events is never retried.
• A read timeout on a write is not retried: the server may have run the call.
• There are at most maxRetries retries (default 2). The wait starts at 0.5s and doubles up to 8s, with jitter. A Retry-After header in seconds replaces it, capped at 30s; an HTTP-date is ignored.
• A call you abort through your own AbortSignal is never retried. A timeout the SDK raised itself (ForgebenchTimeoutError) follows the table.

Governance: audit + metering

const summary = await forgebench.metering.summary();
console.log(`spent $${summary.spent_usd} of $${summary.monthly_limit_usd}`);

const entries = await forgebench.audit.list({ limit: 20 });

entries[i].prev_hash and row_hash form a tamper-evident hash chain per workspace.

API keys

const created = await forgebench.keys.create({ name: "ci", scopes: ["chat:write"] });
console.log(created.api_key); // shown ONCE, store it now

const keys = await forgebench.keys.list();
await forgebench.keys.revoke(created.id);

keys.list() returns the safe view: no secrets.

Next