Tool Registry

The Tool Registry controls which tools an agent's model may propose calling. You register a tool by the exact name the model emits in tool_calls[].function.name and bind it to agents. Forgebench checks every response against those bindings before the agent sees it.

There is no server behind a registry entry. Forgebench doesn't call or run the tool; your code does.

Register a tool

1. Name it exactly

   Must match the string the model emits, character for character. You can't change it later.

2. Describe it

   Helps colleagues decide whether to bind an agent to it.

3. Confirm the payload location

   The standard OpenAI shape (tool_calls[].function.name / .arguments) is detected automatically. If your response puts tool calls elsewhere, set Payload location to Custom paths and enter the paths to the call list, the name and the arguments. Blank fields use the standard shape.

To register in bulk, upload a CSV with columns name, description, tool_calls_path, name_path and arguments_path. Only name is required. Duplicate or invalid rows are skipped.

A registered tool's page has Overview, Bindings and Activity tabs, and a Disable everywhere action, the kill switch.

Handle it in your code

Your calling code only needs to handle the new refusals. A blocked response raises before your tool_calls handling runs.

The Python and TypeScript SDKs don't have tools or tool_choice parameters, but Forgebench forwards them. Send them through extra_body in Python, or a widened type in TypeScript. In Java, use tools(...) on the request builder and extraBody(...) for tool_choice. In Python, tool_calls is a list of plain dicts, so index into it; in Java it is a List<Map<String, Object>>.

• PermissionDeniedError (tool_not_authorized): the model proposed a tool this agent isn't bound to. Bind it; don't retry.
• RateLimitError: the tool's call-volume ceiling, or the repeat-offender breaker (three unauthorized attempts in ten minutes).

from forgebench import Forgebench, PermissionDeniedError, RateLimitError

client = Forgebench(api_key="sk_...", base_url="https://api.forgebench.ai")
messages = [{"role": "user", "content": "Find last quarter's invoices."}]

try:
  resp = client.chat.completions.create(
      model="gpt-4o",
      messages=messages,
      extra_body={"tools": [...]},  # "search", "send_email", etc.
  )
except PermissionDeniedError as e:
  log.error("tool not authorized: %s", e)
except RateLimitError as e:
  log.warning("tool call rate-limited: %s", e)
else:
  for call in resp.choices[0].message.tool_calls or []:
      fn = call["function"]
      result = your_tool_dispatch(fn["name"], fn["arguments"])

import { Forgebench, PermissionDeniedError, RateLimitError, type ChatCompletionRequest } from "@seedlinglabs/forgebench-sdk";

const forgebench = new Forgebench({ apiKey: "sk_...", baseUrl: "https://api.forgebench.ai" });
const messages = [{ role: "user" as const, content: "Find last quarter's invoices." }];

try {
const res = await forgebench.chat.create({
  model: "gpt-4o",
  messages,
  tools: [...], // your tool schema
} as ChatCompletionRequest & Record<string, unknown>);
for (const call of res.choices[0]?.message.tool_calls ?? []) {
  const result = await yourToolDispatch(call.function.name, call.function.arguments);
}
} catch (e) {
if (e instanceof PermissionDeniedError) {
  console.error("tool not authorized", e);
} else if (e instanceof RateLimitError) {
  console.warn("tool call rate-limited", e);
} else {
  throw e;
}
}

import ai.forgebench.ChatCompletionRequest;
import ai.forgebench.ChatMessage;
import ai.forgebench.Forgebench;
import ai.forgebench.errors.PermissionDeniedException;
import ai.forgebench.errors.RateLimitException;
import ai.forgebench.types.ChatCompletion;
import java.util.List;
import java.util.Map;

Forgebench client = Forgebench.builder().apiKey("sk_...").baseUrl("https://api.forgebench.ai").build();
List<ChatMessage> messages = List.of(ChatMessage.user("Find last quarter's invoices."));

try {
  ChatCompletion resp = client.chat().completions().create(ChatCompletionRequest.builder()
      .model("gpt-4o")
      .messages(messages)
      .tools(tools) // "search", "send_email", etc.
      .build());
  List<Map<String, Object>> calls = resp.getChoices().get(0).getMessage().getToolCalls();
  for (Map<String, Object> call : calls == null ? List.<Map<String, Object>>of() : calls) {
      Map<?, ?> fn = (Map<?, ?>) call.get("function");
      Object result = yourToolDispatch((String) fn.get("name"), (String) fn.get("arguments"));
  }
} catch (PermissionDeniedException e) {
  log.error("tool not authorized: {}", e.toString());
} catch (RateLimitException e) {
  log.warn("tool call rate-limited: {}", e.toString());
}

curl -X POST https://api.forgebench.ai/v1/chat/completions \
-H "Authorization: Bearer sk_..." \
-H "Content-Type: application/json" \
-d '{
  "model": "gpt-4o",
  "messages": [{"role": "user", "content": "Find invoices from last quarter."}],
  "tools": [{
    "type": "function",
    "function": {
      "name": "search",
      "description": "Search the KB",
      "parameters": {"type": "object", "properties": {"q": {"type": "string"}}}
    }
  }]
}'

With curl, the refusal is a 403 or 429 with a JSON body; see What a blocked call looks like to the agent.

Which agents may call it

An agent with no binding for a tool can't call it. Bindings take effect on the agent's next call. You can bind from either side:

1. From the tool: open it, then its Bindings tab

   Click Bind agent and choose an agent. Remove one with Unbind on its row.

2. Or from the agent: open its Tools tab, Registry sub-tab

   Click Bind tools and tick the tools it needs. (MCP allowlist next to it is the separate MCP allowlist.)

Disabling a tool (enabled: false) is the kill switch: it blocks every bound agent at once without changing bindings.

Set a call-volume ceiling

A binding decides whether an agent may call a tool. A ceiling optionally caps how often, in calls per window, not dollars.

1. Open the tool, then its Ceilings tab

   Lists the tool's ceilings with current usage.

2. Add ceiling

   Choose a window (per minute, hour, day or month) and a maximum call count. Leave the agent field blank to cap all agents combined, or pick one agent.

3. Save

   The ceiling applies immediately.

A call over the ceiling is refused with 429 (ceiling_exceeded), not 403.

What gets checked

Forgebench checks the response, after the model call. It matches every entry in tool_calls against the agent's bindings. If any entry is unauthorized, the whole response is blocked. Entries that would have passed on their own are recorded as withheld.

Every attempt is recorded in the tool's activity log, including unauthorized ones.

What a blocked call looks like to the agent

A blocked response returns an error; Forgebench doesn't strip tool_calls silently.

• Unauthorized (403): the agent isn't bound to the tool the model tried to call.
• Repeatedly unauthorized (429): the same agent and tool were blocked three or more times in the last ten minutes. This refuses before calling the provider, so retry loops stop costing money. Binding the tool clears it.
• Ceiling reached (429): authorized, but the agent and tool have hit their volume cap for the current window.

Change many tools at once

Select tools in the list to set Tags or a Description, Enable or Disable them, Set ceiling, Grant to agents, or Delete. Disable a tool before deleting it. See Tags and bulk actions.

Next