Bonito Docs

Getting Started

Bonito is a unified AI gateway that connects your AI providers (AWS Bedrock, Azure OpenAI, Google Cloud Vertex AI, OpenAI Direct, Anthropic Direct, and Groq) and lets you manage all your models from a single dashboard. You get one API endpoint, one place to track costs, and one control plane for your entire AI stack.

Quick start (5 minutes)

1. 1Sign up at getbonito.com/register — one account covers your entire organization.
2. 2Go to Providers → Add Provider and connect at least one cloud provider (AWS, Azure, or GCP).
3. 3Bonito validates your credentials and syncs all available models automatically.
4. 4Enable the models you want — click Enable on any model or use bulk activation for up to 20 at once.
5. 5Go to Gateway → Create Key to generate an API key.
6. 6Point any OpenAI-compatible SDK at gateway.getbonito.com/v1 with your new key.

# Make your first request through Bonito
curl -X POST https://getbonito.com/v1/chat/completions \
  -H "Authorization: Bearer YOUR_BONITO_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "anthropic.claude-3-sonnet-20240229-v1:0",
    "messages": [{"role": "user", "content": "Hello!"}],
    "max_tokens": 256
  }'

Provider Setup

Connect your cloud provider accounts so Bonito can discover your available models, route requests, and track costs. Each provider requires different credentials.

AWS Bedrock

To connect AWS, you need an IAM Access Key ID and Secret Access Key. Bonito validates them using STS and checks Bedrock permissions automatically.

Azure OpenAI

Azure requires a service principal with access to your Azure OpenAI resource. The endpoint must be a custom-subdomain URL (e.g., https://your-resource.openai.azure.com/), not a generic regional endpoint.

Google Cloud (Vertex AI)

GCP requires your Project ID and a Service Account JSON key file. Paste the entire JSON contents — Bonito validates the format in the browser before sending.

OpenAI Direct

For teams using OpenAI directly (not through Azure). Connect with just an API key to access GPT-4o, GPT-4o mini, and other OpenAI models.

Anthropic Direct

For teams that want to use Claude models directly through Anthropic without going through AWS Bedrock. Connect with just an API key.

Groq

Ultra-fast inference for open-source models like Llama 3.3 and Mixtral. Groq's LPU hardware delivers the lowest latency in the industry. Connect with just an API key.

Permissions & IAM

Bonito supports two IAM setup modes for every provider. Choose based on your security requirements.

AWS Bedrock permissions

In Enterprise mode, each capability has its own policy so you only grant what you need:

Example IAM policy (core only — minimum to get started):

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "bedrock:ListFoundationModels",
        "bedrock:GetFoundationModel",
        "bedrock:InvokeModel",
        "bedrock:InvokeModelWithResponseStream"
      ],
      "Resource": "*"
    },
    {
      "Effect": "Allow",
      "Action": ["sts:GetCallerIdentity"],
      "Resource": "*"
    }
  ]
}

Azure permissions

Quick Start: Assign Cognitive Services Contributor on the Azure OpenAI resource.

Enterprise: Create a custom role with only the exact permissions Bonito uses — account read, deployments read/write/delete, models read, and inference actions.

az role definition create --role-definition '{
  "Name": "Bonito AI Operator",
  "Actions": [
    "Microsoft.CognitiveServices/accounts/read",
    "Microsoft.CognitiveServices/accounts/deployments/read",
    "Microsoft.CognitiveServices/accounts/deployments/write",
    "Microsoft.CognitiveServices/accounts/deployments/delete",
    "Microsoft.CognitiveServices/accounts/models/read"
  ],
  "DataActions": [
    "Microsoft.CognitiveServices/accounts/OpenAI/deployments/chat/completions/action",
    "Microsoft.CognitiveServices/accounts/OpenAI/deployments/completions/action",
    "Microsoft.CognitiveServices/accounts/OpenAI/deployments/embeddings/action"
  ],
  "AssignableScopes": ["/subscriptions/YOUR_SUBSCRIPTION_ID"]
}'

Optionally add Cost Management Reader at subscription scope for spend visibility.

GCP permissions

Quick Start: Assign roles/aiplatform.user to the service account.

Enterprise: Create a custom role with discovery (publishers.get, publisherModels.get), invocation (endpoints.predict), endpoint management (create/get/list/update/delete/deploy/undeploy), model metadata (models.list, models.get), and project validation (resourcemanager.projects.get).

Model Management

Once a provider is connected, Bonito automatically syncs all available models. You can view, search, filter, and enable models from a single catalog.

One-click model activation

Models with a 🔒 icon exist in your provider's catalog but aren't yet enabled in your cloud account. Instead of switching to each provider's console, enable them directly from Bonito:

1. 1Go to the Models page and find the model you want to enable.
2. 2Click the Enable button on the model card.
3. 3Bonito handles the provider-specific activation (Bedrock entitlements, Azure deployments, GCP API enablement).
4. 4Some models may require approval from the provider and won't activate instantly.

Playground

Test any enabled chat model directly in the browser. The Playground supports single-model chat and side-by-side comparison mode (up to 4 models). Token usage and cost appear after each response. Only chat-capable, enabled models are shown in the picker.

Deployments

Deploy AI models directly into your cloud from the Bonito UI — no console-hopping required. Bonito creates real deployments in your cloud account.

Gateway API

Bonito provides an OpenAI-compatible API endpoint so you can use any connected model with tools that support the OpenAI format. One API key, all your providers.

Endpoint

POST https://getbonito.com/v1/chat/completions

Authentication

Generate API keys from the Gateway page in the dashboard. Include your key in the Authorization header:

Authorization: Bearer YOUR_BONITO_API_KEY

Example: Python (OpenAI SDK)

from openai import OpenAI

client = OpenAI(
    base_url="https://getbonito.com/v1",
    api_key="YOUR_BONITO_API_KEY"
)

response = client.chat.completions.create(
    model="anthropic.claude-3-sonnet-20240229-v1:0",
    messages=[{"role": "user", "content": "Hello!"}]
)

print(response.choices[0].message.content)

Example: curl

curl https://getbonito.com/v1/chat/completions \
  -H "Authorization: Bearer YOUR_BONITO_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "anthropic.claude-3-sonnet-20240229-v1:0",
    "messages": [{"role": "user", "content": "Hello!"}],
    "max_tokens": 256
  }'

Model names

Use the provider-native model IDs shown on the Models page in Bonito. For example: anthropic.claude-3-sonnet-20240229-v1:0 for AWS Bedrock, gpt-4o for Azure, gemini-1.5-pro for GCP.

When using a routing policy, pass the policy name as the model field instead of a specific model ID.

Rate limits from the provider

When the model provider rate-limits a chat completion and failover has nothing left to try, the gateway answers 429 with a short Retry-After header (the provider's own value when it sent one, otherwise 2 seconds, never more than 10), so your client can back off and retry. Before, this came back as a 502. A provider billing or credit error (for example OpenAI's insufficient_quota) is not a rate limit and still returns 502.

Access Tokens

Bonito has four authentication methods. Choose the right one for your use case:

Gateway Keys (bn-...)

Gateway keys authenticate LLM proxy requests to /v1/* endpoints only (chat completions, embeddings, images, video). They resolve to an org but carry no user identity. Create them in Settings → Gateway API Keys.

Limit a key to some providers or models

A key can be limited to certain providers, models, or both. A provider-limited key only ever routes to those providers: model choice, failover, images, video, embeddings and /v1/models all stay inside them, and a model they don't serve is refused with the reason. Name providers by type (groq, aws) or by connection id to pin one account. Browser tokens minted from a limited key inherit its limits. Keys without limits work exactly as before.

# Only Groq, 120 requests a minute
bonito gateway keys create --name fraud-agent --providers groq --rate-limit 120

# Change it later; the key's secret stays the same
bonito gateway keys update fraud-agent --providers groq,deepinfra
bonito gateway keys update fraud-agent --any-provider

Personal Access Tokens (bp-...)

PATs carry your full user permissions and work on all endpoints — both /api/* (platform) and /v1/* (gateway). They are ideal for CI/CD pipelines, scripts, and CLI automation.

1. 1Go to Settings → Personal Access Tokens
2. 2Enter a name and click Generate Token
3. 3Copy the token immediately — it is shown only once
4. 4Use it as: Authorization: Bearer bp-...

CLI: Create tokens with bonito auth token create --name my-token, then authenticate with bonito auth token login --token bp-....

Tier limits: Free = 2 PATs, Pro = 10, Enterprise+ = unlimited. Tokens expire after 90 days by default (max 365). Revoke anytime in Settings or via bonito auth token revoke <id>.

Project Tokens (bj-...)

Project tokens are scoped to a single project. They are created by an org admin and restrict access to only the agents, knowledge bases, and resources within that project. Ideal for giving external teams or CI pipelines access to a specific project without exposing the full org.

Requires: Pro plan or above. Create via API: POST /api/projects/{id}/tokens.

Session Tokens (JWT)

Session tokens are used by the dashboard automatically when you log in. You can also obtain them via bonito auth login (CLI) or POST /api/auth/login. They expire after 24 hours and auto-refresh.

Routing Policies

Routing policies let you automatically select the best model for each request based on your priorities. Create policies from Routing → Create Policy in the dashboard, or declare them in bonito.yaml and run bonito deploy.

routing:
  - name: production
    strategy: failover          # cost | latency | balanced | failover | ab_test
    targets:
      - { provider: aws, model: us.anthropic.claude-sonnet-4-6-v1:0, priority: 1 }
      - { provider: azure, model: gpt-4o, priority: 2 }
  - name: internal-cheap
    strategy: cost
    targets:
      - { provider: gcp, model: gemini-2.5-flash }
    rules: { max_tokens: 2000 }

Policies are matched by name, so deploying again updates them instead of creating copies. A target names a model by its model ID or display name; add provider when the same model is on more than one provider, and provider_id to pin one of several accounts on the same provider. The first target by priority is the primary, the rest are fallbacks. A/B targets take a weight, and the weights must sum to 100. The block can also sit under gateway:.

Cross-Region Inference

Bonito automatically creates cross-region inference profiles for AWS Bedrock models. If us-east-1 is overloaded or experiencing downtime, requests seamlessly route to us-west-2 or eu-west-1 without any action on your part.

How it works

• Bonito handles the us. prefix routing at the gateway level, so cross-region inference profiles are created and managed automatically.
• No configuration needed — just connect your AWS provider and Bonito takes care of the rest.
• If a primary region is throttled or returns errors, requests are retried in an alternate region transparently.

True high-availability

Cross-region inference combines with multi-provider failover for true high-availability. For example, if Claude on Bedrock fails across all regions, Bonito can fall back to Anthropic Direct automatically. No other platform (Portkey, LiteLLM, Helicone) offers automatic cross-region inference profiles with intelligent failover built in.

Data Residency

Pin an organization's model calls to one geography: CA (Canada), US or EU. Bonito then loads only the provider connections whose region is in that geography. Routing, failover, agents, the playground and knowledge-base embeddings all draw from that set, so a request has no path to a provider outside it. It is off by default.

Turn it on

# CLI (org admins)
bonito gateway residency set CA
bonito gateway residency show      # every provider, whether it serves, and why

# API
curl -X PUT https://api.getbonito.com/api/gateway/config \
  -H "Authorization: Bearer $TOKEN" -d '{"data_residency": "CA"}'

gateway:
  data_residency: CA
  providers:
    - name: aws
      region: ca-central-1
      access_key: ${AWS_ACCESS_KEY_ID}
      secret_key: ${AWS_SECRET_ACCESS_KEY}
    - name: gcp
      region: northamerica-northeast1
      project_id: ${GCP_PROJECT}
      service_account_file: ./vertex-sa.json

In the dashboard it is under Settings, Data Residency. Self-hosted installs can set BONITO_DATA_RESIDENCY=CA on the backend to lock it for every organization.

Which connections count as in-region

• AWS Bedrock: the connection's region (ca-central-1 and ca-west-1 are CA). Models AWS only serves through a cross-region profile are used when a profile exists inside the geography (us. for US, eu. for EU). AWS has no Canada profile, so under CA only models served in the region itself are routable.
• Google Vertex: the connection's region (Montreal and Toronto are CA). Every model is called in that region; the global endpoint is never used.
• Azure OpenAI: set the connection's region (for example canadaeast). Only Standard and Provisioned deployments are used; Global and Data Zone deployments process elsewhere and are skipped.
• OpenAI, Anthropic, Groq, OpenRouter, DeepInfra, DeepSeek, Qwen, Kimi, MiniMax, GLM, TypeSafe and managed providers make no residency guarantee and are skipped.

Zero Retention

For teams that keep their own conversation history and do not want a second copy held by Bonito. Set a project's retention to none and its agents run exactly as before, but nothing they were told or said is stored: run records keep the model, tokens, cost, latency and tool names, without the message text or tool arguments, and conversation memory is not written. Because nothing is kept, each run starts without history, so send any earlier context you want the agent to see with the message. The default is full.

# CLI
bonito projects update <project-id> --retention none

# API (settings is replaced as a whole, so include your other keys)
curl -X PUT https://api.getbonito.com/api/projects/<project-id> \
  -H "Authorization: Bearer $TOKEN" -d '{"settings": {"retention": "none"}}'

retention: none

Bring Your Own Retriever

Keep a knowledge base's passages in your own database. Point the knowledge base at a search endpoint you run, and every search of it (an agent's per-turn context, its knowledge search tool and the search API) asks your endpoint instead of reading passages stored on Bonito. Agents, routing, governance and analytics work exactly as before; the passages travel per search and are not stored.

# CLI (the secret is read from an environment variable, never a flag)
bonito kb retriever <kb> --url https://search.example.com/bonito --secret-env KB_RETRIEVER_SECRET
bonito kb retriever <kb> --test
bonito kb retriever <kb> --remove      # search the passages stored on Bonito again

# API (a test search must succeed before anything is saved)
curl -X PUT https://api.getbonito.com/api/knowledge-bases/<kb-id>/retriever   -H "Authorization: Bearer $TOKEN" -d '{"url": "https://search.example.com/bonito", "secret": "..."}'

knowledge_bases:
  support-docs:
    retriever:
      url: https://search.example.com/bonito
      secret_env: KB_RETRIEVER_SECRET

Bonito sends a signed POST and expects your top passages back. Verify the signature, reject timestamps older than five minutes, and return at most limit results with a score between 0 and 1.

POST <your url>
X-Bonito-Timestamp: 1760000000
X-Bonito-Signature: v1=<hex HMAC-SHA256(secret, "<timestamp>.<raw body>")>

{"knowledge_base_id": "...", "query": "when do we close on Thursdays", "limit": 5}

200 {"results": [{"content": "We close at 4:15 pm on Thursdays.", "score": 0.61,
                  "source_name": "hours.md", "document_id": "...", "chunk_index": 0}]}

Agents in Your Clouds

AWS and Google each govern the agents built on their own platform. Bonito lists them across both, next to your Bonito agents, using the AWS and Google Cloud connections you already have. It reads and never changes anything: AgentCore runtimes and gateways, Bedrock Agents, Agent Registry agents and MCP servers, and Agent Engine deployments. Prompts, environment variables and secrets are never stored. Available on Pro and above; the list refreshes every 6 hours or when you scan.

bonito agents inventory --sync          # scan now, then list
bonito agents inventory --findings      # only what needs attention

curl -X POST https://api.getbonito.com/api/agent-inventory/sync -H "Authorization: Bearer $TOKEN"
curl https://api.getbonito.com/api/agent-inventory?finding=any -H "Authorization: Bearer $TOKEN"

gateway:
  providers:
    - name: aws
      access_key: ${AWS_ACCESS_KEY_ID}
      secret_key: ${AWS_SECRET_ACCESS_KEY}
      region: us-east-1
      inventory_regions: [us-west-2, ca-central-1]   # scanned as well as us-east-1
    - name: gcp
      project_id: my-project
      service_account_file: ./sa.json
      inventory: false                               # don't scan this one

Each item is checked against your own rules. Findings: an agent outside your data residency geography, a model your model-access policy does not allow, an AgentCore gateway that accepts calls with no sign-in or has no policy engine (or one that only logs), nothing updated in 90 days, and anything removed from the cloud.

Let a Bonito agent call them

Connect a Bonito agent to an AgentCore runtime or an Agent Engine agent and it appears in that agent's invoke_agent tool next to your Bonito agents. Every call goes through Bonito: only connected agents can be called, the agent's tool policy applies, an agent outside your data residency geography or removed from the cloud is refused, and the hop is recorded in the calling agent's transcript and trace. Bonito signs each call with that cloud's own credentials (AWS signature, Google OAuth) and tells AgentCore which Bonito agent is calling. Point the cloud agent's model at the Bonito gateway and its model calls get routing, failover and cost tracking too. Agents in Agent Registry that publish an A2A address (for example on Cloud Run) are called with A2A message/send, with a Google ID token when the address is Google-hosted.

Failover between agents. Give a connection a fallback: another connected cloud agent or a Bonito agent in the same project. If the call fails, or data residency refuses it, Bonito asks the fallback instead and the result says so (failover: from, to, reason). Point a US agent's fallback at its Canadian twin and a Canada-only org is answered in Canada.

The other direction. An agent on another cloud calls a Bonito agent with POST /api/agents/{id}/execute and a project token (bj-, limited to one project, rate-limited, revocable); pass caller and it is recorded on the session. That Bonito agent can in turn call agents on a third cloud.

bonito agents cloud-agents <bonito-agent-id> --add <inventory-id> [--fallback <bonito-agent-id>|cloud:<inventory-id>]
curl -X PUT https://api.getbonito.com/api/agents/<agent-id>/cloud-agents/<inventory-id> -H "Authorization: Bearer $TOKEN"

agents:
  concierge:
    system_prompt: "Ask the claims agent about claims and the payments agent about payments."
    cloud_agents:
      - claims-assistant                              # name as the inventory lists it
      - { name: payments, cloud: aws, fallback: payments-backup }   # a Bonito agent in this file, or another cloud agent

Agents calling each other across clouds

The gateway routes model calls; the relay routes agent calls. An agent on AgentCore, Agent Engine or any A2A host calls another agent with POST /v1/agents/{id}/invoke and its own project token. Bonito checks that a route allows that caller to reach that agent, refuses a target outside your data residency or removed from the cloud, holds a burst while the target scales (each calling token is limited to its own rate limit per minute; calls beyond a target's in-flight cap wait their turn; the cloud's busy answers are retried, all within 110 seconds; see below), answers from the route's fallback when the target fails or stays busy, signs the call with the target cloud's credentials, and records every hop (caller, target, cloud, region, status, time) without keeping the content. Chains stop after 4 hops.

# allow the claims agent's token to call the payments agent, with a fallback
bonito agents relay add --token <project-token-id> --target <inventory-id> --fallback <bonito-agent-id>

# from the calling agent
curl -X POST https://api.getbonito.com/v1/agents/cloud:<inventory-id>/invoke   -H "Authorization: Bearer bj-..." -d '{"message": "Has claim C-3310 been paid?"}'
# -> {"answer": "...", "agent": "payments", "runs_on": "aws:agentcore_runtime", "chain_id": "...", "hop": 0}

# an agent called this way calls onward with the chain_id and hop it was given
bonito agents relay calls --chain <chain-id>

Or declare it in a version: "2" bonito.yaml, by name: tokens mints each calling agent's project token once (its secret is shown once, after apply; an existing token of that name is reused), and relay sets which agents each token may call. Agents are named as the inventory lists them, with cloud when two share a name, or by resource (the ARN or Google resource name, or its last part). A fallback is an agent in the same file (that wins over a cloud agent of the same name) or another cloud agent. A token has one route to each target. Routes not in the file are left alone. Needs bonito-cli 0.15.0 or later; older versions reject the two sections.

version: "2"
name: claims-cluster
tokens:
  - name: claims-runtime                 # the Google claims agent calls with this
    rate_limit: 120
agents:
  payments-backup:
    model_id: claude-haiku-4-5
    system_prompt: Answer payment status questions in one sentence.
relay:
  - caller: claims                        # shown on the route; the call log names the token
    cloud: gcp
    token: claims-runtime
    targets:
      - { name: payments, cloud: aws, fallback: payments-backup }
      - { resource: "runtime/fraud-check-x1" }

How many calls at once

Cloud runtimes scale out but do not queue, so Bonito caps the calls it has in flight to each cloud agent and holds the rest until a slot frees. The cap follows what the target can run: an Agent Engine deployment gets its max instances x container concurrency (9 per instance when the spec sets max instances only), an AgentCore runtime gets 16, and anything else, including an Agent Engine deployment with no known spec and A2A agents, gets 8. A cap worked out this way is at most 64. An org admin can set a cap of their own, from 1 to 256, which wins until it is cleared. The inventory shows each callable agent's cap and where it came from (relay.max_in_flight, relay.source, relay.override).

The cloud's refusals at the front door (before the agent runs) are retried with backoff; an error from inside a run is never retried, because the agent may already have acted. A quota refusal from Agent Engine pauses every call to that Google project and region for a few seconds (2s, doubling to at most 16s, at most 3 retries) instead of spending more quota. When the target stays busy, or a pause is longer than the time left, the relay answers 429 with a Retry-After header and the body says why: reason is queue, quota, throttle, capacity or time, next to retry_after in seconds. When the route has a fallback, the fallback answers instead of a 429. The same cap, retries and pause apply to every Bonito call to a cloud agent: the relay, its cloud fallback, and invoke_agent from a Bonito agent (which waits at most 20 seconds for a slot).

bonito agents inventory                                  # "Relay cap" column
bonito agents relay capacity <inventory-id> --max-in-flight 40
bonito agents relay capacity <inventory-id> --clear    # back to the cap worked out from the runtime

curl -X PUT https://api.getbonito.com/api/agent-inventory/<inventory-id>/relay-capacity \
  -H "Authorization: Bearer $TOKEN" -H "Content-Type: application/json" -d '{"max_in_flight": 40}'
# {"max_in_flight": null} removes the override

In a version: "2" bonito.yaml, a top-level cloud_agents list sets the override by name (with cloud when two share a name) or by resource. Agents left out of the list keep what they have. Needs bonito-cli 0.16.0 or later.

version: "2"
name: claims-cluster
cloud_agents:
  - { name: payments, cloud: aws, max_in_flight: 40 }
  - { resource: "reasoningEngines/1234567890", max_in_flight: 18 }

Decisions (Jev)

Some model calls are really decisions your code consumes: which team gets this ticket, is this message urgent, how risky is this action. Connect a TypeSafe provider and send those to POST /v1/systemone with the same bn- key. Jev reads a state and answers typed questions with calibrated probabilities: noul (probability of yes), choice (one of up to 255 options) and score (an ordered rubric). Every question in a request is evaluated in parallel against the same state. Jev does not generate text; a chat request to a Jev model returns 400.

bonito providers add typesafe --api-key $TYPESAFE_API_KEY

curl https://api.getbonito.com/v1/systemone \
  -H "Authorization: Bearer bn-..." -d '{
    "model": "jev-latest",
    "state": "Help! My payouts have been failing for 3 days.",
    "questions": {
      "urgent": {"type": "noul", "instructions": "Does this convey urgency?"},
      "team": {"type": "choice", "instructions": "Which team should handle this?",
                "criteria": {"billing": "Payments, refunds", "technical": "Bugs, outages"}}
    }
  }'

bonito gateway decide -f request.json   # same request from the CLI

Requests are logged, costed (input tokens only) and counted against your plan like any other gateway call. Use the confidence on each answer to decide in code whether to act, escalate to a larger model, or ask a person.

Model Aliases

Bonito supports shorthand model aliases that resolve to provider-specific versioned model IDs. This means you can reference models by simple names and switch providers without changing your code.

How aliases work

When you send a request with an alias like claude-sonnet, Bonito resolves it to the correct model ID for your active provider. On Bedrock, that becomes anthropic.claude-3-sonnet-20240229-v1:0. On Anthropic Direct, it becomes claude-3-sonnet-20240229.

• Switch providers without changing a single line of code in your application.
• Aliases are updated automatically when new model versions are released.
• Works with routing policies — aliases resolve before routing rules are applied.

from openai import OpenAI

client = OpenAI(
    base_url="https://getbonito.com/v1",
    api_key="YOUR_BONITO_API_KEY"
)

# Use an alias instead of a provider-specific model ID
response = client.chat.completions.create(
    model="claude-sonnet",  # Resolves automatically
    messages=[{"role": "user", "content": "Hello!"}]
)

Notifications

Bonito sends in-app notifications for important events across the platform so you never miss a deployment status change or cost alert.

Notification types

• Deployment lifecycle — creation, scaling, completion, and failure alerts for deployments across all providers.
• Spend alerts — get notified when costs approach or exceed your configured budget thresholds.
• Model activation — confirmation when models are enabled or if activation requires provider approval.
• Provider health — alerts when a provider connection has issues or needs credential rotation.

The notification bell in the dashboard header shows your unread count. Click to see the full list with read/unread states. You can configure alert rules for budget thresholds with email and in-app delivery preferences.

Cost Management

Monitor AI spending across all connected providers from a single dashboard. Bonito pulls real cost data from your cloud accounts and shows breakdowns by model, provider, and time period.

What you get

• Aggregated costs across AWS, Azure, and GCP with daily/weekly/monthly views.
• Cost forecast with projected spending and confidence bounds.
• Per-model and per-provider breakdowns to identify expensive workloads.
• Budget alerts — set thresholds and get notified before you exceed them.
• Optimization recommendations — Bonito suggests cheaper model alternatives and cross-provider routing opportunities.

CLI Tool

bonito-cli is a Python CLI for managing your Bonito resources from the terminal. It's useful for scripting, CI/CD pipelines, and terminal-first workflows.

Installation

pip install bonito-cli

Authentication

# Login with your Bonito credentials
bonito auth login

# Or set your API key directly
export BONITO_API_KEY=your-key-here

Common commands

# List connected providers
bonito providers list

# List available models
bonito models list

# Create a gateway API key
bonito gateway keys create --name "my-key"

# List routing policies
bonito routing list

# Check costs
bonito costs summary

# Preview a bonito.yaml stack, then apply it
bonito deploy -f bonito.yaml --plan
bonito deploy -f bonito.yaml

Run bonito --help for the full list of commands and options.

Integrations

Claude CoworkOpenClaw

Claude Cowork is Anthropic's agentic desktop application. The Bonito plugin gives Claude deep knowledge of your AI infrastructure, so you can deploy providers, create agents, configure routing, and analyze costs through natural conversation.

Install the Plugin

Install from the Claude plugin marketplace or via Claude Code:

claude plugin install bonito

What the Plugin Adds

6 domain skills that Claude draws on automatically when relevant:

• deploy-stack: Deploy infrastructure from a bonito.yaml config
• manage-providers: Connect and manage cloud AI providers
• create-agent: Create BonBon agents and Bonobot orchestrators
• gateway-routing: Configure failover, cost-optimized routing, A/B testing
• cost-analysis: Analyze spending and recommend optimizations
• debug-issues: Troubleshoot gateway, provider, and agent problems

MCP Server

The plugin connects to the Bonito MCP server, which exposes 38 tools: providers, models, gateway keys, data residency, projects, agents (create, update, run, read a queued run), knowledge bases (create, upload, search, link), cloud agents, the relay (routes, call records, per-agent capacity), and stacks (plan_stack and apply_stack for a whole bonito.yaml). Use a personal access token (bp-) as the key; it works for both the management API and the gateway. Install via PyPI or Docker:

pip install bonito-mcp

Or run with Docker:

docker run -e BONITO_API_KEY=your-key -p 8080:8080 bonitoai/mcp-server

Claude Desktop Configuration

Add this to your Claude Desktop MCP config:

{
  "mcpServers": {
    "bonito": {
      "command": "bonito-mcp",
      "env": {
        "BONITO_API_KEY": "bp-your-personal-access-token"
      }
    }
  }
}

Tip

The MCP server works with any MCP-compatible client including Cursor, Windsurf, and other tools that support the Model Context Protocol.

Learn More

• Claude Cowork
• Bonito MCP on PyPI
• Bonito MCP on Docker Hub
• Plugin source on GitHub

Declarative Config (bonito.yaml)

Define your entire AI stack in a single bonito.yaml file. Providers, agents, MCP servers, knowledge bases, routing and gateway keys, all in one place. Review the plan, then deploy with a single command.

bonito deploy -f bonito.yaml --plan     # what would change; writes nothing
bonito deploy -f bonito.yaml   # apply it

What the file can say

Top-level sections: gateway (providers, data_residency, keys), routing, knowledge_bases, mcp_servers, agents, integrations, secrets, retention, and in version 2 tokens, relay and cloud_agents (see Agents in your clouds). An agent takes model_id, system_prompt (inline, or a path to a file), model_config, tools (mode and the engine tool names: search_knowledge_base, search_web, http_request, invoke_agent, delegate_task, use_skill, save_artifact, send_notification), rag, mcp_servers, delegates, cloud_agents, rate_limit_rpm and scaling.

A deploy is repeatable: the stack deploys into a project named after it, a second deploy updates providers, agents, routing policies, keys and knowledge bases in place, and nothing is created twice. version: "2" files refuse unknown keys; version 1 files warn about them and deploy the rest.

Example: AWS Bedrock stack

A complete stack using AWS Bedrock with two agents, a knowledge base and an MCP server:

version: "2"
name: my-ai-stack

gateway:
  providers:
    - name: aws
      access_key: ${AWS_ACCESS_KEY_ID}
      secret_key: ${AWS_SECRET_ACCESS_KEY}
      region: us-east-1

routing:
  - name: production
    strategy: failover
    targets:
      - { provider: aws, model: anthropic.claude-sonnet-4-5-20250929-v1:0, priority: 1 }
      - { provider: aws, model: anthropic.claude-haiku-4-5-20251001-v1:0, priority: 2 }

knowledge_bases:
  company-docs:
    description: Internal documentation
    sources:
      - type: directory
        path: ./docs/
        glob: "**/*.md"

agents:
  support-bot:
    display_name: Support Agent
    model_id: anthropic.claude-sonnet-4-5-20250929-v1:0
    system_prompt: |
      You are a helpful support agent. Answer from the knowledge base and
      escalate to a human when you are not sure.
    model_config:
      temperature: 0.3
      max_tokens: 1024
    tools:
      mode: allowlist
      allowed: [search_knowledge_base]
    rag:
      knowledge_base: company-docs

  code-reviewer:
    display_name: Code Reviewer
    model_id: anthropic.claude-sonnet-4-5-20250929-v1:0
    system_prompt: prompts/code-reviewer.md
    mcp_servers: [github]

mcp_servers:
  github:
    url: https://mcp.example.com/github
    auth:
      token: ${GITHUB_MCP_TOKEN}

Gateway keys (version 2)

version: "2" files can declare gateway keys. Keys are matched by name: a new key is created and its secret printed once, and an existing key is updated to match the file (its secret never changes). Keys not in the file are left alone. Version 1 files deploy exactly as before.

version: "2"
name: my-ai-stack

gateway:
  keys:
    - name: fraud-agent
      providers: [groq]          # only ever routed to Groq
      rate_limit: 120
    - name: claims-agent
      models: [claude-haiku-5-5]  # only this model, any provider serving it

Example: Multi-provider stack

A stack using Groq, Anthropic, OpenAI, and GCP with multi-agent orchestration:

version: "2"
name: multi-provider-stack

gateway:
  providers:
    - name: groq
      api_key: ${GROQ_API_KEY}
    - name: anthropic
      api_key: ${ANTHROPIC_API_KEY}
    - name: openai
      api_key: ${OPENAI_API_KEY}
    - name: gcp
      project_id: ${GCP_PROJECT_ID}
      service_account_file: ./gcp-service-account.json
      region: us-central1

routing:
  - name: everyday
    strategy: cost
    targets:
      - { provider: gcp, model: gemini-2.5-flash }
      - { provider: openai, model: gpt-4.1-mini }

agents:
  orchestrator:
    display_name: Operations Center
    model_id: claude-sonnet-5
    system_prompt: |
      You run operations. Hand triage to fast-responder and analysis to deep-analyst.
    delegates:
      - agent: fast-responder
        domains: [triage, alerts, summaries]
      - agent: deep-analyst
        domains: [code-review, analysis, research]

  fast-responder:
    display_name: Fast Responder
    model_id: openai/gpt-oss-120b
    system_prompt: You answer triage questions in one short paragraph.

  deep-analyst:
    display_name: Deep Analyst
    model_id: claude-sonnet-5
    system_prompt: prompts/deep-analyst.md
    mcp_servers: [github, jira]

mcp_servers:
  github:
    url: https://mcp.example.com/github
    auth: { token: ${GITHUB_MCP_TOKEN} }
  jira:
    url: https://mcp.example.com/jira
    auth: { type: api_key, key: ${JIRA_MCP_KEY} }

Plan, then apply

bonito deploy --plan shows what a deploy would create, change or leave alone, without changing anything. It runs the same engine as bonito deploy against your live org and records each write instead of sending it, so the plan is exactly what the deploy will do. Add --json for CI.

bonito deploy -f bonito.yaml --plan

                                Plan - claims-stack
┏━━━┳━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃   ┃ Resource       ┃ Name          ┃ Change                                      ┃
┡━━━╇━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│ + │ agent          │ Stack Billing │ model anthropic.claude-haiku-4-5, in        │
│   │                │               │ project claims-stack                        │
│ ~ │ gateway key    │ claims-key    │ rate_limit: 30 -> 60                        │
│ ~ │ agent          │ Stack Claims  │ model_config.temperature: 0.2 -> 0.5        │
│ ↻ │ provider       │ groq          │ credentials re-sent and re-verified; stored │
│   │                │               │ secrets can't be compared                   │
│ = │ knowledge base │ claims-kb     │ exists; its settings and documents are left │
│   │                │               │ as they are                                 │
│ = │ agent          │ Stack Triage  │                                             │
└───┴────────────────┴───────────────┴─────────────────────────────────────────────┘

1 to create, 2 to change, 1 to refresh, 2 unchanged. plan 2eb2c671a077511e

• +Created.
• ~Changed. Each line is field: from -> to. Only the fields the file sets are compared; a block such as model_config is replaced whole, so a key the file drops shows as going to null.
• ↻Refreshed. Provider credentials and retriever secrets are re-sent and re-verified on every apply, because stored secrets can't be read back to compare.
• =Unchanged.

Secrets always show as (secret). A new gateway key's secret appears once, after apply.

The plan ends with a hash of its changes. When you apply from the dashboard's Stacks page, the API or Claude, the server plans again first and refuses if the result no longer matches the plan you reviewed, so nothing you did not see gets applied.

Over the API, send the file, the environment variables it references and the files it points at (prompt files, knowledge base documents) to POST /api/stacks/plan, then the same body plus plan_hash to POST /api/stacks/apply. The server reads nothing from its own disk or environment: a ${VAR} resolves only from env, and a path resolves only from files (base64 for binary files).

{
  "yaml": "version: \"2\"\nname: claims-stack\n...",
  "env": { "GROQ_API_KEY": "gsk_..." },
  "files": {
    "prompts/triage.md": "Route each claim to the right specialist.",
    "docs/policy.pdf": { "content": "JVBERi0xLjcK...", "encoding": "base64" }
  },
  "plan_hash": "2eb2c671a077511e"
}

Apply answers with the per-resource results, the agent ids, any new gateway key and agent token secrets in new_keys (shown once; agent tokens carry kind: "project token") and the deploy log. It returns 409 plan_changed with a fresh plan when the org changed since your plan, 409 apply_running while another apply for your org is running, and 422 plan_invalid when the file has errors.

With Claude, bonito-mcp 1.5.0 adds plan_stack and apply_stack (the server now has 38 tools). Ask Claude to deploy a stack: it plans first, you review the changes, and it applies with that plan's hash. Secrets stay as ${VARS}: list their names in BONITO_STACK_ENV in the MCP server's config and set the values there. Only those variables, and only files inside the stack's folder, are ever sent.

BonBon Agents

BonBon is Bonito's managed agent service. Create AI agents with custom system prompts, connect them to knowledge bases for RAG, and deploy them as embeddable chat widgets or API endpoints — all without managing infrastructure.

Agent tiers

Creating an agent

1. 1Go to Agents → Create Agent in the dashboard.
2. 2Choose a tier (Simple or Advanced) and give your agent a name.
3. 3Write a system prompt that defines your agent's personality, constraints, and behavior.
4. 4Optionally attach a knowledge base for RAG-powered responses.
5. 5Select the backing model (or models for Advanced tier).
6. 6Deploy — Bonito gives you a widget embed code and an API endpoint.

# Create an agent via CLI
bonito agents create \
  --name "Support Bot" \
  --tier simple \
  --model "anthropic.claude-3-sonnet-20240229-v1:0" \
  --system-prompt "You are a helpful support agent for Acme Corp..."

# List your agents
bonito agents list

# Get agent details
bonito agents get --id ag_abc123

System prompts

System prompts define how your agent behaves. Write clear instructions about the agent's role, tone, constraints, and what it should or shouldn't do. You can update the system prompt at any time without redeploying.

You are a customer support agent for Acme Corp.

Rules:
- Only answer questions about Acme products and services.
- If you don't know the answer, say so and offer to connect with a human.
- Be friendly, concise, and professional.
- Never make up pricing or feature information — use the knowledge base.
- Respond in the same language as the customer.

RAG integration

Connect a knowledge base to give your agent access to your documents. When a user asks a question, Bonito retrieves relevant chunks from your KB and includes them in the context before the model generates a response.

# Attach a knowledge base to an agent
bonito agents update --id ag_abc123 \
  --knowledge-base kb_xyz789

# Attach multiple KBs (Advanced tier only)
bonito agents update --id ag_abc123 \
  --knowledge-base kb_xyz789 \
  --knowledge-base kb_docs456

Widget embedding

Every BonBon agent gets an embeddable chat widget. Add it to any website with a single script tag:

<!-- Add to your website -->
<script
  src="https://getbonito.com/widget.js"
  data-agent-id="ag_abc123"
  data-theme="dark"
  data-position="bottom-right"
  async
></script>

Bonobot Orchestrator

Bonobot is Bonito's multi-agent orchestration layer. It acts as a front-door agent that classifies user intent, delegates to specialized sub-agents, and synthesizes their responses into a unified reply. Think of it as a dispatcher that routes conversations to the right expert.

How it works

1. 1A user sends a message to the Bonobot endpoint.
2. 2The orchestrator classifies the user's intent using a fast classification model.
3. 3Based on the intent, it delegates the request to one or more specialized BonBon agents.
4. 4Each sub-agent processes its part using its own system prompt, model, and knowledge base.
5. 5Bonobot synthesizes the responses and returns a single, coherent answer.

Delegation map

The delegation map defines which sub-agents handle which intents. Configure it as a JSON mapping of intent patterns to agent IDs:

{
  "delegation_map": {
    "billing": {
      "agent_id": "ag_billing01",
      "description": "Handles billing, invoices, and payment questions"
    },
    "technical_support": {
      "agent_id": "ag_techsup01",
      "description": "Handles technical issues, bugs, and troubleshooting"
    },
    "sales": {
      "agent_id": "ag_sales01",
      "description": "Handles pricing, demos, and feature inquiries"
    },
    "general": {
      "agent_id": "ag_general01",
      "description": "Fallback for anything that doesn't match a specific intent"
    }
  }
}

Creating a Bonobot

# Create an orchestrator
bonito bonobot create \
  --name "Customer Hub" \
  --classifier-model "anthropic.claude-3-haiku-20240307-v1:0" \
  --delegation-map ./delegation.json

# Update the delegation map
bonito bonobot update --id bot_abc123 \
  --delegation-map ./updated-delegation.json

# Test intent classification
bonito bonobot classify --id bot_abc123 \
  --message "I need a refund for my last invoice"

Response synthesis

When a request touches multiple intents (e.g., "I want to upgrade my plan and fix a bug"), Bonobot can delegate to multiple agents in parallel and merge their responses. Enable multi-delegation in the orchestrator settings:

{
  "multi_delegation": true,
  "synthesis_model": "anthropic.claude-3-sonnet-20240229-v1:0",
  "synthesis_prompt": "Combine the following specialist responses into a single, coherent answer."
}

Agent Burst Limits and Queue

Every agent has a per-minute limit, rate_limit_rpm (default 30). With autoscaling on, that limit stretches while the agent is busy and shrinks back when it is quiet, and requests over it wait in an overflow queue instead of being refused. Nothing extra is started: the same agent takes more requests per minute. The API keeps its existing names (autoscale_enabled, max_replicas), where max_replicas is the multiplier on the base limit.

How the limit stretches

1. 1Each request is counted against the agent's current minute.
2. 2When the count reaches capacity_threshold (default 60%) of the current limit, the limit doubles, up to max_replicas x rate_limit_rpm.
3. 3Every 30 seconds Bonito checks raised agents. One whose use is below scale_down_threshold (default 30%), and that has not scaled for scale_down_cooldown_seconds (default 300), has its limit halved, never below rate_limit_rpm.
4. 4A request over the limit is queued (autoscaling on) or refused with 429 and a Retry-After that runs to the next minute (autoscaling off, queue full, or a zero-retention project).

Example: a triage agent with rate_limit_rpm: 30 and max_replicas: 5. At 18 requests in a minute it moves to 60, at 36 to 120, and it stops at 150. After the burst it steps back down to 30.

Configuration via bonito.yaml

agents:
  triage-router:
    model_id: gpt-4o-mini
    system_prompt: agents/triage-router/system-prompt.md
    rate_limit_rpm: 30
    scaling:
      enabled: true
      capacity_threshold: 0.6           # double the limit at 60% use
      scale_down_threshold: 0.3         # halve it below 30%
      max_replicas: 5                   # limit never above 5 x 30 = 150 per minute
      scale_down_cooldown_seconds: 300  # wait 5 minutes after a change before halving
      mode: virtual                     # the only mode

Configuration via API

# Turn it on (Enterprise and Scale)
curl -X POST https://api.getbonito.com/api/agents/{agent_id}/scaling/configure \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "enabled": true,
    "capacity_threshold": 0.6,
    "scale_down_threshold": 0.3,
    "scale_down_cooldown_seconds": 300,
    "max_replicas": 5,
    "mode": "virtual"
  }'

# Turn it off (any plan)
curl -X POST https://api.getbonito.com/api/agents/{agent_id}/scaling/configure \
  -H "Authorization: Bearer $TOKEN" -H "Content-Type: application/json" -d '{"enabled": false}'

# Current limit and scaling events
curl https://api.getbonito.com/api/agents/{agent_id}/scaling -H "Authorization: Bearer $TOKEN"
curl https://api.getbonito.com/api/agents/{agent_id}/scaling/events -H "Authorization: Bearer $TOKEN"

# Double or halve the limit by hand (409 while autoscaling is off)
curl -X POST https://api.getbonito.com/api/agents/{agent_id}/scaling/manual \
  -H "Authorization: Bearer $TOKEN" -H "Content-Type: application/json" \
  -d '{"direction": "up"}'

The same settings are accepted as autoscale_enabled and autoscale_config when creating or updating an agent. Unknown keys and values out of range are refused with 422.

CLI commands

bonito agents scaling configure <agent-id> --threshold 0.6 --scale-down 0.3 \
  --max-replicas 5 --cooldown 300 --enable
bonito agents scaling configure <agent-id> --disable
bonito agents scaling status <agent-id>
bonito agents scaling events <agent-id> --limit 20
bonito agents scaling manual <agent-id> up

Settings

Execution responses report the limit in the security object: rate_limit_remaining, effective_rpm and scaling_active.

Overflow queue

Only POST /api/agents/{id}/execute queues, and only for an agent with autoscaling on. A request over the limit gets 202 Accepted and a ticket. The queue runs as the agent's minute window frees up, so most tickets run within the next minute. Tickets in the same session run one at a time, in order. A project with zero retention is never queued, because the queue would hold the message and the answer; it gets 429 with Retry-After instead.

// POST /api/agents/{id}/execute  ->  202 Accepted
{
  "status": "queued",
  "ticket_id": "3f0c...",
  "position": 4,
  "estimated_wait_seconds": 8,
  "poll_url": "/api/agents/{id}/queue/3f0c...",
  "message": "Agent at capacity. Request queued at position 4."
}

Polling for results

Poll the ticket with the same credentials. A project token (bj-) can poll tickets and read the queue depth for agents in its own project.

curl https://api.getbonito.com/api/agents/{agent_id}/queue/{ticket_id} \
  -H "Authorization: Bearer $TOKEN"

# waiting, or running (position 0)
{ "status": "queued", "ticket_id": "3f0c...", "position": 2, "estimated_wait_seconds": 4, "requeues": 0 }
{ "status": "processing", "ticket_id": "3f0c...", "position": 0, "estimated_wait_seconds": 0, "requeues": 0 }

# done
{ "status": "completed", "ticket_id": "3f0c...",
  "result": { "content": "...", "tokens": 812, "cost": "0.0021", "turns": 1, "model_used": "...",
              "effective_rpm": 60, "scaling_active": true } }

# failed, with the reason
{ "status": "failed", "ticket_id": "3f0c...", "error": "expired: waited more than 50 minutes in the queue" }

# queue depth
curl https://api.getbonito.com/api/agents/{agent_id}/queue -H "Authorization: Bearer $TOKEN"
{ "agent_id": "...", "queue_depth": 12 }

bonito agents execute <agent-id> "Triage ticket 4411"            # waits for a queued run (--wait-timeout, default 180s)
bonito agents execute <agent-id> "Triage ticket 4411" --no-wait  # prints the ticket and returns
bonito agents queue get <agent-id> <ticket-id> --wait
bonito agents queue depth <agent-id>

Queue behaviour

Code Review

Bonito's GitHub App provides AI-powered code review on pull requests. Install it from the Bonito dashboard, connect your GitHub repos, and get automatic reviews on every PR.

What it does

• Automatically reviews PRs for security vulnerabilities, performance issues, and code quality.
• Posts structured findings directly as PR comments with clear explanations and suggestions.
• Multiple review personas available — default professional tone, or fun characters for a lighter touch.
• Configure which repos and branches trigger automatic reviews from the dashboard.

Getting started

1. 1Go to Code Review in the Bonito dashboard.
2. 2Click Install GitHub App and authorize access to your repositories.
3. 3Select which repos and branches should trigger automatic reviews.
4. 4Open a pull request — Bonito reviews it and posts comments within minutes.

MCP Integration

MCP (Model Context Protocol) lets your BonBon agents call external tools — databases, APIs, code execution, file systems, and more. Register MCP servers with Bonito and connect them to your agents so they can take actions, not just answer questions.

Supported transports

Registering an MCP server

{
  "name": "github-tools",
  "transport": "sse",
  "url": "https://mcp.example.com/github/sse",
  "headers": {
    "Authorization": "Bearer ghp_xxxxxxxxxxxx"
  },
  "tools": ["create_issue", "search_repos", "get_pull_request"]
}

# Register via CLI
bonito mcp register \
  --name "github-tools" \
  --transport sse \
  --url "https://mcp.example.com/github/sse"

# List registered MCP servers
bonito mcp list

# Test a tool call
bonito mcp test --server "github-tools" --tool "search_repos" \
  --params '{"query": "bonito"}'

stdio configuration

For stdio-based MCP servers, provide the command and arguments to launch the server process:

{
  "name": "sqlite-tools",
  "transport": "stdio",
  "command": "npx",
  "args": ["-y", "@modelcontextprotocol/server-sqlite", "./data/mydb.sqlite"],
  "env": {
    "NODE_ENV": "production"
  }
}

Connecting MCP to agents

Once registered, attach MCP servers to your BonBon agents. The agent's model will automatically see the available tools and can call them during conversations:

# Connect MCP server to an agent
bonito agents update --id ag_abc123 \
  --mcp-server "github-tools" \
  --mcp-server "sqlite-tools"

# Remove an MCP server from an agent
bonito agents update --id ag_abc123 \
  --remove-mcp-server "sqlite-tools"

Knowledge Bases

Knowledge Bases power RAG (Retrieval-Augmented Generation) for your BonBon agents. Upload documents, and Bonito chunks, embeds, and indexes them so your agents can retrieve relevant context at query time.

Creating a knowledge base

1. 1Go to Knowledge Bases → Create in the dashboard, or use the CLI.
2. 2Give it a name and optional description.
3. 3Choose an embedding model (defaults to a high-quality model on your connected providers).
4. 4Configure chunking strategy (size, overlap).
5. 5Upload your documents.

# Create a knowledge base
bonito kb create --name "Product Docs" \
  --embedding-model "amazon.titan-embed-text-v2:0" \
  --chunk-size 512 \
  --chunk-overlap 50

# Upload documents
bonito kb upload --id kb_xyz789 ./docs/*.pdf
bonito kb upload --id kb_xyz789 ./faq.md
bonito kb upload --id kb_xyz789 https://example.com/api-reference.html

# Check indexing status
bonito kb status --id kb_xyz789

Supported file formats

Bonito accepts PDF, Markdown, plain text, HTML, DOCX, and CSV files. Each file is parsed, split into chunks, and embedded using your chosen embedding model.

Chunking strategies

Embedding models

Bonito uses embedding models from your connected providers. Any enabled embedding model can be used:

• AWS Bedrock: amazon.titan-embed-text-v2:0, cohere.embed-english-v3
• Azure OpenAI: text-embedding-ada-002, text-embedding-3-small, text-embedding-3-large
• GCP Vertex AI: textembedding-gecko, text-embedding-004

Querying

You can query a knowledge base directly to test retrieval before connecting it to an agent:

# Query a knowledge base directly
bonito kb query --id kb_xyz789 \
  --query "What is the refund policy?" \
  --top-k 5

# Connect KB to an agent (see BonBon Agents section)
bonito agents update --id ag_abc123 \
  --knowledge-base kb_xyz789

Managed Inference

Managed Inference gives you zero-config access to AI models without connecting any cloud provider. No API keys, no cloud accounts, no setup — just start making requests. Bonito handles provider selection, routing, and billing.

How it works

1. 1Sign up for Bonito and create a gateway API key — that's it.
2. 2Use any supported model by name in your API requests.
3. 3Bonito routes your request to the optimal provider automatically.
4. 4You're billed through Bonito based on token usage — no separate cloud bills.

from openai import OpenAI

# No cloud provider setup needed — just your Bonito key
client = OpenAI(
    base_url="https://getbonito.com/v1",
    api_key="YOUR_BONITO_API_KEY"
)

# Use any supported model
response = client.chat.completions.create(
    model="claude-3-sonnet",  # Bonito routes to the best provider
    messages=[{"role": "user", "content": "Explain quantum computing"}]
)

print(response.choices[0].message.content)

Supported models

Managed Inference supports a curated set of popular models across providers. Use simplified model names — Bonito resolves them to provider-specific IDs:

When to use Managed vs BYOC

Triggers

Triggers let you invoke BonBon agents automatically based on events — incoming webhooks, cron schedules, slash commands, or custom events. Instead of waiting for a user to open a chat widget, triggers bring your agents into workflows programmatically.

Trigger types

Creating triggers

# Create a webhook trigger
bonito triggers create \
  --agent-id ag_abc123 \
  --type webhook \
  --name "GitHub PR Review"

# Output: Webhook URL → https://getbonito.com/hooks/tr_wh_abc123

# Create a scheduled trigger (daily at 9 AM UTC)
bonito triggers create \
  --agent-id ag_abc123 \
  --type cron \
  --schedule "0 9 * * *" \
  --name "Daily Summary" \
  --input "Generate a summary of yesterday's support tickets"

# Create a slash command trigger
bonito triggers create \
  --agent-id ag_abc123 \
  --type slash-command \
  --platform slack \
  --command "/ask-support" \
  --name "Slack Support"

# List triggers for an agent
bonito triggers list --agent-id ag_abc123

Webhook payload

When a webhook trigger fires, the HTTP request body is passed to the agent as context. You can define a template to extract specific fields:

{
  "trigger_id": "tr_wh_abc123",
  "payload_template": "New {{event}} from {{repository.full_name}}: {{pull_request.title}}",
  "headers_to_forward": ["X-GitHub-Event"],
  "secret": "whsec_xxxxxxxx"
}

Observability

Bonito provides built-in observability for every request flowing through the gateway and every agent interaction. Track tokens, latency, costs, and errors across your entire AI stack without any additional tooling.

Request tracing

Every API request gets a unique trace ID. View the full lifecycle of a request — from gateway receipt through provider routing to response delivery:

# View recent requests
bonito logs list --limit 20

# Get details for a specific trace
bonito logs get --trace-id tr_xxxxxxxxxxxx

# Filter by model, status, or time range
bonito logs list \
  --model "claude-3-sonnet" \
  --status error \
  --since "2024-01-15T00:00:00Z"

What's tracked

Per-agent analytics

Each BonBon agent has its own analytics dashboard showing conversation volume, average response time, token usage, cost breakdown, and error rates over time.

# Get agent analytics
bonito agents analytics --id ag_abc123 \
  --period 7d

# Export analytics as CSV
bonito agents analytics --id ag_abc123 \
  --period 30d \
  --format csv > agent-analytics.csv

Cost monitoring

Observability integrates with Cost Management to show real-time spend. Set budget alerts per agent, per model, or globally:

# Set a per-agent budget alert
bonito alerts create \
  --scope agent \
  --agent-id ag_abc123 \
  --threshold 500 \
  --period monthly \
  --notify email,in-app

# Set a global daily spend alert
bonito alerts create \
  --scope global \
  --threshold 100 \
  --period daily

Troubleshooting

"Connected! Found 0 models"

Your credentials connected successfully, but model listing failed silently. Check:

• Azure: Make sure your Endpoint URL is an Azure OpenAI resource endpoint with a custom subdomain, not a generic regional endpoint. Also verify the resource group is correct.
• GCP: Ensure the Vertex AI API is enabled in your project.
• AWS: Verify your IAM user has Bedrock permissions and you're in a region where Bedrock is available.

Models showing a 🔒 lock icon

The model exists in your provider's catalog but isn't enabled. Click the Enable button in Bonito, or enable it directly in your cloud console (AWS Bedrock → Model Access, Azure → create a deployment, GCP → enable the Vertex AI API).

Playground returns a 500 error

This typically means the model isn't a chat model (embedding or completion-only models can't be used in the Playground), the model isn't enabled in your account, or the model isn't available in your region.

Rate limit or timeout errors

When a 429 carries a Retry-After header, wait that many seconds and try again; otherwise wait a few seconds. If 502 errors persist, contact support@getbonito.com.