Skip to content

In active development · self-hostable today

One endpoint for every AI provider.

Free-tier-first, self-hosted, with automatic failover. Add your provider keys once; the gateway picks the model, balances your quotas, and keeps working when a vendor does not.

Get Started Documentation
LLM request routing diagram A client request enters the gateway router, which checks provider health and free-tier status, then directs the request to one of the configured provider accounts — for example a local Ollama model, Groq, or Gemini — cycling between them over successive requests, while also exposing health, status, metrics and model-list endpoints. CLIENT POST /v1/* ROUTER free + health Ollama Groq Gemini OBSERVABILITY GET /health GET /v1/providers/status GET /metrics GET /v1/models

Illustrative request path; see verified endpoints in the docs below.

// overview

AI Bakery Gateway

A self-hosted LLM gateway: one OpenAI- and Anthropic-compatible endpoint, free-tier-first routing, and automatic failover across the providers you own.

The problem What the gateway does
Every vendor has its own API shape, SDK quirks and error format. One OpenAI- and Anthropic-compatible API. Existing clients work by changing the base URL.
Free tiers run out mid-task. Spreads load across all your accounts by remaining quota and fails over to the next one.
A provider has an outage. The failing model is taken out of rotation and the request moves on to the next candidate.
Fear of an accidental bill. FreeOnly ships as the default. A request is never sent to a model the gateway can't currently prove is free.
"Why did it answer from that model?" Every response says which provider, account and model served it, and a CLI command explains the routing decision afterwards.
Several people share the same keys. Per-team keys, budgets and routing profiles, plus a usage view per team.

Unified

One OpenAI-compatible API, plus OpenAI Responses and Anthropic Messages, across 14 provider kinds: cloud vendors, your own Ollama, or any OpenAI-compatible server.

Free-first and safe

Ships configured FreeOnly, with evidence-based checks and a billing tripwire that raises an alert if a vendor ever charges for a call the gateway believed was free.

Operable

A live dashboard, explainable routing, alerts to webhook/Slack/Telegram, Prometheus and Grafana, a CLI, and an MCP server — all running in your own Docker Compose stack.

01 · #how-it-works

How it works

Request flow, smart routing, free-tier protection, resilience and observability.

AI Bakery Gateway sits between your clients and the provider accounts you’ve connected. Your app, agent or coding tool sends an OpenAI Chat Completions, OpenAI Responses, or Anthropic Messages request to one address; the gateway resolves it to an account and model, calls it, and fails over if that call doesn’t succeed.

The request flow

  1. Resolve the model: a specific model name, a logical model, auto, or a named routing profile such as free-code.
  2. Apply hard rules: capability fit, free-tier proof (under FreeOnly), and budget.
  3. Rank candidates: quota headroom, availability, and measured quality.
  4. Reserve quota, then call the best-ranked account.
  5. Fail over on failure, and record enough detail to explain every decision afterwards.

Smart routing, not round-robin

  • auto: the gateway scores candidates on capability fit, measured quality, availability, remaining quota and freshness, and picks one. You don’t name a model.
  • Routing profiles ask for the kind of work, not a model name — see the 8 below.
  • Quota-aware balancing: with several accounts for the same model, traffic goes to whichever has the most headroom, so one account isn’t drained while another sits idle.
  • Bounded failover: a request walks a ranked candidate list up to a configured attempt limit and a first-token deadline, so one slow or dead vendor can’t stall it indefinitely.
  • No silent substitution: ask for a specific model and you get that model, or a clear failure — never a quiet swap to something unrelated.
  • Equivalent-model matching: the same model offered by two vendors is recognised as one logical model and used as a fallback for the other.

Free-tier protection

FreeOnly ships as the gateway’s default: a request is never sent to a model the gateway cannot currently prove is free. It keeps re-checking that picture — re-reading vendor model lists and recomputing “is this model free for me?” — rather than assuming a past answer still holds. A billing tripwire watches for the case evidence alone can’t catch: if a vendor ever reports a charge on a model the gateway believed was free, the gateway stops using that assignment and raises a critical alert. Expiring trial credit (such as a Cerebras trial) is tracked, and an account with no credit left is excluded before it can fail a request.

Resilience

  • Circuit breaker per account and model: a failing unit is taken out of rotation for a cooldown that depends on why it failed — rate limit, timeout, auth, or a withdrawn model.
  • The vendor’s own Retry-After is respected.
  • Streaming safety: a request can fail over before the first token. After the first token, the gateway never silently switches model mid-answer.
  • Client cancellation never counts against a provider’s health.
  • Honest errors: 402 means no free model exists for this request under policy; 503 means nothing could serve the request right now; 429 means a tenant used its own budget.

Explainability and operations

Every response carries X-Scheduler-Provider, X-Scheduler-Account and X-Scheduler-Model headers, plus a request id. The bakery command-line tool’s why <request-id> shows every candidate that was considered, why it ranked where it did, and what happened to it. The same CLI covers setup, asking a model directly, listing models, usage, and probing an account (bakery setup, ask, models, usage, probe, tenants, accounts).

A live admin dashboard (Blazor, English and Russian) shows usage, fleet health and routing, a Try page for test requests, a Connect page with ready-to-copy client configuration, and a setup wizard. Alerts go to a webhook, Slack, or Telegram when a free model is lost or gained, a model is withdrawn, or a vendor bills a call unexpectedly. Prometheus metrics, OpenTelemetry tracing, and a ready-made Grafana dashboard with alert rules are included. An MCP server (bakery-mcp) lets an MCP-capable client — Claude Code, OpenCode, Claude Desktop — inspect and diagnose a running gateway directly.

View the provider list

Routing profiles

  • free-best

    Balanced default

  • free-fast

    Low latency

  • free-smart

    Answer quality

  • free-code

    Coding

  • free-tools

    Reliable tool calling

  • free-long

    Long context

  • free-vision

    Image input

  • free-safe

    Availability, no previews

02 · #providers

Providers

14 supported provider kinds, several accounts per provider, and how the fleet is configured.

  • Ollama Cloud

    Hosted Ollama models

  • Ollama (Local)

    On the gateway's own machine

  • Ollama (Local Network)

    On another machine in your LAN

  • Groq

  • Cerebras

    Trial credit balance and expiry tracked

  • Gemini

  • OpenRouter

  • NVIDIA NIM

  • Mistral AI

  • Cohere

  • SambaNova Cloud

  • Z.ai (Zhipu GLM)

  • Cloudflare Workers AI

  • Custom (OpenAI-compatible)

    Your own server — LM Studio, vLLM, llama.cpp, or any OpenAI-compatible endpoint

Curious which of these currently have a free tier? See the current rundown →

AI Bakery Gateway manages a fleet of provider accounts, so you can mix local models and cloud APIs behind one address.

Several accounts per provider

Add as many API keys as you own for a given provider. Each account gets its own quota bucket and its own circuit breaker — except where a vendor meters one shared allowance per account, in which case the gateway shares one bucket across that account’s models. With several accounts for the same model, traffic goes to whichever account has the most headroom, so one account isn’t drained while another sits idle.

Local models are first-class

Run Ollama, or any OpenAI-compatible server, alongside cloud vendors. Local inference costs nothing per call, and once you confirm an account as free in the dashboard it takes part in FreeOnly routing the same as a cloud account.

Equivalent-model matching

The same underlying model offered by two vendors — for example a model available both directly and through OpenRouter — is recognised as one logical model, so the gateway can use the second as a fallback for the first.

Embedding providers

Separate embedding providers handle vectorization tasks for retrieval-augmented generation and similar workloads.

Fleet configuration

Keys are entered in the dashboard or the CLI, never in a config file. For each account you can set priority, rate limits and timeouts. At least one enabled account is required before the gateway can process a chat, responses, messages or embeddings request.

Free-tier availability, limits and prices are set by each vendor and change over time — check each vendor’s current terms. The gateway enforces the limits you configure and learns the rest from the vendor’s own responses.

Read the quick start

03 · #quick-start

Quick Start

Run the gateway with Docker Compose and send your first request in about 5 minutes.

Up and running in about 5 minutes

Run it with Docker Compose and connect your first provider account.

1

Generate your secrets

Writes a `.env` file with a fresh encryption key, database password and client key. It refuses to overwrite an existing `.env`. On Windows, run `scripts/init-env.ps1` in PowerShell instead.

./scripts/init-env.sh
2

Start the stack

Starts the gateway together with PostgreSQL and Redis.

docker compose up -d
3

Open the dashboard and connect a provider

Open http://localhost:18080, choose a provider, and paste its API key. The key is checked, then encrypted at rest, and its models are discovered automatically. The Setup wizard at /setup walks you through it.

4

Verify it's ready

The ready check passes once at least one model is routable. A fresh install starts with an empty fleet, so this fails — expected — until you've added an account.

curl http://localhost:18080/health/ready
5

Send your first request

Point any OpenAI-compatible client at the gateway, or send a curl request directly.

You need Docker with Compose v2. No .NET SDK and no local PostgreSQL to install yourself.

Using curl

curl http://localhost:18080/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model":"auto","messages":[{"role":"user","content":"Hello"}]}'

The X-Scheduler-Provider, X-Scheduler-Account and X-Scheduler-Model response headers tell you who answered. "model":"auto" asks the gateway to pick; name a specific model, or a routing profile such as "free-code", once you know what your fleet publishes via GET /v1/models.

Using the OpenAI SDK (Python)

from openai import OpenAI

client = OpenAI(base_url="http://localhost:18080/v1", api_key="local")  # any value works until data-plane auth is enabled
reply = client.chat.completions.create(
    model="auto",
    messages=[{"role": "user", "content": "Hi"}],
)
print(reply.choices[0].message.content)

Open WebUI, Continue, and Cline: add an OpenAI-compatible provider with base URL http://localhost:18080/v1. From another container, use http://host.docker.internal:18080/v1 instead of localhost.

Going beyond localhost

Set SCHEDULER_ADMIN_TOKEN in .env before letting teammates open the dashboard remotely, and set SCHEDULER_AUTH_ENABLED=true before any API client outside this machine should reach the gateway — clients then send the generated BAKERY_API_KEY as Authorization: Bearer <key>. Enable both before exposing the port beyond this machine.

Back up what matters

Keep .env together with your database dump — it holds the encryption key for your stored provider keys, and without it they can’t be decrypted.

04 · #self-hosting

Self-hosting

Docker deployment options, security defaults, and team mode for self-hosting AI Bakery Gateway.

AI Bakery Gateway runs in your own infrastructure via Docker. There is no published container image to pull yet — see the changelog — so the documented path builds the image from the repository with Docker Compose.

Team mode

One gateway can serve several people or services. Each tenant gets its own API key, request and token budgets (per minute, day and month), its own routing profile and cost policy, and its own line in the usage view — the CLI’s tenants and usage commands show it per tenant. One tenant can never spend another’s budget.

Licence

AI Bakery Gateway is distributed under the AIProvidersScheduler Free Use License 1.0. It’s free to download, run and use — commercially included — in your own infrastructure. Calling it through its documented APIs from your own apps carries no extra licence obligations. It’s proprietary software: no source code, no redistribution or re-publishing of the image, no resale of the gateway itself, and it may not be used as the engine of a competing hosted gateway service. For OEM, hosted-service or redistribution rights, contact the copyright holders directly.

Back up what matters

Deployment options

Full stack

The gateway together with PostgreSQL and Redis, via `docker compose up -d`. Redis is only needed for shared rate limits and budgets across multiple gateway instances.

Lite

One container with PostgreSQL embedded and no Redis — the quickest way to try it, and the same code, migrations and database as the full stack if you grow out of it later.

Demo profile

Starts a small local Ollama model alongside the gateway so you can try it with no provider account at all. The gateway still refuses to route to it until you confirm in the dashboard that it's free.

Security by default

  • Port bound to 127.0.0.1 only

    PostgreSQL and Redis aren't published at all.

  • Admin plane closed until opened

    The dashboard and admin API answer only this machine until `SCHEDULER_ADMIN_TOKEN` is set; teammates then sign in at `/_bakery/login`. Wrong credentials are throttled.

  • Data-plane auth is opt-in

    Off by default, anonymous calls accepted from this machine only. `SCHEDULER_AUTH_ENABLED=true` requires every client to send `BAKERY_API_KEY` as a Bearer token.

  • Provider keys encrypted at rest

    AES-256-GCM envelope encryption; keys are never returned by the API.

  • Safe request handling

    Only an allow-list of request fields reaches a vendor, so a caller can't steer it into a paid model or an unexpected charge.

  • Reverse-proxy aware

    Send `X-Forwarded-For`; proxied requests are treated as remote, so they need the admin token like any other remote client.

05 · #docs

Documentation

Routes, configuration, and what this site does not cover for AI Bakery Gateway.

Configuration for a Docker deployment lives in the .env file scripts/init-env.sh/.ps1 generates in the gateway repository. This page lists the settings and routes you look up most often. It is not a full reference to every option.

Routes

  • POST /v1/chat/completions returns a chat completion in the OpenAI shape.
  • POST /v1/responses returns a response in the OpenAI Responses shape.
  • POST /v1/messages and POST /v1/messages/count_tokens accept the Anthropic Messages shape.
  • POST /v1/embeddings returns embeddings.
  • GET /v1/models lists the published providers ordered by priority, with catalog aliases and classes.
  • GET /health/live and GET /health/ready report liveness and readiness; ready is gated on a reachable database and a non-empty, routable provider pool.
  • GET /v1/providers/status and GET /v1/providers/embeddings/status report provider health and circuit state.
  • GET /metrics exposes Prometheus metrics.
  • GET /swagger serves the Swagger UI, and only while the gateway runs in the Development environment.

Configuration

  • Secrets and ports: generated into .env by scripts/init-env.sh/.ps1 — encryption key, database password, and the client key (BAKERY_API_KEY). Change the published port with BAKERY_PORT.
  • Admin access: closed to this machine until SCHEDULER_ADMIN_TOKEN is set.
  • Data-plane auth: off by default; turn it on with SCHEDULER_AUTH_ENABLED=true.
  • Provider accounts: entered in the dashboard or the CLI, never in a config file — see Providers.
  • Tenants: per-tenant request/token budgets, routing profile and cost policy — see Self-Hosting.
  • Routing and cost: choose a routing profile, or let auto decide; FreeOnly ships on by default — see How it Works.
  • Alerts: a webhook, Slack, or Telegram destination for fleet/billing/model-availability events.

Request shapes

POST /v1/chat/completions takes the OpenAI Chat Completions request shape, POST /v1/responses the OpenAI Responses shape, and POST /v1/messages the Anthropic Messages shape — so each provider’s own API reference describes most of the fields you send. Check GET /v1/models for the model names your deployment publishes before you send a request.

What this site does not cover

This site does not publish a full configuration reference, an API reference, or a deployment guide beyond the quick start. The gateway’s own repository documentation, and the CLI’s own --help output, are the record for everything not named above.

06 · #changelog

Changelog

What AI Bakery Gateway does today. No version numbers or release dates — see why below.

This page lists what the gateway does today. It does not track releases: no version numbers and no release dates appear here, because no gateway release has a confirmed public version tag yet — only its MCP server is independently versioned.

What the gateway does today

  • Three protocols, one pipeline: POST /v1/chat/completions, POST /v1/responses and POST /v1/messages (plus /count_tokens) all run the same routing, failover and quota handling.
  • 14 provider kinds: Ollama (Cloud, Local, Local Network), Groq, Cerebras, Gemini, OpenRouter, NVIDIA NIM, Mistral, Cohere, SambaNova, Z.ai, Cloudflare Workers AI, and a generic OpenAI-compatible custom provider.
  • auto and 8 named routing profiles: free-best, free-fast, free-smart, free-code, free-tools, free-long, free-vision, free-safe.
  • FreeOnly ships as the default, with evidence-based re-checks and a billing tripwire.
  • Team mode: per-tenant keys, budgets, routing profile and usage view.
  • A CLI (bakery): setup, ask, models, usage, probe, why, tenants, accounts.
  • An MCP server (bakery-mcp) for inspecting a running gateway from an MCP-capable client.
  • Alerting to webhook, Slack, or Telegram; Prometheus metrics, OpenTelemetry tracing, and a Grafana dashboard with alert rules.
  • Docker Compose deployment, with a full-stack, lite, and no-provider-account demo option.

The first request is described in the quick start.

Want to hear about new releases? Email us to get notified →

07 · #about

About

What AI Bakery Gateway is, who it's built for, and its licence.

AI Bakery Gateway is a self-hosted LLM gateway and smart router. It gives your apps, agents and coding tools one stable, OpenAI- and Anthropic-compatible address, then spreads requests across the provider accounts you own, watches rate limits, and fails over automatically when a vendor is slow, rate-limited or down.

Project goals

  • Interoperability: one address for OpenAI Chat Completions, OpenAI Responses, and Anthropic Messages, so existing client libraries work by changing only the base URL.
  • Free-tier safety: ship configured FreeOnly, with evidence-based checks and a billing tripwire, rather than trusting a one-time assumption about what’s free.
  • Control: make routing, cost and team budgets explicit and configurable, not hidden behind a hosted service.
  • Operability: explainable routing, a CLI, a dashboard, alerts and Grafana, so running it stays a known quantity, not a black box.

Licence

AI Bakery Gateway is distributed under the AIProvidersScheduler Free Use License 1.0: free to download, run and use — commercially included — in your own infrastructure. It’s proprietary software, with no source access, no redistribution, and no use as the engine of a competing hosted gateway service. See Self-Hosting for the full summary, or the licence file itself for the authoritative text.

Source

The public URL of the gateway’s source repository is not confirmed, so this site does not link to it.