Multi-provider routing.
Chat, embeddings, images, audio, files, and batch jobs across providers, behind one OpenAI-compatible API.
Frosty Gateway
A self-hosted gateway for all your LLM traffic. Routing, budgets, caching, and MCP tools behind one OpenAI-compatible endpoint.
What it is
Frosty Gateway is one self-hosted gateway for all your LLM traffic. Every application in your organization talks to a single OpenAI-compatible endpoint; behind it, Frosty routes requests to the right provider, enforces budgets and rate limits, caches what can be cached, and gives operators a control plane to see and govern all of it.
Once a company uses AI in more than one place, the questions arrive fast. Who is spending what? Which team is using which model? What happens when a provider goes down or a key leaks? Frosty answers those questions in one place instead of five. Virtual keys and a team hierarchy map spend to the people responsible for it, a pricing catalog turns token counts into money, and budgets stop surprises before the invoice does.
Because it is self-hosted, all of this runs on your infrastructure. Your prompts, your usage data, and your keys never route through a third-party SaaS.
Who it is for
Platform and IT teams governing multi-provider LLM usage across an organization, and anyone who wants a single, accountable front door for AI traffic.
What it does
Virtual keys and team hierarchy.
Per-team and per-key budgets, rate limits, and spend attribution.
Pricing catalog.
Token usage translated into cost, per model and per team.
Two-layer response cache.
Exact-match and semantic caching backed by pgvector, cutting cost and latency on repeated questions.
MCP aggregation and serving.
Frosty aggregates MCP tool servers for its clients and exposes an MCP surface of its own, with a gated Code Mode.
Operator control plane.
A same-origin React UI for keys, teams, budgets, and traffic.
Under the hood
A single Deno 2 process combining the inference gateway, the control plane, and the MCP client and server surfaces, with durable state in PostgreSQL. Currently pre-1.0, and unusually honest about its performance: v0.9.0 ships a measured benchmark report with a documented methodology of 3,000 requests per wave and preserved historical results.
Interested?
If your LLM usage is growing faster than your visibility into it, a gateway is the fix, and we can run it on your terms.