A team at a mid-size fintech shipped a support-ticket summarizer in March. It worked fine in staging. Three days after launch, one customer’s integration retried a failed webhook every 400 milliseconds, each retry firing a fresh call to Claude. The bill for that one afternoon came out to $4,200. Nobody had built a rate limiter in front of the model calls, because in staging nobody needed one. That’s the whole argument for an AI gateway in six sentences.
What a gateway actually does
An AI gateway is a proxy layer sitting between your application code and the model providers — Anthropic, OpenAI, whoever. It’s not a new idea; it’s the same reverse-proxy pattern API teams have used for REST services since the 2000s, aimed at a workload that happens to be metered by tokens instead of requests. The gateway does three jobs: it limits how fast callers can hit the model, it tracks what every call costs in near-real time, and it decides what happens when a caller exceeds their budget or the provider throttles you back.
The rate-limiting piece almost always uses a token bucket, not a fixed window counter. Fixed windows have a well-known edge case: a caller can burn their entire quota in the last second of one window and the first second of the next, doubling their effective rate at the boundary. A token bucket avoids that. Each API key gets a bucket that holds up to N tokens (say, 50) and refills at a steady rate (say, 10 per second). Every request costs some number of tokens — often the actual LLM token count, not a flat “1 request = 1 token” charge, since a 50-token prompt and a 4,000-token prompt do very different amounts of work downstream. Request comes in, gateway checks the bucket, if there’s enough capacity it deducts and forwards the call; if not, it returns a 429 immediately, before the request ever reaches the model provider.
That last detail matters more than it sounds. Rejecting locally, in Redis, costs you a few milliseconds. Rejecting after the provider call has already gone out costs you the full latency of that call — and, worse, you’ve already been billed for the tokens. A gateway that rate-limits after the fact isn’t a rate limiter, it’s a cost report.
Cost tracking runs in parallel. Every response from the provider comes back with usage metadata — input tokens, output tokens, sometimes cached tokens billed at a different rate. The gateway multiplies those against a price table (which you have to keep updated by hand or by scraping the provider’s pricing page, because there’s no standard API for “tell me your current prices”) and writes a ledger entry: which API key, which model, how many tokens, what it cost. Aggregate that ledger by key, by team, by hour, and you get a live spend dashboard instead of a surprise on the monthly invoice.
Where this gets subtle
Common knowledge: everyone building on LLM APIs eventually adds some rate limiting, usually a naive per-minute counter, usually after a bill scare.
Rare knowledge: token buckets need per-key buckets, not one global bucket, or your best customer gets throttled because a different customer is misbehaving. Getting this wrong is the single most common gateway bug — I’ve seen it ship to production at three different companies.
Advanced insight: at real scale, the bucket state itself becomes a bottleneck. A single Redis instance doing a check-and-decrement on every request for thousands of keys per second will show up as your p99 latency floor before the model call even starts. Sharding buckets by key hash, or using Redis Cluster with client-side hashing, fixes it — but now you’ve got a distributed rate limiter, which has its own consistency headaches (two gateway nodes racing to decrement the same bucket).
Strategic impact: cost control isn’t just finance hygiene. A gateway that tracks spend per feature lets you kill a runaway feature without killing the whole product — set a hard budget cap on the “auto-summarize” endpoint specifically, and when it hits zero, that endpoint degrades to a cached or non-LLM fallback while checkout, search, and everything else keeps working. Without per-feature attribution, your only lever is an all-or-nothing kill switch on the whole gateway.
Implementation nuance: decide up front whether rate limits apply to requests or tokens per minute. Most providers rate-limit you on both, separately, and your gateway’s limits should mirror that shape — otherwise you’ll pass your own gateway’s check and get throttled by the provider anyway, which is a confusing failure mode to debug at 2 a.m.
