Medium · reference design

Distributed rate limiter

Cap how many requests each client can make, across a fleet of gateways, without adding noticeable latency or becoming a single point of failure.

1 Requirements

Functional

Limit requests per client, identified by API key, user id or IP address.
Support different rules per endpoint and per plan (for example, 100 requests a minute on the free plan).
Reject requests over the limit with HTTP 429 and tell the client when it can retry.

Non-functional

Low overhead: a few milliseconds at most added to each request.
Highly available: the limiter failing must not take the API down.
Accurate across many gateway instances, not just per instance.
Scale to millions of requests per second and hundreds of millions of clients.

Out of scope: Defending against volumetric DDoS, which belongs at the network edge. Billing and quota purchases.

2 Core entities

Rule: Who it applies to, which endpoints, the limit and the window.
Client: The identity being limited: API key, user id or IP address.
Bucket: The live state per client and rule: tokens left and last refill time.

3 API

internal check(clientId, ruleId) → { allowed, remaining, resetAt }

Called by the gateway on every request before it is forwarded.

headers RateLimit-Limit · RateLimit-Remaining · RateLimit-Reset · Retry-After

Returned to clients so well-behaved ones can slow down before they are cut off.

4 Data flow

A request reaches the API gateway.
The gateway identifies the client and finds the rules that apply.
It atomically takes a token from the client bucket in a shared store.
If a token was available the request goes to the backend, otherwise the client gets a 429.

5 High-level design

Press play to follow one request through the system. Every step explains what happens and the trade-off behind the choice.

ClientRequest + API key

API gatewayEvery request

RulesCached in memory

LimiterGateway middleware

Redis clusterBuckets by client

429 responseWith Retry-After

BackendOnly allowed traffic

Follow one request from the client to either the backend or a 429. Press play, or step through the stages to see what happens and what each choice costs.

6 Deep dives

How does the store scale to millions of checks per second?

Option	Gives	Costs
One Redis primary	Simple, strongly consistent, easy to operate.	One node caps throughput and is a single point of failure.
Redis Cluster, sharded by client key	Throughput grows with nodes; a client always lands on the same shard, so its count stays exact.	More nodes to run; a very hot client still lands on a single shard.
Local counters, synced periodically	No network call in the hot path.	Approximate: between syncs every gateway is blind to the others.

Choice. Shard by client key. Each check touches one key, so sharding is natural and exact. For a few very hot clients, add a small local pre-limit in each gateway.

How do concurrent requests avoid double-spending a token?

Option	Gives	Costs
Read, then write	Easy to write.	Two gateways can read the same value and both allow the request.
Atomic script in the store	Refill and take happen in one step, so the count is always correct.	The logic lives in a script that has to be versioned and tested.

Choice. An atomic script. Correctness is the whole point of a limiter, and the script is a few lines.

What about multiple regions?

Option	Gives	Costs
One global store	One exact count worldwide.	Every request pays cross-region latency.
A store per region, with the limit split between regions	Fast, and each region keeps working on its own.	A client spread across regions can briefly exceed the global limit.

Choice. A store per region. A small, bounded overshoot is cheaper than adding cross-region latency to every request.

In one sentence

A token bucket per client in a sharded Redis, checked atomically at the gateway, failing open with a local fallback: accurate where it matters and invisible when it works.