Medium · reference design
Distributed cache
Put a shared, in-memory cache in front of a database so reads take a millisecond instead of tens, without serving data that stays wrong, melting the database on a miss storm or losing everything when a node dies.
1 Requirements
Functional
- Get, set and delete values by key, with an optional time to live.
- Serve reads for many application servers from one logical cache.
- Keep cached data close to the database: changes show up quickly.
Non-functional
- Reads in about a millisecond at hundreds of thousands of requests per second.
- Scale out by adding nodes, moving as few keys as possible.
- Stay available when a node fails, and never let a miss storm take the database down.
- Bounded staleness: seconds, not hours.
Out of scope: Using the cache as the system of record: the database stays the source of truth. Cross-region replication.
2 Core entities
- Entry
- Key, value, expiry and a version or timestamp.
- Shard
- A primary node and its replicas, owning part of the key space.
- Hash ring
- The map from keys to shards, with virtual nodes for balance.
3 API
internal get(key) → value | missUsed by the application on every read, before the database.
internal set(key, value, ttl) · delete(key)Set after loading from the database; delete when the data changes.
4 Data flow
- The application asks its small local cache, then the shared cache shard that owns the key.
- On a hit it returns; on a miss one request loads from the database and fills the cache.
- Writes go to the database and delete the cached key; a change stream catches writes from elsewhere.
5 High-level design
Press play to follow one request through the system. Every step explains what happens and the trade-off behind the choice.
Follow one read from the application to the cache and, on a miss, to the database; then see how writes, evictions and failures keep the cache honest. Press play, or step through the stages.
6 Deep dives
Which caching pattern?
| Option | Gives | Costs |
|---|---|---|
| Cache-aside (lazy loading) | Only what is read gets cached; the cache can fail without losing data. | The first read of each key is a miss, and the application owns the logic. |
| Write-through | The cache is always populated after a write. | Every write pays for the cache too, and caches data that may never be read. |
| Write-behind | Writes are very fast. | A cache failure can lose writes the database never received. |
Choice. Cache-aside with delete on write. It keeps the database as the only source of truth and degrades to "slower" rather than "wrong" when the cache has problems.
What about a single very hot key?
| Option | Gives | Costs |
|---|---|---|
| Do nothing | Simple. | One shard takes all the traffic for that key and becomes the bottleneck. |
| Local cache in each app server | The hot key is served from memory on every server. | A few seconds of staleness for that key. |
| Replicate the key under several names | Spreads one key across many shards. | Every write must update or delete all copies. |
Choice. A short-lived local cache for keys detected as hot, and copies across shards only for the rare key that still overloads one node.
How do we avoid caching a stale value after a delete?
| Option | Gives | Costs |
|---|---|---|
| TTL only | Nothing extra to build. | A stale value lives for the whole TTL. |
| Versioned values | A fill carrying an older version than the stored one is rejected. | Needs a version or timestamp from the database on every row. |
| Delete again shortly after the write | Cheap; catches a reader that filled the old value in between. | Narrows the window instead of closing it. |
Choice. Versioned values where staleness matters, plus the TTL as the last line of defense everywhere else.
In one sentence
A local cache for hot keys, consistent hashing to replicated shards, single-flight loads with jittered TTLs on a miss, delete-on-write plus a change stream for freshness, and a database that never has to absorb a stampede.