Caching — in depth
The summary said "keep a copy near whoever keeps asking". That is the easy half. This page covers the five places a cache can live and what each one costs you, the four write strategies and when each is right, how invalidation fails, the three ways an empty cache takes down a database, hot keys, and how to decide what your hit rate needs to be.
Where the cache lives, and what each location costs
A cache is not one thing. There are five places to put one, and they differ in speed, in how many copies exist, and in whether you can ever take something back.
The instinct is to reach for Redis. Often the right answer is two layers, a small in-process cache in front of a shared one, and being able to explain why is worth more than naming a product.
- In-process is thousands of times faster than a shared cache, but every server has its own copy, so consistency between servers is gone.
- The two-layer pattern (a small in-process cache in front of Redis) absorbs hot keys and softens a Redis outage. That is the answer to "what if the cache dies".
- A CDN is a cache. Treat it as part of this decision rather than a separate topic.
- The browser is the one layer you cannot invalidate, which is why versioned filenames exist.
| Layer | Speed | Copies | The catch |
|---|---|---|---|
| Browser | Free, zero latency | One per user | You cannot recall it. A wrong long-TTL response lives on that device. |
| CDN edge | ~10 ms | Hundreds | Purging is not instant. Private data here is a disclosure, not a bug. |
| In-process | ~100 ns | One per server | Servers drift apart, so two users can get different answers. |
| Shared (Redis) | ~0.5 ms | One | A network hop, plus a dependency that can fail and take everything with it. |
| Database buffer pool | Fast | One per node | Real, and you do not control it. Do not count on it in a design. |
The four write strategies
Reading from a cache is obvious. What happens on a write is where the designs differ, and this is the part interviewers probe.
- Cache-aside (the default): the application checks the cache, and on a miss reads the database, writes the value into the cache, and returns it. Simple, and the cache failing degrades you rather than breaking you.
- Read-through: the cache itself fetches on a miss. Cleaner application code, and now the cache is on the critical path for correctness rather than just for speed.
- Write-through: write to cache and database together. The cache is never stale, every write pays both costs, and you cache things nobody will read.
- Write-behind: write the cache now, the database shortly after. Fast writes, and you can lose data in the gap. Only acceptable when you can afford to lose the data.
- —Application controls everything, so failure behaviour is obvious.
- —The cache going down means slow, not broken.
- —Only requested data is cached, so nothing is wasted.
- —Every read path must remember to do it, and one that forgets fails without an error.
- —There is a window on a miss where two requests both fetch from the database.
- —The cache is not stale after a normal write, so there is little invalidation logic.
- —Every write pays cache latency as well as database latency.
- —You cache everything written, including data nobody ever reads.
- —A cache write failing means either a failed write or an inconsistency.
Invalidation
There are two basic mechanisms. Both are compromises, and production systems usually use them together.
TTL: the data is wrong for at most that long, and you never have to think about it again. Explicit deletion on write: it is right almost always, and every code path that writes has to remember.
The reason to use both is that they fail differently. A TTL fails by being slightly stale. Explicit deletion fails by being wrong indefinitely when someone forgets, and that bug is hard to reproduce. So the TTL is the safety net that bounds the damage from a forgotten delete.
- TTL alone: simple, and your staleness window is the TTL. Fine for most things.
- Explicit deletion alone: correct until someone adds a write path and forgets. Ordinary tests rarely catch this.
- Both: explicit deletion for freshness, TTL as the bound on how wrong you can be. This is the answer to give.
- Jitter the TTL. A million keys written together will otherwise all expire in the same second.
- Deleting the key is safer than updating it: an update built from a stale read writes stale data back into the cache, and it stays there.
Going from 90% to 99% removes 90% of the remaining database load. That is why "we have a cache" is not an answer and the hit rate is. It is also why a cache that drops from 99% to 90% sends ten times the traffic to your database, not 9% more.
The three ways an empty cache kills you
This section separates a design from a diagram. A cache rarely hurts you by being slow. It hurts you by being empty at the wrong moment, because the database behind it was never sized for the traffic the cache was absorbing.
- Stampede (also called dog-piling): one popular key expires, and the thousand requests that were being served from it all miss at once and all query the database. The fix is request coalescing: the first miss fetches, and the others wait for that result. Done in-process, this gives one query per server rather than one overall, which is usually enough.
- Cold start: the cache is restarted or a new node joins, so the whole hot set is missing and every request goes to the database at once. The fix is warming before taking traffic, and adding nodes during quiet periods.
- Penetration: requests for keys that do not exist anywhere miss every time, because there is nothing to store. Someone probing random ids drives your miss rate to 100%. The fix is caching the "not found" result briefly, or a bloom filter of existing ids.
- All three have the same shape: your database suddenly receives the traffic your cache was absorbing, and at the ratios that justified adding a cache, it will not survive it.
Hot keys, and why sharding does not help
A hot key is one item everybody wants at once: a viral post, a flash-sale product, the configuration blob every request reads. Consistent hashing spreads *keys* evenly, and a hot key is one key, so it lands on one node no matter how elegantly you assigned it.
The answer is replication of that key rather than better placement, and there are three levels of it.
- Level one: an in-process cache in front of the shared cache. The hottest key is answered from local memory on every server, so the shared cache sees a trickle. This alone solves most hot key problems.
- Level two: write the hot key under several suffixed names and have readers pick one at random, spreading it across nodes. Costs a fan-out on invalidation.
- Level three: detect hot keys at runtime (a count-min sketch is the standard tool) and promote them automatically rather than hard-coding a list.
- Say out loud that even key distribution does not mean even traffic distribution. That sentence is the core of the answer.
Deciding what to cache, and admitting when not to
Caching everything is a failure mode of its own: memory spent on items requested once, invalidation complexity for data nobody reads, and a hit rate that looks fine while the useful keys are being evicted by useless ones.
- Cache what is read far more often than it is written, and expensive to produce. Both conditions, not either.
- Do not cache the value a commit decision is based on when people compete for it. Seats, stock and balances must be read inside the transaction that changes them.
- Check your eviction policy matches the access pattern. Least-recently-used is right for most things and wrong for a workload that scans everything periodically, since the scan evicts your hot set.
- Measure hit rate per key pattern, not overall. An aggregate of 95% can hide one important pattern sitting at 20%.
- A cache that never gets a hit is pure cost plus a new failure mode. Be willing to say a cache is not needed here.
The cache that took down the database at 9am every Monday
Every Monday at 09:00 the site slows to a crawl for about four minutes, then recovers on its own. Database CPU hits 100%, application error rate spikes, and by the time anyone opens a dashboard it is already fixing itself. Redis is healthy throughout. This has happened for six weeks.
- 1Find what is special about 09:00 Monday
It is not traffic. Monday peak is only slightly above other days, and nothing like a fourfold spike. The clue is that Redis is healthy but the database is saturated, which means requests are missing the cache. So the question is not "why is there more traffic" but "why did the hit rate collapse".
- 2Graph the hit rate, not the traffic
Hit rate is 99.2% all week and drops to 61% for four minutes every Monday at 09:00. That is the incident: at 99.2% the database sees 0.8% of reads, and at 61% it sees 39%. That is close to fifty times the database traffic, with no change in user traffic at all.
- 3Find why the keys expired together
A weekly job repopulates product data on Sunday night, writing about two million keys with a fixed 12-hour TTL. They were all written within a few minutes of each other, so they all expire within a few minutes of each other, at roughly 09:00 on Monday. A synchronised mass expiry, created by the cache-filling job itself.
- 4Fix it in the cheapest place
Add jitter: the TTL becomes 12 hours plus a random 0 to 90 minutes. The same two million keys now expire spread over an hour and a half instead of over four minutes, and the database barely notices. It is one line of code, and it removes this incident.
- 5Then make the class of failure survivable
Jitter fixes this occurrence; it does not fix the next mass eviction. So also add request coalescing, so a thousand simultaneous misses on the same key become one database read. And add a short in-process cache in front, so even a completely empty Redis does not send full read load to the database. Now the database is protected from an empty cache generally, rather than from this specific Monday.
- 6Alert on the leading indicator
The alert was on database CPU, which fires when the damage is already happening. Add an alert on cache hit rate dropping below a threshold, which fires before the database is in trouble and points directly at the cause instead of at the symptom.