Partitioning (sharding)
Split the data so different machines own different pieces. It is how you scale writes past one machine.
What problem does partitioning solve?
When one machine can no longer hold the data, or take the write rate, you split it. Each machine owns a slice, so ten machines take roughly ten times the writes.
Everything then rests on one decision: what you split by. Pick well and the system grows without drama for years. Pick badly and you get one overloaded machine while nine sit idle, and fixing it means moving all the data.
What partitioning costs you
Every concept in here states a cost. A technique with no price is a technique you have not understood yet — and naming the cost out loud is most of what separates a senior answer from a junior one.
When to reach for partitioning, and when not to
Use it when
- Write volume is past what one machine can commit, and you have the number to prove it.
- The data does not fit, or backups and recovery have become unmanageable.
- You need to limit blast radius, or keep certain data inside certain countries.
Do not, when
- You have not filled one machine yet. Most systems never need this, and doing it early costs you on every future feature.
- Your queries are unpredictable and touch everything. You will be asking every shard on every query.
How partitioning actually works
By range: shard A holds keys a–h, shard B holds i–p, and so on. Range queries are efficient, because neighbouring keys live together. The danger is uneven load. Sharding by timestamp puts all of today's writes on one shard while the rest sit idle. That is the classic mistake.
By hash: hash the key and take the remainder. Load spreads evenly, and you lose range queries completely, because neighbouring keys land on unrelated machines. Plain modulo has a second problem. Adding a machine changes almost every mapping, so nearly all the data has to move. Consistent hashing exists to fix that.
Hot partitions are the failure most sharded systems meet eventually. Sharding a social product by user id is even, until one user has 200 million followers and their shard melts. There are two defences. Give the hot key extra sub-partitions by adding a small random suffix, then read from all of them. That is more read work for everyone, in exchange for surviving the outlier. Or detect the hot key and handle it on a separate path, which is where many large systems end up. Say out loud that spreading keys evenly does not mean spreading traffic evenly.
Choosing the key. It should appear in almost every query, so most requests hit one shard. It should have many different values. And its traffic should be roughly even. User id and tenant id are usually good. Timestamps are usually bad. A combined key like (tenantid, entityid) is often the right shape.
Resharding is where the difficulty hides. Doing it with no downtime means writing to both old and new layouts, backfilling history, checking the two agree, switching reads over, then removing the old path. That is weeks of work. The way to avoid it is to start with far more logical partitions than machines, say 1024, and map many logical partitions onto each physical machine. Then growing means moving whole logical partitions, with no re-hashing at all. Mentioning this in an interview is a strong signal.
- Sharding by timestamp. Every write goes to the newest shard.
- Choosing a key that is not in your main query, so every read hits every shard.
- Assuming rebalancing is easy. Plan the logical-partition trick up front, or budget weeks later.
The partitioning follow-up an interviewer will ask
"One user has 200 million followers. What breaks?"
The shard holding that user, and then everything that depends on it. If I copy each post out to every follower, one post from them creates 200 million writes. That floods the queue and delays everyone else's timeline, not just theirs. If I keep their follower list on one shard, that shard is hot for reads too. The answer is to stop treating them like a normal user. Above a follower threshold, do not copy the post out. Leave it in place and have readers pull it when they load their timeline, merging it with the posts that were copied to them. Most users get the cheap read. The handful of huge accounts get the expensive read, and there are few enough of them to cache heavily. Cost: two code paths for the same feature, forever, plus a threshold that needs tuning. And timelines now come from two sources, so ordering needs care.
Your turn
You shard a messaging system by message id, hashed. What query just became expensive, and what would you shard by instead?
The shard key that looked fine for two years
A B2B analytics product shards its events table by event_id, hashed. It has run happily for two years. Now the biggest customers are complaining that their dashboards take 40 seconds to load, while smaller customers see them load instantly. The cluster is not saturated: average CPU across 32 nodes is 30%.
- 1Notice that the resource graphs are lying
Average CPU at 30% with slow queries means either the load is uneven or the queries themselves are expensive by design. Every partition is roughly equally busy, so this is not a hot partition. That rules out the usual suspect and points at the query.
- 2Look at what the dashboard query does
The query is "all events for customer 419 in the last 30 days". Events are hashed by event id, so that customer's events are scattered uniformly across all 32 partitions. Every dashboard load asks all 32 nodes, each scans for matching rows, and the coordinator merges. For a small customer that returns a few thousand rows and feels instant. For a large one it is millions of rows across 32 machines, and the merge is the 40 seconds.
- 3Name the mistake
The shard key is not in the hot query. Hashing by event id gave a perfectly even write distribution, which is the thing that looked good for two years, and turned the most important read into a cluster-wide scatter-gather whose cost grows with customer size. Even distribution was optimising the wrong thing.
- 4Choose the key that should have been there
Partition by
(customer_id, timestamp): hash on customer, range on time within it. Now one customer's events live on one partition, and "last 30 days" is a single continuous read. The known risk is that the largest customer becomes a hot partition, which is a real problem affecting a handful of accounts rather than a structural problem affecting every read. - 5Handle the hot customer on purpose
For the few accounts big enough to saturate a partition, sub-partition by adding a bucket suffix (customer 419 becomes 419-0 through 419-7) and fan reads across those eight. Those customers pay a small fan-out; everyone else keeps the single-partition read. Two paths, with a threshold tuned from measurement.
- 6Migrate without downtime
Dual-write to the old and new layouts. Backfill two years of history in batches with pauses, tracking progress so it can be stopped and resumed. Verify with row counts and checksums per customer. Shift dashboard reads to the new layout one percent at a time, watching latency. Only then stop dual-writing, and drop the old table a week later. Six weeks of work, most of it waiting for backfill.
Learn it from the source
This page gets you interview-ready. These take you to where the idea is actually specified or measured — read them when you want the real depth.
- MongoDB — Sharding ↗Shard keys, chunk splitting and balancing, including how a bad shard key ruins you.