Follower timeline — a full worked answer

Assumptions: a few hundred million users, global, mostly mobile. Posts are small text plus optional media references. Reads dominate: people open the app far more than they post. Most people follow a few hundred accounts, and a handful of accounts have tens of millions of followers. Auth is solved elsewhere.

First question, and it is the one that changes the most boxes: is the timeline strictly reverse-chronological, or ranked by relevance? Ranked means I cannot simply precompute and append: every read may need scoring, and I need a feature store and a model serving path. I will assume reverse-chronological with a light recency-and-affinity ordering applied at read time, which keeps precomputation viable.

Second question: must a post be visible in followers' feeds immediately, or is a few seconds acceptable? This decides synchronous versus asynchronous fan-out. It sounds like a product detail, but it is the central architectural constraint. I will assume a few seconds is fine.

Scope I propose: post, follow and unfollow, and render the home timeline. Out of scope: ranking models, direct messages, notifications, and media storage. I will treat media as a reference to a separate service rather than pretend it does not exist.

Actors: the author, the followers who read, and the system. The system makes the decisions that matter: whether to fan this post out or leave it to be pulled, when a post is "delivered", what to do with followers who have not opened the app in months, and when to stop trying.

A post's life: composed → accepted and durable → queued for fan-out → written into follower inboxes → read → possibly deleted.

Failure branches. At accepted: the post is stored but the fan-out event is lost, so it never reaches anyone. That is why the event must be written in the same transaction as the post, via an outbox, rather than published separately. At queued: the author has 30 million followers, so this single job is 30 million writes and will flood the queue and delay everyone else's posts. The branch: above a threshold, do not fan out at all, and let readers pull instead. At writing to inboxes: the job crashes after 8 million of 30 million writes, so it retries and rewrites the first 8 million. That is safe only because inbox writes are idempotent on post id, which makes idempotency a requirement here. At read: a follower's inbox is missing entries because their fan-out is still in flight, so their timeline is briefly short rather than wrong, which is acceptable and worth saying. At deleted: the post is now in millions of inboxes and cannot be deleted from all of them quickly. So deletion is a tombstone plus filtering at read time, and the inbox entries are cleaned up lazily.

A follow also has a life worth a sentence: followed → backfill decision. Following someone does not retroactively insert their old posts into your inbox, because that would be another enormous write job. Their older posts appear via the pull path or are not shown. Saying this out loud is a good signal, because it is easy to overlook.

Assume 200 million daily active users. Say each posts 0.5 times a day on average: 100 million posts a day, about 1,000 posts per second, peak maybe 3,000. That is a modest write rate on its own.

Now the amplification, which is the number that matters. Average followers, say 300. Fan-out on write means every post becomes 300 inbox writes, so 1,000 posts a second becomes 300,000 inbox writes a second, peaking near a million. That number drives the design, and it is three hundred times larger than the one you would have quoted if you stopped at "1,000 posts per second".

The tail is worse than the average. One account with 30 million followers posting once produces 30 million writes. At 100,000 inbox writes a second of spare capacity, that single post takes five minutes to deliver and blocks everyone else's posts behind it. The average hides this.

Reads: 200 million users opening the app 10 times a day is 2 billion timeline reads a day, roughly 20,000 per second, peak 60,000. Each read returns maybe 50 posts.

Storage: 100 million posts a day at ~300 bytes is 30 GB a day for posts, which is small. The inboxes are the cost: 100 million posts times 300 followers is 30 billion inbox entries a day, and at ~30 bytes each that is about 900 GB a day, growing with posts times followers. Bound it by capping each inbox at the most recent few hundred entries.

So the hard part here is write amplification, and specifically its tail. Not the post rate, not storage, not read throughput. The design that follows is aimed at the gap between 300 followers and 30 million.

The core decision is push versus pull. Fan-out on write (push) precomputes each user's inbox when the author posts: reads are one sequential lookup, which is what a 20,000-reads-per-second budget wants, and writes are amplified 300x. Fan-out on read (pull) stores posts once and assembles the timeline by querying everyone you follow at read time: writes are trivial, reads are 300 queries plus a merge, which at 20,000 reads a second is impossible. Neither works alone, so use both.

The hybrid. For ordinary accounts, push: fan out on write into follower inboxes. For accounts above a threshold, say 100,000 followers, do not fan out at all. Their posts stay in their own timeline, and at read time each user's feed is their precomputed inbox merged with the recent posts of the handful of huge accounts they follow. Most users follow only a few such accounts, so that merge is a small, cacheable read. The huge accounts' recent posts are the most cacheable data in the system, because millions of people want the same rows.

Justification by number: the 100,000 threshold comes from the amplification maths. Below it, a post costs at most 100,000 writes, about a second of the fleet's spare capacity; above it, the job would monopolise the pipeline. It is a tunable number, and I would tune it by watching queue latency rather than by picking a round figure and defending it.

The inbox itself: a list per user, keyed by user id, holding post ids and timestamps only. Storing post content there would duplicate every post hundreds of times and make an edit a rewrite of every copy. Capped at the most recent 500 entries, because few users scroll past that and unbounded inboxes are how this design runs out of disk. Partitioned by user id, so one user's timeline is one partition and one sequential read. Held in a store built for this shape: a wide-column store or Redis lists, not a relational table with an index.

The write path: post accepted and written to the post store, with a fan-out event written in the same transaction via an outbox. A relay publishes it. Fan-out workers read the event, look up the author's followers, and write inbox entries in batches, idempotent on post id, because these workers will retry. Justified by the Stage 2 branch where a job dies partway through.

The read path: timeline service reads the user's inbox (cached), merges in recent posts from the large accounts they follow (also cached), sorts, then hydrates post ids into full posts from a post-content cache. Hydration is a batch get, not 50 individual lookups.

The author's own post: written to their own inbox synchronously, before the response returns. This costs one extra write and removes a common complaint in this kind of product: posting something and not seeing it.

Counts (likes and replies) do not live in the post row. They are batched and aggregated separately, because a popular post would otherwise turn into a single contended row taking thousands of increments a second.

Hard part one: the huge-account path, in detail. Above the threshold nothing is fanned out, so at read time I need the recent posts of every large account a user follows. I keep a per-author recent-posts list, capped at the last few hundred, and cache it aggressively. This is the highest-value cache in the system: a large account's recent posts are requested by millions of users, so the hit rate approaches 100% and one row serves a huge amount of traffic. Cost: a reader sees a new post from a large account only once that cached list refreshes, so its visibility depends on the cache TTL rather than on fan-out. I accept that, because the alternative is a five-minute fan-out.

Merging the two sources correctly is the fiddly part. The inbox is ordered by time; so are the pulled lists. I merge and take the top N by timestamp. Pagination cannot be an offset, because the two sources shift independently. It has to be a cursor holding the timestamp and post id of the last item shown, and the next page asks both sources for items older than that cursor. Cost: a cursor is more work than an offset and cannot express "jump to page 5", which nobody wants in a feed anyway.

Hard part two: fan-out backpressure. If the queue backs up, timelines silently get stale, and because nothing errors, the team may not notice until users complain. Defences: separate queues by priority so a celebrity backfill cannot delay ordinary posts; alert on queue age rather than queue depth, because age is the number that maps to user pain; and scale workers off backlog growth rather than absolute size. If it backs up badly, I would drop fan-out for inactive users (someone who has not opened the app in 60 days does not need their inbox maintained in real time) and rebuild their timeline via the pull path when they return. Cost: a returning user's first load is slower, which is a good trade against delaying everyone else.

Inactive users are worth raising unprompted. Maintaining inboxes for accounts that never open the app can be a large fraction of total fan-out work, and it produces nothing. Skipping them is one of the biggest savings available here, and the cost is the slower first load above.

Hot key on read: a very large account's recent-posts row is read by millions. One cache key means one hot node. I would replicate that key across several cache nodes with a suffix and read a random one. Cost: N times the memory for those keys and slightly more complex invalidation. Both are small, since there are only a few thousand such accounts.

Consistency, said per feature: the timeline is eventually consistent and may be seconds behind, which is fine and matches how users perceive it. The author's own post is read-your-own-writes, guaranteed by the synchronous self-write. Like counts are eventually consistent and approximate. Follows are strongly consistent, because unfollowing someone and still seeing their posts is a trust problem, not a latency one. So the follow write is synchronous, and the read path checks the follow graph before showing inbox entries.

Deletion, revisited: a deleted post is tombstoned, filtered at hydration time, and its inbox entries are removed lazily. Cost: for a few seconds the post id sits in inboxes and is filtered out on read, which means a timeline can return 49 items instead of 50. Acceptable, and much cheaper than chasing millions of rows synchronously.