Web crawler — a full worked answer
The whole thing, start to finish, in the order you would actually say it. Read it to learn the shape of a strong answer — then go and produce one yourself, because reading is not the same skill as producing.
The question
Design a crawler that fetches a large portion of the web, stores the pages for later processing, and keeps them reasonably fresh. It must not overload the sites it visits and must not fetch the same content repeatedly.
Crawling and ingestion. The work is unbounded, the sources are hostile or fragile, and you are a guest on someone else's machine.
There is no fixed input size: the queue of work generates more work. So the interesting questions are what you refuse to fetch, how you avoid fetching the same thing twice, and how you avoid hurting the sites you depend on.
Before anything: where is the slack?
How long the system has before a person notices. Ask this before you draw a single box — it decides more of the design than scale does.
Nobody is waiting for any single page. This group has the most slack in the app, and most good decisions here come from spending it.
No human is watching a fetch. If one page takes thirty seconds, or fails and is retried in an hour, no user notices. So everything can be queued, batched, retried patiently and processed out of order: the opposite of a request path.
What has no slack is politeness. A site you are crawling has a request path, and their users are waiting. Hitting one domain a thousand times a second is an accidental denial-of-service attack. It gets you blocked, and possibly sued. So your own generous budget has to be converted into a strict per-domain delay.
Freshness has a range rather than a number, and that range is a design decision. A news homepage is stale in minutes. A 2011 blog post is unlikely to change again. Crawling both on the same schedule wastes most of your capacity re-fetching pages that have not changed.
So: Everything is a queue, nothing is synchronous, and the scheduler is built around per-domain politeness rather than global throughput. Recrawl frequency is adaptive, because a uniform schedule spends most of its budget on pages that never change.
Requirements
about 5 minWhy this stage exists: Because the question is deliberately vague, and every box you draw before this is a guess.
Assumptions: this feeds a search index, we want a broad crawl of a few billion pages, HTML only for now with media referenced but not fetched, and we obey robots.txt. That last one is a constraint that shapes the design.
The question that changes a box: how fresh does this need to be? A one-off archive crawl is a much simpler system than one that keeps a few billion pages current. The second needs change detection, per-page recrawl scheduling, and a way to decide what is worth revisiting. I will assume continuous recrawling with adaptive frequency, because that is what a search product needs.
Second question: do we need JavaScript rendering? That changes the cost per page by an order of magnitude or more, because a headless browser uses far more CPU and memory than an HTTP fetch. I will assume a hybrid: plain fetch by default, rendering only for pages where the plain fetch returns almost no content, which is a small fraction.
Scope I propose: discovering URLs, scheduling politely, fetching, deduplicating content, storing raw pages, and extracting links to continue. Out of scope: the index itself, ranking, and interpreting the content. That is a separate system that consumes what this one produces.
What an interviewer is ticking off in this stage
- Established the purpose, because it changes what "done" means
- Asked about freshness and tied it to recrawl scheduling
- Asked about JavaScript rendering and named the cost multiplier
- Committed to robots.txt as a design constraint, not an afterthought
- Proposed a scope that ends where the index begins
Actors and lifecycle
about 6 minWhy this stage exists: Because the happy path is the easy half. The failure branches are where the design actually lives.
Actors: the seed source that gives you starting URLs; every website you visit (millions of actors, each with its own rules, capacity and tolerance for you); the DNS system, which you will hit very hard; and the crawler itself. The crawler makes constant decisions that nobody sees: is this URL worth fetching, is it the same page as one I already have, how long until I look again, and should I give up on this domain entirely.
The chain: discovered → normalised → deduplicated against what we know → checked against robots → scheduled by domain → fetched → content deduplicated → stored → links extracted, which feeds back to discovered → scheduled for recrawl.
The failure branches are most of this system. At normalised: the same page has five URLs with different tracking parameters and a trailing slash, so without normalisation you crawl it and store it five times. At robots: fetching robots.txt is itself a request, so it is cached per domain with a TTL, or you double your traffic to every site. At fetch: timeouts, 404s, 500s, redirect chains that loop, and pages that are 40 MB of nothing. Each needs a limit and a give-up rule. At content dedup: two different URLs return byte-identical or near-identical content, which is common with mirrors and printer-friendly pages, so you need a content fingerprint as well as a URL check.
The branches the system owns: a site that starts returning errors, where continuing to hammer it is both rude and pointless. A per-domain circuit breaker backs off entirely. And a crawler trap: a calendar page that generates an infinite sequence of links, or a site with a URL pattern that expands forever. Without a per-domain page cap and a depth limit, one such site can consume your crawler indefinitely.
What an interviewer is ticking off in this stage
- URL normalisation before deduplication — the same page has many URLs
- robots.txt cached per domain, since fetching it is itself traffic
- Limits on redirect chains, response size and timeouts
- Content fingerprinting separate from URL deduplication
- A per-domain breaker for a site that starts failing
- Crawler traps named, with a depth limit and per-domain cap
The lifecycle, drawn
Numbers
about 6 minWhy this stage exists: Because one part of this is hard and the rest is easy, and they look identical until you do the arithmetic.
Target: 4 billion pages, recrawled on average monthly. That is 4 billion fetches per month, divided by 2.6 million seconds, so about 1,500 pages per second sustained. Not extreme, but it has to run continuously for years.
Storage: an HTML page is about 100 KB raw, roughly 25 KB compressed. 4 billion pages compressed is about 100 TB for one copy of the web. That goes to object storage, not a database. At list prices that is roughly a few hundred dollars a month in an infrequent-access archive tier and about two thousand in standard storage. Worth saying, because candidates often assume storage is the expensive part.
The number that shapes the design: the seen-URL set. 4 billion URLs at 100 bytes each is 400 GB if stored as strings. That is too big to hold comfortably in memory on one machine, and link extraction means checking far more URLs than you fetch (perhaps 50,000 per second), which is a lot of random lookups for a disk-backed database. A bloom filter of 4 billion entries at a 1% error rate is about 5 GB, which fits in memory comfortably. The cost is that about 1% of new URLs are wrongly reported as already seen and never crawled, which is acceptable for a broad crawl.
DNS: 1,500 fetches a second across many domains means a lot of lookups, and resolution becomes a bottleneck if uncached. Aggressive DNS caching is a specific, named component here rather than an implementation detail.
So the hard part here is the seen-set at 4 billion entries and per-domain politeness, not fetch throughput. One is a membership problem and the other a scheduling problem; neither is about bandwidth.
What an interviewer is ticking off in this stage
- Turned a target into a sustained pages-per-second number
- Estimated compressed storage and put it in object storage, not a database
- Calculated the seen-set size and showed why a plain set does not fit
- Chose a bloom filter and named the false-positive consequence
- Named DNS as a real bottleneck at this rate
- Finished the sentence: the seen-set and politeness, not throughput
The numbers, drawn
So the hard part is membership at 4 billion entries, plus per-domain politeness. Storage is 100 TB compressed in object storage, which costs less than candidates tend to expect and is not the constraint.
High-level design
about 12 minWhy this stage exists: Because now — and only now — you have a reason for every box you are about to draw.
The frontier is the core of the system, and it is not one queue. It is a set of per-domain queues, plus a scheduler that only releases a URL from a domain when that domain's politeness delay has elapsed. The unit of scheduling is the domain, not the URL. Workers ask the frontier for work and are handed URLs from domains that are ready, so no worker is ever waiting on a delay.
Partitioning by domain follows from that. All URLs for one domain are handled by one frontier partition, so the politeness state (last fetch time, current delay, breaker status) lives in one place with no coordination. This is the same "partition by the thing that needs ordering" move as chat, applied to rate rather than sequence.
The seen-set service holds the bloom filter, sharded by URL hash, and answers "have we seen this" for the tens of thousands of extracted links per second. New URLs are added and passed to the frontier. It is fronted by an exact store for the recently added, so the filter can be rebuilt periodically without losing the last hour.
Fetchers are stateless and numerous. Each has a timeout, a response size cap, a redirect limit, and a per-domain breaker. They resolve DNS through a heavily cached resolver, fetch, and hand the raw response on. They never decide what to fetch next; the frontier does.
Content processing computes a fingerprint of the normalised text, checks it against a content-seen store, and drops near-duplicates. The raw page goes to object storage keyed by content hash, which means identical content from ten URLs is stored once. Extracted links go back to the seen-set service. Metadata (URL, fetch time, status, content hash, change signal) goes into a database that the recrawl scheduler reads.
The recrawl scheduler is what makes freshness affordable. Each page carries a next-fetch time, adjusted by whether the content changed last time: pages that change get revisited sooner, and pages that do not decay toward a long interval. Most pages fall into the second group, which is why this box earns its place.
What an interviewer is ticking off in this stage
- The frontier is per-domain queues plus a scheduler, not one global queue
- Partitioned by domain, so politeness state needs no coordination
- Seen-set as a sharded bloom filter service with an exact recent layer
- Fetchers stateless, with timeouts, size caps and per-domain breakers
- Raw pages in object storage keyed by content hash, so duplicates store once
- Adaptive recrawl scheduling driven by observed change
The architecture, drawn
Deep dive and tradeoffs
about 14 minWhy this stage exists: Because naming what a decision costs is the single clearest signal of seniority there is.
Hard part one: politeness against throughput. These are in direct conflict. My fleet can fetch 1,500 pages a second, but a single site might tolerate one request every two seconds. So a domain with 500 million pages would take 30 years at that rate. For large sites I need a middle position: honour crawl-delay where robots.txt specifies one, and otherwise adapt. Start conservative, watch response times and error rates, and speed up only while the site is answering quickly. If latency rises or errors appear, back off immediately. That is a control loop, not a constant. Cost: complexity in the scheduler, and I will never fully crawl the largest sites. So coverage is a decision, and I would prioritise by page importance rather than pretending completeness is possible.
Hard part two: what to recrawl. I cannot refetch 4 billion pages daily, so priority has to be computed. Signals: how often this page changed in the past, how important it is by inbound links, and how important its site is. A news homepage that changed every one of the last ten fetches goes on a minutes-to-hours cycle. A page unchanged for two years decays to quarterly. Cheap change detection helps a lot. A conditional request (If-Modified-Since or If-None-Match with the stored ETag) gets a small 304 "not modified" response instead of the page from servers that support it, so many recrawls cost almost nothing in bandwidth. Cost: per-URL state for billions of URLs, which needs a proper database, and a scoring model that can be wrong. It will miss a page that suddenly starts changing, so I would keep a floor on recrawl frequency rather than letting anything decay to never.
Traps and hostile sites. A calendar with infinite next-month links generates URLs forever. Defences: a maximum crawl depth, a per-domain page budget, and detection of URL patterns that keep producing near-identical content. Without these, one site can fill the frontier without any error being raised. Cost: legitimate deep sites get truncated, so the budget needs to scale with site importance rather than being fixed.
Operating it. This runs for years, so the metrics that matter are pages per second by status code, frontier depth per partition, seen-set false-positive rate, and the ratio of not-modified to full fetches. That last one tells me whether my recrawl scheduling is working. Cost: small, and skipping it means never knowing that a scheduling change made things worse.
What an interviewer is ticking off in this stage
- Named politeness against throughput as a direct conflict, with a control loop as the answer
- Accepted that the largest sites will never be fully crawled, and prioritised instead
- Adaptive recrawl driven by observed change, with a floor so nothing decays to never
- Conditional requests so unchanged pages cost almost no bandwidth
- Depth limits and per-domain budgets for crawler traps
- Named the metric that tells you whether recrawl scheduling is working
And then they push back
A complete answer is where the interview starts, not where it ends. These are the questions this design invites — the interview is decided here far more often than in the diagram.
One key gets 90% of the traffic. What breaks, and what do you do?
"Consistent hashing spreads the load evenly." It spreads keys evenly, not traffic. All requests for one key go to one place no matter how the keys are hashed. This answer confuses two different problems.
"Whichever node owns that key is saturated while the rest idle, and no partitioning scheme fixes it because partitioning distributes keys, not requests for a single key. Three options. Replicate the hot key across N nodes with a suffix and read a random one. It costs N times the memory and a fanned-out invalidation, and it is the right answer for a read-hot key. For a write-hot key, shard the counter and sum on read, accepting that the total is now a computed value. Or handle it on a separate path: an in-process cache on every server absorbs a read-hot key almost completely. I would also detect hot keys automatically with a count-min sketch rather than maintaining a manual list, because yesterday's hot key is not today's."
A third-party API you depend on starts taking 30 seconds to respond. What happens?
"We would retry." Retrying against something already overloaded makes it worse, and retrying a 30-second call means holding your resources for a minute or more. This answer usually turns one slow dependency into your own outage.
"Slow is more dangerous than down, because down fails fast and slow holds resources. Without a timeout, every request calling that API holds a thread and a connection for 30 seconds, my pools exhaust, and endpoints that never touch that API start failing. One slow dependency has become a total outage. So: a timeout set from their real latency distribution, not a round number; a circuit breaker that opens on failure rate so I stop waiting entirely once it is clearly bad; a bulkhead giving that dependency its own limited pool so it cannot consume capacity the checkout path needs; and a defined fallback — cached data, a degraded response, or dropping an optional feature. The fallback has to be decided before the incident, per dependency, or I fail fast into a blank page. And retries need backoff with jitter and a cap, or I contribute to keeping them down."
Your storage bill doubles every six months. What do you do?
"We would delete old data." Which data, and who agreed you could? Retention is usually a product and legal decision, not an engineering one.
"First, find out what is actually growing, because it is rarely uniform — usually one dataset, often logs or raw events or derived copies nobody remembers creating. Then, in order of cheapness: lifecycle policies moving cold data to colder tiers, which is cheap to implement and can cut the bill sharply when most data is rarely read after its first week; compression and columnar formats, which for event data is a 5-10x reduction; tiered granularity, keeping recent data fine-grained and rolling older data up; and only then deletion, which needs a retention policy agreed with the product and legal side rather than decided by me. I would also check for duplication — a derived index, a backup of a backup, an old migration copy — because a surprising share of storage growth is copies. The one thing I would not do is delete raw data that lets me reprocess when a job turns out to be wrong, because that is the safety net that makes everything else correctable."
How do you know when you are about to run out of capacity?
"We would monitor utilisation." Of what resource, and against what threshold? Utilisation of the wrong resource is how people get surprised.
"I would identify the constraining resource per component, because it is different for each and rarely CPU. For the connection servers it is memory and file descriptors. For the database it is write throughput and connection count. For the queue it is consumer throughput against arrival rate. Then track headroom against those specific limits, and trend them, because the useful alert is not 'we are at 70%' but 'at the current growth rate we hit the ceiling in three weeks'. That gives time to act rather than a surprise. I would also load test to find the real ceiling rather than assuming a number, because the actual failure point is usually lower than the theoretical one and often in an unexpected place: a connection pool, a lock, a single-threaded component. And I would test the degraded case: what capacity do I have with a third of the fleet gone, because that is the number that matters during an incident."
Could you build this with half the components? Which would you cut?
"Every component is necessary." Rarely true, and it signals that you added things by pattern rather than by requirement.
"Yes, and I would cut in this order. The queue on the write path goes first — at my write rate a synchronous call is simpler and gives the user a real answer. The separate search index goes next, because the built-in full-text search handles this data size and one system beats two. The in-process cache layer goes, keeping just the shared cache, because the extra hit rate is not worth servers disagreeing with each other. What I would not cut: the shared cache, because the database cannot take the read load, and the idempotency keys, because at-least-once delivery is not optional. If I were shipping this in a week, I would build the database, the cache and the stateless services, and add the rest when a measurement told me to. Most of what I drew is a response to scale I have estimated but not yet reached."
Now it must support scheduled actions — do this thing at 8am next Tuesday.
"We would use a cron job." Cron per user does not scale, does not survive a machine dying, and has no answer for what happens when a scheduled time is missed.
"I would store scheduled items in a table indexed by due time, and have workers poll for items due now. That gives durability, survives restarts, and lets me see and cancel what is pending, none of which cron gives me. The interesting parts are the failure branches. If the system is down at 8am, the item is late rather than lost, and I need a policy: run it late, or skip it, and that differs per action — a reminder is worth sending late, a market order is not. Duplicate execution must be harmless, so each scheduled item carries an id and execution is idempotent. Time zones and daylight saving cause bugs here: '8am next Tuesday' means 8am in someone's local zone, which is a different UTC instant depending on the date, so I store the intent rather than a resolved timestamp. And scheduled times cluster on round hours like 9am, so the due-time index gets hot and I would jitter execution by a small window where the exact second does not matter."
A worker dies halfway through processing a batch. What is the state of the world?
"The job would be retried." Retried from where? If the first half already had side effects, retrying repeats them, and now you have sent 5,000 emails twice.
"It depends on whether I made the work idempotent, which is a decision I have to have made beforehand. The message was not acknowledged, so it is redelivered. That is at-least-once delivery, and it is the only guarantee I can get. So a retry reprocesses the first half. If each item's effect is idempotent — an upsert by a natural key, setting a status rather than incrementing — the retry is harmless and this is a non-event. If the effects are external, like sending an email or charging a card, each one needs its own idempotency key carried to the provider, and I need to record progress per item rather than per batch so a retry skips what is done. The design lesson is to record completion in the same transaction as the work where possible, so there is no window where the work happened and the record of it did not."
You grow 10x every year for three years. What do you rebuild?
"The design already scales to any size." No design does, and claiming so means you have not identified where your assumptions run out.
"Year one, nothing. I sized the current design with headroom and it absorbs 10x. Year two, the single-primary database stops being enough and I have to partition, which is the expensive one: it means choosing a shard key, dual-writing, backfilling and cutting over, and it takes a quarter. That is why I chose the data model on day one so the natural shard key was already the primary access key. The migration is painful, but it is not a redesign. Year three, the thing I expect to hurt is not capacity but blast radius and operations: one global system serving 1,000x the original traffic needs regional isolation so one failure does not take everything down. So I rebuild the storage layer once, and I plan for it by picking the key correctly now."
The topics this answer used
You have read a good answer. That is worth very little until you can produce one on a blank page — so go and write this problem without looking, and compare afterwards.