Dev Learning
Crawling and ingestion

Web crawler

Design a crawler that fetches a large portion of the web, stores the pages for later processing, and keeps them reasonably fresh. It must not overload the sites it visits and must not fetch the same content repeatedly.

coreBloom filtersDNS — how a name becomes a serverMessage queues and backpressureRate limitingObject storage and presigned uploadsPartitioning (sharding)Circuit breakers and retries with jitter

Where’s the slack?

Per page: unlimited. Full recrawl: days. Freshness for news: minutes.

Nobody is waiting for any single page. This group has the most slack in the app, and most good decisions here come from spending it.

No human is watching a fetch. If one page takes thirty seconds, or fails and is retried in an hour, no user notices. So everything can be queued, batched, retried patiently and processed out of order: the opposite of a request path.

What has no slack is politeness. A site you are crawling has a request path, and their users are waiting. Hitting one domain a thousand times a second is an accidental denial-of-service attack. It gets you blocked, and possibly sued. So your own generous budget has to be converted into a strict per-domain delay.

Freshness has a range rather than a number, and that range is a design decision. A news homepage is stale in minutes. A 2011 blog post is unlikely to change again. Crawling both on the same schedule wastes most of your capacity re-fetching pages that have not changed.

So: Everything is a queue, nothing is synchronous, and the scheduler is built around per-domain politeness rather than global throughput. Recrawl frequency is adaptive, because a uniform schedule spends most of its budget on pages that never change.

Work it through

Five stages, gated. You write your answer for each stage before its model answer unlocks. That is the core of the method.

Loading your progress…

Reference material for this problem. The diagrams below are here to check your own against once you have drawn it, not to read first.

The lifecycle

Discoveredlink extractionNormalised + deduplicatedsystemScheduled by domainfrontierFetchedfetcherContent deduplicated + storedsystemScheduled for recrawlsystemSame page, five URLsnormalise parameters and trailing slashes before the seen-set checkrobots.txt disallows itcached per domain with a TTL, or fetching it doubles your trafficInfinite calendar linksdepth limit and per-domain page budget, or one site eats the frontierTimeout, 40 MB page, redirect loopexplicit caps on time, size and redirect count — each is a separate limitSite starts returning 500sper-domain breaker backs off entirely; hammering is rude and pointlessByte-identical to a mirrorcontent fingerprint, stored once by content hash
A URL's life. At this scale, most branches below happen many times a day, and each one is a limit you have to set explicitly.

The numbers

Sustained fetch rate~1,500 pages / sec
Seen-set as plain strings~400 GB — will not fit in memory
Seen-set as a bloom filter~5 GB at 1% error — fits easily

So the hard part is membership at 4 billion entries, plus per-domain politeness. Storage is 100 TB compressed in object storage, which costs less than candidates tend to expect and is not the constraint.

The seen-set, not the bandwidth, is what decides the data structures on this board.

The architecture

ready?new linksunseen onlyrecrawl dueFrontierper-domain queuesSchedulerpoliteness delayFetchersstateless, breakersDNS cacheThe webProcessorfingerprint + linksSeen-setbloom, ~5 GBObject storagekeyed by content hashURL metadatadrives recrawl
The frontier schedules domains, not URLs. The seen-set answers membership in memory. Pages land in object storage keyed by content hash, so mirrors cost nothing.

Follow-ups you should expect

Many interviews are decided in the follow-ups, not the first diagram.

One key gets 90% of the traffic. What breaks, and what do you do?A third-party API you depend on starts taking 30 seconds to respond. What happens?Your storage bill doubles every six months. What do you do?How do you know when you are about to run out of capacity?Could you build this with half the components? Which would you cut?Now it must support scheduled actions — do this thing at 8am next Tuesday.A worker dies halfway through processing a batch. What is the state of the world?You grow 10x every year for three years. What do you rebuild?

Same shape, other products

The work is unbounded, the sources are hostile or fragile, and you are a guest on someone else's machine.

There is no fixed input size: the queue of work generates more work. So the interesting questions are what you refuse to fetch, how you avoid fetching the same thing twice, and how you avoid hurting the sites you depend on.

Others with this shape: Web crawler · Feed and API ingestion · Data scraping pipelines

Before you leave
Try this problem again in a week without reading anything. Comparing what you write then with what you wrote today shows you what stuck.
Stuck, or new to this?Read the full worked solutionAll five stages end to end, with the diagrams and the costs, written out the way you would say it. Best read after you have had a go.Read it →

How it is really built

Write your own answer first. Then read how teams who run this at scale describe it. Their constraints were real, and the write-ups often say what their choices cost.

  • Percolator (Google, OSDI 2010) ↗How Google moved its index from batch rebuilds to incremental updates as pages are recrawled. Useful for the "what happens after the fetch" follow-up.