Design a web crawler

Design a system that crawls the web: fetches pages, extracts links, and keeps going. Assume you want to crawl billions of pages and keep them reasonably fresh.