Design a Web Crawler: Architecture and Data Flow

Most web crawler explanations describe components in isolation. What actually matters in a system design interview is the data flow between them — what gets passed from stage to stage, in what format, and what happens when a stage fails or falls behind. This article traces a single URL’s journey from seed to storage through five components: the URL frontier queue, fetcher workers, the content parser, the dedup store, and sitemap-based discovery, and shows how backpressure and failure propagate through the pipeline.

The Five-Component Pipeline

At a high level, data flows in a loop:

Seed URLs / Sitemaps → URL Frontier Queue → Fetcher Workers → Content Parser → Dedup Store
                              ↑                                      |
                              └──────────── new URLs extracted ──────┘

Every component in this loop is independently scalable, and each boundary between components is a queue or a store — this is the detail that turns a whiteboard sketch into a credible distributed system.

Component 1: URL Frontier Queue

The frontier is not a single FIFO queue — it’s a priority-and-politeness-aware structure. Each entry carries:

• url (normalized: lowercase host, resolved relative paths, stripped tracking params)
• priority score (from PageRank estimate, sitemap <priority> hint, or business signal)
• earliest fetch time (enforces per-host rate limits)
• depth (crawl depth from seed, used to cap runaway link chains)

Data flow in: new URLs arrive from two sources — the content parser (links extracted from crawled pages) and the sitemap discovery service (bulk URL lists). Both write into the same normalization + dedup-check step before entering the frontier, so a URL is never queued twice.

Data flow out: fetcher workers pull batches keyed to their assigned host-shard, respecting the earliest-fetch-time ordering (typically a min-heap per shard).

Component 2: Fetcher Workers

A fetcher worker does exactly four things per URL: resolve DNS (via a caching resolver), open a connection, issue the HTTP request with a well-identified User-Agent, and stream the response body up to a byte cap (to avoid a 10GB file stalling a worker).

Data flow out: on success, the worker emits a (url, http_status, headers, raw_body, fetch_timestamp) tuple to the parser’s input queue. On failure (timeout, DNS error, 5xx), it emits a (url, error_type, retry_count) tuple to a retry queue with exponential backoff, distinct from the main frontier so failing hosts don’t crowd out healthy ones.

This separation — success path to parser, failure path to retry queue — is the single most important data-flow detail to state explicitly, because it’s what prevents one slow or broken host from degrading throughput for the whole crawl.

Component 3: Content Parser

The parser receives raw bytes and does three jobs: detect content type (HTML vs. PDF vs. binary — route non-HTML to specialized extractors or skip), extract the visible text and metadata for storage, and extract outbound <a href> links plus canonical tags and <meta robots> directives.

Data flow out splits three ways:

1. Extracted content + metadata → dedup store (content-level check) → if unique, persisted to the blob store.
2. Extracted links → normalization → frontier queue (loop closes here).
3. Extracted canonical URL / robots directives → applied back to the URL’s metadata record, so future re-crawls respect noindex or canonicalization without re-parsing.

Component 4: Dedup Store

Two lookups happen here, and it’s worth stating both explicitly rather than treating “dedup” as one step:

• URL-level: before a URL is even queued into the frontier, check a Bloom filter (in-memory, low false-positive rate) backed by a persistent hash set for confirmed hits. This happens at frontier-insertion time, not at parse time.
• Content-level: after parsing, hash the normalized body (SimHash for near-duplicate tolerance) and compare against recently-seen fingerprints. This happens at parse time, after the bytes are already fetched — you can’t avoid the fetch cost, only the storage and link-extraction cost of a duplicate.

Component 5: Sitemap Discovery

Sitemaps are an underused shortcut candidates rarely mention, and naming it signals real-world crawler experience. Rather than relying purely on link-following, a discovery service periodically fetches robots.txt for known hosts, follows the Sitemap: directive to sitemap.xml (or a sitemap index referencing multiple child sitemaps), and bulk-inserts the listed URLs directly into the frontier with the <priority> and <lastmod> hints applied to scoring. This gives near-complete coverage of a site in one request instead of discovering pages incrementally through crawled links, and <lastmod> directly informs the re-crawl scheduler.

Data Flow Comparison: Push vs. Pull Between Stages

Handling Backpressure

If the parser falls behind fetchers (common when content is unusually large or JS-heavy), the fetcher→parser queue grows. The correct response is to throttle fetcher concurrency, not drop messages — dropping a fetched-but-unparsed page wastes the fetch cost and can leave duplicate detection blind to content the crawler already downloaded. Size the fetcher→parser queue with a high-water mark that pages fetcher workers to slow down, mirroring standard producer-consumer backpressure patterns from streaming systems.

Failure Propagation

Each component should fail independently:

• A dead fetcher worker: URLs it held are re-queued by the frontier’s lease-timeout mechanism (similar to a visibility timeout in a message queue).
• A parser crash mid-batch: the fetcher→parser queue is durable (not in-memory only), so unprocessed items survive a parser restart.
• A dedup store outage: fetchers can continue running (fetch cost is sunk either way), but the parser should buffer or pause final persistence rather than skip the dedup check and risk storing rampant duplicates.

Closing the Loop

The full architecture is a closed loop with five independently-scalable stages connected by durable queues, each boundary chosen deliberately as push or pull based on which side is the bottleneck. For a deeper worked example — including capacity estimates and a sample interview transcript for this exact question — see The 0-to-1 SWE Interview Playbook (Amazon: https://www.amazon.com/dp/B0H256Z1MF?tag=sirjohnnymai-20), which walks through the architecture diagram stage by stage the way an interviewer expects to see it built up on a whiteboard.

Practice Prompt

Try sketching this architecture from memory, labeling each arrow with what data crosses it (not just “URLs flow here”) and whether that boundary is push or pull. That level of precision — not just naming the five boxes — is what separates a pass from a borderline result in a 45-minute system design loop.