System design interview questionMediumScheduling / Crawl
Design a Web Crawler
Design a web crawler: discover URLs, fetch pages asynchronously, store content/metadata for later search/index use.
Interview scope
Producer/Client → Gateway/LB → App → Message Queue; workers (App) consume Queue; App → DB and Object Storage. No Client → Queue/DB.
Functional requirements
- Enqueue URLs to crawl.
- Workers fetch pages and extract new links.
- Persist page content or metadata.
Scale and quality goals
- Crawl must be async — don’t block on every HTTP fetch in one request thread.
- Scale by adding workers consuming a queue.
- Politely avoid hammering one host forever (high level).
Capacity assumption
Design for this scale
Assume ~100M pages/month ≈ ~40 pages/s average; bursts higher — queue + workers absorb spikes.
Practice more than a diagram
Place components on the Froquiz canvas, connect the data flow, and see which decisions the scenario-specific review finds missing. The full exercise and review unlock with Premium.
Explore Premium →