Designing a Web Crawler: URL Frontiers, Politeness Policies, and Distributed Crawling at Scale

Most engineers understand what a web crawler does. Fewer have thought through what it takes to build one that runs at scale without getting banned, wasting compute on duplicate content, or hammering servers into the ground. This guide walks through the decisions that matter: frontier management, politeness enforcement, deduplication, distributed coordination, and re-crawl scheduling.

The URL Frontier

The frontier is the queue of URLs waiting to be crawled. At small scale a simple FIFO queue works. At scale you need something more structured because naive FIFO breaks two things: it lets you send too many requests to a single host too quickly, and it gives you no control over which URLs get crawled first.

A production frontier separates concerns into two layers: a priority queue that ranks URLs by importance, and a politeness queue that groups URLs by host and enforces per-host rate limits.

interface FrontierURL {
  url: string;
  host: string;
  priority: number;       // higher = crawl sooner
  scheduledAt: number;    // unix ms, earliest allowed fetch time
  depth: number;
  discoveredFrom: string | null;
}

class URLFrontier {
  // Priority queue ordered by (priority DESC, scheduledAt ASC)
  private priorityQueue: MinHeap<FrontierURL>;
  // Per-host queues for politeness scheduling
  private hostQueues: Map<string, FrontierURL[]>;
  // Tracks the next allowed fetch time per host
  private hostNextFetch: Map<string, number>;

  enqueue(url: FrontierURL): void {
    const host = url.host;
    if (!this.hostQueues.has(host)) {
      this.hostQueues.set(host, []);
    }
    this.hostQueues.get(host)!.push(url);
    this.priorityQueue.push(url);
  }

  dequeue(): FrontierURL | null {
    const now = Date.now();
    // Walk candidates in priority order, skip those whose host isn't ready
    while (!this.priorityQueue.isEmpty()) {
      const candidate = this.priorityQueue.peek()!;
      const nextAllowed = this.hostNextFetch.get(candidate.host) ?? 0;
      if (nextAllowed <= now) {
        this.priorityQueue.pop();
        return candidate;
      }
      // All top candidates are host-blocked: sleep or try next worker
      break;
    }
    return null;
  }

  markFetched(host: string, crawlDelayMs: number): void {
    this.hostNextFetch.set(host, Date.now() + crawlDelayMs);
  }
}

The key invariant: you never issue two requests to the same host without waiting the required delay between them.

Priority Assignment

Priority signals tell the crawler what to crawl first. Common inputs:

• PageRank or link in-degree: pages with more inbound links are more likely to be valuable.
• Freshness decay: pages that change frequently should be revisited sooner.
• Depth: pages discovered near the seed set are usually higher quality.
• Domain authority: authoritative domains get priority over newly discovered ones.

function computePriority(url: FrontierURL, linkInDegree: number, changeRate: number): number {
  const depthPenalty = Math.max(0, 1 - url.depth * 0.05);
  const linkScore = Math.log1p(linkInDegree) / 10;
  const freshScore = changeRate; // 0-1, estimated from past crawls
  return depthPenalty * 0.4 + linkScore * 0.35 + freshScore * 0.25;
}

Politeness Policies

A crawler that ignores politeness will get IP-banned within hours. There are three things you must handle: robots.txt, per-host crawl delays, and connection limits.

Parsing robots.txt

robots.txt specifies which paths are off-limits and, optionally, a crawl delay. Parse it once per domain and cache the result.

interface RobotRules {
  disallowedPrefixes: string[];
  crawlDelayMs: number;
  sitemapUrls: string[];
}

function parseRobotsTxt(content: string, userAgent: string): RobotRules {
  const lines = content.split('\n').map(l => l.trim());
  const rules: RobotRules = { disallowedPrefixes: [], crawlDelayMs: 1000, sitemapUrls: [] };

  let applicable = false;

  for (const line of lines) {
    if (line.startsWith('#') || line === '') continue;

    if (line.toLowerCase().startsWith('user-agent:')) {
      const agent = line.split(':')[1].trim();
      applicable = agent === '*' || agent.toLowerCase() === userAgent.toLowerCase();
      continue;
    }

    if (line.toLowerCase().startsWith('sitemap:')) {
      rules.sitemapUrls.push(line.split(':').slice(1).join(':').trim());
      continue;
    }

    if (!applicable) continue;

    if (line.toLowerCase().startsWith('disallow:')) {
      const path = line.split(':')[1].trim();
      if (path) rules.disallowedPrefixes.push(path);
    }

    if (line.toLowerCase().startsWith('crawl-delay:')) {
      const delay = parseFloat(line.split(':')[1].trim());
      if (!isNaN(delay)) rules.crawlDelayMs = delay * 1000;
    }
  }

  return rules;
}

function isAllowed(url: string, rules: RobotRules): boolean {
  const path = new URL(url).pathname;
  return !rules.disallowedPrefixes.some(prefix => path.startsWith(prefix));
}

Cache robots.txt per domain for at least 24 hours. Refetch it on a longer interval (once per week for stable domains). If the fetch fails, back off and assume everything is disallowed until you can confirm.

Per-Host Rate Limiting

Beyond the Crawl-Delay directive, you should apply your own defaults. A reasonable floor is one request per second per host. For hosts that return 429s, apply exponential backoff:

class HostRateLimiter {
  private backoffMs: Map<string, number> = new Map();
  private baseDelayMs = 1000;

  getDelay(host: string): number {
    return this.backoffMs.get(host) ?? this.baseDelayMs;
  }

  onSuccess(host: string): void {
    // Slowly recover toward base delay after successful responses
    const current = this.backoffMs.get(host) ?? this.baseDelayMs;
    this.backoffMs.set(host, Math.max(this.baseDelayMs, current * 0.9));
  }

  onRateLimited(host: string): void {
    const current = this.backoffMs.get(host) ?? this.baseDelayMs;
    this.backoffMs.set(host, Math.min(current * 2, 60_000));
  }
}

Content Deduplication

The web is full of near-identical pages: printer-friendly versions, session ID variants, tracking parameter URLs, and scraped mirrors. Without deduplication you waste crawl budget and pollute your index.

URL Normalization

Before enqueueing, normalize every URL:

function normalizeURL(raw: string): string {
  const url = new URL(raw);

  // Force lowercase scheme and host
  url.hostname = url.hostname.toLowerCase();

  // Remove tracking parameters
  const trackingParams = ['utm_source', 'utm_medium', 'utm_campaign', 'ref', 'fbclid', 'gclid'];
  for (const param of trackingParams) {
    url.searchParams.delete(param);
  }

  // Sort query parameters for canonical form
  url.searchParams.sort();

  // Strip default ports
  if ((url.protocol === 'https:' && url.port === '443') ||
      (url.protocol === 'http:' && url.port === '80')) {
    url.port = '';
  }

  // Remove trailing slash from paths (except root)
  if (url.pathname.length > 1 && url.pathname.endsWith('/')) {
    url.pathname = url.pathname.slice(0, -1);
  }

  // Drop fragment identifiers
  url.hash = '';

  return url.toString();
}

Store normalized URLs in a Bloom filter for fast membership checks before doing any network work. A Bloom filter with a 1% false positive rate uses roughly 10 bits per element. For 10 billion URLs that is about 12 GB, which fits comfortably in memory on a single machine.

SimHash Fingerprinting for Near-Duplicate Detection

URL normalization catches exact duplicates. SimHash catches near-duplicates: pages with the same content but different headers, footers, or ads.

SimHash produces a 64-bit fingerprint. Two documents with Hamming distance below a threshold (typically 3) are considered duplicates.

function simhash(tokens: string[]): bigint {
  const v = new Array(64).fill(0);

  for (const token of tokens) {
    const h = fnv1a64(token); // 64-bit FNV-1a hash
    for (let i = 0; i < 64; i++) {
      if ((h >> BigInt(i)) & 1n) {
        v[i]++;
      } else {
        v[i]--;
      }
    }
  }

  let fingerprint = 0n;
  for (let i = 0; i < 64; i++) {
    if (v[i] > 0) {
      fingerprint |= (1n << BigInt(i));
    }
  }
  return fingerprint;
}

function hammingDistance(a: bigint, b: bigint): number {
  let diff = a ^ b;
  let count = 0;
  while (diff) {
    diff &= diff - 1n;
    count++;
  }
  return count;
}

function isDuplicate(a: bigint, b: bigint, threshold = 3): boolean {
  return hammingDistance(a, b) <= threshold;
}

For lookup at scale, split the 64-bit fingerprint into four 16-bit segments and build an inverted index per segment. Documents sharing any segment are near-duplicate candidates. This reduces the search space from O(N) to O(k) where k is the number of documents sharing a segment.

Distributed Crawler Architecture

A single-machine crawler tops out around 50-200 requests per second due to DNS resolution time, TCP handshake overhead, and connection limits. Beyond that, you need horizontal distribution.

Partition-Based URL Assignment

Assign URLs to workers by consistent hashing on the host. This keeps all requests for a given host routed to the same worker, which simplifies politeness enforcement: each worker owns its own HostRateLimiter and does not need coordination for per-host delays.

import { createHash } from 'crypto';

function assignWorker(host: string, workerCount: number): number {
  const hash = createHash('sha256').update(host).digest();
  // Use first 4 bytes as uint32
  const n = hash.readUInt32BE(0);
  return n % workerCount;
}

When workers are added or removed, consistent hashing minimizes reassignment: only URLs on the moved partition need to migrate.

Worker Coordination

Workers pull from a shared frontier backed by Redis or a purpose-built queue like Apache Kafka. The frontier partitioned by host means each worker reads only its assigned partitions:

interface CrawlTask {
  url: string;
  host: string;
  priority: number;
  scheduledAt: number;
}

class DistributedWorker {
  private workerId: number;
  private workerCount: number;
  private rateLimiter: HostRateLimiter;
  private robotsCache: Map<string, RobotRules>;

  async run(): Promise<void> {
    while (true) {
      const task = await this.fetchTask();
      if (!task) {
        await sleep(100);
        continue;
      }

      const allowed = await this.checkRobotsPolicy(task);
      if (!allowed) continue;

      const delay = this.rateLimiter.getDelay(task.host);
      await sleep(delay);

      try {
        const result = await this.fetch(task.url);
        this.rateLimiter.onSuccess(task.host);
        await this.processResult(result);
      } catch (err) {
        if (isRateLimitError(err)) {
          this.rateLimiter.onRateLimited(task.host);
        }
        await this.requeueWithBackoff(task);
      }
    }
  }

  private isMyTask(host: string): boolean {
    return assignWorker(host, this.workerCount) === this.workerId;
  }
}

DNS Resolution Caching and Connection Pooling

DNS resolution is a hidden bottleneck. At 100 req/s, uncached DNS resolution can add 20-100ms of latency per request and become the dominant cost. Cache DNS results with a TTL that matches the domain’s actual DNS TTL, but floor it at 60 seconds.

class DNSCache {
  private cache: Map<string, { addresses: string[]; expiresAt: number }> = new Map();

  async resolve(hostname: string): Promise<string[]> {
    const cached = this.cache.get(hostname);
    if (cached && cached.expiresAt > Date.now()) {
      return cached.addresses;
    }

    const result = await dnsResolve(hostname);
    const ttlMs = Math.max(result.ttl * 1000, 60_000);
    this.cache.set(hostname, {
      addresses: result.addresses,
      expiresAt: Date.now() + ttlMs,
    });
    return result.addresses;
  }
}

Pair DNS caching with an HTTP connection pool that reuses TCP connections per host. Node’s http.Agent supports this natively:

import http from 'http';
import https from 'https';

const httpAgent = new http.Agent({ keepAlive: true, maxSockets: 4 });
const httpsAgent = new https.Agent({ keepAlive: true, maxSockets: 4 });

Keep per-host socket counts low (2-8). Opening too many parallel connections to a single host is both impolite and likely to trigger bans.

Handling JavaScript-Rendered Pages

A growing share of the web requires JavaScript execution to produce meaningful content. For a general-purpose crawler, you need a headless browser tier alongside your standard HTTP fetcher.

Route URLs to the appropriate fetcher based on signals from prior crawls or explicit configuration:

type FetcherType = 'http' | 'headless';

interface FetchResult {
  url: string;
  statusCode: number;
  body: string;
  fetcherUsed: FetcherType;
  renderedAt: number;
}

async function fetchPage(url: string, preferHeadless: boolean): Promise<FetchResult> {
  if (!preferHeadless) {
    const result = await httpFetch(url);
    // Heuristic: if body is tiny or contains known SPA markers, retry with headless
    if (result.body.length < 500 || result.body.includes('id="root"></div>')) {
      return headlessFetch(url);
    }
    return result;
  }
  return headlessFetch(url);
}

Headless rendering (via Playwright or Puppeteer) is 10-50x more expensive than plain HTTP fetches. Use it selectively. A two-pass approach works well: fetch with HTTP first, promote to headless only when the response body fails a content quality check.

Incremental Re-Crawling

You cannot crawl the entire web once and be done. Pages change. New content appears. A re-crawl strategy decides when to revisit each page.

The simplest model uses change frequency estimates derived from prior crawls. If a page changed on 3 of the last 5 visits, its estimated change rate is 0.6. Schedule the next visit proportionally:

interface CrawlRecord {
  url: string;
  lastCrawledAt: number;
  changeCount: number;
  visitCount: number;
  contentHash: string;
}

function nextCrawlAt(record: CrawlRecord, now: number): number {
  const changeRate = record.changeCount / Math.max(record.visitCount, 1);

  // High change rate: revisit every hour. No changes: revisit weekly.
  const minIntervalMs = 60 * 60 * 1000;       // 1 hour
  const maxIntervalMs = 7 * 24 * 60 * 60 * 1000; // 7 days

  const intervalMs = minIntervalMs + (1 - changeRate) * (maxIntervalMs - minIntervalMs);
  return now + intervalMs;
}

function updateRecord(record: CrawlRecord, newHash: string, now: number): CrawlRecord {
  const changed = newHash !== record.contentHash;
  return {
    ...record,
    lastCrawledAt: now,
    changeCount: record.changeCount + (changed ? 1 : 0),
    visitCount: record.visitCount + 1,
    contentHash: newHash,
  };
}

For news sites or social feeds where freshness matters, augment frequency estimates with explicit signals: sitemap <lastmod> dates, Last-Modified and ETag headers. A conditional GET with If-None-Match or If-Modified-Since lets you check for changes without re-downloading the full body.

Crawling Strategy Tradeoffs

Strategy	Order	Pros	Cons	When to Use
Breadth-first	Layer by layer from seeds	Finds high-authority pages fast, good link graph coverage	Memory-heavy frontier, slower depth discovery	General-purpose crawls, building link graphs
Depth-first	Follow links deeply before backtracking	Low memory frontier, good for focused topic crawls	High risk of spider traps, poor coverage of diverse domains	Focused crawlers targeting known site structures
Priority-based	Rank by score (PageRank, freshness, domain authority)	Best crawl budget efficiency, finds valuable pages first	Requires score computation infrastructure, more complex	Production search indexers, news crawlers
Domain-focused	Enumerate one domain fully before moving on	Easy politeness enforcement, deep coverage per domain	Poor at discovering cross-domain link patterns	Site monitors, archival crawlers
Sitemap-first	Parse sitemaps before following links	Respects site owner intent, faster coverage of canonical URLs	Sitemaps are often incomplete or stale	Enterprise web crawlers, SEO auditing tools

Production Considerations

Trap detection. Crawler traps are infinite URL spaces generated by calendar widgets, sort parameters, or session IDs. Detect them by capping the maximum number of URLs accepted from a single domain relative to your expectation, and by refusing to enqueue URLs with an unusual number of query parameters or path segments beyond a threshold.

Redirect handling. Follow up to 5 redirects. After that, mark the URL as a trap or broken link. Track redirect chains to detect redirect loops. Update your URL normalization to point to the final destination.

Crawl budget allocation. If you are crawling the open web, dedicate most of your budget to high-priority frontier entries and keep a smaller fraction for re-crawl maintenance. A 70/30 split between new URL discovery and re-crawl works as a starting point.

Monitoring. Track per-host error rates, frontier queue depth, and pages crawled per second per worker. A sudden spike in 403/429 responses from a host means your politeness policy may be too aggressive or your IP has been flagged. Track your effective crawl delay per host and alert when it drifts significantly from target.

Legal and ethical boundaries. Always respect robots.txt. Identify yourself with a real user agent string that includes a URL explaining your crawler’s purpose. If a site operator contacts you to stop crawling, stop immediately. These are not optional courtesies; they are the baseline for operating a crawler that others will tolerate.

A web crawler is one of the few systems where the design constraints come from three directions at once: data structures and algorithms for scale, distributed systems for coordination, and social norms for responsible operation. Getting any one wrong is enough to make the whole thing fail.