Distributed Crawler Rate Limits: One Budget per Site
Keep a fleet-wide request budget consistent as workers scale, distinguishing admission rate, in-flight concurrency and bounded queue waiting.
KAIROXO
Distributed crawler rate limiting prevents a fleet from multiplying its traffic budget every time another worker starts. If five workers each interpret a rule as ten requests per second, the destination may receive fifty. The limit must describe the collective traffic to the target, not simply the behavior of one process. This matters even when every worker uses a correctly configured proxy.
Consider a permitted documentation collection spread across several workers. One host accepts a small steady request rate, while another supports more capacity. The collector needs distinct per-host budgets, fair access to those budgets and a clear response when coordination is unavailable. Those are application decisions; an IP pool alone cannot enforce them.
One destination, one shared admission decision
Think of each outbound request as asking for admission before it reaches the network client. The identity of the budget might include the target hostname, a documented API account and a particular operation class. Every worker sharing that budget must derive the same key. Otherwise a shared store merely holds many independent counters, and the aggregate limit still grows unnoticed.
Keep the key stable and non-secret. Do not use raw credentials or arbitrary full URLs as coordinator keys. Unbounded key variation creates both privacy and resource problems. A host-level policy can also need exceptions for endpoints with distinct documented limits, but those exceptions must be explicit rather than accidental differences in URL spelling.
Rate and concurrency answer different questions
A rate limit controls admissions across time. A concurrency limit controls the number of operations currently in flight. Ten slow requests may occupy ten connections for a long time without exceeding a low admission rate. Conversely, many very fast requests can respect a concurrency cap while exceeding a destination's per-second policy. A reliable collector often needs both controls.
Queue capacity is a third boundary. Waiting for a permit should not allow unbounded pending jobs to accumulate. Give queued work a deadline and a maximum waiting depth. When those limits are reached, return an explicit deferred or rejected result. Do not report a job as running indefinitely when it no longer has a realistic chance to start.
Choose the policy before the storage technology
A fixed time window is simple but can permit a burst on both sides of a window boundary. Sliding windows smooth that boundary at the cost of more state. Token-style policies separate a sustained refill rate from a burst allowance. Choose the model based on the target's rules and workload instead of selecting an algorithm because a code snippet is short.
CALLR's engineering article describes shared Redis-based admission for distributed operations and explains why local counters cannot represent aggregate traffic. Its examples come from telephony infrastructure. Apply the coordination principle to crawling, but do not import its traffic values as recommended web-request rates. There is no universally polite requests-per-second number for every website.
Coordinate atomically and account for failures
The read, decision and update must form one indivisible transition. Separate reads and increments let workers observe the same remaining capacity and spend it twice. A production coordinator can use supported atomic primitives, but the exact implementation belongs in a separately reviewed infrastructure component. This guide does not require changing the website backend or executing Redis commands.
Define what happens if admission succeeds and the worker crashes before sending. Charging that admission conservatively wastes some capacity but avoids exceeding the agreed limit. Refunding blindly can over-admit when the request actually left the process before the failure. Also specify whether a retry consumes a fresh permit; for outbound traffic accounting, another network attempt normally should.
Make waiting and coordinator outages explicit
Return a retry-eligible time or bounded delay from the coordinator, then let workers rejoin fairly rather than spin in a tight loop. Apply jitter where appropriate so a group of denied workers does not wake simultaneously. A delay is not a reservation: another worker may obtain the available capacity first. Avoid promising precise start times unless the coordinator actually implements reservations.
Choose outage behavior deliberately. For a collector whose priority is honoring destination limits, pausing or deferring new admissions is safer than silently falling back to an unrestricted local counter. Distinguish coordinator failure from target-site failure in logs. A proxy timeout should not be diagnosed as a quota-store outage, and neither should be counted as a completed record.
Observe the budget from the destination's perspective
Track admissions, denials, waiting time, in-flight requests and actual attempts per target. Compare the sum across workers with the configured fleet policy. Testing one worker is insufficient. A useful local simulation starts several virtual workers together, advances a shared fake clock and verifies that admitted work stays within the chosen window and burst rules.
Keep retries visible in those measurements. See retry budgets for controlling repeated attempts and proxy benchmarks for separating useful results from network activity. More available proxy exits do not automatically authorize more traffic to the same destination.