Web Scraping Retries: Budgets, Backoff and Retry-After
Bound retry attempts and elapsed time, distinguish temporary failures and respect server pauses without amplifying load across workers.
KAIROXO
Web scraping retries should recover temporary failures without turning a slow destination into an overloaded one. A retry consumes another request, another connection slot and part of the job's deadline. Counting only the successful final response hides that cost. Design the retry policy around the entire operation, including requests already retried by a library or upstream service.
Imagine a permitted collector reading a public parts catalog. Most pages respond normally, several return 503 during maintenance, and a rate-limited group returns 429. Repeating every failure immediately gives the busiest pages the most traffic. A bounded policy classifies failures, respects server timing and leaves an explicit outcome when the budget runs out.
Separate attempts from useful results
An attempt is one network operation; a result is a validated record. Keep those counters separate. A page delivered after four attempts may be a useful result, but it did not cost one request. Include the final HTTP status, attempt count, elapsed time and termination reason in sanitized job metadata. Avoid logging complete query strings or authenticated proxy URLs.
Decide which layer owns retries. If a scraping service already retries internally, your outer loop should use that service's documented behavior when choosing its own limits. Four outer attempts multiplied by several hidden inner attempts can produce much more traffic than intended. Increasing concurrency at the same time makes the amplification difficult to diagnose.
Implementation steps
Classify failures before waiting
Network interruptions and selected temporary server responses can justify another attempt. A malformed URL, invalid credentials or a missing page generally requires a different action. A 407 points to proxy authentication; repeatedly sending the same credentials is not recovery. A 403 needs investigation of the access contract rather than automatic escalation into unlimited traffic.
Treat returned bodies separately from transport status. A 200 page may still fail your content contract, while an intentionally empty resource may be valid. Define those checks for the destination. Do not implement a universal rule that any short page needs another proxy, another browser or another retry; each of those actions consumes additional resources.
Set both attempt and elapsed-time limits
Choose a maximum number of attempts and a total operation deadline. Before every attempt, calculate the remaining budget and give the request a timeout no longer than that remainder. Response-body consumption also belongs within the operation's timeout; receiving headers does not mean the work is complete. Stop cleanly when cancellation arrives from the caller.
The example accepts a request adapter and injectable timing functions. It is intended for idempotent GET retrieval. The adapter owns the HTTP or proxy client configuration and must observe the supplied abort signal. Keep parsing outside this helper and give body reading its own remaining-time boundary. Non-idempotent operations require their own safe-repetition contract.
Codejavascript export async function retryGet(fetchOnce, { attempts = 4, budgetMs = 12000, now = Date.now, sleep = ms => new Promise(resolve => setTimeout(resolve, ms)), random = Math.random,} = {}) { const deadline = now() + budgetMs; for (let attempt = 0; attempt < attempts; attempt++) { const remaining = deadline - now(); if (remaining <= 0) throw new Error('retry budget exhausted'); // The adapter must respect this signal, including during body consumption. let response; try { response = await fetchOnce(AbortSignal.timeout(remaining)); } catch (error) { if (error.name === 'AbortError' || error.name === 'TimeoutError') throw error; if (attempt + 1 === attempts) throw error; } if (response && ![408, 429, 500, 502, 503, 504].includes(response.status)) return response; if (response?.body) await response.body.cancel(); if (attempt + 1 === attempts) throw new Error('attempt limit reached'); const value = response?.headers.get('retry-after'); const serverDelay = value == null ? NaN : /^\d+$/.test(value.trim()) ? Number(value) * 1000 : Date.parse(value) - now(); const jitter = Math.floor(random() * Math.min(4000, 250 * 2 ** attempt)); const delay = Number.isFinite(serverDelay) ? Math.max(0, serverDelay) : jitter; if (delay >= deadline - now()) throw new Error('retry budget exhausted'); await sleep(delay); } throw new Error('no attempts configured');}Honor Retry-After without extending the deadline
Retry-After can contain seconds or an HTTP date. Parse both forms, tolerate an invalid header and calculate dates against a consistent clock. If the required wait is longer than the remaining job budget, stop or reschedule the job for later. Shortening that wait to squeeze another request into the current job ignores the server's requested pause.
When no usable server delay exists, exponential backoff spreads successive attempts farther apart. Add jitter so multiple workers do not wake at exactly the same instant. AWS's discussion explains why deterministic backoff can still produce synchronized clusters. Its simulation results are evidence about that model, not a performance guarantee for your catalog or proxy profile.
Release resources and preserve failure evidence
Discard or cancel a response body before waiting for another attempt so the old response does not retain resources indefinitely. Keep a bounded worker queue: sleeping requests should not invite an unlimited number of new tasks into memory. Separate per-host pacing from per-request retry timing so one troubled destination cannot monopolize all workers.
On exhaustion, save enough context for a repair run: stable request identity, last status, failure category and the next eligible time when known. Do not label exhaustion as a successful empty record. A later scheduled repair is different from an invisible retry loop that never finishes. Review proxy error diagnosis before changing credentials or route settings.
Verify with a virtual clock
Test retry behavior without real sleeping or a production target. Inject a sequence such as 503, 429, 200 and inspect the timing decisions. Cover Retry-After dates, missing headers, permanent errors, cancellation and a delay exceeding the remaining budget. Test whether the total number of requests matches the intended limit, including the first attempt.
Use deterministic random values in tests to exercise jitter bounds. A test that merely waits for the helper to finish can miss an extra request or an ignored server pause. Assert calls and delays directly. After local checks, a small authorized probe can verify the actual client's timeout and response-release behavior without creating a load test.