Resumable Crawlee Queues and Idempotent Results
Recover interrupted crawls with durable request identities, deliberate refresh windows and result writes that tolerate repeated processing.
KAIROXO
Resumable web crawling starts with a question that a successful HTTP request cannot answer: what happens when the process stops halfway through its URL list? A restart should recover unfinished work without silently duplicating the records already delivered. That requires a durable work queue and a deliberate output policy, not simply another retry around the downloader.
Consider an authorized supplier-directory crawl with a discovery stage and a detail stage. The directory contains many categories pointing to the same suppliers. Workers may restart during maintenance, and a supplier page may be rediscovered tomorrow. We will separate discovery, request identity and result identity so each event has a clear meaning.
Define a run and its durable state
A run is an execution interval; it is not necessarily the lifetime of the crawl. Keep the seed list, crawl policy version and output dataset identifier with a stable collection identity. Record when each run started and ended, including an interrupted status when completion was never confirmed. Otherwise a dashboard can mistake a stopped worker for a finished collection.
Persist pending requests outside the worker's disposable memory. Crawlee provides request-queue abstractions; Apify's named queues can retain work across Actor runs. These are related but distinct choices: using Crawlee locally does not automatically give you Apify's hosted storage behavior. Choose the storage implementation and verify its persistence before promising restart recovery.
Implementation steps
Give requests explicit identities
Determine whether two URLs represent the same fetch. Tracking fragments may be irrelevant, while a language, category or page parameter may select genuinely different content. Use an explicit unique key when the default URL identity does not express your collection contract. Preserve the original URL alongside that key for diagnostics.
In the supplier example, two category pages can enqueue the same supplier detail URL. Queue deduplication prevents repeated discovery from creating unlimited pending copies. It does not mean that a future scheduled refresh should be skipped forever. Include a collection window in the identity or deliberately requeue handled requests when your refresh policy calls for it.
Separate discovery from detail extraction
Let discovery handlers add approved detail URLs, and let detail handlers assemble supplier records. Carry small routing metadata rather than entire HTML documents in queued requests. Restrict discovered URLs to the agreed host and path scope before enqueueing; a navigation link should not expand the task to a new website merely because it appears on an allowed page.
Track counts for discovered candidates, accepted unique requests, completed detail pages and failed detail pages. These counts answer different questions. A queue becoming empty establishes that no queued work remains, but it does not prove that discovery found every supplier. Compare with available category totals or another independent inventory where practical.
Make result writes idempotent
Use a stable supplier identifier, not an auto-incremented output row number, as the result key. Define whether a later observation replaces an earlier record or creates a versioned observation. Include source URL, observation time and collection window. Then a repeated detail handler can produce the same intended result instead of an accidental second supplier.
The difficult crash happens after the result is saved but before the request is marked handled. A restarted worker may process that request again. Queue locks coordinate work ownership, but they are not a transaction spanning your destination storage. The output operation must tolerate that repetition. If appending observations, use a deterministic observation key and reject duplicates at the destination.
Recover in a controlled order
Restart with the same durable queue and compatible routing logic. First inspect pending, handled and failed counts; do not immediately create another queue with the same seeds. Allow the queue implementation's lock or lease recovery to release work held by the stopped process. Avoid manually clearing active locks while another worker could still be processing them.
Run a small recovery test before increasing workers. Stop after saving a known result, restart, and confirm that the final dataset contains one logical supplier record. Also test a request that repeatedly fails: it should enter a visible terminal-failure path with an explanation, rather than circling through fresh queues indefinitely. Keep the failed URL available for a later deliberate repair run.
Keep refreshes different from restarts
A restart completes unfinished work in the same collection window. A refresh asks for a new observation of already known entities. Mixing these operations can either suppress fresh data or create duplicate historical observations. Record the reason for requeueing and the extraction-policy version, especially when selectors or required fields have changed between runs.
Use KAIROXO's configured proxy profile at the access layer and keep it independent of supplier identity. An exit address changing should not create a new logical supplier. Review proxy error diagnosis when access fails, while keeping persistence failures and parser failures in their own categories.