Scrapy Proxies: Middleware, Concurrency and Throttling
Assign a proxy once, keep retry budgets bounded and use per-domain concurrency and AutoThrottle to build a measured public-data collection job.
KAIROXO
A Scrapy proxy is one component of a collection job. The scheduler decides what to request, downloader middleware assigns the route, throttling controls pressure on destinations and the parser determines whether the response becomes useful data. A reliable pilot makes each responsibility visible rather than increasing proxy count whenever the job slows down.
Start with a small set of authorized public pages and a clear stopping rule. Record the required fields, allowed domains and acceptable freshness. A proxy supplies connectivity; permission to collect and the site's published access rules remain part of the job design. The example keeps robots handling enabled and uses deliberately conservative concurrency.
Pick one configuration owner
Scrapy's HTTP proxy downloader middleware consumes the request's proxy metadata. Assign that value in one place: either individual requests or a custom middleware with an explicit policy. If both a spider and several middleware components overwrite it, diagnosing the final route becomes difficult.
For KAIROXO, copy the actual gateway and credentials from the selected network product. The sample keeps the endpoint and credentials separate in environment variables, then encodes the credential components when building the connection URI. Do not send this URI to a target as an ordinary header or include it in exported records.
Implementation steps
Build a minimal diagnostic spider
Use a supported Python environment and record the Scrapy version used for validation. The example uses the asynchronous start method available in recent Scrapy releases. Save it as a standalone spider, install Scrapy in an isolated environment and run it with the normal runspider command after supplying your own local environment variables.
The custom middleware runs before Scrapy's HTTP proxy middleware. It assigns the route while leaving protocol handling to Scrapy. The diagnostic only exports status and whether the expected field exists. Inspect the origin locally to verify the exit, then remove temporary evidence before moving the job into regular operation.
Codepython import osfrom urllib.parse import quote, urlsplitimport scrapy class ExplicitProxyMiddleware: def __init__(self): endpoint = urlsplit(os.environ["PROXY_URI"]) if endpoint.scheme != "http" or not endpoint.hostname or endpoint.username: raise ValueError("Use an HTTP proxy endpoint without credentials") user = quote(os.environ["PROXY_USERNAME"], safe="") password = quote(os.environ["PROXY_PASSWORD"], safe="") self.proxy = "http://" + user + ":" + password + "@" + endpoint.netloc def process_request(self, request, spider): request.meta["proxy"] = self.proxy class DiagnosticSpider(scrapy.Spider): name = "proxy_diagnostic" custom_settings = { "DOWNLOADER_MIDDLEWARES": {__name__ + ".ExplicitProxyMiddleware": 543}, "ROBOTSTXT_OBEY": True, "CONCURRENT_REQUESTS": 4, "CONCURRENT_REQUESTS_PER_DOMAIN": 2, "DOWNLOAD_DELAY": 1, "AUTOTHROTTLE_ENABLED": True, "AUTOTHROTTLE_TARGET_CONCURRENCY": 1.0, "AUTOTHROTTLE_START_DELAY": 1.0, "AUTOTHROTTLE_MAX_DELAY": 30.0, "DOWNLOAD_TIMEOUT": 20, "RETRY_TIMES": 1, "CLOSESPIDER_PAGECOUNT": 10, "LOG_LEVEL": "ERROR", } async def start(self): yield scrapy.Request("https://httpbin.org/ip", callback=self.parse) def parse(self, response): diagnostic = response.json() # Inspect the exit locally; keep feed output free of credentials and IPs. yield { "http_status": response.status, "origin_field_present": isinstance(diagnostic.get("origin"), str), }Add concurrency and adaptive delay
CONCURRENT_REQUESTS constrains the downloader globally, while CONCURRENT_REQUESTS_PER_DOMAIN limits parallel work for a destination domain. DOWNLOAD_DELAY adds spacing and AutoThrottle adjusts delay from observed response latency. These controls interact, so the target concurrency is a tuning goal rather than a promise that exactly that many requests will always run.
The sample values are safe starting points for a diagnostic, not a recommended production rate for every website. Check the target's requirements and your authorization, then measure the actual request cadence. Increasing the number of proxy exits does not remove the need to respect a destination's overall rate limits.
Keep retries and rotation independent
Set an attempt budget that includes automatic downloader retries and any external job retry. A failed request can otherwise be repeated once by Scrapy, again by a queue worker and again by a scheduler. Count attempts as well as distinct URLs so increased traffic remains visible.
Choose a session policy according to the task. Pagination or a sequence tied to a website session may require continuity, whereas independent public pages may tolerate changing exits. A retry is not automatically a reason to switch identity. Review 407 as authentication configuration, and handle destination throttling by slowing or pausing rather than repeatedly resubmitting the same work.
Validate the extracted record
After the diagnostic, replace the URL and parser with a small authorized sample. A response is usable only when required fields, content type and basic domain-specific checks pass. A missing price, an empty result list or a login page should have a distinct outcome from a network timeout.
Store a sanitized task identifier, collection time, parser version and validation result with the record. Keep raw evidence only where necessary, with limited access and retention. Do not place proxy credentials, account cookies or unrelated personal information in feed exports. When a parser changes, reprocess a small known fixture before launching a full collection run.
Observe bottlenecks before expanding
Break elapsed time into scheduling delay, network work and parsing or output work where your instrumentation allows it. If requests finish quickly but exported records arrive slowly, adding network capacity may not help. Inspect item pipelines, output storage and CPU pressure as well as the proxy route.
Run a fixed pilot URL set at a fixed window, then change one setting at a time. Track useful records per minute, failures by category and total attempts. Keep an explicit page or time stop during experiments. The sample's page-count limit is a small guardrail, not a complete spending or runtime limit for every deployment.