aiohttp Proxy: Sessions, Timeouts and Controlled Concurrency

Implementation steps

1. 1

   Define one explicit access profile

   Start with an approved target URL and a proxy endpoint copied from your own account. Keep the gateway scheme, hostname, port and authentication method separate from the target site's URL. An HTTPS destination can use an HTTP proxy tunnel; that does not imply the client-to-proxy connection itself is encrypted. Preserve certificate verification for the destination.

   Choose one locale, account context and task identity for the session. Record the installed aiohttp version alongside this profile because authentication and session defaults can evolve. The reference blogs are useful starting points, but their examples may target older releases. Never copy a public proxy address or disable TLS verification simply because a tutorial does so.

2. 2

   Reuse the session and bound active work

   ClientSession owns connection-pooling behavior and normally a cookie jar. Create it around the group of requests that should share that state, then close it when the group finishes. A connector limit controls simultaneous connections, whereas an application semaphore limits active request work. Neither value is a requests-per-second budget. Add a separate scheduler when the target requires spacing.

   import asyncioimport osimport aiohttp async def collect(urls):    proxy = os.environ['KAIROXO_PROXY_URL']    timeout = aiohttp.ClientTimeout(total=30, connect=10, sock_read=10)    gate = asyncio.Semaphore(4)    connector = aiohttp.TCPConnector(limit=4, limit_per_host=2)    async with aiohttp.ClientSession(connector=connector, timeout=timeout) as session:        async def one(url):            async with gate:                async with session.get(url, proxy=proxy, allow_redirects=False) as response:                    response.raise_for_status()                    if response.status != 200:                        raise ValueError('unexpected status')                    body = bytearray()                    async for chunk in response.content.iter_chunked(65536):                        body.extend(chunk)                        if len(body) > 2_000_000:                            raise ValueError('decoded response exceeds budget')                    return bytes(body)        return await asyncio.gather(*(one(url) for url in urls)) # Use approved test URLs and an environment variable such as# KAIROXO_PROXY_URL=http://USER:PASS@PROXY_HOST:PROXY_PORT# await collect(urls) from your application's async entry point.

   The example uses an explicit proxy for each request, keeps at most four active operations and sets a per-host connection limit. It deliberately refuses automatic redirects so the caller can apply a destination policy. Its return values are bytes, not validated business records. Parsing and saving should happen only after the expected page type is confirmed.

3. 3

   Distinguish timeout stages before retrying

   The total timeout bounds the operation inside the client. The connect timeout can include waiting for a pooled connection; sock_connect concerns establishing a new socket, and sock_read limits waiting for subsequent data. A slow job is therefore not automatically evidence that the gateway is unreachable. Time spent waiting outside the request, such as behind a semaphore, needs its own job deadline.

   Measure queue wait, request start, response headers and completion separately. If all workers wait on the connector, reduce unnecessary concurrency before changing the proxy. Treat authentication failure as a configuration problem. An unexpected content type, a redirect or a successful response containing a challenge needs classification rather than an immediate identical retry.

4. 4

   Bound response consumption and cancellation

   An asynchronous client still consumes memory. The example caps the body bytes exposed by aiohttp before returning them to a parser. Because clients can decompress responses automatically, that cap should be described as a decoded-body cap, not as the billed network volume. For large objects, stream into a bounded temporary artifact and publish it only after validation.

   On cancellation, let context managers close responses and the session. Do not swallow cancellation inside an endless retry loop. Start with a small URL group; an enormous gather call creates many waiting tasks even when the semaphore limits active work. A worker queue is more suitable when inputs are unbounded or arrive continuously.