HTTPX Async Proxy: Control Concurrency and Pool Timeouts
Use one asynchronous HTTPX client per access profile, with explicit proxy settings, connection limits and a separate bound on active work.
KAIROXO
An HTTPX async proxy workflow needs two kinds of limits: how many connections the client may hold and how much work the application may start. Those limits solve different problems. Setting a connection pool size does not stop your program from allocating thousands of pending tasks, and starting more tasks does not make an overloaded target respond faster.
For a collector that reads approved public pages, start with one access profile and a deliberately small queue. Establish which requests reached the network, which waited for a connection, and which returned usable content. This makes asynchronous execution observable instead of turning every slow result into an unexplained proxy timeout.
Check the installed client API
Current HTTPX configuration uses proxy= for a client-wide proxy. Older examples using a proxies argument may not work with the installed release. Check the version and official documentation before copying a sample. HTTP proxying and HTTPS destinations are compatible when the proxy supports CONNECT; the destination scheme does not require you to change an HTTP proxy URL into an HTTPS proxy URL.
Use the SOCKS extra only when you intentionally need SOCKS support, and check the DNS behavior of the chosen scheme. Record the access profile without logging its credential-bearing URL. Environment-derived proxies can make local and deployed behavior differ, so the example disables trust_env and supplies its proxy explicitly.
Implementation steps
Share a client for one access profile
Create AsyncClient outside the per-URL function and close it with an asynchronous context manager. Reusing that client preserves the opportunity to reuse connections. Repeated construction inside a hot loop adds setup overhead and makes results difficult to compare. Separate clients can be appropriate for different credentials, session policies or incompatible proxy routes.
Set max_connections and max_keepalive_connections for the client. These values define a pool budget, not a global traffic entitlement. If several clients share the same destination or account, their combined load still matters. Maintain an application-wide view of concurrency rather than assuming that each independently reasonable pool creates a reasonable total.
Bound active work and network phases
Use a semaphore to limit the number of tasks actively doing work, and bound the producer or input batch as well. A semaphore around requests still leaves memory exposed if you eagerly create an enormous list of tasks. For larger workloads, a bounded queue with a fixed worker count gives more explicit backpressure than one giant gather call.
HTTPX distinguishes connect, read, write and pool timeouts. A pool timeout means the task waited too long for a connection; it is not evidence that the remote proxy was slow. A read timeout limits inactivity while reading, not necessarily the whole lifetime of the job. The example adds a separate outer deadline for the small batch.
Codepython import asyncioimport osimport httpx async def probe(client, semaphore, url): async with semaphore: try: async with client.stream("GET", url) as response: size = 0 async for chunk in response.aiter_bytes(): size += len(chunk) if size > 1_048_576: return {"result": "body_limit"} return {"result": "response", "status": response.status_code, "bytes": size} except httpx.PoolTimeout: return {"result": "pool_timeout"} except httpx.TimeoutException: return {"result": "network_timeout"} except httpx.RequestError: return {"result": "request_error"} async def main(): timeout = httpx.Timeout(connect=8, read=12, write=8, pool=5) limits = httpx.Limits(max_connections=4, max_keepalive_connections=4) semaphore = asyncio.Semaphore(2) async with httpx.AsyncClient( proxy=os.environ["PROXY_URL"], trust_env=False, follow_redirects=False, timeout=timeout, limits=limits, ) as client: urls = [os.environ["PROBE_URL"]] # only authorized probe targets results = await asyncio.wait_for( asyncio.gather(*(probe(client, semaphore, url) for url in urls)), timeout=30 ) for result in results: print(result) if __name__ == "__main__": asyncio.run(main())Inspect bounded responses and failures
The example streams the response and stops after a small body limit. This is useful for a probe that should never download an unexpectedly large page. Closing a stream releases resources even on an error or early return. A body-limit result belongs in your measurements; silently counting it as success would hide incomplete data.
Classify PoolTimeout separately from other timeout exceptions and other request errors. Then evaluate HTTP status and content outside the transport layer. A response can be delivered successfully but contain a login page, an empty shell or an unexpected format. Check the expected field or record count before marking the task useful.
Increase capacity from measured evidence
Run a small authorized batch and record queued time, active time, status and extraction outcome. Increase concurrency only after the current batch completes cleanly within your budget. If pool waits dominate, inspect how long responses occupy connections and whether bodies are closed. If the target slows down or signals a limit, reduce pressure rather than adding more clients.
Retry only failures that your operation can safely repeat, with a bounded attempt count and delay. An authentication problem needs configuration repair, while a rate-limit response needs the destination's guidance and your own schedule adjustment. Keep cancellation effective across retries so a job that has expired cannot continue consuming its original resource budget.
- Keep a limit on queued inputs as well as active requests.
- Close clients and response streams through context managers.
- Record transport, HTTP and content outcomes separately.
- Recheck behavior after changing credentials, destinations or client versions.