Compressed HTTP Responses: Bound Encoded and Decoded Bytes
Treat compression as a transport feature, measure the bytes your client exposes and set limits before decoding large responses.
KAIROXO
Compressed HTTP responses create two sizes that a collector must not confuse: the bytes transferred in an encoded representation and the bytes available after decoding. A small download can expand into a much larger document. Conversely, a client may already have decompressed the body before your application sees it, even though diagnostic headers still describe the encoded response.
Compression is useful, but it does not remove the need for request, memory and parsing budgets. This guide adds that distinction to KAIROXO's download-integrity topic. Integrity checks ask whether the expected bytes arrived; compression budgets ask whether obtaining and interpreting those bytes is safe for the worker's resources.
Implementation steps
Identify the byte layer exposed by the client
Read your HTTP client's documentation and test a known compressed fixture. Determine whether its normal body API returns encoded bytes or decoded content, and whether automatic decoding changes which headers remain available. Do not run a decompressor solely because a Content-Encoding header exists; double decompression can corrupt a perfectly valid response.
Content-Length, when present, describes the message's representation at its own layer, not necessarily the application buffer. It may also be absent. Avoid promising that a declared size protects the process: actual streamed bytes need a limit. Keep encoded transfer measurements separate from decoded body measurements in both logs and cost analysis.
Set independent transport and expansion limits
Choose a maximum encoded body size, a maximum decoded size and a deadline. Size limits answer different questions from timeouts. An expansion-ratio threshold can be useful, but a hard decoded-byte ceiling remains necessary even for a response with a modest ratio. Include parser memory and text decoding in the overall worker budget.
The example helper below accepts raw encoded bytes. It is not meant to wrap an HTTP client's already-decoded text method.
Codejavascript import { gunzipSync, brotliDecompressSync } from "node:zlib"; function decodeBounded(raw, encoding, encodedLimit, decodedLimit) { if (raw.length > encodedLimit) throw new Error("encoded budget"); let output; if (!encoding || encoding === "identity") output = raw; else if (encoding === "gzip") output = gunzipSync(raw, { maxOutputLength: decodedLimit }); else if (encoding === "br") output = brotliDecompressSync(raw, { maxOutputLength: decodedLimit }); else throw new Error("unsupported encoding"); if (output.length > decodedLimit) throw new Error("decoded budget"); return output;}For large responses, use streaming decompression with bounded input and output sinks instead of first buffering the entire download.
Negotiate only formats the pipeline understands
Advertise supported encodings through the client's documented mechanism. If the installed runtime does not support an encoding, do not claim it does. The example handles a single gzip or Brotli layer and identity; it deliberately rejects unsupported values. A general implementation would need to parse multiple content codings and decode them in the specified reverse order.
Do not confuse content encoding with the document's character encoding. Decompression restores bytes; UTF-8 or another text decoder interprets those bytes afterward. Preserve the raw and decoded digests separately if both are relevant to an audit. Each digest must say which representation it identifies.
Classify failures before retrying
A truncated compressed stream, a size-limit rejection and an unsupported encoding are different failure classes. Repeating the same oversized response through another IP does not reduce its decoded size. Retrying a corrupt transfer may be reasonable within a bounded request budget, but never turn decompression failure into an empty successful document.
Discard incomplete temporary files and do not send their content to downstream extraction. Record the encoding, actual byte counts, limit that was crossed and a sanitized source identifier. If a response is intentionally excluded because it is too large, keep that status distinct from “source contained no records.”
Build fixtures that expose hidden assumptions
Test an ordinary identity body, valid gzip, valid Brotli, truncated input, unknown encoding, a declared length smaller than actual content and a very small file that expands beyond the output limit. Include Chinese and Vietnamese text so the test also checks that decompression does not silently damage multilingual data.
A transport mock can verify limit behavior without touching production sites. It cannot establish a provider's billed byte definition. Reconcile application metrics with the actual product's usage records before attributing a savings figure to compression. Compression ratios vary by content, and results from one provider's article are not performance promises for KAIROXO.