Proxy File Downloads: Bounds, Ranges and Integrity Checks

Implementation steps

1. 1

   Bound the transfer before accepting it

   Set connection and operation timeouts, a maximum byte count and a redirect policy. Content-Length is useful metadata, not permission to allocate unlimited memory. Servers can omit it, and a decoded response body may differ from the encoded transfer length when compression is involved. Apply the limit to the bytes your application actually consumes.

   Check the response status and media type before writing a final artifact. A response with text/html can be an error page even when the requested path ends in .pdf. Filenames from URLs or Content-Disposition are untrusted input; remove path components and use an application-controlled destination rather than allowing remote text to select a local path.

2. 2

   Hash bytes as they arrive

   For large files, write chunks to a temporary file while updating a hash incrementally. Promote it only after the complete read and all checks succeed. On failure, keep a clearly marked partial file only if your resume policy needs it; otherwise remove it within the controlled work directory. Do not expose partial output to downstream parsing as a finished document.

   The bounded illustration below collects a small body in memory and hashes it while reading. It rejects partial-content responses because it does not implement range assembly. A production large-file variant should stream to controlled storage instead of increasing this memory limit indefinitely. The helper is transport-independent: a configured proxy client supplies the response.

   import { createHash } from 'node:crypto'; export async function readVerifiedBody(response, { maxBytes = 1024 * 1024, expectedSha256 } = {}) {  if (!response.ok || response.status === 206) throw new Error('expected complete successful response');  if (!response.body) throw new Error('missing body');  const reader = response.body.getReader(), chunks = [], hash = createHash('sha256');  let size = 0;  try {    while (true) {      const { done, value } = await reader.read();      if (done) break;      size += value.byteLength;      if (size > maxBytes) throw new Error('body size limit exceeded');      hash.update(value); chunks.push(value);    }  } catch (error) { await reader.cancel().catch(() => {}); throw error; }  finally { reader.releaseLock(); }  const sha256 = hash.digest('hex');  if (expectedSha256 && sha256 !== expectedSha256.toLowerCase()) throw new Error('checksum mismatch');  return { bytes: Buffer.concat(chunks), sha256, size };} export function decodeKnownEncoding(bytes, encoding) {  // Explicit encoding is required. This does not implement HTML charset sniffing.  if (!encoding) throw new Error('encoding required');  return new TextDecoder(encoding, { fatal: true }).decode(bytes);}

3. 3

   Resume only when object identity is stable

   A resume request asks for bytes starting at an offset, but the server may ignore Range and return the entire object with 200. Appending that response to an existing partial file corrupts the result. Check for 206 and validate Content-Range before writing at an offset. Treat 416 as a reconciliation case, not automatic proof that the local file is complete.

   Use an appropriate validator such as a strong ETag with If-Range when the server supports it. If the object changed, restart deliberately rather than combining bytes from different versions. Keep range boundaries and object identity together, and calculate the final checksum over bytes in final file order, not the order that parallel requests happened to finish.

4. 4

   Verify before making the file visible

   At completion, compare the received byte count with the expected representation length where applicable, compare the trusted checksum if available, and perform a lightweight format check. No single check replaces all the others. A matching length does not prove correct content; a correct file signature does not establish that the full document arrived.

   Save a receipt with final URL, observation time, byte count, checksum and validation outcome. Keep proxy credentials out of receipts and logs. If the publisher provides no trusted digest, say that explicitly rather than labeling the local SHA-256 as publisher-verified. These receipts make a later parser failure easier to distinguish from a transfer problem.