Web Scraping Character Encoding: Preserve Multilingual Text

Implementation steps

1. 1

   Capture the original response evidence

   Save a bounded sample of raw bytes together with Content-Type and relevant response metadata before decoding. Preserve the final URL and selected language context. The first diagnostic question is whether corruption exists in the received bytes or was introduced by your decoder, parser, serializer or display layer. A screenshot of wrong text does not identify that boundary.

   Keep raw samples only within your retention and privacy rules. For public fixtures, use synthetic strings that exercise Chinese characters, Vietnamese tone marks and ordinary English punctuation. Record which decoder was chosen and why, rather than storing only the final string and an unexplained confidence score.

2. 2

   Determine the decoder deliberately

   Inspect the byte-order mark, transport declaration and early HTML charset declaration using the rules appropriate to your parser. Browser HTML encoding detection has specific precedence and label mapping; an HTTP library may apply different defaults. Passing an already-decoded string to an HTML parser can prevent it from considering the original byte declarations at all.

   When the encoding is known from a reliable source contract, use it explicitly. When declarations conflict, flag the conflict and apply a documented rule rather than silently switching until the output looks plausible. Statistical detection is a fallback, particularly uncertain on short samples. A detector returning a label does not prove that label is correct.

3. 3

   Reject silent corruption in critical fields

   Use strict decoding where loss is unacceptable, then route failures for review or a controlled fallback. Replacement decoding may be useful for a human preview, but it loses information and should not be mistaken for a verified supplier name. Count replacement characters and suspicious control characters as signals, not universal proof of the exact original encoding.

   The example provides explicit TextDecoder use with fatal errors. It does not implement HTML charset sniffing and must not be presented as a complete browser decoder. The adjacent bounded reader keeps bytes available for reprocessing. Tests verify that Chinese and Vietnamese survive UTF-8 and that invalid UTF-8 fails visibly.

   import { createHash } from 'node:crypto'; export async function readVerifiedBody(response, { maxBytes = 1024 * 1024, expectedSha256 } = {}) {  if (!response.ok || response.status === 206) throw new Error('expected complete successful response');  if (!response.body) throw new Error('missing body');  const reader = response.body.getReader(), chunks = [], hash = createHash('sha256');  let size = 0;  try {    while (true) {      const { done, value } = await reader.read();      if (done) break;      size += value.byteLength;      if (size > maxBytes) throw new Error('body size limit exceeded');      hash.update(value); chunks.push(value);    }  } catch (error) { await reader.cancel().catch(() => {}); throw error; }  finally { reader.releaseLock(); }  const sha256 = hash.digest('hex');  if (expectedSha256 && sha256 !== expectedSha256.toLowerCase()) throw new Error('checksum mismatch');  return { bytes: Buffer.concat(chunks), sha256, size };} export function decodeKnownEncoding(bytes, encoding) {  // Explicit encoding is required. This does not implement HTML charset sniffing.  if (!encoding) throw new Error('encoding required');  return new TextDecoder(encoding, { fatal: true }).decode(bytes);}

4. 4

   Decode HTML entities only at the right layer

   Character encodings and HTML entities are not the same thing. An HTML parser can turn a character reference into its intended character during parsing. Repeatedly applying entity decoding until a string stops changing can alter text that deliberately displays a literal entity. Likewise, JSON escape sequences belong to JSON parsing, not an HTML unescape loop.

   Unicode normalization is another separate choice. Visually similar Vietnamese text may use composed or combining characters. Keep the original string and choose a normalization form for comparison only when the downstream contract needs it. Removing accents to simplify matching changes names and search intent; it is not an encoding repair.