Crawler URL Normalization Without Losing Query Semantics

Implementation steps

1. 1

   Resolve relative links with a real URL parser

   Use the platform URL parser with the page's effective base URL. String concatenation fails on parent paths, query-only links and protocol-relative references. When the document includes a base element, decide whether your extractor respects it and validate the resulting destination. A successful parse only establishes syntax; it does not establish that the URL belongs to the allowed crawl.

   Restrict schemes to those your collector supports, typically HTTP and HTTPS. Reject embedded user credentials rather than allowing a discovered link to introduce a new authentication context. Match the parsed hostname and port against your scope rules. Do not use substring checks that accept an unrelated hostname containing the permitted name.

2. 2

   Preserve meaningful query parameters

   Create a per-origin list of parameters proven irrelevant to the required representation. Tracking parameters may be candidates, but there is no universal safe deletion rule. A parameter named source could select a data source, not a campaign. Keep language, pagination, filters and variant selectors unless you have evidence that they do not change the resource you need.

   Query strings can contain repeated keys, empty values and a significant order. Signed links may validate their exact encoding. Sorting all parameters or replacing plus signs can invalidate those links or change server behavior. The example removes only explicitly listed parameter names and leaves other ordering untouched. Use the generated value as a comparison key, not a universal request rewriter.

   export function comparisonKey(input, { base, ignoredParameters = [] } = {}) {  const url = new URL(input, base);  if (!['http:', 'https:'].includes(url.protocol)) throw new Error('unsupported protocol');  if (url.username || url.password) throw new Error('credential-bearing URL');  url.hash = '';  const ignored = new Set(ignoredParameters);  const parts = url.search.slice(1).split('&');  const kept = parts.filter(part => {    // Decode the name for comparison only; retain every remaining raw field.    const name = new URLSearchParams(part).keys().next().value;    return !ignored.has(name);  });  if (kept.length !== parts.length) url.search = kept.length ? `?${kept.join('&')}` : '';  // Do not sort query parameters, lowercase paths, or decode reserved escapes.  return url.href;}

3. 3

   Avoid destructive path transformations

   Hostnames have case-insensitive semantics, but paths often do not. Lowercasing the whole URL can merge two distinct manuals. Likewise, an encoded slash is not automatically interchangeable with a path separator. Let the standards-based parser perform its defined work; avoid a second blanket percent-decoding pass that changes reserved delimiters.

   A fragment usually identifies a location inside a retrieved document and is not sent in the HTTP request. Removing it from a fetch key is often useful, but browser applications can interpret fragments as client-side routes. If the task depends on rendered fragment routes, use a different identity policy and test it explicitly rather than applying the plain-HTTP rule.

4. 4

   Bound redirect and canonical handling

   Record the final URL after redirects and verify that it remains in scope. A redirect is a network instruction; a canonical link is publisher-provided metadata about preferred identity. They are not interchangeable. A canonical may be wrong, point outside the approved domain or describe a different localized version. Do not automatically replace every discovered page with its canonical target.

   Set redirect limits and detect loops. When multiple discovered URLs reach the same final representation, preserve the mapping so audit records still explain their origins. If the final page differs in language or selected variant, keep that distinction. Deduplication should reduce redundant work while preserving the evidence needed to explain what was actually collected.