Web Scraping Data Quality: Validate, Quarantine and Review

Implementation steps

1. 1

   Step 1: Write a concrete data contract

   Describe the record with required fields, types, allowed values and limits. For this example, each document needs an identifier, title, HTTPS source URL, language, observation timestamp and a tags array. An empty array means no tags were extracted; an absent array means the parser did not fulfill the contract. These states should not be treated as interchangeable.

   Agree on timestamp semantics and identifier scope. An observation timestamp describes collection, not publication. A document identifier can be unique within one source without being globally unique. Document these rules beside the parser and use representative fixtures to review them with the consuming team. JSON Schema is useful for expressing structural rules; additional application checks are still needed for business meaning.

2. 2

   Step 2: Validate before admitting records

   The following dependency-free Node example implements a deliberately small contract directly. It is not a general JSON Schema engine and does not verify that a URL is safe to fetch. Its URL check validates record format only. Keep destination authorization and network controls in the collection layer, where they can be applied before any request is sent.

   export function validateRecord(record) {  const errors = [];  if (!record || typeof record !== 'object' || Array.isArray(record)) return ['record: expected object'];  for (const field of ['id', 'title', 'sourceUrl', 'language', 'observedAt']) {    if (typeof record[field] !== 'string' || !record[field].trim()) errors.push(`${field}: required non-empty string`);  }  if (typeof record.title === 'string' && record.title.length > 500) errors.push('title: maximum 500 characters');  try {    const source = new URL(record.sourceUrl);    if (source.protocol !== 'https:' || source.username || source.password) errors.push('sourceUrl: expected HTTPS without credentials');  } catch { errors.push('sourceUrl: invalid URL'); }  if (!['en', 'zh', 'vi'].includes(record.language)) errors.push('language: unsupported value');  const date = record.observedAt;  const iso = /^\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2}\.\d{3}Z$/;  if (typeof date !== 'string' || !iso.test(date) || !Number.isFinite(Date.parse(date)) || new Date(date).toISOString() !== date) {    errors.push('observedAt: expected valid UTC ISO timestamp with milliseconds');  }  if (!Array.isArray(record.tags) || record.tags.some(tag => typeof tag !== 'string' || !tag.trim())) errors.push('tags: expected array of non-empty strings');  return errors;} export function partitionRecords(records) {  const accepted = [], quarantined = [], seen = new Set();  for (const [index, record] of records.entries()) {    const errors = validateRecord(record);    const key = JSON.stringify([record?.sourceUrl, record?.id]);    if (!errors.length && seen.has(key)) errors.push('id: duplicate within source');    if (errors.length) quarantined.push({ index, record, errors });    else { seen.add(key); accepted.push(record); }  }  return { accepted, quarantined };}

   The timestamp rule accepts one canonical UTC representation and rejects impossible dates that JavaScript might otherwise normalize. The duplicate key combines source URL and source identifier for this demonstration. Adapt that scope to your real source registry instead of assuming two publishers use coordinated identifiers. The validator returns field-level reasons so a failed row can be inspected without disappearing from the run history.

3. 3

   Step 3: Quarantine failures and preserve context

   Send invalid records to a separate review collection with their source, run identifier, parser version and validation reasons. Keep valid output independent from that queue. Do not insert an empty title merely to make a required-field check pass, and do not carry forward an old value as if it had been observed again today.

   Preserve a bounded source fixture when permitted, or a minimal excerpt sufficient to diagnose the field. Store credentials outside evidence and redact sensitive request headers. A reviewer should be able to distinguish a missing source value, a parser failure and an unsupported format. Repairing extraction and replaying saved evidence is usually more precise than guessing what the missing value ought to be.

4. 4

   Step 4: Test quality across the batch

   Record-level checks cannot detect every coverage problem. A batch of ten valid rows may still be wrong if a formerly complete source normally provides hundreds. Track expected source checks, pagination completion and accepted versus quarantined rows. Use historical ranges as signals for investigation, not as instructions to manufacture enough records to satisfy a target.

   Create isolated tests for valid records, missing text, unsupported languages, invalid timestamps and source-scoped duplicates. Include a case where one invalid row does not discard the rest of the batch. Keep a holdout set of representative pages for extraction review. After a parser release, compare both output validity and field values against that set before increasing collection volume.