Website Change Detection: Reduce Noise and Keep Evidence
Monitor public pages with stable comparison regions, versioned normalization and explicit states for changes, failures and missing content.
KAIROXO
Website change detection should answer a specific question: did the information we care about change? Comparing whole HTML files often answers a noisier question about timestamps, banners, rotating recommendations or generated identifiers. A useful monitor defines its scope before choosing a hash function or alert channel.
Consider a team watching public product documentation, service announcements and a small set of partner pages. The desired output is a reviewable change record, not a stream of notifications whenever markup moves. Keep collection, normalization, comparison and notification as separate stages so a failure in one stage cannot masquerade as a business change.
Define the observation and its states
Choose the exact page region or fields that matter. A documentation monitor may care about supported options and examples, while an announcement monitor may care about new entries and their publication dates. Write down expected selectors or structural markers, along with what counts as missing content. A selector returning nothing should not automatically mean the page has become empty.
Use explicit states such as baseline, unchanged, changed and collection_error. If the expected region disappears, classify that separately for review rather than replacing the last valid baseline. This keeps a timeout, challenge page or parser regression from generating a false deletion event. An absent alert is meaningful only when the check actually completed successfully.
Keep a stable, versioned baseline
Store the source identifier, collection time, relevant request context, extraction rule version and normalized content with the baseline. Preserve a bounded raw snapshot where appropriate so you can reprocess it after a parser correction. The baseline is a known valid observation, not simply the last response received from the network.
When extraction or normalization rules change, treat the migration explicitly. Comparing text produced by two different rules can create a large artificial diff even if the page is unchanged. Recompute comparable baselines from saved evidence or initialize a new version. Keep an audit record of that decision so downstream reviewers know why the comparison history restarted.
Remove noise without removing meaning
Begin with deterministic rules: scope the main region, remove known decorative elements, normalize whitespace and preserve meaningful order. Ignore a rotating banner only if it is outside the question being monitored. Removing every date or number because they often change would also remove release dates, quantities and prices that might be the most important signal.
Choose the comparison method according to the content. Exact hashes efficiently detect identical normalized text. Line or field diffs show what changed. Similarity scores can help prioritize large queues, but their thresholds need representative examples. A low score does not explain the business meaning, and a high score can still hide one critical changed sentence.
Use conditional requests carefully
Where the source provides validators, ETag or Last-Modified may help avoid transferring an unchanged representation. Their meaning belongs to the requested representation and request context. Keep relevant headers, language and authentication scope consistent. Do not treat a validator observed for one representation as universal evidence that every variant of the page is unchanged.
A 304 response is useful only when you have the corresponding valid cached content and comparison state. If the extraction rules changed, the stored representation may need reprocessing. If the baseline was lost, there is nothing reliable to compare. Conditional requests optimize collection; they do not replace a complete baseline and versioning strategy.
Deliver an alert that can be reviewed
An alert should contain the source, observation time, monitored scope and a concise before-and-after difference. Include the underlying evidence or an approved reference to it. Distinguish new content, modified content and removal, and avoid making an automated summary the only surviving record. Reviewers should be able to confirm the claim without reconstructing an entire collection run.
Control alert frequency independently of polling frequency. Related edits can be grouped into one event, while collection errors may require a different operational channel. Use deduplication keys and delivery state so retries do not send the same alert repeatedly. Test the unhappy paths, including missing selectors and failed delivery, before relying on silence as a signal of stability.
Measure monitor quality
Review examples of true changes, false alarms and missed changes. Track successful checks, extraction failures, change-review outcomes and notification delivery separately. If the page layout changes, repair the extraction rule and replay saved evidence before trusting the new baseline. A monitor that sends fewer alerts is not necessarily more accurate; it may simply be failing to observe the page.
For a small pilot, use a known set of public pages and a few intentional local fixture changes. Verify that whitespace noise is ignored, meaningful edits are detected and missing content is reported as an error. These tests are more useful than a large initial crawl with no expected outcomes, because they exercise the actual question the monitor is meant to answer.