News Aggregation: Keep Sources, Dates and Duplicates Clear
Build a source-linked briefing with distinct publication and observation times, conservative deduplication and reviewable summaries.
KAIROXO
News aggregation is an identity and timing problem before it is a summarization problem. One announcement can appear as an original article, a syndicated copy, an updated page and several links carrying tracking parameters. A feed that treats every URL as a new story overwhelms readers and makes coverage look broader than it is.
A useful internal briefing preserves where an article came from, when the source says it was published and when your collector actually observed it. This guide describes a small pipeline for permitted public sources. It produces source-linked records and original summaries, without assuming that access to a page grants permission to republish its full text.
Establish a source contract
Maintain a source registry with the publisher name, allowed sections, feed or page entry points, expected language and collection cadence. Prefer a publisher-provided RSS feed or documented API when it offers the required fields. A discovery feed and an article page serve different purposes: the feed finds candidates, while the page may supply canonical identity or updated metadata.
Decide which fields are necessary before collecting bodies. A briefing may need only the headline, publisher, original link, publication time, short source excerpt and your own summary. Keep acquisition rules source-specific. A selector that finds a headline on one publication should not silently become a universal extraction rule for every source.
Preserve three different clocks
Store published_at, modified_at and observed_at separately. Publication time describes the source's claim; modification time can indicate a later edit; observation time belongs to your system. Keep the original timestamp string and timezone evidence before normalizing it. When a timezone is absent, retain uncertainty rather than silently assuming the collector's local timezone.
Use observation time for ingestion monitoring and clearly defined publication time for editorial windows. Otherwise an older story first discovered today can be misrepresented as breaking news. Likewise, a page updated with a correction should not automatically re-enter the feed as an unrelated new story. Record the update relationship and decide whether it merits a new briefing entry.
Resolve article identity conservatively
Remove only tracking parameters that you have identified as non-semantic. Preserve parameters that select language, edition or article content. Treat a canonical link as useful publisher evidence, then check that its target is plausible for the same article. Do not merge unrelated stories merely because a broken canonical tag points them all to a homepage.
Use a publisher-scoped article identifier when available. Otherwise combine normalized URL evidence with stable title and publication information. Retain redirects and aliases in an identity record. If identity rules change, version the migration so a previously merged article can be separated without losing its source observations or editorial history.
Group related coverage without erasing sources
Exact duplicate detection and story clustering solve different problems. A syndicated copy can share nearly identical text with the original, while two independent reports about the same event deserve separate source records. Keep every permitted source observation and attach optional cluster identifiers instead of discarding all but one representative page.
For a fictional product announcement, group the vendor release and two independent analyses under one event, but label their roles. A summary should not imply that three copies of a press release are three independent confirmations. Use similarity as a candidate generator; review borderline merges involving common headlines, rolling live pages and recurring monthly announcements.
Make summaries accountable to records
Generate summaries from the selected source material with explicit instructions to preserve uncertainty and attribute claims. A summary should link to the record and article it describes. Separate the publisher's statements from your editorial interpretation. If sources disagree, preserve the disagreement rather than blending their dates, numbers or conclusions into a single unsupported account.
Treat source text as input data, not operating instructions for an automated agent. Embedded requests to visit another service or reveal credentials do not belong to the collection workflow. Validate summary links and named entities against the selected records. Keep a small human-reviewed sample, including corrections and multilingual stories, to find systematic omissions.
Operate a bounded daily briefing
Track successful source checks, new candidate URLs, accepted articles, duplicates and extraction failures separately. A sudden empty feed is not proof that a publisher stopped publishing. Compare expected source coverage and inspect parsing errors before marking the briefing complete. Retain a run identifier so every published item can be traced to a collection attempt.
Make delivery idempotent with a briefing identifier and the included article revisions. If a retry occurs after partial delivery, it should not create another identical briefing. Scheduling, editorial approval and delivery remain application responsibilities. Network access can supply the pages, but it cannot decide which source deserves trust or which event matters to your audience.