Pagination Scraping: Stop Rules and Record Deduplication

Implementation steps

1. 1

   Choose an explicit stopping rule

   Prefer the site's documented end signal: no next link, a null cursor or a false continuation flag. An empty result may also be terminal, but only when that is the observed contract. Some services return temporarily empty batches with a continuation cursor. Conversely, a disabled button may still exist in the DOM and should not be clicked indefinitely.

   Always add a safety cap on pages and elapsed time. Reaching that cap means incomplete collection, not successful completion. Preserve the last cursor and stop reason so a later run can resume or investigate. A log saying only that the loop ended cannot distinguish a complete catalog from a job stopped by its own limit.

2. 2

   Detect repeated navigation state

   Keep a set of processed cursor values or resolved page URLs. If the next value repeats, stop with a loop error. A server may ignore an expired cursor and return the first page; a page link can also point back to itself. Counting requests alone will not reveal this problem until a large safety cap is reached.

   Consider a page fingerprint when the navigation state advances but the returned record IDs do not. That is useful evidence, though a repeated set can occasionally be legitimate. Report it as a suspected navigation problem rather than silently discarding pages. Distinguish transport retries of one page from advances to another page in your metrics.

3. 3

   Deduplicate records without losing observations

   Use the site's stable entity ID whenever available. Names and row positions are weak substitutes: two suppliers can share a name, and the same supplier can move between pages. Decide how to handle repeated IDs. A current-state dataset can keep the latest observation; an audit dataset may keep versions with observation times and collection identifiers.

   The small example makes its policy explicit: duplicate IDs replace earlier values, and only a null next cursor confirms completion. The injected page loader handles routing, authentication and retries separately. This separation lets tests simulate loops and overlapping pages without sending traffic to a real catalog.

   export async function collectPages(loadPage, { first = 'start', maxPages = 100 } = {}) {  const seenCursors = new Set(), records = new Map();  let cursor = first;  for (let page = 0; page < maxPages; page++) {    if (seenCursors.has(cursor)) throw new Error('pagination cursor repeated');    seenCursors.add(cursor);    const result = await loadPage(cursor);    if (!Array.isArray(result.items)) throw new Error('invalid page shape');    for (const item of result.items) {      if (typeof item.id !== 'string' || !item.id) throw new Error('missing stable ID');      records.set(item.id, item); // Deliberate policy: latest observation wins.    }    if (result.next === null) return { records: [...records.values()], pages: page + 1, complete: true };    if (typeof result.next !== 'string' || !result.next) throw new Error('missing or invalid next cursor; collection incomplete');    cursor = result.next;  }  throw new Error('page budget exhausted; collection incomplete');}

4. 4

   Reconcile the completed collection

   Compare unique record count with a reliable total when the site exposes one. Keep in mind that a visible total may be approximate, filtered differently or changing during the crawl. Record the filters, sort order, region and time window with the count. A mismatch should trigger investigation, not an arbitrary change to the expected number.

   For moving lists, a stable sort with a tie-breaker or a server-provided snapshot can reduce gaps. Neither is always available. State that limitation when reporting completeness. A second pass with overlapping boundaries may reveal missed IDs, but it cannot prove a continuously changing source was captured at one perfectly consistent instant.