Sitemap URL Discovery with Robots Rules and Crawl Boundaries
Build a bounded candidate inventory from robots.txt and sitemap indexes, keeping discovery, permission and verified page coverage separate.
KAIROXO
Sitemap URL discovery gives a crawler a structured starting point instead of relying entirely on page navigation. A published sitemap can expose manuals or articles that are awkward to reach through menus. Robots.txt provides a separate set of crawling preferences. Together they help plan a bounded collection, but neither grants legal permission nor guarantees that every listed URL is current, relevant or accessible.
Consider an authorized crawl of a public help center. The agreement covers one domain and its documentation paths. A sitemap index points to several child sitemaps, while robots.txt excludes administrative and search pages. We need to gather candidate documentation URLs without following every link or accidentally expanding to unrelated hosts.
Separate discovery, permission and retrieval
A sitemap is an inventory hint, robots.txt expresses crawling rules, and the task's authorization defines what your collector may do. Treat these as independent inputs. A URL appearing in a sitemap is not a reason to ignore the agreed scope. Likewise, a robots rule is not an authentication system protecting private resources.
Google's Robots Refresher explains the purpose and location of robots.txt and discusses sitemap discovery. Use the current Robots Exclusion Protocol and sitemap documentation to implement parsing rather than copying an informal line-splitting example. The details of grouping, path matching, redirects and retrieval errors matter once the crawler runs unattended.
Implementation steps
Establish a bounded crawl scope
Write down allowed schemes, hostnames, ports and path areas before fetching the first discovery file. Use parsed URLs for comparisons. A hostname merely containing the approved name is not an allowed host. Set limits on redirect depth, sitemap count, decompressed bytes and accepted candidate URLs so a surprising inventory cannot create an unlimited job.
Choose a descriptive crawler user agent and the robots policy that applies to it. Fetch robots.txt for the relevant origin, not once for every unrelated website. Record retrieval time and the policy decision. Reuse that policy according to a bounded freshness rule, and refresh it when the task lasts long enough for changes to matter.
Parse rules without guessing their meaning
Use a maintained parser that follows the chosen standard. User-agent groups and path rules require more than searching for the word Disallow. Preserve a reason for every excluded URL so operators can distinguish a robots exclusion from a path-scope exclusion. If policy retrieval fails, apply a documented error policy rather than silently treating every failure as unrestricted access.
Do not confuse robots.txt with page-level noindex directives. One controls crawling behavior; the other concerns indexing by systems that implement it. This collection produces a dataset, not a public search index. Any indexing or publishing use must respect its own requirements and the source's permissions rather than borrowing assumptions from a search engine tutorial.
Expand sitemap indexes carefully
Read sitemap declarations and any explicitly approved sitemap location. Distinguish an index from a URL set, then follow child sitemap links within the allowed discovery policy. Track visited sitemap URLs to stop cycles. XML parsers must not fetch external entities, and compressed files need a decompressed-size limit as well as a download-size limit.
Validate every loc value individually. An index may point to another hostname, and a URL set may contain expired links or an unexpected language area. Keep candidate discovery separate from retrieval so you can inspect counts and rejection reasons before launching many requests. Do not treat lastmod as independent proof that the underlying page changed exactly then.
Filter, deduplicate and reconcile candidates
Apply the allowed path and robots policy to each candidate, then deduplicate using a conservative identity rule. Preserve the original URL and the sitemap that supplied it. A page appearing in several child inventories should not create multiple current records solely because discovery found it twice. See URL normalization for avoiding destructive query changes.
Compare accepted candidates with a small sample of navigation links. A sitemap can omit old pages, include removed pages or lag behind a deployment. Report what was discovered and what was actually retrieved. A finite, processed sitemap is evidence of completing that inventory, not proof that every relevant page on the website was collected.
Verify the discovery process offline
Build fixtures for a simple URL set, a nested index, a repeated child, an external hostname and a malformed document. Test a URL that is in scope but excluded by robots and one allowed by robots but outside the contract. They should produce different reasons. Add an oversized or expanding compressed input to verify the parser cannot grow work without limit.
Start network testing with a tiny authorized sample. KAIROXO can supply the configured access route, while your application owns robots decisions, XML parsing and work admission. Record status and final URL for discovery documents just as carefully as ordinary pages. A proxy response that looks like XML but is actually an error document should fail format validation.