Website to Markdown: Prepare Data for an AI Knowledge Base
Turn authorized public pages into traceable knowledge documents with clean structure, source metadata, deduplication and a deliberate refresh policy.
KAIROXO
A website-to-Markdown pipeline should produce documents a reader can trace, not just text a model can ingest. Clean formatting is useful, but a knowledge base also needs source identity, collection time, language, stable document boundaries and a way to replace stale material.
Start with an authorized public documentation collection and a small question set. Define which pages belong in the corpus and which should stay out. This guide describes an application-owned preparation workflow around a verified collection service; it does not assume that every KAIROXO endpoint natively returns Markdown or manages a complete retrieval system.
Define the corpus and the evidence contract
List approved source domains, included paths, content languages and exclusion rules. Separate main documentation from navigation, account pages, temporary search results and unrelated downloads. Preserve the original source URL even if the crawler discovers a canonical or redirected address.
Decide what every prepared document must carry: source URL, title, language, actual collection time, source revision when known, content hash and transformation version. A page's claimed publication time and the time you fetched it are different facts. Keep unknown values unknown instead of inferring an update date from a footer.
Acquire the page through a verified path
Use a direct authorized request or a collection API that supports the required source and response format. Confirm whether the desired content is available in retrieved HTML or needs supported rendering. Separate network failure, inaccessible content and a valid empty page before attempting conversion.
The KAIROXO Web Crawler API can be evaluated as the acquisition component. Map its actual response into your preparation workflow. If Markdown is not part of the verified output contract, perform conversion in your own application and describe that responsibility explicitly.
Preserve structure while removing boilerplate
Extract the meaningful content region, then retain heading hierarchy, paragraphs, lists, links, tables and code fences that affect understanding. A conversion that keeps every navigation link can pollute retrieval, while an aggressive cleaner can remove warnings, prerequisites or examples that make an answer correct.
Resolve relative links against the page's actual URL and preserve source references. Keep code examples intact and distinguish them from surrounding prose. For tables, verify that headers remain associated with values; a plain sequence of cells may lose the relationship the reader needs. Use a documented Markdown dialect such as CommonMark and test the constructs your renderer supports.
Normalize and deduplicate deliberately
Apply conservative whitespace and text normalization, then compute a hash for the normalized content. Keep the original and canonical URL relationship rather than treating every discovered URL as an unrelated document. Identical copies can share content handling while retaining their provenance.
Do not merge pages solely because their titles match. Different versions or languages may contain materially different instructions. Keep language and version boundaries visible. Store the transformation version with the hash so a cleaning-rule change can be distinguished from a source update and old documents can be reprocessed deliberately.
Prepare chunks and a searchable index
Split around meaningful sections before applying the size limits required by your retrieval stack. Keep a heading path and source identity with each chunk, and avoid dividing a short prerequisite from the instruction it qualifies. Preserve useful context around code and tables instead of relying on character count alone.
The embedding model, index and retrieval policy are separate application decisions. Validate their actual input constraints and version them with the preparation process. Assign stable document and chunk identities so a refresh can replace an older version without leaving stale fragments mixed with new material.
Validate retrieval with real questions
Create a small set of questions whose answers are present in known source sections, plus questions that the corpus cannot answer. Check that retrieved passages contain the needed evidence and that the answer can point back to the correct source. A well-formatted document does not prove that retrieval selects the right passage.
Treat collected page content as untrusted data. A document can contain instructions, advertisements or malicious text, but those are source material rather than authority to operate tools or change application behavior. Keep source content separate from the system's operating instructions and restrict downstream actions to the application's intended policy.
Refresh without losing the history
Choose refresh frequency from source importance and how quickly stale information becomes harmful. Re-fetch within the source's permitted limits, compare content and replace changed documents together with their chunks. Track removed pages as well as updated ones so deleted guidance does not remain available indefinitely.
Record freshness, conversion failures, duplicate rates and unanswered-question examples. A failed refresh should preserve a visible status and the age of the last accepted document. Do not label an old snapshot as current merely because a scheduled job ran. Keep only the source evidence and operational metadata needed for the knowledge use case.