HTTP Conditional Requests: ETags, 304 and Freshness
Reduce repeated document transfers while retaining cached bodies, language context and a clear distinction between stale and revalidated content.
KAIROXO

HTTP conditional requests let a collector ask whether a stored representation is still current before transferring the entire body again. They are useful when an authorized crawl revisits stable documents: specifications, manuals or public reference pages. The benefit is reduced repeated transfer, not a guarantee that a site will provide validators or that a cached document answers a new question.
Consider a technical-manual collection refreshed every morning. Most pages are unchanged, but some exist in several languages. A cache must distinguish those representations and retain the body that a 304 response refers to. Otherwise a bandwidth optimization becomes missing text or a language mix-up downstream.
Understand what is being validated
An ETag is an opaque validator assigned by the origin to a representation. Store it as received, including quotes and any weak prefix. It is not necessarily a file hash, and it should not be treated as proof that a download came from a trusted publisher. Last-Modified is another validator, with date semantics and different precision limitations.
Google's discussion of crawler caching describes If-None-Match and If-Modified-Since as ways to avoid transferring unchanged content. That is the relevant principle for our collector. It does not imply that your custom crawler has Google's scheduling behavior or that every origin will honor the same conditional request.
Implementation steps
Define the cache key before storing bodies
At minimum, identify the URL and the request context that selects content. Language can matter, along with authentication scope, accepted format or region. Respect Vary and other cache controls when building a general cache. Do not share personalized responses between users merely because their URL strings match.
For the manual example, store English and Vietnamese separately. Use an explicit policy for URLs with query parameters rather than removing parameters that look inconvenient. Keep fetched-at and validated-at timestamps distinct: checking that a body is unchanged today does not mean that its text was first downloaded today.
Retain a complete initial representation
On the first successful response, validate status and expected content type, read the complete bounded body, and save it with the validator. Do not cache partial reads, parse failures or error pages as though they were valid manuals. A timeout halfway through the body is still a failed retrieval even when headers arrived successfully.
Honor no-store by avoiding persistence, including removal of a previously saved entry when appropriate. A private cache and a shared cache have different rules. The following demonstration focuses only on a per-language application cache for known public pages; it deliberately does not claim to implement all HTTP caching semantics.
Codejavascript export async function revalidate(url, language, cache, fetchOnce) { const key = JSON.stringify([url, language]); const saved = cache.get(key); const headers = { 'Accept-Language': language }; if (saved?.etag) headers['If-None-Match'] = saved.etag; const response = await fetchOnce(url, { headers }); const control = response.headers.get('cache-control') ?? ''; if (response.status === 304) { if (!saved) throw new Error('304 without cached representation'); if (/\bno-store\b/i.test(control)) cache.delete(key); return { ...saved, revalidated: true }; } if (response.status !== 200) throw new Error(`unexpected status ${response.status}`); // Demo bound: callers must set transport and body-size limits for real targets. const text = await response.text(); const item = { text, etag: response.headers.get('etag'), revalidated: false }; if (!/\bno-store\b/i.test(control)) cache.set(key, item); else cache.delete(key); return item;}Revalidate and handle 304 correctly
Send the saved ETag in If-None-Match when revalidating the same representation. A 304 normally has no response body: use the corresponding stored body instead. If that body is absent, treat the response as unusable and perform a deliberate unconditional recovery request under the normal request budget. Do not replace your document text with an empty string.
If the server returns 200, validate and replace the stored representation and its validator. If it returns an error, retain the old body separately if policy allows, but mark it stale rather than presenting it as freshly validated. Stale data may be useful to a reader, but a downstream system must know the difference.
Choose freshness for the task
Freshness is a business requirement, not a universal number. A specification archive and a live availability feed tolerate different delays. Document how long a stored body can be used without contacting the origin, when revalidation is mandatory, and whether a failed check allows serving a clearly labeled older version.
Avoid unnecessary cache busting. Adding random query strings can bypass useful origin caches and increase load while selecting a different URL identity. Conversely, never assume that a cache hit means fresh enough: inspect the age and the policy that produced it. Record cache hits, validated hits and fresh transfers as distinct events.
Keep regional and proxy context visible
A proxy changes the network path and can also affect a region-sensitive representation. If the target shows different manuals, notices or downloads by location, include that selection in your cache design. Do not reuse one region's body simply because another request has the same path. A validator observed under one representation may not describe another.
Use KAIROXO access profiles to keep routing consistent during your experiment, and review website change detection when you need semantic differences after retrieval. Conditional caching reduces transfer; detecting meaningful edits is a separate step with its own comparison policy.