Hiring Market Data: Normalize Roles, Employers and Locations
Build comparable public job records with clear employer identities, location rules and observation dates.
KAIROXO
Hiring market data becomes useful when records describe comparable openings, rather than merely pages that contain a job title. A search for a company can also return agencies, integration partners and employers mentioning that company in their descriptions. Counting every result as a vacancy creates a convincing chart with the wrong denominator.
Consider a weekly research project covering several employers in three countries. Its question is which teams are hiring, where those roles are based and which skills appear in the requirements. The output should be a reproducible observation of public postings. It should not claim to measure actual hires, undisclosed headcount or the complete labor market.
Define the record before collecting pages
Start with employer identity, source URL, source job identifier, title, location text, employment type, posting date and observation time. Keep requested skills separate from benefits and company descriptions. A technology mentioned in an employer introduction is not automatically a required skill. Record the section from which each extracted field came.
Use explicit unknown values for missing salary, seniority or closing dates. Do not convert an absent salary into zero or infer a seniority level from compensation. Preserve the original text alongside normalized values so an analyst can challenge a mapping. Schema.org JobPosting is a useful field vocabulary, but a page using that vocabulary can still contain stale or incomplete data.
Separate the employer from the search match
A company name in a search result is a discovery clue. Confirm the listed hiring organization, its public domain and the destination of the application link. Record agency roles separately when the ultimate employer is undisclosed. Maintain an alias table for genuine name variations, with a reviewer and evidence for each accepted mapping.
For example, a fictional vacancy that says experience with Acme payments is required must not be assigned to Acme. The advertiser remains the employer shown on the posting. Likewise, a subsidiary and its parent may share a brand while operating different hiring programs. Keep their original identities before creating an optional group-level report.
Normalize location without inventing eligibility
Keep city, region, country and remote policy in separate fields. Remote, hybrid and onsite are work arrangements, not geographic locations. A role advertised as remote in Vietnam is not necessarily open to applicants everywhere. Preserve eligibility restrictions exactly as stated and avoid transforming a country-specific remote role into a worldwide opportunity.
Normalize common location spellings with a versioned lookup, but retain ambiguous values for review. Several cities share names, and a single posting can list multiple possible offices. Represent that as one opening with multiple locations unless the source explicitly identifies separate positions. Otherwise location expansion can multiply the apparent number of vacancies.
Distinguish postings from observations
Assign a durable posting key using source identity and its job identifier where available. Store each collection as an observation attached to that key. A changed description is a new observation, not automatically a new job. Across sources, use employer identity, application destination, title and location as matching evidence; keep uncertain matches unresolved.
If a previously seen posting disappears, mark it not observed in the latest successful scan. A request failure or incomplete pagination cannot prove that a vacancy closed. Require a documented rule, such as an explicit closed message or repeated complete scans, before assigning a closed state. This distinction protects trend reports from collection outages.
Compare like periods and explain coverage
Publish the source list, observation window and inclusion rules with each report. Compare records collected through the same scope and normalization version. If a new employer or country is added, show that coverage change separately from hiring growth. A larger dataset can reflect better collection rather than more openings.
Use a small review sample from every source, including multi-location jobs, staffing agencies, missing dates and translated titles. Measure employer-match precision and missing-field rates before scaling. Keep original titles in their source language; translated titles support cross-language analysis but should not silently replace evidence. Reports should label counts as observed postings, not employees or completed hires.
Design a restrained collection workflow
Prefer public career feeds or documented APIs when they provide the required data. For permitted public pages, request only the needed scope, bound concurrency and stop on persistent access errors. Keep applicant information, résumés and account-only content outside this research dataset. A recruiting intelligence project usually needs employer-side publications, not personal profiles.
Separate access failures, parsing failures and validation failures in the run summary. Save the last successful coverage record without treating it as a fresh scan. When a selector changes, repair and replay a bounded fixture sample before resuming collection. This makes a weekly refresh operationally useful without pretending that every source always works.