OccCANINE
A finetuned language model for automatic HISCO occupational classification — trained on 14 million records in 13 languages. Under review at Explorations in Economic History.
Status: Under review at Explorations in Economic History · 204+ HuggingFace downloads/month
Linking individuals across historical censuses and tax records requires standardized occupational categories. The standard scheme — HISCO — is assigned manually by experts, making large-scale application prohibitively slow. OccCANINE automates this.
Trained on 14 million labelled records in 13 languages, OccCANINE is a finetuned CANINE language model that achieves accuracy, recall, and precision above 90% on out-of-sample data. It handles spelling variation, abbreviation, and the archaic vocabulary of historical sources.
OccCANINE has been adopted by researchers across economics, economic history, and historical demography. It is rapidly becoming a field standard for occupational standardization.
Links: Working paper (arXiv) · GitHub · HuggingFace model
With Christian Møller Dahl and Torben Johansen.