Transformer based named entity recognition for place name extraction from unstructured text
Cillian Berragan; Alex Singleton; Alessia Calafiore; Jeremy Morley (2023). International Journal of Geographical Information Science, 37(4), 747-766. DOI: 10.1080/13658816.2022.2133125
Abstract
Place names embedded in online natural language text present a useful source of geographic information. Despite this, many methods for the extraction of place names from text use pre-trained models that were not explicitly designed for this task. Our paper builds five custom-built Named Entity Recognition (NER) models and evaluates them against three popular pre-built models for place name extraction. The models are evaluated using a set of manually annotated Wikipedia articles with reference to the F1 score metric. Our best performing model achieves an F1 score of 0.939 compared with 0.730 for the best performing pre-built model. Our model is then used to extract all place names from Wikipedia articles in Great Britain, demonstrating the ability to more accurately capture unknown place names from volunteered sources of online geographic information.
Extended Summary
This research asks whether custom-built language models can more accurately identify place names in unstructured text than the generic, pre-built tools currently used in geographic information science. Place names appear constantly in online text, from Wikipedia to social media, but most existing natural language processing (NLP) systems were trained to recognise broad categories like ‘persons’ or ‘organisations’ rather than a dedicated ‘place name’ entity, causing many genuine place references to be missed or misclassified.
To address this gap, the study built five custom named entity recognition (NER) models using 200 manually annotated Wikipedia articles about locations in Great Britain, sourced via DBpedia. These included two simpler Bidirectional LSTM models and three transformer-based models—BERT, RoBERTa, and DistilBERT—which use modern ‘attention’ mechanisms allowing them to weigh relationships between all words in a sentence simultaneously. These custom models were then compared against three widely used pre-built NER systems (SpaCy small and large, and Stanza), commonly relied upon by existing geoparsing tools such as GeoTxt and Mordecai.
Results showed a substantial performance gap. The best custom transformer model, BERT, achieved an F1 score (a balanced measure of precision and recall) of 0.939, significantly outperforming the best pre-built model, Stanza, which scored only 0.730. The pre-built models particularly struggled with recall, meaning they frequently failed to detect genuine place names, especially when place references were used in unusual or context-dependent ways. For example, generic models often misclassified place names as ‘persons’ or ‘organisations’, or incorrectly extracted place names used metonymically (such as ‘China competed’ referring to a national team rather than the country itself). The fine-tuned transformer models, by contrast, correctly distinguished between these figurative and literal uses based on surrounding context.
Applying the best-performing efficient model, DistilBERT, to over 42,000 Wikipedia articles across Great Britain, the research extracted more than 614,000 place name mentions, comprising almost 100,000 unique names. Notably, over 62,000 of these unique place names did not appear in the GeoNames gazetteer, a standard geographic reference database. These missing names included hyper-local references such as road names, informal or alternative names, and organisational names used in place-related contexts, revealing a significant volume of geographic information invisible to conventional gazetteers.
The findings carry important implications for geographic information science and volunteered geographic information (VGI) research. They demonstrate that task-specific training data, tailored explicitly to place name recognition rather than general-purpose entity categories, produces markedly better results than adapting off-the-shelf models. This approach offers a pathway toward building gazetteers from the bottom up, capturing vernacular and informal place names directly from public text sources rather than relying solely on formal administrative databases. The methodology could be extended to other unstructured text sources beyond Wikipedia, potentially enriching gazetteers and geographic databases with previously unrepresented, community-recognised place names.
Key Findings
- Custom transformer-based NER models (BERT) achieved an F1 score of 0.939, far exceeding the best pre-built model (Stanza) at 0.730
- Pre-built NER models suffered predominantly from low recall, frequently failing to detect genuine place names in text
- Fine-tuned models correctly distinguished metonymic place name usage (e.g., country names representing sports teams) from literal geographic references
- Extraction from 42,222 Wikipedia articles yielded over 614,000 place name mentions, including 62,178 unique names absent from the GeoNames gazetteer
- Task-specific training data tailored to place name recognition substantially outperforms generic annotation schemes used by off-the-shelf NER tools
Citation
@article{berragan2023transformer,
author = {Cillian Berragan; Alex Singleton; Alessia Calafiore; Jeremy Morley},
title = {Transformer based named entity recognition for place name extraction from unstructured text},
journal = {International Journal of Geographical Information Science},
year = {2023},
volume = {37(4)},
pages = {747-766},
doi = {10.1080/13658816.2022.2133125}
}