Each of these datasets was built to answer a narrower question, by people who never expected their work to end up joined to others. Building a settlement typology out of them is where an LLM came in. I did not use it to generate the findings, but it facilitated my work in merging disparate community sources into one. It helped by converting them into spreadsheets, importing them into a relational database (Neo4j), filtering this database for relevant information, standardizing the data against chronological change, and then effectively and accurately querying the results to answer new questions asked of old material. A census enumerator’s tally sheet, a grain elevator inspector’s license log, and a local history society’s book were never built to talk to each other. Now they can.
In my MA thesis, I built one example of how GraphRAG can work for historians. From 20,000 early modern documents, I constructed a knowledge graph of 218,000 entities and 692,000 relationships, then tested eight retrieval systems on thirty historical questions of escalating difficulty and then grading each of them. Plain RAG scored 48.4 percent against the metrics of groundedness, completeness, accuracy, synthesis, and usefulness. The best off-the-shelf system reached only 61.6 percent, an unacceptable result for a careful historian.
Part of the reason those systems created these kinds of errors was that they were not created for historical research. However, looking at these errors from a historian’s lens allowed me to create a better solution for history. Additionally, I noticed that those systems had a tendency to repress those entities which appear only rarely in the corpus. Sometimes, the most valuable entities, the obscure people and the once-mentioned places, are the very heart of where historical discovery is found. Fixing the systemic problems, and surfacing these obscure entities, were my goals in creating my own solution.
