Each of these datasets was built to answer a narrower question, by people who never expected their work to end up joined to others. Building a settlement typology out of them is where an LLM came in. I did not use it to generate the findings, but it facilitated my work in merging disparate community sources into one. It helped by converting them into spreadsheets, importing them into a relational database (Neo4j), filtering this database for relevant information, standardizing the data against chronological change, and then effectively and accurately querying the results to answer new questions asked of old material. A census enumerator’s tally sheet, a grain elevator inspector’s license log, and a local history society’s book were never built to talk to each other. Now they can.
In my MA thesis, I built one example of how GraphRAG can work for historians. From 20,000 early modern documents, I constructed a knowledge graph of 218,000 entities and 692,000 relationships, then tested eight retrieval systems on thirty historical questions of escalating difficulty and then grading each of them. Plain RAG scored 48.4 percent against the metrics of groundedness, completeness, accuracy, synthesis, and usefulness. The best off-the-shelf system reached only 61.6 percent, an unacceptable result for a careful historian.
Part of the reason those systems created these kinds of errors was that they were not created for historical research. However, looking at these errors from a historian’s lens allowed me to create a better solution for history. Additionally, I noticed that those systems had a tendency to repress those entities which appear only rarely in the corpus. Sometimes, the most valuable entities, the obscure people and the once-mentioned places, are the very heart of where historical discovery is found. Fixing the systemic problems, and surfacing these obscure entities, were my goals in creating my own solution.
The outputs of many OCR technologies were developed for training LLMs, but that are frequently of little use in the humanities. They can be used by people with specific data mining and data science purposes, like researchers doing Named Entity Recognition. But these are still rather niche, and the majority of archival researchers and historians who want to access these OCR outputs would instead benefit from the document processing of OCR resulting in searchable archives. This is another area where AI-driven OCR is helpful, as it has the capacity to easily transform OCR outputs into specific and bespoke formats for the needs of the researchers who are using these outputs.
talkie’s poor representation of history is a side effect of a larger problem. The model was conceptually built as an experimental tool to see if language models can accurately predict the future, also known as “forecasting.” The blog post assertively claims the models predictive capabilities, yet is devoid of evidence enabling researchers to reproduce their experiment. The version that could serve serious research is gated behind compute, and is the same version released with no guardrails at all.
Nastasha Sartore Earlier this year, over forty people logged in to Zoom to attend a CHA webinar titled “Generative AI and the Practice of History.” Introduced as a “show and tell” for AI tools and strategies, the webinar included a panel of three expert historians who each presented on a distinct topic related to agentic AI, the tech that’s been… Read more »
By Jessica van Horssen Historians aren’t really made to be on film. Or at least not beyond a 15-second “talking head” clip in a documentary, and even then, we can be woefully thrown off-course. As a fan of Diane Morgan’s Philomena Cunk, I’m well-versed in the risks of historians on the screen. This is why I was both excited and… Read more »
Mack Penner and Edward Dunsworth In his case for “steering a middle course” on the use of artificial intelligence (AI) in the history classroom, written partially as response to earlier pieces by each of us, Mark Humphries makes a number of points with which we agree. First among those points of agreement are the value of a historical education and… Read more »
Edward Dunsworth Two approaches dominate discussion about how professors should handle generative “artificial intelligence” in the classroom: give up or give in. Give up. Faced with a powerful new technology custom-cut for cheating, many professors are throwing up their hands in despair. This was the dominant mood of last month’s widely shared New York Magazine article. “Everyone is cheating their… Read more »
Commercial DNA tests have had many different impacts, from confirming existing research, breaking down brick walls, and uncovering long-hidden family secrets. DNA has become an essential component of genealogical research.
