In my MA thesis, I built one example of how GraphRAG can work for historians. From 20,000 early modern documents, I constructed a knowledge graph of 218,000 entities and 692,000 relationships, then tested eight retrieval systems on thirty historical questions of escalating difficulty and then grading each of them. Plain RAG scored 48.4 percent against the metrics of groundedness, completeness, accuracy, synthesis, and usefulness. The best off-the-shelf system reached only 61.6 percent, an unacceptable result for a careful historian.
Part of the reason those systems created these kinds of errors was that they were not created for historical research. However, looking at these errors from a historian’s lens allowed me to create a better solution for history. Additionally, I noticed that those systems had a tendency to repress those entities which appear only rarely in the corpus. Sometimes, the most valuable entities, the obscure people and the once-mentioned places, are the very heart of where historical discovery is found. Fixing the systemic problems, and surfacing these obscure entities, were my goals in creating my own solution.
The outputs of many OCR technologies were developed for training LLMs, but that are frequently of little use in the humanities. They can be used by people with specific data mining and data science purposes, like researchers doing Named Entity Recognition. But these are still rather niche, and the majority of archival researchers and historians who want to access these OCR outputs would instead benefit from the document processing of OCR resulting in searchable archives. This is another area where AI-driven OCR is helpful, as it has the capacity to easily transform OCR outputs into specific and bespoke formats for the needs of the researchers who are using these outputs.
Edward Dunsworth While a small but vocal minority of historians have embraced the use of generative artificial intelligence in their research and teaching,[1] many of us find ourselves trying to hold the pedagogical line against what we perceive as serious threats to students’ ability to learn, reason, and express themselves intelligently.[2] In this post, I share my attempts to “teach… Read more »
talkie’s poor representation of history is a side effect of a larger problem. The model was conceptually built as an experimental tool to see if language models can accurately predict the future, also known as “forecasting.” The blog post assertively claims the models predictive capabilities, yet is devoid of evidence enabling researchers to reproduce their experiment. The version that could serve serious research is gated behind compute, and is the same version released with no guardrails at all.
Mack Penner and Edward Dunsworth In his case for “steering a middle course” on the use of artificial intelligence (AI) in the history classroom, written partially as response to earlier pieces by each of us, Mark Humphries makes a number of points with which we agree. First among those points of agreement are the value of a historical education and… Read more »
Edward Dunsworth Two approaches dominate discussion about how professors should handle generative “artificial intelligence” in the classroom: give up or give in. Give up. Faced with a powerful new technology custom-cut for cheating, many professors are throwing up their hands in despair. This was the dominant mood of last month’s widely shared New York Magazine article. “Everyone is cheating their… Read more »
To the extent that we as historians accept as settled the first order questions about AI and instead opt to talk about nuanced details of implementation, I think we risk a very serious mistake. Here, then, I want to publicly state my view of AI and its use in history, and to do so without any qualification. I hate AI.
by Carly Ciufo “Do museum workers do human rights work?” I ask ChatGPT. The artificial intelligence’s (AI) answer is longer and, honestly, more robust than I expect:
Sean Kheraj In their 2005 article in First Monday, Daniel J. Cohen and Roy Rosenzweig recount the story of a remarkably prescient colleague, Peter Stearns, who “proposed the idea of a history analog to the math calculator, a handheld device that would provide students with names and dates to use on exams—a Cliolator, he called it, a play on the… Read more »
