Is this the new normal? Let's hope not.

By Ian Milligan

The government claims that Library and Archives Canada needs to be modernized so all Canadians can access archival services. Yet the state of Canada’s online collections are small and sorely lacking when compared to their expansive on-site collections. LAC does need to modernize, and the goal of expanding access beyond just Ottawa is actually a laudable one. But what they’re doing here, under the guise of ‘modernization’, is simply cutting services and diminishing our access to Canada’s past. In this post, I want to show you how small and insignificant LAC’s online collections are, why they haven’t taken them seriously, and that if we’re fighting for better on-site access, we might as well fight for better online access too! They are, after all, despite the rhetoric of LAC and the government, not incompatible in the slightest.

What has been happening? Canada’s archives are under attack. Announcements made on May 1st confirmed that Library and Archives Canada will lose 20% of their workforce, that appointments will be needed to access the reference desk, and the shuttering of the National Archival Development Program. LAC is spinning this as a “new approach to service delivery” and makes the following point by way of justification:

LAC’s service statistics provide a vivid illustration of this digital revolution. Our website now gets close to half a million visits per month. In contrast, LAC’s in-person service hub located at 395 Wellington Street, receives about 2,000 visits per month. These two service points are also trending in opposite directions, with online consultations increasing rapidly, and in-person visits declining slowly but steadily.

Despite rhetoric of modernization and looking forward to the digital era, LAC has been comparatively slow in aggressively preparing for the next generation. It has been pointed out by several people that it will be even slower now that 50% of its digitization staff will be cut!

We need to contextualize LAC’s digital collections. Let’s quickly see what the Library of Congress (LOC) in the United States has been up to on this front. Their print collection is already dwarfed by their newest collection of archived born-digital sources. If each and every one of their 26 million books was scanned and digitized at 8MB per book, the collection would be about 200TB (a figure that you can now conceivably store at home). Just from websites alone, the LOC now has 254TB of data, adding 5TB a month. They’re taking the internet seriously. Library and Archives Canada, on the other hand, only collects a “representative sample of Canadian websites,” notably the government, and has about 7TB of data. While some recent pronouncements suggest that LAC is taking born-digital sources seriously are encouraging and should be celebrated, it is a pin compared to a much larger haystack of archival information.

Firstly, there simply isn’t that much there that has been digitized. LAC has an incredible on-site collection for Canadian historians: be it political history, social history, cultural history, military history, etc. If we go to their “about the collection” page, we see that they have: 71,000 films, 2.5 million architectural drawings, millions of books, 21.3 photographs, not to mention the thousands upon thousands of boxes. In that list I provide, they also note that as of 2007 they had “3.18 million megabytes of information in electronic formats.” Sounds, impressive eh? That’s only a bit over 3TB of data. Presumably that doesn’t include their web archive, which dwarfs that, but it starts to give you a sense of the mismatch in size between conventional and digitized historical sources.

They also don’t take many steps to make their online collections (what they have there) fully accessible to researchers. Canadiana.ca provides an Application Programming Interface (a way for a computer program to talk directly to another computer, and speed up research – and let you do new, cool things with the data). Library and Archives Canada does not.

If LAC was really serious about modernization, if they put more of their collection online in a comprehensive manner, if they were open to new forms of research, and if they didn’t do this at the expense of their on-site collections, this would be a good thing.

But, given the state of their online collections, I don’t see any reason to be happy here.

So, as we begin our fight to Save Library and Archives Canada, taking action by sending letters to the Minister of Canadian Heritage as well as the Librarian and Archivist of Canada, why don’t we also call out for better online services. There’s a kernel of truth to some government pronouncements: all Canadians, not just those who can come in person to Ottawa, deserve access to their national archive. But instead of using that chip as a talking point to justify cuts, let’s actually mean it. All Canadians deserve robust archives, be it on-site or online.

Ian Milligan is co-editor of ActiveHistory.ca and is also a postdoctoral fellow with Western University’s Department of History.

We’re happy to announce that ActiveHistory.ca is getting a new look! Over the next few days, we will be implementing some major changes to our website. This process should take about a week or so, so things may be in some flux.

If you have any comments about our new site, such as any features that may have been moved during our migration, or things you’d love to see, please let us know below.

By Ian Milligan

The splash page for the Globe and Mail's "Canada's Heritage Since 1844" website.

Online digitized newspapers are great. If you have access (either through a free database or via a personal or library subscription), you can quickly find the information you need: a specific search for a last name might help you find ancestors, a search for a specific event can find historical context for it (i.e. the Christie Pits Riots, or a certain strike), and generally the results are beautiful, render relatively well, and are – crucially – immediate.

In some ways, however, poor and misunderstood use of online newspapers can skew historical research. In a conference presentation or a lecture, it’s not uknown to see the familiar yellow highlighting of found searchwords on projected images: indicative of how the original primary material was obtained. But this historical approach generally usually remains unspoken, without a critical methodological reflection. As I hope I’ll show here, using Pages of the Past uncritically for historical research is akin to using a volume of the Canadian Historical Review with 10% or so of the pages ripped out. Historians, journalists, policy researchers, genealogists, and amateur researchers need to at least have a basic understanding of what goes on behind the black box.

An example of a "results list" from the Globe and Mail's newspaper database. It all seems so orderly and systematic.

And the ensuing results, a newspaper article focused on the Artistic Woodwork Strike of 1973

An amazing array of information at your fingertips (but…)

In Canada, when one thinks of online digitized newspapers, the Toronto Star’s Pages of the Past and the Globe and Mail’s “Canada’s Heritage from 1844″ often come to mind. There are other wonderful collections, of course, notably the incredible historical newspapers of British Columbia collection, but the Star and the Globe are most commonly used.

The Star and Globe can be accessed through an institutional or personal subscription (you can also access these two databases through libraries like the Toronto Public Library – with a valid library card). You can search by a specific word, or a specific phrase, and narrow it down by a date range. A keyword search (such as for “Artistic Woodwork” at right) and a date range can quickly take you to a seemingly systematic, quantified, and perhaps even complete listing of relevant articles. History laid before you, neatly ordered, from the comfort of your home, library, or office. Another click, and you’re brought to a PDF version of the scanned document: complete with placement, accompanying advertisements, etc.

An example of a feature, front-page, above-the-fold article on the Artistic Woodwork strike that does not appear in a keyword search.

But we need to use these databases with greater caution. In the example at right, for example, the Globe and Mail‘s database has correctly found a large feature article on the Artistic Woodwork strike of 1973. Yet it is a continuance of an article from Page One. That headline, the first page of the newspaper, does not appear in the search list. If one just uses the search engine, you miss this vivid headline, picture, and entire story.

Why?

Primarily, the issue lies in faulty optical character recognition (OCR). This issue is not just limited to these newspapers, and is an inherent flaw in large projects. Tim Hitchcock has described the uncritical use of digitized sources as “roulette dressed up as scholarship,” as historians are “not even bothering to apply the kind of critical approach that historians built their professional authority upon.”

What about the specific case of the Toronto Star and the Globe and Mail online? These databases were assembled at the turn of the present century, and indeed, the Toronto Star is heralded on Paper of Record’s (the company responsible for the database creation) as the “first newspaper in the world to have its entire history … digitized.” It was created quickly, as Bruce Gillespie reported in 2003 in his “All the News That’s Fit to Scan”:

 Using technology developed in-house, Cold North Wind [Paper of Record's parent company] converts documents stored on rolled microfilm into digital computer files. It is an automated process that works quickly-Mr. Huggins says two million pages from The Toronto Star’s 110-year history were archived in less than four months.

This incredible speed and the use of microfilm originals comes at a cost, however. The former means that basic OCR is used: hyphenations are not covered (problematic in smaller columns, where Woodwork might be hyphenated as Wood-work across two lines), if microfilm streaks obscure a letter, if it was slightly tilted, or if the OCR just plain misses a character. This is currently unavoidable with large-scale digitization projects: I am currently OCRing a large collection of word processed documents from 1997 onwards – about as perfect a sample as you can get, and while the OCR under these ideal circumstances is well above 99%, it can never be perfect. Quite frankly, without human proof-reading and additional layers, you can never be completely convinced of your accuracy. Furthermore, comprehensive database use requires some limited understanding of Natural Language Processing (NLP). NLP is a complicated field of research, and a proper search query would also need to be formulated to pick up alternates such as ‘Woodworking,’ etc. without unnecessarily duplication of results.

Another issue lies in the proprietary nature of the Star and Globe databases: I have been trying to track down their technical support team to discuss a research project, to no avail. E-mails often bounce back from the addresses provided on their search portals, and they can be a bit impenetrable. This is understandable, in a way: unlike other national newspaper projects, they are run by private companies.

So what can we do?

Now, with a strike (as in my example above), one could pop the date ranges in, go through each newspaper throughout the period, and explore specific events. This would avoid the above problem. But studies that purport to trace social or cultural trends over a long period of time can fall into the habit of relying on these databases without critical reflection. That’s not to say that they should not use them – we can find most articles, especially by the postwar period and its attending better image quality. Indexes are hardly perfect alternatives. History has always had an element of serendipity.

Indeed, we cannot and should not abandon our use of digitized online databases. Despite their faults, they allow us to cover large swaths of time and space on a realistic timeline, and are much quicker than using microfilm. They also open up new frontiers of large-scale data and textual processing, although the current user interface and databases are not terribly amenable to this form of work.

But we do need to be cognizant. Dissertations and articles that extensively rely on these databases need to be up-front about the issue and at least mention how they have dealt with or recognized the very real and concrete limitations inherent in this form. In my on-going survey of English-language dissertations and other historical work, I have found that while these databases appear to be having some impact on citation counts, few scholars note their database use. Doctoral supervisors, journal editors, bloggers, public historians, etc. need to realize how these databases are potentially shaping professional and amateur historical inquiry in Canada.

So next time you’re using the databases, think about what’s going on. Are you getting everything? Are you missing something? Should you do some digging around a hotspot of hits on a given date? In all cases, we should be more up-front about the tools we’re using and how they might be shaping our research.

By Ian Milligan

Today’s Air Canada wildcat strikes, which led to widespread delays and cancellations at Toronto’s Pearson and Montreal’s Dorval airports, surprised many Canadians. That it could all begin with a seemingly minor issue – the suspension of a number of workers who sarcastically applauded Labour Minister Lisa Raitt as she debarked from a flight last night – is, however, familiar when compared to the “wildcat wave” that was in full swing throughout the summer of 1966.

Indeed, the events of the last 36 hours are reminiscent of several large events that swept the Canadian industrial scene throughout that hot summer of labour unrest. In this post, I’ll take us back to that wild summer of unrest, and help show that the Air Canada wildcat strike is hardly a unique phenomenon.

The “wildcat wave” of 1965-66 was without any statistical precedent. The comparatively quiescent labour movement suddenly exploded with roughly 575 strikes over those two years, with somewhere between 20 and 50 percent of them being illegal wildcat strikes (the statistics are notoriously opaque). I can’t do justice to it all in a short blog post, but I can give two stories that show how two very large case studies started.

SUDBURY – 14 July 1966

The spark was lit 2,200 feet below the surface in the Levack Mine (map), a remote and undesirable outpost of the International Nickel Company near Sudbury, Ontario, when a group of low-seniority young men opened their lunch pails in July 1966. Due to rising tensions between their union, Local 6500 of the United Steelworkers, and their employer over a recently expired collective agreement – languishing in Toronto-based conciliation hearings – they were forbidden to gather on the job. Their foreman stopped them from gathering, told them to close their lunch pails and carry on to work without eating; as the few “old hands” explained to the newer workers, this was a severe provocation. Miners worked “collar to collar,” or surface-to-surface, and work assignments were always to be given underground while they took an initial break. This was violating a twenty-year old tradition.

The miners refused and were sent to the surface. Word spread, the entire shift throughout the mine began returning to the surface, gathering at the cages that would return them to the surface: taunting foremen, banging lunch pails, singing songs and chants. By the next morning, word had spread, picket lines thrown up, and the union had lost control. It would take almost a month of significant union and police efforts to bring these young workers under control, and lead to enduring changes in union discipline within one of Canada’s most important trade unions.

Why did they strike? It was about sandwiches, but that is not all – that was the spark. There had been tense contract negotiations throughout 1966, a union that was seen as perhaps being weak (the United Steelworkers of America had only won the right to represent workers instead of the communist Mine Mill union in 1962 and had faced a re-vote in 1965), work-to-rule was ongoing, and graffiti throughout the facility argued “No contract, no work – - July 10.” Young people were gathered together, several had been brought in from Newfoundland en masse and were living together, growing angrier and angrier at a company that was perceived to be exploiting them, and the situation was ripe on many levels for an explosion. So here, we must understand the wildcat as part of a broader phenomenon: growing anger, structural issues, and ongoing concern.

Thanks to the role of youth, it was a unique event. As one anonymous striker reported to a researcher, “It was like a festival – there were a lot of people around the gates – there were a lot of ‘Newfies’ … around with their guitars and there was drinking and singing and dancing in the streets.” While the strike eventually ended with a mass vote of the workers to return, the contract made Inco workers better paid, although some gains would be lost in their drive to make sure everybody got rehired.

HAMILTON – 3 August 1966

Young workers took a militant lead elsewhere. Indeed, that same month, almost sixteen thousand members of Steelworkers Local 1005 wildcatted at Hamilton’s Steel Company of Canada (Stelco). Young workers had similarly been concentrated in an undesirable low-seniority position, and after being provoked by a foreman, ignored their stewards in walking out.

Stelco was dramatically expanding through the 1960s, growing from 7,258 employees in 1960 to 11,762 on the eve of the wildcat strike in July 1966. Young workers were thus getting hired en masse, and then put into the crappiest, lowest seniority positions en masse – a pressure cooker of tension and resentment. Tensions were growing throughout 1966, a bargaining year. The usual pressure for a settlement was compounded by fears surrounding technological change and automation. Low-seniority men, who would have the most to lose in automation-related job loss, felt this tension acutely. On the warm summer evening of 3 August 1966, negotiators met with the provincial Conciliation Board, a central instrument of postwar labour legislation designed to cool down tensions. It would not work here.

That night, young men gathered in the lunchroom of the Hot Strip Finishing Department, a unit comprised largely of low-seniority men. They were heatedly discussing the negotiating team’s progress, decrying their lack of information. As tensions rose, one young worker burst in. A foreman had told him that “you guys haven’t got the guts to walk out,” he declared, setting the room off. “Let’s show the f——!,” the young men declared, as they began shutting down equipment, and gathering others together. A union steward came over to tell them to “go back and settle it the union way” to no avail, as members streamed out of the plant and quickly set up picket lines. The first picket line was about 200 men strong and grew, especially after the overnight shift arrived and honoured it. Through the night, the line grew to about 3,000 members. Union leaders publicly declared the strike illegal, denouncing it as “irresponsible” and “futile,” and ordered members back to work, but were ignored.

This was, needless to say, an enormous industrial action. Some 10,992 workers were now on strike. Workers continued to honour pickets. The local president, John Morgan, came down to the lines himself to make his stance clear: the men had to return to work, the lines had to dissipate, and that penalties would be severe if they did not listen to him. “We’re fed up with you, we don’t want you,” shouted one picketer, and several physically assaulted Morgan. Hamilton Police placed a tearful Morgan into protective custody, before 300 officers (out of a total police force of 420) broke the picket line to allow management into the plant. Afterwards, picketers even managed to stop a freight train from entering Stelco through the force of their numbers. The street scene was remarkable. About five hundred men now milled around in front of Stelco, moving in and out of nearby restaurants and bars. Cars were burned. Thirty-three mostly young men were arrested for variously obstructing or assaulting police, or causing a disturbance. At one point, picketers sat down en masse, obstructing traffic.

It would take another five days for the wildcat to end, thanks to a majority vote of union members (many of whom decried the role of young workers). Police eventually restored order, the action moved off the streets, and a City Clerk-supervised vote at the Civic Stadium (today’s Ivor Wynne) saw 4,319 members favouring a return to work, 1,142 against, and hundreds more leaving without voting. Wives held the line while the Local held its largest union meeting in recent history.

CONCLUSIONS

Wildcats aren’t new. But in a context of growing pressure on the job, both from management and government (especially acute in the case of Air Canada), small provocations and sparks can unsurprisingly lead to massive eruptions. While today’s Air Canada’s strike was not on the scale of the Hamilton or Sudbury explosions, it does remind us of an earlier period where such labour unrest was relatively common.

An inspiring historical visualization of Napoleon's 1812 campaign (please click to see it).

I have recently been trying to figure out good ways of representing large amounts of historical information in a way that makes sense to everybody who might stumble across my work! I think that a good graphic has the ability to draw readers into what we do, letting us convey the scope, joy, or horror of history without needing to read through often dense prose. In this post, I want to give a sense of what I think works, what doesn’t, and why we should start thinking about cool maps, graphs, and charts!

What is arguably the finest historical visualization ever is at right. Charles Joseph Minard, a French engineer, produced this 1869 chart visualizing Napoleon’s 1812 Russian campaign. In this image, which I encourage you to click on to zoom in, you see how BIG the army is going into Russia, the path it took, where it had to begin retreating, just how few soldiers returned from the campaign, and the plummeting and wretched winter conditions along the path. Take a second, look at this chart. From this, you learn a great deal, and the human scale of suffering and tragedy can be captured in a way that a paragraph of text might not (at least to us visual learners).

Why should we pay so much attention to how we make our graphics? Firstly, I think people learn in a variety of ways. Some can sit through a three-hour lecture with rapt attention, while others snooze. Some can read a 400-page book, devouring every nook and cranny, while others get bogged down. Still more draw inspiration from graphics, visual ways to see the past. But, perhaps we should leave it to Mark Howard Moss and his book Toward the Visualization of History: The Past as Image, who argues that “individuals access images more readily than words. Seeing is a central feature of modern soceity and as Mitchell Stevens cogently puts it, ‘Moving Images use our senses more effectively than do black lines of type stacked on white pages.’” Heck, think of chemical elements: chances are, your mind immediately flashes to a visualization, the Periodic Table of the Elements.

From Canada: A National History, a top notch-history textbook! Thumbnail used for the purposes of critical analysis.

Visualizations, as Minard’s map shows, have tremendous power (another famous one that had power at its time is of course John Snow’s map of contaminated wells and cholera outbreaks). For more, see “12 Great Visualizations that Made History.” But we can also do them very poorly. Flipping open a random Canadian history textbook, I am confronted with the visualization at left. Do we get the sense of the turmoil of the late Second World War? The flow of modern warfare, or the human suffering? One can read the prose, certainly, but the picture contributes little.

To be fair, I’ve made my share of very poor visualizations (you can click here to see the worst one that I suspect a Canadian historian has ever made, as compared to a much clearer one here that really shows what I want it to), but I think that historians – if we want to be active, engage with the public, need to begin thinking about how to visualize information.

How?

• We can take it seriously! This goes for professionals and amateurs. We can read and absorb it all. Maybe that’s reading Edward Tufte’s beautifully bound and comprehensive books, thinking about how to produce really cool and engaging visualizations. Or maybe it is simply seeing what else works, as in the beautiful and inspiring “50 Great Examples of Data Visualization.”
• Popularize fun, easy ways to visualize information. Anybody with a computer can use ‘Wordle.’ They’re not perfect visualizations, BUT they are a lot of fun. When I’ve used them in presentations, people are really engaged. During my doctoral defence, handouts with a word cloud gave people a quick sense to see some of the themes, and encouraged deeper questions. As instructors, amateur historians, etc., why not start out simple with these ways forward?
• Teach it. If a great visualization is around, use it in a lecture or seminar. Of course, the problem is, there aren’t too many visualizations out there that we can immediately grab!
• Practice it. Let’s start creating this massive database of visualizations so our teachers and researchers can draw on them. Much of this will probably come from programmers, tinkerers, and amateurs – let’s embrace it. There are tons of online resources, such as “How to create a visualization.”

But most importantly, just think about it. The power of images, the ability to draw people in, to make connections that you might otherwise miss.

What could we use to make these, in terms of off-the-shelf solutions? I’ll provide a few, and if you can think of any please let us know in the comments!

• Wordle: Creates word clouds that can quickly give you a sense of word frequency in a document. While they lack contextualization (you can’t see if people love Elvis at a glance or hate him, just that they’re talking about him, for example), they’re beautiful and very easy to use. I’ve discussed these elsewhere here at ActiveHistory.ca.
• Timelines: If you have technical skills, there are free open-source options like SIMILE Timeline. That said, it does require a basic understanding of HTML and Javascript (there is a very good ‘getting started tutorial‘). An easier alternative, at minimal cost, is BEEDOCS Easy Timeline, which presents interactive timelines in 3D!
• Maps: From a simple annotated Google Map to more advanced solutions, digital maps can provide an interactive sense of space and time. Tom Peace, who uses maps extensively, draws on “Magic Maps” (providing free Canadian topographic maps) and DIVA-GIS.

In the comments below, please feel free to discuss other options, their various pros and cons, and other ideas to make history come alive…

The Programming Historian

Depending on your vantage point, we have a looming opportunity – or a looming problem. Historical digital sources have reached a scale where they defy conventional analysis and now call out for computational analysis. The Internet Archive alone has 2.9 million texts, there are 2.6 million pages of historical newspapers archived at the Chronicling America site of the US Library of Congress, the McCord Museum at McGill University has over 80,000 historical photographs, and Google Books has now digitized fifteen million books out of their total goal of 130 million. Archives are increasingly committed to preserving cultural heritage materials in digital, rather than more traditional analog, forms. This is perhaps best exemplified in Canada by digitization priorities at Library and Archives Canada. The amount of accessible digital information continues to grow daily, making digital humanities projects increasingly feasible, and for that matter, necessary.

In this post, I will do two things. Firstly, I will give a sense of how much information is out there, and make the case for why Canadian historians need to start thinking about it. Secondly, I will introduce readers to the Programming Historian, a wonderful resources that at least puts you on the right track to a programming frame of mind.

TMI?

Too much information? (Photo of FEMA Publications Warehouse, WikiMedia Commons - http://bit.ly/zjmlYc

Information overload is not new. People have long worried about the impact of too much information. In the 16th century, the German priest Martin Luther decried that the “multitude of books [were] a great evil,” in the 19th century Edgar Allan Poe bemoaned that “[t]he enormous multiplication of books in every branch of knowledge is one of the greatest evils of this age,” and as recently as 1970, American historian Lewis Mumford lamented that “the overproduction of books will bring about a state of intellectual enervation and depletion hardly to be distinguished from massive ignorance.” The rise of born-digital sources must thus be seen in this continuous context of hand wringing around the expansion and rise of information.

Despite the frustrations of microfilm for today’s historians, as well as the pitfalls of separating the wheat from the chaff amongst rising numbers of modern sources, historians have undoubtedly benefitted from these technical developments. This is perhaps disproportionately for those engaged in social and cultural pursuits. Historians will profit meaningfully from born-digital sources. These, however, do present added – albeit surmountable – challenges due to their scope and production processes. Sources do not always have attributable or reliable authorship, are often undated, but in aggregate can give a sense of the zeitgeist of a time.

Library of Congress (Photo from WikiMedia Commons - http://bit.ly/ArU8YZ)

Storage price is falling. For example, James Gleick [in his book, The Information: A History, a Theory, a Flood] estimates that the Library of Congress collection is around 10TB (although the LOC itself claims around 200TB). These would previously have been unimaginable figures; I can now pick up 10TB of data storage for under a thousand dollars. Born-digital collections are larger, of course: the LOC’s digital collection is 254TB, larger than their print holdings, and the Internet Archive now has 3 Petabytes (PB) of information, growing at 12TB/month! In Canada, LAC has about 4TB of federal government web information and 7TB in its own internet archive. Information is also being preserved through programs such as the Roy Rosenzweig Center for History and New Media’s September 11th Digital Archive, the Hurricane Digital Memory Bank (focusing on Hurricanes Katrina and Rita), and, as of writing, the #Occupy archive. Online content is curated and preserved en masse: photographs, news reports, blog posts, and now tweets. These complement more traditional efforts at collecting and preserving oral histories and personal recollections, which are then geo-tagged, transcribed, and placed online.

What can we do about this conventional and especially born-digital deluge? There are no simple answers, but historians must begin to conceptualize new additions to their traditional research and pedagogical toolkits.

Where to Start: Programming

By the end of the Programming Historian, you'll have a basic know-how of Python and will be able to tackle projects requiring textual analysis.

One important thing we can do with this deluge of information is learn how to interact with digital information on a mass scale. Luckily, we have a tremendous resource available to us: The Programming Historian, by William Turkel and Alan MacEachern, hosted on the Network in Canadian History & Environment (NiCHE) site. Why might you want to open up this free, open-access website book?

• If you were to try to deal with born-digital sources in a traditional manner, you would spend A LOT of time flicking through websites. Much of it hasn’t been curated, and realistically, you could not read every blog comment published on a given day in Canada, navigate the tweets, or so forth. For this, you will need computational analysis.
• The same holds true for the conventional array of information discussed above: if you want to use 2.6 million newspaper pages to their full potential, there must be a way to “distant read” it.
• Digital history is ‘hot.’ The American Historical Association, meeting right now, is full of panels and twitter has been afire with the field. Even if you do not necessarily see yourself using programming languages, it behooves you to be able to understand it.
• And, most importantly, it isn’t that hard, and it doesn’t take that much time. You could move through the whole guide in a weekend, or – better yet – break it into small chunks, spending 20-30 minutes here and there.
• Finally, I believe we’ll also have to equip the next generation of historians, as I’ve written about elsewhere on ActiveHistory.ca.

The Programming Historian is very straight forward, but by the end of it, you’ll be able to do the following:

• In an automated, systematic fashion, you will be able to take a website and extract all of the words from it for further analysis.
• Establish word frequency, similar to what a Wordle word cloud displays (the possibile utility of this is discussed elsewhere on this site). Indeed, you will be able to make your very own tag clouds!
• Move beyond word frequency to see the keyword-in-context – i.e. you see that the word ‘aboriginal’ appears a hundred times in a given site, so why not see where it has appeared. This enables you to move very quickly to the relevant information.
• Download and harvest information automatically. Say you find a large collection of a hundred websites. Rather than clicking repeatedly through each to download the information, a simple script can do it for you!

Conclusion (and a proviso about why we don’t all have to be programmers!)

Don’t be afraid. It’s New Years, so why not make it your resolution as a historian to figure out some of these very basic steps. It could make you a better historian, or in any case, will equip you to figure out what’s going on. In any case, it’s an additional tool in one’s toolkit. Unlike earlier social science histories of counting with computers in the 1970s (which did revolutionize areas of historical inquiry), it is important to remember that we can use broad analysis to find issues, but then move dynamically down into context.

That all said, historians will not all have to become programmers. Just as not all historians need a firm grasp of Geographical Information Systems (GIS), or a developed understanding of the methodological implications of community-based oral history, or in-depth engagement with cutting edge demographic models, not all historians have to approach their trade from a computational perspective. Nor should they. Computational history – to use only a few examples – does not replace close reading, traditional archival inquiry, or going into communities to uncover notions of collective memory or trauma. Indeed, computational historians will play a facilitative role and provide a broader reading context; yet there will still be historians, collecting relevant primary and secondary sources, analyzing and contextualizing them, situating them in convincing narratives or explanatory frameworks, and disseminating their findings to wider audiences.

Surveying all of cosmic history using ChronoZoom: you can't even see human history up there in the upper right corner.

Historians aren’t always the best at crossing the hall to the sociologists across the way, let alone the astronomers, physicians, or geologists across campus. Scientists who study the Big Bang, however, are engaged in history – just a (very) different kind. Similarly, those who study the very long-term geographical forces that have shaped Earth, those who study evolutionary processes across flora and fauna, even those who study broader, galactic or universal phenomena, are often seen as very distinct from historians.

Big History, a new and emerging field, seeks to bridge these very real but also occasionally artificial disciplinary boundaries. It can be hard, however, to really establish how we can go forward and what a Big History approach might look like in real, deliverable terms (Bill Gates and David Christian have a great project also looking at how to teach these concepts to classrooms). Look no further: ChronoZoom, from the University of California-Berkeley’s Department of Earth and Planetary Science, has a working model that gives us a sense of what this might look like.

What is ChronoZoom? Just as historians cannot reasonably access millions of books without heavy computational aids, humans have difficulty even conceiving of the scope of ‘big’ human history that covers billions of years. ChronoZoom, “a tool to aid the comprehension of time relationships between events, trends, and themes,” aids in this. The most developed big history project yet in existence, ChronoZoom ties together extant online resources (scholarly articles, photographs, audio-visual media, etc.) by placing them along a constant timeline, stretching along a 13.7 billion year continuum from the Big Bang until the present day. While this would be initially overwhelming, its use of Microsoft’s Seadragon Deep Zoom technology – a smooth means of transitioning around an extremely large and high-resolution file without overwhelming the computer (similar to how one navigates Google Maps) – will allow a user to swap between an overview of natural history, a snapshot of human history, to a more focused overview of major events in the 19th century. Historians use time as a primary frame of analysis, and ChronoZoom represents the first major search engine project to recognize it as a constant base. Imagine using this in a classroom, to give students a sense of how long ago events truly were. Or, just sitting at home, using it almost as a “total perspective vertex” as Douglas Adams humorously foresaw in his Hitchhiker’s Guide to the Galaxy.

ChronoZoom after literally zooming into all of human history (3500BC onwards)

ChronoZoom isn’t fully fleshed out yet, but a very good working demo let’s us see some of the potential. Please fire it up yourself here (you’ll need to download Microsoft Silverlight for it to fully work). At first, it looks like a static poster (pictured above): sophisticated, to be sure, but nothing too exceptional. In the lower left hand of the poster we have the Big Bang, and, in the upper right, we have human history. However, you can’t see any human history at this point – it’s too small. But if you click on the ‘human history’ bar at the top, we begin to rapidly and fluidly zoom into the upper right hand of the corner. Cosmic history recedes into the foreground, Earth & Life history as well, pre-history appears and disappears.. and then we see human history. This process really needs to be played out for yourself: watching billions of years fly by, receding, our ‘human history’ a mere dot in this billions of years.

December 2000: on the very same chart that contained all of Cosmic History!

And from here, we can zoom in even more: to the 2nd Millenium A.D. for example, to the 20th Century alone, to its 10th Decade, to the year 2000, until finally, in the very upper-right corner of the map, we see New Year’s Eve of the year 2000 (and the beginning of our 3rd Millenium). This is a fantastic way to describe, deliver, and allow people to dynamically manipulate an otherwise inconceivable amount of historical information covering an absurd amount of time.

It seems weird to be describing, in text, such a dynamic website. What are you waiting for? Get out there and check it out! What do you think about it? I’d love to hear in the comments below.

If Big History fascinates you, please watch this TED video by David Christian on Big History. I guarantee you’ll find it entertaining and provocative.

The network of links stemming from ianmilligan.ca (activehistory.ca alone was too big!). This gives you a visual sense of the power behind hyperlinked information!

We will need to make dramatic changes to history undergraduate curriculums by aggressively implementing digital literacy programmes. This will benefit both our students and the historical profession.

Why? Let’s imagine how a future historian will tackle the question of what everyday life was in September 2011 – today. She will have a tremendous array of sources at her fingertips: the standard newspaper and media reports and oral interviews that we use today, but also a ton of added sources that would help give a sense of the flavour of daily life. Two hundred million tweets are sent every day. Hundreds of thousands of blog posts. Incredible arrays of commentary, YouTube videos, online comments, viewership and readership numbers will all hopefully be available to this historian.

But how will she read it all? Realistically, nobody is ever going to be able to get through all the tweets for even just one day: let alone categorize, analyze, and meaningfully interact with it. She’ll need to use digital tools. We are at a crossroads. This sort of history won’t be the be all and end all of future historical research, but I believe that somebody is going to do this sort of social history. Let’s make sure our future students are ready for it!

Using Mathematica, I have been able to track the rise and fall of the terms "war" (red) and "peace" (blue) across a comprehensive Top-40 Lyrics Database.

We need to begin thinking about how we are going to train historians of the future, today. Somebody is going to do this work. They are probably sitting in high school or elementary school classrooms today. When they show up at the university, let’s make sure that we’re ready to train them to write the history of today.. tomorrow. This is not simply for historians who fashion themselves as social scientists, as opposed to those who see themselves as pure humanists. It’s about deploying a tool which can provide information through which we can drape our stories, our interviews, our human anecdotes, etc.

Historians need to begin thinking about digital literacy and writing programs that will help access these sources. What’s going on right now? Tons. I have previously discussed one of the biggest current projects, the Culturomics project and their accessible Google n-gram viewer. You can see the rise and fall of a word or phrase (an ngram) and see how much it has been used across several centuries. It’s an incredible project, albeit not without some caution needed in how it is approached. There are also several digital history projects ongoing, some of which has garnered considerable attention (such as the Criminal Intent project in the New York Times).

This is just a hint of what’s to come. We need to be able to populate these future projects with even more historians. Which means thinking about how to train them today. Training these people by graduate school is simply too late, however. We need to begin training undergraduates in their first year. Indeed, as a recent study carried out at the University of Rochester indicates, students won’t adopt new technologies by the time they get to graduate work – the risks are too high. Let’s get them as undergrads.

What could a digital literacy programme look like for the next generation of historians, so that they’re ready to begin thinking and tackling these issues?

• We’ll need a firm grasp of the historiographic context of this shift – i.e. the old school quantitative historians, who crunched the census of Hamilton for example, or poured considerable time and effort into understanding demographic shift.
• Basic digital tools: What is cloud computing? How can we secure and backup our data?
• How can we digitally organize conventional sources? I’ve discussed this before in my post on Zotero.
• Grasping the SHEER SCOPE of large digital depositories. It’s one thing to say that Google Books has fifteen million books. It’s another to really grasp this. And to further realize what a drop in the bucket that is compared to other repositories of automated data being collected every day.
• Basic programming? The ‘Programming Historian‘ is a great start. What most of us will have to do won’t be so complicated and we need to be able to do it ourselves. While well-funded projects may be able to raise the funds to recruit teams of programmers to join them, or others may form collaborative and interdisciplinary work-teams, many historians will not be able to do so. They should be self-sufficient in this regard, at least for more simple and routine data mining exercises.

Our students should be able to come out of undergraduate history programs and be truly equipped for our knowledge economy and for the future demands of the profession. This will help teaching, research, and labour market outcomes. Information is increasingly being generated by the internet, written on the internet, and being consumed by internet users. People need to be able to create it, interact with it, in a fluent, comfortable manner.

What do you think? Should historians make this shift? Or are there disadvantages that I’m overlooking in my enthusiasm for this field of research? I’d love to hear from you all, especially as I begin my next project (a digital history of postwar English-Canadian youth).

What do you want to “calculate or know about,” asks Wolfram Alpha. Voted the best computer innovation of 2009 in Popular Science‘s “Best of What’s New,” Wolfram Alpha lets users interact with over 10 trillion pieces of information curated by a large research team. You just type in what you want to know, the engine tries to figure out what you’re asking it, and you’re presented with a remarkable array of information (as well as ways to refine your subsequent searches). This has tremendous historical applications, both for teaching and for historical research. I’ll show off some of these possibilities in this post, and hope that you take a moment to try it out yourself. If you find anything of particular interest, please let us know in the comments below.

Grow, Canada, Grow!

Wolfram Alpha likes to help you out. By simple typing ‘history’ into the answer engine, you’ll get a page demonstrating all the cool things you can do for historical topics. This is a reliable source of simple information for anybody wanting to quickly access basic facts. As noted by Léon Robichaud in the comments, however, however, the data on Canadian Prime Ministers is faulty – Meighen was the 9th Prime Minister but had the 3rd shortest term; similarly, 1st Prime Minister returns Harper! For example, if you type ’3rd Prime Minister of Canada’ into the engine, you’ll see it parsed as ‘Canada’ ‘Prime Minister’ ’3rd.’ And then you would [erroneously] learn that the third Prime Minister of Canada was Arthur Meighen, he governed for 2 months and 27 days, he was born in Toronto 137 years ago and died 51 years ago at the age of 86.

What’s going on with the Prime Minister section? Turns out that Wolfram is parsing the information in a peculiar way! It’s interpreting the list of prime ministers by term-length. You can see this by pulling up the list (press ‘more‘ on sequence to see Alpha’s take on it). There you have the correct sequence, but the data parses it by focusing on the LENGTH field as opposed to the sequence you have. This appears to be a unique manifestation, as opposed to the American list of presidents.

This is perhaps a more valuable lesson than the original post entailed! In any case, I’ll report this to Wolfram Alpha as it’s clearly not returning the data that we want. That way they can refine their results better. There is a ‘feedback’ button on the bottom of every page.

Say that you’ve always wondered what your ‘third cousin’ really is: a simple search demonstrates a genealogical chart! Population information abounds in the database: you can learn the population growth percentages of Canada between 1945 and 1960, or the population circa 1911 (7.21 million people, the 33rd largest country then in the World, and probably weighing approximately 504 462 metric tons). You can also access basic biographic (Harriet Tubman) or comparative biographic information (Stephen Harper and Pierre Trudeau), information on historical events (D-Day), or even defunct empires (Austro-Hungary).

The showdown everybody has been waiting for: the name Ian vs. the name Edith!

For contemporary information, you can do a search such as ‘Income Inequality of Canada vs United States‘ and quickly learn out Gini indexes, income share held by the various fifths of the population. How frequent does your own name appear? I learn that ‘Ian’ is the 72nd most popular name in the United States, and it has been steadily increasingly in popularity since 1960 or so. It’s a young name. Edith, on the other hand, is pretty unpopular these days – although it was pretty popular back in the day. Indeed, about 2% of people in their late 80s are named ‘Edith’ (as opposed to less than half a percentage under 10 these days). You can even do showdowns: “Ian vs. Edith” to see the relative ranges [my apologies to any of our readers named Edith - quite frankly, we both pale in comparison to the most popular US given names!]. Or simpler, everyday things: exchange rates, both historically and today (right now $2500 Canadian gets you $2538.38 US).

You can have a lot of fun with this information. For each result, you can see what sources were used to generate the information. If anything, it’s a snapshot into the future: refined, intelligent knowledge acquisition.

What do you think? Toy or tool? Would you encourage your undergraduates to check it out and play with it? How trustworthy is the information? Are you just thrilled to have a simple way to figure out how you’re related to your grandmother’s aunt?

[Text in red was modified and added from the original post]

Google Books even has the full text of LIFE magazine!

Organizations, activists, and laypeople are trying to put the sum of all printed knowledge on the internet. They’re facing copyright issues, ethical and moral debates, but it’s marching on nonetheless. Why should we have to travel to archival repositories, especially if they’re in an already convenient form like microfilm? Shouldn’t everybody have access to information, not just the select few who happen to have institutional affiliations? When it comes to access to information, we should be on an even playing field. Lay people interested in history, undergraduates, cash-strapped professional researchers, and all can benefit from several internet resources that put an incredible amount of information at your finger tips.

In this post, I’ll introduce people to the Internet Archive, the Haithi Trust, and Google Books. I hope to show you that there are incredible numbers of primary sources, digitized books, internet snapshots, among other things, out there. From an 1888 report on the Knights of Labor by a Canadian Legislative Committee, to the music video for the “first rap single ever released in Canada,” to American prohibition speeches, they’re all out there – free, accessible, and often downloadable.

The Internet Archive Webpage

What’s the Internet Archive? In a word, it’s incredible. It has collections of videos (530k+), live music (over 94k concerts), audio (over 914k recordings), and texts (2.8m+). It also has 150 BILLION old internet pages. For Canadians, it has a large collection of digitized microfilms from Library and Archives Canada (the above Knights of Labor report, for example), as well as radio clips, community videos, concerts, and others. An activist project, everything on the Internet Archive is free of charge. It continues to grow. If you use firefox, you can also put an Internet Archive extension into your browser.

The WayBackMachine – part of the Internet Archive – is an invaluable resource as well, both for researching and nostalgia. What did cbc.ca look like in the past? You can go all the way back to 1996 on a variety of dates to see what news issues appeared on the front page, how the layout of the page has changed, and maybe even do a systematic study of how it appeared. It’s the newspaper archive of the internet. It’s also nostalgic… look at this York University webpage from 1997!

François-Xavier Garneau's History of Canada - a free book, fully downloadable on Google Books!

Google Books, which is increasingly popping up in all of our web searches, has over fifteen million books scanned – and has the jaw-dropping goal of digitizing every single unique book existing in the world by 2020. They’ve run up against publishers and institutions. For copyright reasons, many books can only provide snippets – a percentage to give you a sense of the book, maybe a quotation or two, but you still have to buy access to it. Books in the public domain (approx. 2 million) are fully accessible and you can even download it as a very high quality PDF! Outside of the United States – like in Canada – many books have murky copyright status to Google so we need to go to the Internet Archive where public domain works are. There have been concerns raised, however.

To do a full text search of Google Books, make sure you select the ‘Full View’ tab on the left column of your search. For example, ‘Canada History’ will bring up 464,000 full text works! You can download these, and read it on your computer or even your e-reader! For example, say you want to read François-Xavier Garneau’s History of Canada (1866). Click here (PDF link), download it, and you’re good to go. And that’s just one small snippet.

Finally, Haithi Trust is a partnership of several major university libraries to digitize material (some by themselves, others from Google Books who copied their books – they demanded them back, apparently). They have almost nine million digitized volumes, of which 2.4 million are in the public domain. For Canadians, they have a decent collection. Search using the ‘only full view’ option and you can see a vast array of primary and secondary sources.

I see these resources as goods, although we need to continue to consider the intellectual rights of creators (especially those who don’t have state support for universities) as well as the importance to understand that it’s not all on there… yet.