‘A map and some pins’: open data and unlimited horizons

8681035823_72f8c57ae0_m

This is the text of my keynote address to the Digisam conference on Open Heritage Data in the Nordic Region held in Malmö on 25 April 2013. You can also view the video and slides of my talk, or experience the full interactive experience by playing around with my text/presentation in all its LOD-powered glory. (Use a decent browser.)


The Australian poet and writer Edwin James Brady and his family lived for many years in the isolation of far eastern Victoria — in a little town called Mallacoota.

Edwin James Brady

Edwin James Brady (NLA: nla.pic-vn3704359)

Here, from time to time, Brady amused himself by taking a map of Australia down from the wall and sticking pins in it. The pins, Brady explained in 1938, included labels such as ‘Hydro-electric supply base’, ‘Irrigation area, and ‘Area for tropical settlement’. The map and its pins were one expression of Brady’s life-long obsession with Australia’s potential for development — for progress.

Maps and pins are probably more familiar now than they were in Brady’s time. We use them routinely for sharing our location, for plotting our travel, for finding the nearest restaurant. Maps and pins are one way that we document, understand and express our relationship to space.

Brady, however, was interested in using his pins to highlight possibilities. In the late nineteenth and early twentieth centuries size mattered. With the nations of Europe jostling for land and colonial possessions, space become an index of power. When the Australian colonies came together in 1901 to form a nation, maps and spatial calculations abounded. Australia was big and so it’s future was filled with promise.

Australia was big with promise.

Australia was big with promise.

In his travels around Australia, EJ Brady started to catalogue ways in which its vast, ‘empty’ spaces might be turned to productive use. A hardy yeomanry armed with the latest science could transform these so-called ‘wastes’, and Brady was determined to bring these opportunities to the attention of the world.

This evangelical crusade reached its high point in 1918, with the publication of his huge compendium, Australia Unlimited — 1139 pages of ‘Romance History Facts & Figures’.

National Archives of Australia:  A659, 1943/1/3907, page 135

National Archives of Australia:
A659, 1943/1/3907, page 135

Space may no longer be invested with the same sense of swelling power, but our maps and pins still figure in calculations of progress. Now it is the data itself that awaits exploitation. Our carefully plotted movements, preferences and passions may well have value to governments, planners or advertisers. Data, according to some financial pundits, ‘is the new oil’.

Whereas Brady traveled the land documenting its untapped riches, we can take his work and life and mine it for data — looking for new patterns and possibilities for analysis.

Brady wasn’t the first to use the phrase ‘Australia Unlimited’, though he did much to make it familiar. By exploring the huge collection of digitised newspapers available through the National Library of Australia’s discovery service, Trove, we can track the phrase through time.

australia_unlimited

Brady was a skilled self-publicist and his research trips in 1912 were eagerly reported by the local press such as the Barrier Miner in Broken Hill and the Cairns Post.

In 1918 the book was published, receiving a generally positive reception — as an advertising leaflet showed, even King George V thought the book was ‘of special interest’. In 1926, a copy of the book was presented to a visiting delegation from Japan.

NAA: A659, 1943/1/3907, page 55

NAA: A659, 1943/1/3907, page 55

Over the years Brady sought to build on his modest successes, planning a variety of new editions and even a movie. But while his hopes were thwarted, the phrase itself lived on.

In 1938 and 1952, there are references to a radio program called ‘Australia Unlimited’, apparently featuring young musical artists. Also in 1952 came the news that the author of Australia Unlimited, EJ Brady, had died in Mallacoota at the age of 83.

Sydney Morning Herald, 23 July 1952

Sydney Morning Herald, 23 July 1952

Unfortunately copyright restrictions bring this analysis to a rather unnatural end in 1954. If we were able to follow it through until the present, we could see that from 1957 the phrase was used by a leading daily newspaper as the title of an annual advertising supplement detailing Australia’s possibilities for development. In 1958, it was adopted as a campaign slogan by Australia’s main Conservative party. In 1999 it was resurrected again by a major media company for a political and business symposium. Even now it provides the focus for a ‘branding’ initiative supported by the federal government.

Graphs like this are pretty easy to interpret, but of course we should always ask how they were made. In this case I simply harvested the number of matching search results from the Trove newspaper database for each year. The tool I built to do this has been through several different versions and is now a freely-accessible web application called QueryPic. Anyone can quickly and easily create this sort of analysis just by entering a few keywords.

QueryPic is one product of my experiments with the newspaper database over the last couple of years. I’ve also looked at changes to content of front pages, I’ve created a combination discovery interface and fridge poetry generator, and I’ve even built a simple game called Headline Roulette — which I’m told is strangely addictive.

All of these tools and experiments take advantage of Trove’s API. I think it’s important to note that the delivery of cultural heritage resources in a machine-readable form, whether through a custom API or as Linked Open Data, provides more than just improved access or possibilities for aggregation. It opens those resources to transformation. It empowers us to move beyond ‘discovery’ as a mode of interaction to analyse, extract, visualise and play.

Using freely available tools we can extract named entities from a text, we can look for topic clusters across a collection of documents, we can find places and pin them to a map. With a little bit of code I can take the newspaper reports of Brady’s travels in 1912 and map them. With a bit more time I could take another of Brady’s travel books, River Rovers, available in digitised form through the Internet Archive, and plot his journey along Australia’s longest river, the Murray.

Such transformations help us see resources in different ways — we can find new patterns, new problems, new questions. But transformation is a lossy business. I can put a pin in a map to show that Brady stopped off in Mildura on his voyage along the Murray. What is much harder to represent are the emotions that surrounded that visit. While he was there Brady received news of a friend’s death. ‘Bad news makes hateful the most pleasant place of abiding’, he wrote mournfully, ‘I strained to open the gate to go forth again into a wilderness of salt bush and sere sand’. Travel can be a form of escape.

Digital humanist and designer Johanna Drucker has written about the problems of representing the human experience of time and space using existing digital tools. ‘If I am anxious’, she notes, ‘spatial and temporal dimensions are distinctly different than when I am not’. We do not experience our journeys in a straightforward linear fashion — as the accumulation of metres and minutes. We invest each footstep with associations and meanings, with hopes for the future and memories of the past. Drucker calls on humanities scholars to articulate these complexities and work towards the development of new techniques for modelling and visualising our data.

‘If human beings matter, in their individual and collective existence, not as data points in the management of statistical information, but as persons living actual lives, then finding ways to represent them within the digital environment is important.’

In a similar way, Australia Unlimited is not just a catalogue of potentialities, or a passionate plea for national progress. It’s also the story of a struggling poet trying to find some way of supporting his family. Proceeds from the book enabled Brady to buy a plot of land in Mallacoota and build a modest home — I suspect that it wasn’t the same home that was painted by his wife in the 1950s.

nla.pic-an2287718-v

But even this small success was undermined. Distribution of the book was beset with difficulties and disappointments and and despite all his plans financial security remained elusive.

Brady’s youngest daughter, Edna June, was born to his third wife Florence in 1946. What could he leave he leave her? A ‘modern edition’ of Australia Unlimited lay completed but unpublished. ‘If it fails to find a publisher’, he remarked wistfully, ‘the MSS will be a liberal education for her after she has outgrown her father’s nonsense rhymes’. It was, he pondered, ‘a sort of heritage’.

One of the things I love about being a historian is that the more we focus in on the past the more complicated it gets. People don’t always do what we expect them to, and that’s both infuriating and wonderful.

Likewise, while we often have to clean up or ‘normalise’ cultural heritage data in order to do things with it, we should value its intrinsic messiness as a reminder that it is shot through with history. Invested with the complexities of human experience it resists our attempts at reduction, and that too is both infuriating and wonderful.

The glories of messiness challenge the extractive metaphors that often characterise our use of digital data. We’re not merely digging or mining or drilling for oil, because each journey into the data offers new possibilities — our horizons are opened, because our categories refuse to be closed. These are journeys of enrichment, interpretation and creation, not extraction.

We’re putting stuff back, not taking it out.

Cultural institutions have an exciting opportunity to help us work with this messiness. The challenge is not just to pump out data, anyone can do that. The challenge is to enrich the contexts within which we meet this data — to help us embrace nuance and uncertainty; to prevent us from taking the categories we use for granted.

For all it’s exuberant optimism, a current of fear ran through Australia Unlimited. The publisher’s prospectus boldly announced that it was a ‘Book with a Mission’. ‘A mere handful of White People’, perched uncomfortably near Asia’s ‘teeming centres of population’, could not expect to maintain unchallenged ownership of the continent and its potential riches. Australia’s survival as a white nation depended upon ‘Effective Occupation’, secured by a dramatic increase in population and the development of its vast, empty lands — ‘The Hour of Action is Now!’.

National Archives of Australia:  A659, 1943/1/3907, page 208

National Archives of Australia:
A659, 1943/1/3907, page 208

In 1901, one of the first acts of the newly-established nation of Australia was to introduce legislation designed to keep the country ‘white’. Restrictions on immigration, administered through a complex bureaucratic system, formed the basis of what became known as the White Australia Policy.

While the legislation was designed to keep non-white people out, an increase of the white population was seen as essential to strengthen security and legitimise Australia’s claim to the continent. Australia Unlimited was an exercise in national advertising aimed at filling the unsettling emptiness with sturdy, white settlers.

But White Australia always a myth. As well as the indigenous population there were, in 1901, many thousands of people classified as non-white living in Australia. They came from China, India, Indonesia, Turkey and elsewhere. A growing number had been born in Australia. They were building lives, making families and contributing to the community.

Here are some of them…

The real face of White Australia

The real face of White Australia

I built this wall of faces using records held by the National Archives of Australia. If a non-white person resident in Australia wanted to travel overseas they needed to carry special documents. Without them they could be prevented from re-entering the country — from coming home. Many, many thousands of these documents are preserved within the National Archives.

Kate Bagnall, a historian of Chinese-Australia, and I are exploring ways of exposing these records through an online project called Invisible Australians.

To build the wall I downloaded about 12,000 images from the Archives’ website — representing just a fraction of one series relating to the administration of the White Australia Policy. Unfortunately there’s no machine-readable access to this data, so I had to resort to cruder means — reverse-engineering interfaces and screen-scraping.

Once I had the images I ran them through a facial detection script to find and crop out the portraits. What we ended up with was a different way of accessing those records — an interface that brings the people to the front; an interface which is compelling, discomfiting, and often moving.

The wall of faces also raises interesting questions about context. Some people might be concerned by the loss of context when images are presented in this way, although each portrait is linked back to the document it was derived from, and to the Archive’s own collection database. What is more important, I think, are the contexts that are gained.

If you’re viewing digitised files on the National Archives’ own website, you can only do so one page at a time. Each document is separate and isolated. What changes when you can see the whole of the file at once? I’ve built another tool that lets you do just that with any digitised file in the Archives’ collection. You see the whole as well as the parts. You have a sense of the physical and conceptual shape of the file.

National Archives of Australia:  ST84/1, 1908/471-480

National Archives of Australia:
ST84/1, 1908/471-480

In the case of the wall of faces, bringing the images together, from across files, helps us understand the scale of the White Australia Policy and how it impacted on the lives of individuals and communities. These were not isolated cases, these were thousands of ordinary people caught up in the workings of a vast bureaucratic system. The shift of context wrought by these digital manipulations allows us to see, and to feel, something quite different.

And we can go the other way. In another experiment I created a userscript to insert faces back into Archives’ website. A userscript is just a little bit of code that rewrites web pages as they load in your browser. In this case the script grabs images relating to the files that you’re looking at from Invisible Australians.

So instead of this typical view of search results.

Before

Before

You see something quite different.

After

After

Instead of just the record metadata for an individual item, you see that there are people inside.

We also have to remember that the original context of these records was the administration of a system of racial surveillance and exclusion. The Archives preserves not only the records, but the recordkeeping systems that were used to monitor people’s movements. The remnants of that power adhere to the descriptive framework. There is power in the definition of categories and the elaboration of significance.

Thinking about this I came across Wendy Duff and Verne Harris’s call to develop ‘liberatory standards’ for archival description. Standards, like categories, are useful. They enable us to share information and integrate systems. But standards also embody power. How can we take advantage of the cooperative utility of standards while remaining attuned to the workings of power?

A liberatory descriptive standard, Duff and Harris argue: ‘would posit the record as always in the process of being made, the record opening out of the future. Such a standard would not seek to affirm the keeping of something already made. It would seek to affirm a process of open-ended making and re-making’.

‘Holes would be created to allow the power to pour out.’

‘Making and re-making’ — sounds a lot like the open data credo of ‘re-use and re-mix’ doesn’t it? I think it’s important to carry these sorts of discussions about power over into the broader realm of open data. After all, open data must always, to some extent, be closed. Categories have been determined, data has been normalised, decisions made about what is significant and why. There is power embedded in every CSV file, arguments in every API.

This is inevitable. There is no neutral position. All we can do is encourage re-use of the data, recognising that every such use represents an opening out into new contexts and meanings. Beyond questions of access or format, data starts to become open through its use. In Duff and Harris’s words, we should see open data ‘as always in the process of being made’.

What this means for cultural institutions is that the sharing of open data is not just about letting people create new apps or interfaces. It’s about letting people create new meanings. We should be encouraging them to use our APIs and LOD to poke holes in our assumptions to let the power pour out.

There’s no magic formula for this beyond, perhaps, building confidence and creating opportunities. But I do think that Linked Open Data offers interesting possibilities as a framework for collaboration and contestation — for making and challenging meanings.

We tend to think about Linked Open Data as a way of publishing — of pushing our data out. But in fact the production and consumption of Linked Open Data are closely entwined. The links in our data come from re-using identifiers and vocabularies that others have developed. The linked data cloud grows through a process of give and take, by many small acts of creation and consumption.

There’s no reason why that process should be confined to cultural institutions, government departments, business, or research organisations. Linked Open Data enables any individual to talk about what’s important to them, while embedding their thoughts, collections, passions or obsessions within a global conversation. By sharing identifiers and vocabularies we create a platform for communication. Anyone can join in.

So, if we want people to engage with our data, perhaps we need to encourage them to create their own.

I’ve just been working on a project with the Mosman public library in Sydney aimed at gathering information about the experiences of local servicepeople during World War One. There are many such projects happening around the world at the moment, but I think ours is interesting in a couple of ways. The first is a strong emphasis on linking things up.

The are records relating to Australian service people available through the National Archives, the Australian War Memorial, and the Commonwealth War Graves Commission, but there are currently no links between these databases. I’ve created a series of screen scrapers that allow structured data to be easily extracted from these sources. That means that people can, for the first time, search across all these databases in one hit. It’s a very simple tool that I started coding to ease the boredom of a long bus trip — but it has proved remarkably popular with family historians.

Once you’ve found entries in these databases, you can just cut and paste the URL into a form on the Mosman website and a script will retrieve the relevant data and attach it to the record of the the person you’re interested in. Linking to a service record in the National Archives, for example, will automatically create entries for the person’s birthplace and next-of-kin.

The process of linking builds structures, and these structures will themselves all be available as Linked Open Data. Even more exciting is that the links will not only be between the holdings of cultural institutions. The stories, memories, photographs and documents that people contribute will also be connected, providing personal annotations on the official record.

None of this is particularly hard, it’s just about getting the basics right. Remembering that structure matters and that links can have meaning. It’s also about recognising that ‘crowd sourcing’ or user-generated content can be made anywhere. Using Linked Open Data people can attach meanings to your stuff without visiting your website. Through the process of give and take, creation and consumption, we can build layers of description, elaboration, and significance across the web.

What excites me most about open cultural data is not the possibility of shiny new apps or collection visualisations, but the possibility of doing old things better. The possibility of reimagining the humble footnote, for example, as a re-usable container of structured contextual data — as a form of distributed collection description. The possibility of developing new forms of publication that immerse text and narrative within a rich contextual soup brewed from the holdings of cultural institutions.

I want every online book or article to be a portal. I want every blog or social media site to be a collection interface.

What might this look like? Perhaps something like this presentation.

My slides today are embedded within a HTML document that incorporates all sorts of extra goodies. The full text of my talk is here, and as you scroll through it you’ll see more information about the people, places and resources I mention pop up in the sidebar. Alternatively you can explore just the people or the resources, looking at the connections between them and the contexts in which they’re mentioned within my text.

This is part of an ongoing interest to explore different ways of presenting historical narrative, that build a relationship between the text and the structured data that underlies the story.

All of the structured data is available in machine-readable form as RDF — it is, in itself, a source of LOD. In fact the interface is built using an assortment of JavaScript libraries that read the RDF into a little temporary triplestore, and then query it to create the different views. So the whole thing is really powered by LOD.

It’s still very much an experiment, but I think it raises some interesting possibilities for thinking about how we might consume and create LOD simply by doing what we’ve always done — telling stories.

EJ Brady’s dreams were never realised. Australia’s vast spaces remained largely empty, and the poet continued to wrestle with personal and financial disappointment. ‘After nearly eight decades association with the man’, Brady wrote of himself in 1949, ‘I have come to look upon him as the most successful failure in literary history’. This energetic booster of Australia’s potentialities was well aware of his own life’s mocking irony. ‘He has not… made the wages of a wharf laborer out of book writing yet he persists in asserting Australia is the best country in the world!’.

But still Brady continued to add pins to his map.’For half a century I’ve been heaping up notes, reports, clippings, pamphlets, etc. on… all phases of the country’s life and development’. ‘What in hell I accumulate such stuff for I don’t know’, he complained in 1947. As the elderly man surveyed the ‘bomb blasted pile of rubbish’ strewn about his writing tent, he admitted that ‘this collecting is a sort of mania’.

Brady’s map and pins told a complex story of hope and disappointment, of confidence and fear. A story that combined national progress with an individual’s attempts merely to live.

There are stories in our data too — complex and contradictory stories full of emotion and drama, disappointment and achievement, violence and love. Let’s find ways to tell those stories.

Exposing the archives of White Australia

I recently gave a presentation in the Institute of Historical Research’s Digital History Seminar series. The time difference between London and Canberra was a bit of a challenge, so I pre-recorded the presentation and then sat in my own Twitter backchannel while it played. For the full podcast information go to HistorySPOT. You can also play with my slides or peruse the #dhist Twitter archive.

Exposing the Archives of White Australia from History SPOT on Vimeo.

Too important not to try

On Friday 19 October I joined an enthusiastic group of digital humanities explorers at a Deakin University event entitled Dipping a Toe into the Digital Humanities and Creative Arts. @catspyjamasnz has assembled an excellent summary of the day in Storify.

In the morning I told the story of Invisible Australians. You can view the slides of Too or important not to try and listen to my dodgy audio recording via SoundCloud.

In the afternoon I gave a whirlwind workshop which included a headline roulette smackdown and an introduction to the wonders of Zotero.

It’s all about the stuff: collections, interfaces, power and people

This is the full version of a paper I presented at the National Digital Forum, 30 November 2011.

In 1901, one of the first acts of the Commonwealth of Australia was to create a system of exclusion and control designed to keep the newly-formed nation ‘white’. But White Australia was always a myth. As well as the Indigenous population, there were already many thousands of people classified as ‘non-white‘ living in Australia — most were Chinese, but there were also Japanese, Indians, Syrians and Indonesians.

Here are some of them…

The real face of White Australia

The administration of what became known as the White Australia Policy created a huge volume of records, much of which is still preserved within the National Archives of Australia. These photographs are attached to certificates that non-white residents needed to get back into the country if they decided to travel overseas. There are thousands upon thousands of these certificates in the Archives. Thousands of certificates representing thousands of lives — all monitored and controlled.

But is is too easy to see these people as the powerless victims of a repressive system. There were many acts of resistance. Some argued against the need to be identified ‘just like a criminal’. Others exercised control over their representation, submitting formal studio portraits instead of mug shots.

Most commonly and most powerfully, people resisted the policy simply by going ahead and living rich and productive lives.

My partner, Kate Bagnall, is helping to rewrite Australian-Chinese history by overthrowing the stereotype of the culturally isolated Chinese man living a lonely, meagre existence surrounded by gambling and opium dens. By mining the available records, by reading against the grain of contemporary reports and by working with family historians, Kate is documenting their intimate lives — their wives, their lovers, their families and descendants — the sorts of relationships that sent a shudder through the edifice of White Australia. Power can be reclaimed in many subtle and subversive ways.

‘The real face of White Australia’ is an experiment. It uses facial detection to technology to find and extract the photographs from digital copies of the original certificates made available through the National Archives of Australia’s collection database. The photographs you see here come from just one series, ST84/1. There’s no API to the collection so I reverse-engineered the web interface to create a script that would harvest the item metadata and download copies of all the digitised images. There are 2,756 files in this series. On the day I harvested the metadata, 347 of those files had been digitised, comprising 12,502 images. It took a few hours, but I just ran my script and soon I had a copy of all of this in my local database.

Then came the exciting part. Using a facial detection script I found through Google and an open source computer vision library, I started experimenting with ways of extracting the photos. After a few tweaks I had something that worked pretty well, so I pointed my aging laptop at the 12,502 images and watched anxiously as the CPU temperature rose and rose. It took a few emergency cooling measures, but the laptop survived and I had a folder containing 11,170 cropped images. About a third of these weren’t actually faces, but it was easy to manually remove the false positives, leaving 7,247 photos.

These photos. These people.

With my database fully primed and loaded it was just a matter of creating a simple web interface using Django for the backend and Isotope (a jQuery plugin) at the front. Both are open source projects. All together, from idea to interface, it took a bit more than a weekend to create, and most of that was waiting for the harvesting and facial detection scripts to complete. It would be silly to say it was easy, but I would say that it wasn’t hard.

What we ended up with was a new way of seeing and understanding the records — not as the remnants of bureaucratic processes, but as windows onto the lives of people. All the faces are linked to copies of the original certificates and back to the collection database of the National Archives. So this is also a finding aid. A finding aid that brings the people to the front.

According to Margaret Hedstrom the archival interface ‘is a site where power is negotiated and exercised’. Whether in a reading room or online, finding aids or collection databases are ‘neither neutral nor transparent’, but the product of ‘conscious design decisions’. We would like to think that this interface gives some power back to the people within the records. Their photographs challenge us to do something, to think something, to feel something. We cannot escape their discomfiting gaze.

But this interface represents another subtle shift in power. We could create it without any explicit assistance or involvement by the National Archives itself. Simply by putting part of the collection online, they provided us with the opportunity to develop a resource that both extends and critiques the existing collection database. Interfaces to cultural heritage collections are no longer controlled solely by cultural heritage institutions.

It’s these two aspects of the power of interfaces that I want to focus on today.

There are a growing number of examples where the records created by repressive or discriminatory regimes have, in Eric Ketelaar’s words, ‘become instruments of empowerment and liberation, salvation and freedom’. Nazi records of assets confiscated during the Holocaust have been used to inform processes of restitution and reparation. Government records have helped members of Australia’s Stolen Generations trace family members. Descendants of inmates incarcerated by American colonial authorities in what was the world’s largest leprosy colony in the Philippines, have embraced the administrative record as an affirmation of their own heritage and survival. Records can find new meanings. Power can be reclaimed.

Technology can help. Tim Hitchcock has described how something as simple as keyword searching can turn archives on their heads. Recordkeeping systems tend to reflect the structures and power relations of the organisations that create them. The ‘hierarchical and institutional nature of most archives’, Hitchcock argues, ‘contains an ideological component which is sucked in with every dust-filled breath’. But digitisation and keyword searching free us from having to follow the well-worn paths of institutional power. We can find people and follow their lives against the flow of bureaucratic convenience. We can gain a wholly new perspective on the workings of society. ‘What changes’, Hitchcock asks, ‘when we examine the world through the collected fragments of knowledge that we can recover about a single person, reorganised as a biographical narrative, rather than as part of an archival system?’

Projects such as Unknown no longer may help us answer that question.

Unknown no longer

It’s aiming to extract the names and biographical details of slaves from the 8 million manuscript documents held by the Virginia Historical Society. The documents include court records, receipts, wills and inventories. Here is a page from the ‘Inventory of Negroes at Berry Plain Plantation, King George County, Virginia’ for 1855, listing names, occupations and valuations.

Tim Hitchcock is one of the directors of London Lives a project that similarly seeks to find the people in 240,000 manuscript pages documenting the lives of plebeian Londoners in the 17th century.

London Lives

More than three million names have already been extracted from the records of courts, workhouses, hospitals and other institutions. Work is continuing to link these names together, to merge these various shards of identity and piece together the experiences of London’s poorest inhabitants.

Remember me from the US Holocaust Memorial Museum is working with photographs taken by relief agencies in the aftermath of World War Two. The photographs are of displaced children who survived the Holocaust but were separated from families. What happened to them? The project is seeking public help to identify and trace the children.

Remember me

These are all projects about finding people. Finding the oppressed, the vulnerable, the displaced, the marginalized and the poor and giving them their place in history. This is what Kate and I hope to do with Invisible Australians, the broader project of which our faces experiment is part.

Invisible Australians

‘Invisible Australians’ aims to extract more than just photographs. We want to record and aggregate the biographical data contained within the records of the White Australia Policy — to extract the data and rebuild identities.

But we want to do more, we want to link these identities up with with other records, with the research of family and local historians, with cemetery registers and family trees, with newspaper articles and databases we don’t even know about yet. We want to find people, families and communities.

It’s ridiculously ambitious and totally unfunded. But it is possible.

The most exciting part of online technology is the power it gives to people to pursue their passions. As with the faces, we don’t need the help of the National Archives. We need the records to be digitized, but that’s happening anyway and we can afford to be patient. Most of the tools we need already exist, and are free. In the past 12 months, for example, there have been a number of open source tools released for crowd-sourced transcription of manuscript records.

People with passions, people with dreams, people who are just annoyed and impatient, don’t have to wait for cultural institutions to create exactly what they need. They can take what’s on offer and change it.

Interfaces can be modified. It is amazingly easy to write a script that will change the way a web page looks and behaves in your browser. I was frustrated by the standard interface to digitized files in the National Archives of Australia’s Recordsearch database — so I changed it.

Before and after

Not only did make it look a bit nicer, I added new functions. My script lets you print a whole file or a range of pages and display the entire contents of the file on a pretty cool 3d wall.

I’ve shared this script, and a few other Recordsearch enhancements. Anyone can install them with a click and use them.

Wragge Labs Emporium

Interfaces are sites of power and we can claim some of that power for ourselves. Online technologies not only free us from the having to brave the physical intimidation of the reading room, they free us up to engage with the records in new ways. The archivist-on-duty would probably not be pleased if I pulled out some scissors and started snipping photos out of certificates. Or if I pulled a file apart and pasted it’s contents on the wall. But online we are free to experiment.

The power of cultural heritage organisations is perhaps expressed most forcefully in their ability to control the arrangement and description of their collections. ‘Every representation, every model of description, is biased’, note Verne Harris and Wendy Duff, ‘because it reflects a particular world-view and is constructed to meet specific purposes’. Archives, libraries and museums are already starting to share this power, by allowing tagging, or seeking public assistance with description through crowd sourcing projects. But most of the these activities still happen within spaces created and curated by the institutions themselves. Our cathedrals of culture might be opening their doors and inviting the public to participate in their ceremonies, but that doesn’t make them bazaars. The architecture stills speaks of authority.

In any case, people already have a space where they can explore and enrich collections — it’s called the internet.

It would be great to see cultural institutions doing more to watch, understand and support what people are doing with collections in their own spaces — following them as they pursue their passions, rather than thinking of ways to motivate them.

A quick example… You might have heard of Zotero, it’s an open source project that lets you capture, annotate and organize your research materials.

Zotero

One cool thing about Zotero is that you can build and contribute little screen scrapers, called translators, that let Zotero extract structured data from any old collection database. You might not be surprised to learn that I’ve created a translator for Recordsearch. Another cool thing about Zotero is that you can share the stuff that you collect in public groups.

Invisible Australians Zotero group

Put those two cool things together and what do you have? Well to me they spell out user generated finding aids — parallel collection databases created by researchers simply pursuing their own passions.

Linked Open Data greatly increases opportunities for collection description to leak into the wider web. If objects and documents are identified with a unique URL, then anyone can can make and publish statements about them in machine-readable form. These statements can then be aggregated and explored. Initiatives such as the Open Annotation Collaboration will hasten the development of these shared descriptive and interpretative layers around our cultural collections.

And of course all this descriptive and interpretative work can be harvested back to enhance existing collection databases. We could start doing it now — though I will spare you today my rant about the possibilities of mining footnotes.

As well as exploring the possibilities of user-generated content, cultural institutions are starting to open up their collection data for re-use. APIs are great (though Linked Open Data is better), and New Zealand is lucky to have an organisation like DigitalNZ which just gets it. People can and will make cool things with your stuff.

But again, we don’t have to wait for everything to be delivered in a convenient, machine-readable form. If it’s on the web anybody can scrape, harvest and experiment.

You probably all know about the National Library of Australia’s newspaper digitisation project — it’s building a magnificent resource. But I wanted to do more than just find articles. I wanted to explore and analyze their content on a large scale. So I built a screen scraper to extract structured data from search results, and then used the scraper to  power a series of tools. I have a harvester that lets you download an entire results set — hundreds or thousands of articles — with metadata neatly packaged for further analysis.

Harvester

Or what about a script that graphs the occurrence of search terms over time, and allows you to ask questions like When did the Great War become the First World War?.

When did the Great War become the First World War?

In the end I got a bit carried away and built my own public API to the Trove newspaper database.

Unofficial Trove newspapers API

I think it’s important to note that the tools I developed were guided by the types of questions I wanted to ask. While we should welcome APIs and celebrate their possibilities, we should also remain critical. APIs are interfaces, they too embed power relations. Every API has an argument. What questions do they let us ask? What questions do they prevent us from asking?

Even as we move from the age of lumbering, slow-witted data silos into the rapidly-evolving realms of Linked Open Data, we have to constantly question the models we make of the world. Ontologies and vocabularies are culturally determined and historically specific. Yes, they too are interfaces, complete with their own distributions of power and authority. But we can revisit and change them. And we can relate our new models to our old models, capturing complex, long-term shifts in the way we think about the world. That’s incredibly exciting.

All of this hacking, harvesting, questioning, enriching and meaning-making makes me think about the possibilities of grassroots leadership. Online technologies enable people to take cultural institutions into unexpected realms. They can build their own interfaces, ask their own questions, determine their own needs — they can point the way instead of simply waiting to be served.

You might wonder what the National Library of Australia thinks of my various scrapers and harvesters. I can’t speak for them, but I can say that they’ve awarded me a fellowship to explore further the possibilities of text-mining in their newspaper database.

The idea of grassroots leadership brings me back to the title of this talk — ‘It’s all about the stuff’. It seems to me that we tend to model the interactions between cultural institutions and the public as transactions. The public are ‘clients’, ‘patrons’, ‘users’ or ‘visitors’. But the sorts of things I’ve been talking about today give us a chance to put the collections themselves squarely at the centre of our thoughts and actions. Instead of concentrating on the relationship between the institution and the public, we can can focus on the relationship we both have with the collections.

It’s all about the stuff.

It’s all about the respect and responsibility we both have for our collections.

It’s all about the respect and responsibility we both have for people like this.