Resources for documents research, or what Old Pros can teach New Media

By David Perkins

Bless every man and woman who has ever whispered a tip in my ear —and put food on my table. But, in my experience, tips are appetizers. Records are the meat and potatoes of most investigative stories and many other stories.–Pat Stith

Will there ever again be a reporter like Pat Stith? I ask myself that question whenever I teach the use of documents in reporting. This year I’ve wondered about it more strongly than ever. Pat recently retired from the News and Observer (Raleigh, N.C.), where we worked together in the 1980s. He spent 37 years there—most of them quietly and methodically nailing corrupt government officials to the wall.

Pat lived and breathed reporting, especially investigative reporting, especially the aspect of investigative reporting that involves hunting for and finding the incriminating document. One of his proudest moments was when he found a 20-year-old letter from a North Carolina Agriculture Dept. official in a dusty box. That little piece of paper told him all he needed to explain why the N.C. State Fair paid a special $100,000 a year bonus to a Florida carnival owner.

State officials save the darnedest things, he recalled, so I went to the records warehouse looking for proof  of an old,  friendly , relationship between agriculture officials and Strates Shows . The copy of the 20-year-old note I found thanked the carnival owner for his hospitality in Las Vegas–for meeting the North Carolina officials at the airport, for the dinners, the show,  the “free booze,” and the chauffeur. (Pat’s reflections on this and other episodes can be found here.)

After I left the News and Observer, in the late 80s, Pat’s career got a new charge with the arrival of “Computer Assisted Reporting.” He became a pioneer in the field. He’d acquire databases from state agencies, break them down, and find the revealing numbers.  As Pat will tell you, documents and databases have advantages over human sources: There is no arguing over the basis of attribution. They either are, or are not, a public record.  Plus, the job of peeling away hype and spin and getting to the facts has (often) been done for you.  You just need to know what to look for.  And persevere.

Computer analysis was a key tool in “Boss Hog,” the 1996 series Pat wrote, with Joby Warrick, on hog farming in eastern North Carolina. They had found that hog waste was covering thousands of acres, poisoning the ground and air, while regulation was being stymied by the farming interests. The biggest obstacle was the biggest farmer, a former legislator named Wendell Murphy, “Boss Hog” himself. (The whole series, which won a Pulitzer, is here.)

Among several methods he used to establish Murphy’s role, Pat got his hands on a database of every phone call made by someone in state government during a two-year period.  Breaking it down, he found that someone in state government called Murphy, or a member of his family, every hour of every working day during those two years. Pat even traced an individual phone call to Murphy from an office in the Ag. Dept. on the very day a key committee killed a regulatory bill.

Pat’s views on Computer Assisted Reporting are still useful in the post-newspaper age, as is the specific advice he summarized in a talk on “Cop Killers, Hogs, and the Essentials of Good Record Work.”

When Pat retired in October—taking a buyout–the newspaper’s Mandy  Locke wrote a delightful profile that might as well serve as an obituary for old-style newspapering—and for the role the newspaper once played in American family life. (That’s not pessimism, just a healthy recognition that change involves some loss.)

Pat was the son of an Alabama coal miner who had dropped out of school in the seventh grade, but who read every word of the daily paper and, after dinner, quizzed his children with a mix of facts and fibs. If anyone messed up, they were sent to the World Almanac. It was the newspaper that brought the world into that home, and Pat’s Dad saw to it that his kids knew both latitudes and longitude. (I’ll let you draw your own conclusions about whether family life is better or worse off today.)


The N&O has hired someone to take Pat’s place as the paper’s chief investigative reporter, but, with its staff cut by a third in the last five years, from 100 to 65 full-time reporters (20 were cut in September), its managing editor, John Drescher, told me recently it is going to be difficult to allow its beat reporters to do much more than cover meetings and breaking news.   Pat would never criticize the N&O—he’s a company man–but it is no longer the place for him.  

What would be?

Would a blog—even Talking Points Memo, the first blog to win a Polk Award for its reporting– have the resources or the political clout to persuade a government agency (if necessary with a healthy lawsuit) to cough up documents? Could it afford the man hours for someone to parse numbers as painstakingly as Pat did? Do armchair bloggers know where to start?

Yes, there are several excellent nonprofit organizations like ProPublica and the Center for Public Integrity that do good investigative work on the national level and publish their reports on high-profile websites. (Leonard Downie of the Washington Post has talked about future collaborations between the paper and ProPublica.)

What about outside the capital, where corruption also sometimes happens?

There are encouraging signs. A few tough-minded news websites like are emerging and doing Stith-like work. Staff and readership is small–the San Diego site only gets about 18,000 unique visitors a month—but the Voice’s stories about conflicts of interest and misleading crime statistics, for example, have led to investigations and resignations. These stories were picked up by the larger media (where the editors of these sites have often been trained), which assured their impact.  This suggests an emerging “feeder” system where investigative reporting is done by independent reporting teams and then picked up by  corporate/mass media–perhaps eventually for a fee, like a wire service. Other such journalistic terriers include MinnPost in Minneapolis, and the New Haven Independent. The N.Y. Times sees a “fledgling movement.”

The open-source format and world reach of the web ought to allow for an unprecedented collaboration across geographical boundaries between enterprising bloggers and old-line pros like Pat Stith.

If this still seems to be little more than a possibility, a promising tool for such collaboration is in the works.

Earlier this month, the N.Y. Times and ProPublica applied for $1 million from the Knight Foundation to launch an online repository of primary-source documents, to be known as DocumentCloud. This will be open-source, available to all, for download and analysis. The software has already been developed and is used by the Times. The idea is to access the “wisdom of the crowd” in selecting, posting, and tagging documents with keywords.

Using this program, documents that one reporter or team has analyzed—paper or discs that would generally just sit on the floor or be tossed–can be posted and made available for analysis by others.  It will take about three years to create DocumentCloud. (An explanation and their grant proposal is here.  More on it here.)

The only criticism I’ve heard has come from a few bloggers and new media guru Jay Rosen, who says the Times’ proposal is pushing aside worthy requests from smaller innovators. 

There is a major shortcoming to DocumentCloud.  It does not allow for the uploading of databases—only individual documents–for others to absorb and break down, as Pat did those phone numbers.  When I asked Scott Klein, director of Online Development for ProPublica, why there is no “DataCloud,” he said they had thought about it, but other folks are aggregating databases, and they might get to it later. (He mentioned the National Institute of Computer Assisted Reporting/NICAR, and the Sunlight Foundation.)

“For a lot of reasons, including complexity and breadth of impact, we wanted to tackle documents first,” he told me in an email. “The good news is that building the consortium that will help guide DocumentCloud is doing the work of both.”

Perhaps they’ll call Pat for some advice.  He seems to be still at it. He had talked about fishing, etc., in his retirement, but he was recently seen downtown Raleigh, sporting a new beard and no doubt still walking with that bulldog gait.  According to N&O editor Drescher, he had received a phone call from a woman who said she had some juicy news.

“Lady, I’m retired,” Stith said.

“I know, but it’s a really good story,” the woman said.

“All right,” Stith said. “Where do you want to meet?”


Colleagues and readers,

Please feel free to post other web-related resources that relate to the topic of investigative and “accountability” reporting.



About journalismprof

Steve joined the journalism faculty at the University of Massachusetts at Amherst in August 2007 and has been working to incorporate multimedia across the curriculum. Since arriving at UMass, Steve has developed three courses modeled after his multimedia journalism course. The courses allow students to work in teams in a newsroom-like environment where they work on packages -- using video, audio and photos to tell stories. He is also working with students on developing, a news Web site staffed completely by students. Steve has more than 25 years of experience as an editor and reporter for print and online publications, including 10 as an editor at He also edits part-time for with the NFL and college football network.
This entry was posted in Future of Journalism, Uncategorized and tagged , , , . Bookmark the permalink.

11 Responses to Resources for documents research, or what Old Pros can teach New Media

  1. Bob Douglas says:

    Forget about Voice of San Diego. they only get 18,000 visitors and exist solely on money donated by a couple of people. We at 92067 Rancho Santa Fe Free Press at are a template for what award-winning community journalists are doing…and we happen to be in — as the name suggests — Rancho Santa Fe, Ca. We operate on ZERO DOLLARS, yet have it together, and in our own way have it way more together than many . Check us out!!!

  2. BJ Roche says:

    Lots to think about here!

    Guys like Pat Stith – who took quite seriously the role of keeping government accountable– inspired a lot of us to get into journalism. Now we have the technology that makes it easier to do that, but where’s the staff and the will?

    The Globe’s Walter Robinson, another reporting bulldog, is working on a project at Northeastern, called The New England First Amendment Center, that focuses on public docs and public records. Students in his journalism classes have gone out and examined public docs for different stories that have ended up in the Globe.

  3. One quick point: The 18,000 figure cited by the NY Times isn’t correct for’s readership. We weren’t signed up for quantcast, the service the reporter used. We’ve signed up since then and that number has stayed around 60,000. We now have a quantcast badge on our front page that shows the readership.

  4. journalismprofs says:

    It seems quaint to think about how revolutionary CAR was back in the day — if you think about it….almost all reporting today is “computer-assisted” — some more than others. Right?

    You don’t have to look far to find good old-fashioned document research/accountability reporting/investigative reporting today– call it what you want, it’s happening, it’s just being done a little differently.

    Unfortunately, too many of us are still thinking in terms of the sole crusading journalist chewing on a cigar and hammering at a typewriter,

    There’s still a few out there, but today it’s more about working in teams which may include journalists/database editors more in the mold of innovators like Adrian Holovaty, whose most recent project, EveryBlock, takes good old-fashioned document reporting in another direction:

    The other twist on old-fashioned document reporting is the notion of crowdsourcing. I first wrote about this back in 2006:

    I remember talking to News-Press Editor Kate Marymont, who said she was stunned by the level of dedication and involvement from those in the community. Within the community there were attorneys, architects, accountants — the traditional “experts” that journalists often seek to help make sense of documents uncovered during an investigation. Gannett has made a commitment to doing more projects like this and I think David Cohn’s “Spot.Us” project is another place where investigative reporting is being taken in another direction — funding for projects by the crowd.

    We all have our Pat Stith examples. But many of us also keep all those old music “albums.” The tools may change but the need to hear good music doesn’t.

    Technology affords us new ways to do things and included is the use of the crowd. I prefer to look forward and look at all the possibilities out there rather than looking back fondly at the good old days. The key, and I hope students are listening, is to get out there and do it.



  5. David says:

    Look forward, but learn from the masters of the trade.

  6. BJ Roche says:

    Ok, look forward, Steve, but on a more basic level, students also have to know the public records laws and have the courage to go into an office and demand to see these records, even in the face of, say, a disgruntled cop who doesn’t want to give you a document.

    They need to know that you don’t have to show an ID to get a public document, and they don’t need to tell the reason why they’re asking for it. And if a public official asks you this, you have to tell him that. This can be a scary experience for a young person, but a necessary one!

    We’ve all seen cases in our reporting classes where the most basic of public documents, like arrest reports, police logs, minutes of executive sessions once an issue has been decided, have all been withheld.

    In some small towns, the officials themselves don’t even know what’s public, and the knee-jerk reaction is to not give the students the documents. If these public records laws aren’t exercised, we start to lose them.

  7. journalismprofs says:


    I don’t think anyone would disagree about teaching those basic tenets of journalism. Access to public information should be foremost in all we teach. Those basics should continue to be taught, as well as the new tools available in making them easier to compile and put together.

    What I continue to take issue with is the either/or dichotomy where many choose to look back fondly at the good old days and assume that since tools have changed, so has quality. There are new and interesting ways of doing what you’re talking about and we should take advantage of them when teaching and doing journalism.


  8. Art Clifford says:


    I wonder if we teach our students how to find information even when someone (legally) finds a way to deny the information to them? We’ve all been confronted by the brick walls put up by bureaucrats who claim things such as huge “retrieval fees” and documents such as Congressional Research Service reports to members of Congress, which are technically in a state of limbo even though most of them are made public by individual congresspeople.

    How many students know that most towns won’t/don’t create online data bases of registered voters in a community, but that the towns usually sell printed copies each year for about $30? They contain useful information, too — age, address, occupation (I’m listed as a Sherpa Guide). I know UMass students do, but I wouldn’t bet all J students know this.

    Let’s face it, the last eight years have been very unkind to our nation’s sunshine laws. Our students need to learn everything from using a poker face to accessing government archives and developing good, old-fashioned sources to find that one person in the organization who will find time to help that worst-case scenario student journalist who has no ID, and nothing to fall back on but attitude.

  9. David says:


    At my request, Pat sent me an email response to this piece, and had this to say (I think Steve will appreciate his hopefulness):

    “What’s happening to newspapers is sad. Our craft has lost a lot of talented men and woman, and may lose a lot more. But I don’t think investigative reporting is going to be one of the casualities.

    “There may be, probably will be, fewer massive three, four, five-part series. (That’s not all bad.) And I think there will be fewer full time investigative reporters.

    “But the fact is, some reporters just can’t help looking behind the sofa, in the corner, under the rug. You can assign them to cover city hall, or medicine, or the arts, or whatever. You can require them to write dailies. It won’t matter. They’re still going to find the slush fund. Or the sweetheart contract. Or whatever. They can’t help it. They’re cutters and they just can’t help it.

    “And they have a lot more tools to work with than we had 40 years ago, which means more, and much better, investigations can be done with fewer people.

    “In the old days we were almost completely dependent on anonymous sources. There were no databases to be had and, by comparison, few paper records. Now a reporter, with minimal training, can acquire and analyze databases that enable him or her to learn more about an agency than the people who work there. And that database work can be done little by little, bit by bit, while the reporter continues to crank out dailies.

    The Internet, of course, is a gift. You tell me. How much productivity does it add? And someone is going to figure out –maybe they already have– how to consistently enlist the public’s help on investigative stories. A lot of investigation work can’t be done that way –Those that say, don’t know, and those that know, don’t say–but some can. And harnessing and riding that horse is going to be a lot of fun. Maybe that will make up for some of the cuts. I hope so.”

  10. David says:

    And here’s Leonard Downie’s commentary on the limits of investigative journalism in the digital age.

  11. Pingback: news and observer newspaper raleigh nc | Digg hot tags

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s