Resources for documents research, or what Old Pros can teach New Media

By David Perkins

Bless every man and woman who has ever whispered a tip in my ear —and put food on my table. But, in my experience, tips are appetizers. Records are the meat and potatoes of most investigative stories and many other stories.–Pat Stith

Will there ever again be a reporter like Pat Stith? I ask myself that question whenever I teach the use of documents in reporting. This year I’ve wondered about it more strongly than ever. Pat recently retired from the News and Observer (Raleigh, N.C.), where we worked together in the 1980s. He spent 37 years there—most of them quietly and methodically nailing corrupt government officials to the wall.

Pat lived and breathed reporting, especially investigative reporting, especially the aspect of investigative reporting that involves hunting for and finding the incriminating document. One of his proudest moments was when he found a 20-year-old letter from a North Carolina Agriculture Dept. official in a dusty box. That little piece of paper told him all he needed to explain why the N.C. State Fair paid a special $100,000 a year bonus to a Florida carnival owner.

State officials save the darnedest things, he recalled, so I went to the records warehouse looking for proof  of an old,  friendly , relationship between agriculture officials and Strates Shows . The copy of the 20-year-old note I found thanked the carnival owner for his hospitality in Las Vegas–for meeting the North Carolina officials at the airport, for the dinners, the show,  the “free booze,” and the chauffeur. (Pat’s reflections on this and other episodes can be found here.)

After I left the News and Observer, in the late 80s, Pat’s career got a new charge with the arrival of “Computer Assisted Reporting.” He became a pioneer in the field. He’d acquire databases from state agencies, break them down, and find the revealing numbers.  As Pat will tell you, documents and databases have advantages over human sources: There is no arguing over the basis of attribution. They either are, or are not, a public record.  Plus, the job of peeling away hype and spin and getting to the facts has (often) been done for you.  You just need to know what to look for.  And persevere.

Computer analysis was a key tool in “Boss Hog,” the 1996 series Pat wrote, with Joby Warrick, on hog farming in eastern North Carolina. They had found that hog waste was covering thousands of acres, poisoning the ground and air, while regulation was being stymied by the farming interests. The biggest obstacle was the biggest farmer, a former legislator named Wendell Murphy, “Boss Hog” himself. (The whole series, which won a Pulitzer, is here.)

Among several methods he used to establish Murphy’s role, Pat got his hands on a database of every phone call made by someone in state government during a two-year period.  Breaking it down, he found that someone in state government called Murphy, or a member of his family, every hour of every working day during those two years. Pat even traced an individual phone call to Murphy from an office in the Ag. Dept. on the very day a key committee killed a regulatory bill.

Pat’s views on Computer Assisted Reporting are still useful in the post-newspaper age, as is the specific advice he summarized in a talk on “Cop Killers, Hogs, and the Essentials of Good Record Work.”

When Pat retired in October—taking a buyout–the newspaper’s Mandy  Locke wrote a delightful profile that might as well serve as an obituary for old-style newspapering—and for the role the newspaper once played in American family life. (That’s not pessimism, just a healthy recognition that change involves some loss.)

Pat was the son of an Alabama coal miner who had dropped out of school in the seventh grade, but who read every word of the daily paper and, after dinner, quizzed his children with a mix of facts and fibs. If anyone messed up, they were sent to the World Almanac. It was the newspaper that brought the world into that home, and Pat’s Dad saw to it that his kids knew both latitudes and longitude. (I’ll let you draw your own conclusions about whether family life is better or worse off today.)

***

The N&O has hired someone to take Pat’s place as the paper’s chief investigative reporter, but, with its staff cut by a third in the last five years, from 100 to 65 full-time reporters (20 were cut in September), its managing editor, John Drescher, told me recently it is going to be difficult to allow its beat reporters to do much more than cover meetings and breaking news.   Pat would never criticize the N&O—he’s a company man–but it is no longer the place for him.  

What would be?

Would a blog—even Talking Points Memo, the first blog to win a Polk Award for its reporting– have the resources or the political clout to persuade a government agency (if necessary with a healthy lawsuit) to cough up documents? Could it afford the man hours for someone to parse numbers as painstakingly as Pat did? Do armchair bloggers know where to start?

Yes, there are several excellent nonprofit organizations like ProPublica and the Center for Public Integrity that do good investigative work on the national level and publish their reports on high-profile websites. (Leonard Downie of the Washington Post has talked about future collaborations between the paper and ProPublica.)

What about outside the capital, where corruption also sometimes happens?

There are encouraging signs. A few tough-minded news websites like VoiceofSanDiego.org are emerging and doing Stith-like work. Staff and readership is small–the San Diego site only gets about 18,000 unique visitors a month—but the Voice’s stories about conflicts of interest and misleading crime statistics, for example, have led to investigations and resignations. These stories were picked up by the larger media (where the editors of these sites have often been trained), which assured their impact.  This suggests an emerging “feeder” system where investigative reporting is done by independent reporting teams and then picked up by  corporate/mass media–perhaps eventually for a fee, like a wire service. Other such journalistic terriers include MinnPost in Minneapolis, and the New Haven Independent. The N.Y. Times sees a “fledgling movement.”

The open-source format and world reach of the web ought to allow for an unprecedented collaboration across geographical boundaries between enterprising bloggers and old-line pros like Pat Stith.

If this still seems to be little more than a possibility, a promising tool for such collaboration is in the works.

Earlier this month, the N.Y. Times and ProPublica applied for $1 million from the Knight Foundation to launch an online repository of primary-source documents, to be known as DocumentCloud. This will be open-source, available to all, for download and analysis. The software has already been developed and is used by the Times. The idea is to access the “wisdom of the crowd” in selecting, posting, and tagging documents with keywords.

Using this program, documents that one reporter or team has analyzed—paper or discs that would generally just sit on the floor or be tossed–can be posted and made available for analysis by others.  It will take about three years to create DocumentCloud. (An explanation and their grant proposal is here.  More on it here.)

The only criticism I’ve heard has come from a few bloggers and new media guru Jay Rosen, who says the Times’ proposal is pushing aside worthy requests from smaller innovators. 

There is a major shortcoming to DocumentCloud.  It does not allow for the uploading of databases—only individual documents–for others to absorb and break down, as Pat did those phone numbers.  When I asked Scott Klein, director of Online Development for ProPublica, why there is no “DataCloud,” he said they had thought about it, but other folks are aggregating databases, and they might get to it later. (He mentioned the National Institute of Computer Assisted Reporting/NICAR, and the Sunlight Foundation.)

“For a lot of reasons, including complexity and breadth of impact, we wanted to tackle documents first,” he told me in an email. “The good news is that building the consortium that will help guide DocumentCloud is doing the work of both.”

Perhaps they’ll call Pat for some advice.  He seems to be still at it. He had talked about fishing, etc., in his retirement, but he was recently seen downtown Raleigh, sporting a new beard and no doubt still walking with that bulldog gait.  According to N&O editor Drescher, he had received a phone call from a woman who said she had some juicy news.

“Lady, I’m retired,” Stith said.

“I know, but it’s a really good story,” the woman said.

“All right,” Stith said. “Where do you want to meet?”

*****

Colleagues and readers,

Please feel free to post other web-related resources that relate to the topic of investigative and “accountability” reporting.

DP.