By Ariana Giorgi and Christine Zhang 

As journalists who analyze data for stories, we strive to hold ourselves accountable to a high standard of accuracy. But checking our work is rarely a straightforward process. Newsroom editors and fact-checkers might not have enough data expertise. Often, we need an outside opinion. Ideally, we could ask each other for advice, or even turn to experts in other fields for help. In academia, asking for outside comment before publication is broadly referred to as “peer review.”

During SRCCON, we discussed the challenges of applying the concept of peer review to journalism and some ways we’ve make it work for us.

(We are defining peer review here as the practice of having colleagues in a given academic field scrutinize each others’ methods and findings. A more formal definition, which we are not using to here, refers to a committee of referees evaluating a research article before deciding whether or not to publish it in a scholarly journal.)

Here are a few insights that arose from our session.

Basics of Journalistic Peer Review

What does peer review look like, in this context?

Journalistic peer review should be a two-step process: first, vetting your analysis pre-publication and second, posting the data and methods along with the story.

We made an important distinction between verifying your work before running the story and showing your work in tandem with or after publication of the story. While we focused more on pre-publication review, we agreed that the open-sourcing of data analysis, which does not happen enough in academia, is an area where data journalists could take the lead.

What should be peer reviewed?

Stories can include a range of data findings—from a few numbers to supporting charts to in-depth investigations. When considering peer review, you should think about whether your findings warrant outside opinion, or if your numbers could just use a simple double-check. In our session, we decided that it’s most useful to evaluate how much the basis of your story depends on the data analysis. If the entire story hinges on conclusions that you found through an analysis, then it’s worth comparing notes with someone else. This also might include stories that predict or interpolate a finding.

What does peer review look like inside of a newsroom?

There are many ways to checking your numbers internally. It can be as simple as talking to a colleague who understands a bit of the methodology behind the analysis. A stronger approach would be to pair up with a team member before any actual analysis is done: the two of you would independently decide on the appropriate methodology for the given data, then reconvene and look at the strengths and shortcomings of the different approaches. Perhaps if you’re part of a team of other data journalists, you could meet once a week to spot check each other’s work or nerd out about quantitative methodologies.

What about outside of a newsroom?

Sometimes, you don’t have anybody else to swap notes with. Even if you are on a data team, you might still need somebody with more expertise than your co-workers can offer.

In these cases, consider reaching out to other peers who may have struggled with the same decisions or worked with the same dataset. Even if those journalists don’t provide an opinion on your methodology, they can likely point you to someone else who would. They might not see you as “the competition”, since often they have a completely different focus and audience for that particular subject (campaign finance came up as a timely example). If you don’t know where to start reaching out to people, try various topic-specific channels on the NewsNerdery Slack channel or the NICAR-L listserv.

If you’re looking for math and stats experts, you could consider stats.org, a group of statisticians that make themselves available to help journalists interpret findings and discuss computational approaches. Other subject-matter specialists like researchers or professors are another resource for vetting general findings.

Read more