Evaluating the Credibility of a Source: Consistency as a False Positive -By Heinrich Böhmke

In December 2014, Sean Woods received the call editors dread the most. A Rolling Stone reporter told him she no longer stood by her story of a horrible rape supposedly committed by frat boys at the University of Virginia. The feature, published a few weeks earlier, broke readership records at the iconic magazine. It caused a national uproar. It is not hard to see why. It is the dramatic account of a young woman, lured by her date to a fraternity house, there to be gang-raped on broken glass by seven young initiates. The line the story took was that the victim’s experience was emblematic of a ‘culture’ female students faced on campuses across the US. This heinous crime was further compounded by the defensive, almost dismissive, response of University authorities.

Sabrina Erdely is an experienced contributor to Rolling Stone. Her latest story, “A Rape on Campus”, went through a fact checker, an editor and legal counsel. Other writers, however, notably in the Washington Post, had no difficulty exposing her exposé. At best, it was shoddy, prejudicial journalism with the underlying claim still intact. Fair to middling, Erdely was taken in by a teenage lie, which the elementary techniques of investigative reporting could have detected with ease. At worst, Jackie’s account was an inherently improbable hoax. The reason Erdely recounted it at all was that it confirmed her ideological and ‘gothic’ expectation of how entitled male undergraduates were apt to behave. As Cathy Young put it, Erdely’s story flowed from the pen of one possessed of “the troubling zealotry of advocates for whom believing rape claims is somewhat akin to a matter of religious faith”.

Rolling Stone approached the dean of the Columbia School of Journalism with a simple and acute brief. Would Steve Coll help them understand how they had got things so terribly wrong? This review has just been published. It found that the writer and editors were far too deferential to the single pseudonymous source, the victim, Jackie. Unwisely sensitive to her emotional well-being, Rolling Stone permitted Jackie to dictate the scope of the investigation and even their own questions to her. Erdely further failed to verify crucial facts such as that the alleged rapist actually existed or that Jackie’s contemporaneous claims to friends matched what she was saying now. Erdely’s breach of press protocol did not end there. In what a Washington Post writer called an ‘unfathomable deceit’, Erdely and her editor misattributed dialogue and unfairly deprived the targets of her investigation of an opportunity to reply to – or discount – intensely damaging accusations.

The Coll review has itself attracted edifying commentary about the pitfalls of narrative journalism and the shaming by ‘activists’ of those who correctly pointed out the story’s glaring holes. Amidst the condemnation of Rolling Stone, there is a sense that this is a learning moment for the profession as a whole. While some of the faults of the piece flow from Erdely’s unique and perhaps even malicious failings as the author, other errors are common to many in the profession. The error that has attracted the least commentary so far is the idea that Jackie’s credibility was legitimately enhanced because her detailed story stayed consistent over its retellings to the journalist and because of her ‘confident’ demeanour during these narrations. Many a reporter has fallen into these two holes before.

The Coll review states that Erdely “remembered being ‘a bit incredulous’ about the vividness of some of the details Jackie offered, such as the broken glass from the smashed table. Yet Jackie had been ‘confident, she was consistent.’” Erdely repeats the quality of consistency in Jackie’s versions of events in an interview to Slate. This is what convinced her. All those details, staying the same, with every interview.

The fact-checker, who spent four hours on the phone reviewing Jackie’s story was also impressed by her vivid consistency. “She wasn’t just answering, ‘Yes, yes, yes,’ she was correcting me.”

In ascribing credibility to a story because it stays consistent, investigative journalists are committing what amounts to a fallacy in evidentiary terms. Muckraking is the younger profession and journalists are well advised to take heed of the extreme caution, amounting to an exclusionary rule, with which the older profession of lawyering treats prior consistent statements. Contrary to common sense, when a witness repeats an accusation he has made on a prior occasion, this does not increase the credibility of his claim. That witness could be consistently lying or consistently telling the truth. Mere consistency does not tell whether the scale is rigged or not.

Of course, if a witness is accused of recently fabricating a version, the fact that he has long been making the same claim is a fair rebuttal of this imputation. An account given immediately after an event also carries a certain weight. But judges and juries are usually spared the information that a complainant has, thereafter, been repeating themselves because logically, self-corroboration, sheds no light on credibility.

Strange as it may seem at first glance, while prior consistency does not augment credibility, prior inconsistency does validly impeach or discredit a witness. There certainly were material and conspicuous inconsistencies in Jackie’s story, if only Erdely had bothered to find them.

Demeanour is another notoriously unreliable measure of a person’s honesty. Studies show that Erdely had odds little better than flipping a coin in discerning whether Jackie was telling the truth from her ‘confidence’. Confidence means as little as being coy. Indeed, Erdely reads confidence as increasing Jackie’s credibility during some of their interactions, while Jackie’s fear and reticence during other encounters does exactly the same thing for Erdely. This underscores the unreliable nature of demeanour as an objective sign of truth-telling.

Investigative reporting is an inherently contentious business. If the target is important and well resourced, if the stakes are high, there will be significant, coordinated and biased blowback. The greater the impact of a piece, the shriller the shrieks of indignation, the sterner the letters from lawyers and the greater sanctimony from scrutinising commentators. When that kind of heat arrives, it is far better to have forensically examined the facts of your story yourself before publication than to have your adversaries do so afterwards. The Coll review of the retracted Rolling Stone report has supplied useful new guidelines by which the dreaded recantation phone call may be avoided. It turns out that old law books on evidence may be consulted too for additional, cross-over insights on how (not) to assess the credibility of a source.

Heinrich Böhmke is a media and legal trainer and cross-examination expert with the African Network of Centers of Investigative Reporting (ANCIR).