Weighing Consensus – Building Truth on Wikipedia

In a recent piece in the Chronicle of Higher Education's “Chronicle Review” section, Timothy Messer-Kruse criticizes the editing practices of the Wikipedia community. He describes his attempt to correct what he understood to be a factual error in Wikipedia's article on the Haymarket affair, and argues that his experience demonstrates that Wikipedia limits the ability of expert editors, such as himself, to correct factual errors on the site. While Dr. Messer-Kruse believes his experience demonstrates Wikipedia's lack of respect for scholars, I believe it actually demonstrates that Wikipedia holds a deep respect for a collaborative scholarly process that is collectively more capable of producing “truth” than any individual scholar. 

Wikipedia's privileging of the collaborative scholarly process has practical implications for how scholars should, and should not, interact with Wikipedia. Academic Wikipedia editors might have more satisfying Wikipedia editing experiences in the future if they respect this fact.

To understand how and why Wikipedia functions the way it does, we must first understand the day-to-day realities of Wikipedia's editing process. Because they have the responsibility of securing the free encyclopedia against vandals and other bad actors, editors are always on the lookout for certain patterns that, for them, indicate likely vandalism or mischief. For academics, the best analogy might be the way that we scan student papers for patterns that indicate likely plagiarism. This sort of rough pattern recognition is deeply imperfect, and in the case of Messer-Kruse's edits, Wikipedians suffer from a false positive. Nevertheless, just as teachers use rough patterns when scanning giant stacks of student assignments, so too Wikipedia editors have a clear need to be able to quickly detect likely bad actors.

If we look at Messer-Kruse's first interaction with the Wikipedia community, we can see some of the patterns that likely flagged him (incorrectly) as what Wikipedian's sometimes call a “POV pusher.” That is to say, a person with an ax to grind, looking to utilize Wikipedia as a free publishing platform for their own particular pet theories. He starts his engagement with a post to the article's talk page (a special Wikipedia page that permits editors to discuss the process of creating and revising a particular article), writing:

The line in the entry that reads: “The prosecution, led by Julius Grinnell, did not offer evidence connecting any of the defendants with the bombing…” is inaccurate. The prosecution introduced much evidence linking several of the defendants to the manufacture of the bomb, the distribution of the bombs, and an alleged plot to attack the police on the evening of Tuesday, May 4. An eye-witness was put on the stand who claimed to have seen Spies light the fuse of the bomb. Police officers testified that Fielden returned their fire with his revolver. Now these witnesses and this evidence may be disputed, but it is historically wrong to claim it was not introduced. For more specific information, see http://blogs.bgsu.edu/haymarket/myth-2-no-evidence/ (http://en.wikipedia.org/w/index.php?title=Talk:Haymarket_affair&diff=prev&oldid=265725190)

By starting with a post to the talk page, Messer-Kruse follows good Wikipedia etiquette, which encourages new editors to discuss substantial changes they wish to make to pages before making them. (Those who wish to review the full record of Dr. Messer-Kruse's Wikipedia activity may do so, here.)

However, in crafting this talk message, Messer-Kruse has unintentionally engaged in rhetorical patterns that flag him as a potential bad actor in the eyes of experienced Wikipedians. His most significant error is citing a self-published source, his Bowling Green State University blog, in support of his desired changes to the article. This is, as several Wikipedia editors quickly point out, in violation of Wikipedia's reliable source guidelines.

In Messer-Kruse's defense, the tone taken by some editors in these early exchanges represents a serious misstep on their part. They tended to engage in what is known as “wiki-lawyering,” simply spitting the Wiki-shorthand code for the policy he has violated (WP:RS) at him, with little attempt to explain why he has made an error, and no attempt to offer constructive ways in which a compromise solution might be reached. These editors have since been called on the carpet for being unnecessarily hostile to newcomers, or “Biting the Newbies” in Wikipedia-speak, on the article's talk page.

Messer-Kruse, for his part, does not seem to absorb the reason why his blog is unacceptable as a source. After being directed to the reliable source policy by one editor, he retorts, “I have provided reliable sources. See my discussion of the McCormick's strike above in which I cite the primary sources for this information. By what standard are you claiming that http://blogs.bgsu.edu/haymarket/myth-2-no-evidence/ is not a 'reliable source.' It clearly cites primary sources in its rebutal of this myth. Perhaps its [sic] not 'reliable' sources you want but ideologically comfortable ones” (http://en.wikipedia.org/w/index.php?title=Talk:Haymarket_affair&diff=prev&oldid=265740457).

What Messer-Kruse is missing is how the reliable source policy allows Wikipedia to use the larger scholarly process of peer review for its own benefit. By preventing the use of self-published sources, and preferring secondary sources to primary sources, Wikipedia attempts to ensure that information has been subjected to the most vigorous review possible by scholars before being included in the encyclopedia. Does Messer-Kruse really believe that we should abandon this process, and simply allow any individual scholar to make novel claims about truth, regardless of their ability to convince scholarly peers? It is not some faceless herd of editors that Wikipedia defers to when evaluating truth-claims, it is the scholarly process itself. Even now, discussion is unfolding on the Haymarket affair article talk page concerning the larger scholarly response to Messer-Kruse's book. At issue is whether or not, in the eyes of the experts, this very recent book has indeed significantly revised our understanding of the Haymarket affair.

Wikipedia's policies here seem to have frustrated an attempt to add well-researched points to the encyclopedia, which is unfortunate. However, it is important to understand that Wikipedia editors are, every day, confronted by vast numbers of self-styled experts, many claiming academic credentials, referring to a blog or other self-published source that purports to upend this field or that based on a novel review of primary evidence. Climate science, evolutionary biology, and “western medicine,” are all particularly common targets, though I have also witnessed claims to such unlikely discoveries as a grand unified field theory. While Messer-Kruse's claims are not outrageous, his use of a self-published source, and claims to a unique interpretation of historical events flag him in the eyes of Wikipedia editors as a potentially disruptive editor. They thus use the reliable source policy to defer the responsibility for deciding whether or not his claims are true to the larger process of scholarly peer review.

This deference may indeed, as Messer-Kruse points out, render Wikipedia resistant to change. It can also, as I have argued in my previous case study of Wikipedia's coverage of the Gaza conflict (see chapter 6 here for more detail), privilege points of view with greater access to the means of producing “reliable sources.” This is an important potential problem for Wikipedia. It is an even more critical problem for a web-using public that too often allows Wikipedia to serve as their primary, or only, source of information on a given topic. More must be done to ensure the greater visibility of minority opinions on the web, and to prevent so-called “filter bubble” effects that may prevent web users from consuming a diverse set of information sources.

However, I don't think that Messer-Kruse's critique of the “undue weight” policy of Wikipedia, which holds that Wikipedia should base its coverage on academic consensus on a given topic, is the best way of correcting for this potential problem. It is interesting to note that Messer-Kruse himself, in discussing a related edit to the Haymarket affair article, makes a sort of “undue weight” argument of his own. He argues that the article's casualty count for the McCormick riot (an event that would help set the stage for the later events at the Haymarket) should be changed because, “The claim that six men were killed at the McCormick riot is inaccurate. This claim comes from the report written for the anarchist newspaper by August Spies. Chicago cornorer's records and all the other daily newspapers finally settled on two deaths as the correct number” (http://en.wikipedia.org/w/index.php?title=Talk:Haymarket_affair&diff=prev&oldid=265729292). Here, Messer-Kruse is effectively arguing for the exclusion of a fringe opinion, in deference to the weight of consensus found in other sources.

Consensus, then, is an important mechanism by which we judge the validity of certain truth-claims. I believe that one reason academics, like Messer-Kruse, and Wikipedia editors may not see eye to eye is that they have been trained to evaluate consensus in very different ways. Academic training, especially in fields that stress the production of mongraphs like history, tends to privilege the scholar's own individual judgement of the consensus of evidence. Wikipedians, by necessity of the situation Wikipedia finds itself in, understands consensus to be an ongoing process involving a vast number of both scholarly and non-scholarly actors. Rather than asking Wikipedia to hew closer to any one academic's evaluation of “truth,” I would posit that we can more readily improve Wikipedia's accuracy and respect for evidence by engaging with and respecting this ongoing process. By offering our scholarly findings to the Wikipedia community as peers in a larger process of negotiating the truth, we have the best chance of helping to build a Wikipedia that truly reflects the fullest and best picture possible of the always fraught and diverse process of establishing what we know.

 

css.php