When Research and Standards Collide
There were two papers at IPRRC 2013—one by Marianne Eisenmann and Julie O'Neal, and one by Sean Williams—that have significant impact on the Conclave's pursuit of public relations and social media measurement standards.
Eisenmann and O'Neal: Standards for Media Analysis Prove Unreliable—Or Do They?
The paper with the greatest implications for standards-setting efforts was by Marianne Eisenmann and Julie O’Neal: “Testing the reliablility of metrics proposed as standards for traditional media analysis.” The results at first appear to be unpleasant news, but are subject to a couple different interpretations.
The purpose of the research was to test the reliability of coding decisions made by human coders using the standard definitions proposed by the Institute for Public Relations in its June 2012 paper "Proposed Interim Standards for Metrics in Traditional Media Analysis." That paper specified definitions for:
- What constitutes a media “hit”
- Assessing sentiment and tone
- Gauging quality
Eisenmann and O'Neal prepared a detailed set of instructions that specified definitions for each element to be coded. The coding books were reviewed by three PR practitioners experienced in media coding and measurement, who made minor modifications.
Then the researchers collected a systematic random sample of clips about Wal-Mart between July 1, 2011 and June 30, 2012, ultimately analyzing 106 items.
Three graduate students with some experience in public relations were selected to do the actual coding and were trained for about two hours. After the training the three coders and two researchers independently coded seven stories and the results were compared to identify discrepancies. The coding book was modified accordingly. A second set of five items were also subjected to a pretest to make sure that the more subjective qualitative items were coded correctly.
After coding the entire sample of items, reliability results indicated the highest level of agreement between coders on elements like media type, prominence, and shared vs. sole mention. There was low to moderate agreement when coding for the presence of reputational messages, and lowest agreement of all was for sentiment.
One way to interpret these results might be to suggest that the standards as currently written yield unreliable results. However, the way I choose to interpret the results—from the perspective of someone who has managed and trained human coders for more than two decades and has written, reviewed, and tested thousands of coding instructions in that time—is to note that developing accurate coding is a complicated process. It requires a code book that has been tested for at least a month, and the best coders require six months of testing and training before they can be relied upon to code accurately on their own. This training was obviously not within the scope of this study.
Thus I don't recommend using the results of this study to draw too many sweeping conclusions about the fate of the standards. That having been said, I would be seriously remiss if we didn’t report these results.
So, it is possible to draw a couple different conclusions from this study:
1. The definitions of sentiment and the use of latent sentiment analysis are fundamentally flawed, or,
2. The definitions are correct, but it takes more than graduate students and the time available to do this study right.
I’m going with #2.
Williams: Social Media Influence
Sean Williams, like many of us, has long been disturbed by the various claims out there regarding influence in social media. His paper, "Is that all there is? A literature review and potential approach to measuring influence in social media," addressed the following aspects of influence:
- Influence explained via social impact and opinion leadership. This is where the most controversy is.
- Influence applied by a group on its individual members. There is general agreement in the literature on this one.
- Influence as a consequence of position in a social network. There are conflicting results here.
He conducted a massive literature search on the subject, referencing everyone from the 1955 communications theorists Elihu Katz and Paul Lazarsfeld, to Valdis Krebs, who has lead the charge into defining online influence as it relates to social network analysis.
His research dovetails nicely with the current efforts of the Conclave and WOMMA on the topic. What the standards effort has revealed is just how challenging it can be to achieve consensus on such a multi-faceted topic. One can argue that there are 7 billion influencers on the planet, or that there are just a handful—it depends on the topic or product. Williams’ paper outlines a framework for a far more exhaustive research study that could be the first step towards more definitive consensus around this subject.
Learn more at this summary of his paper on the IPR website, where you will find a link to the actual paper.
Katie Delahaye Paine is Chairman, KDPaine & Partners, (a Salience Insight company), and Chief Marketing Officer of News Group International. KDP&P delivers custom research to measure brand image, public relationships, and engagement. Katie Paine is a dynamic and experienced speaker on public relations and social media measurement. Click here for the schedule of Katie’s upcoming speaking engagements. Katie and Beth Kanter are authors of the book “Measuring the Networked Nonprofit,” to be published this year by Wiley.
The Measurement Standard is a publication of KDPaine & Partners, a company that delivers custom research to measure brand image, public relationships, and engagement. Katie Paine, Chairman of KDPaine & Partners, will be glad to talk with you about measurement for your organization.