Advancing communications measurement and evaluation

Why Metrics Don’t Add Up To Evaluation For Public Relations

The Kanizsa illusion made of kiwi fruit.

This is the first of a series of articles by Jim Macnamara on the limitations of quantitative metrics in public relations measurement and evaluation. Coming up next time: “Evaluation Needs More than a One Trick Survey Monkey.”

The world of measurement is experiencing a heightened level of excitement and expectation as ‘analytics’ and ‘metrics’ become the new buzzwords. Furthermore, with digital access to ‘Big Data’, famine has turned to feast in terms of data and numbers. After years of trying to convince management with arbitrary and hypothetical numbers such as impressions and AVEs, digital media have come along with their facility to generate large volumes of metrics – many of them automatically. Now we can give management, who are mostly STEM-orientated (Science, Technology, Engineering and Mathematics), an array of numbers representing views, likes, followers, shares, clickthroughs, downloads, share of voice, and a zillion other things.

But I want to invite readers to participate in a very short exercise that demonstrates an important principle of evaluation. It takes just 60 seconds and comprises three simple steps.

  1. Think of the relationship that is the most important in your life – your most ‘significant other’, such as your husband, wife, partner, child, lover, or best friend. Now put a value on that relationship in either financial terms (e.g., a dollar value) or as a numeric score out of, say, 10.
  1. Second, write down the scientific formula or data that you used to calculate that number.
  1. Third, you must be prepared to show that valuation to your ‘significant other’ and he or she must be convinced it is valid and a reliable evaluation of what they mean to you.

When I presented this exercise to the 2014 International Summit on Measurement in Amsterdam, nervous giggling broke out when I explained step one. By the time I reached step three, most audience members were squirming in their seats, imagining themselves trying to justify whatever number or rating they came up.

This exercise is not a silly or even an unreasonable request for measurement advocates who believe that numbers based on scientific equations and mathematical calculations are the way to describe value. After all, the research question in this exercise asks us to estimate the value of the relationship with which we are most familiar with in the world. If we cannot put a numeric value on relationships that we know best, how can we expect to put a reliable numeric value on relationships with relative strangers such as external stakeholders?

Nevertheless, some might argue that PR cannot be compared with intimate personal relationships. So I gave the Summit another evaluation exercise. What does a home mean to you (think of the home you currently live in or one you and your family want to buy)? Metrics can tell us the median price of homes and recent sales in an area, and valuers can crunch data to produce a number. But is this what you would pay or sell a home for? Is this what your family thinks it is worth? Anyone who has bought or sold a home knows that there is much more involved that statistics.

This exercise brings to the fore a value system that is vitally important in human society beyond the scientifically-based domains of mathematics, physics, chemistry, etcetera, and their derivative fields of practice such as economics and accountancy.

Quantitative methodology dominates the research landscape generally and is emphasized in the dominant paradigm of PR as well corporate and marketing communication. But when we look at human communication and practices such as public relations, we need to recognize that the outtakes and outcomes sought are very often:

  • Awareness;
  • Perceptions, such as reputation and brand attributes;
  • Attitudes, such as goodwill, support, or intention to buy;
  • Opinion, which involves attitudes but is often publicly expressed, whereas attitudes may remain latent;
  • Engagement;
  • Trust;
  • Loyalty;
  • Relationships; as well as
  • Behaviour, such as buying, joining, voting, getting fit, advocacy, and so on.

While behavioural change is an objective of some communication, most of these outtakes and outcomes are human interpretations and feelings, not independently and objectively observable phenomena. Even observable behaviours are heavily influenced by interpretation and affective as well as cognitive processing. That is to say, interpretations and feelings are based on emotion as well as rational logical reasoning. They are subjective, not objective. They are socially, culturally and contextually constructed, not scientific facts. They are infinitely variable and diverse, not stable phenomena governed by natural laws. They are humanistic, not scientific.

As such, these human characteristics do not yield easily to numerical quantification and are simplistically represented in arbitrary scales and ratings – whether they use nominal, ordinal, interval, or ratio numbers expressing counts, levels, percentages, degrees, intervals or other factors. Yet, we try to measure and evaluate these outtakes and outcomes using ‘scientific methods’ and quantitative data.

As I told the 2014 European Summit on Measurement, in focussing predominantly on quantitative research and searching for numbers to define our work and our value, we are in many cases trying to measure air with a ruler.

This is not an argument against science or against empirical quantitative research. Far from it. Few of us would want to live without medical science, computer science, agricultural science and so on. Much of our world can be understood and made better through science. But not all of it.

While we have benefited enormously from a focus on science to understand the world, with its emphasis on empirical data, rational logical reasoning, and quantitative methodology, as John Durham Peters says numbers have “a serene indifference to the world of human things”. What he is referring to is the forgotten humanities and humanistic thinking, the inductive and non-rational, the emotional side of human beings that influences much of what happens in our world, and can only be understood through interpretative qualitative research methods such as in-depth interviewing, detailed textual, narrative and discourse analysis (not just counts of key words), ethnography (close observation), and ethnomethodology (conversation analysis).

Being a pragmatist, I support the drive for reliable quantitative data to measure PR and corporate communication, but I advocate mixed method research, incorporating as often as possible qualitative techniques and recognition of the humanistic as well as the scientific. Value itself is ultimately a perception – not a scientifically deduced fact.

###  (thanks to Mighty Optical Illusions for the illustration, a example of the Kanizsa Illusion)

Jim Macnamara

Jim Macnamara

Jim Macnamara PhD, FAMI, CPM, FAMEC, FPRIA is Professor of Public Communication at the University of Technology Sydney, a position he took up in 2007 after a 30-year career working in journalism, PR and media research which culminated in selling the CARMA Asia Pacific franchise that he founded to iSentia (formerly Media Monitors) in 2006. He is the author of 15 books, including his latest, Organizational Listening: The Missing Essential in Public Communication (Peter Lang, 2015), as well as Public Relations Theories, Practices, Critiques (Pearson, 2012); The 21st Century Media (R)evolution: Emergent Communication Practices (Peter Lang, New York, 2010, 2014); and Journalism and PR: Unpacking ‘Spin’, Stereotypes and Media Myths (Peter Lang, New York, 2014).
Jim Macnamara
728 Ad Block

Related posts

1 Comment

  1. Brendan Cooper

    It’s the ‘what is the ROI of your granny’ argument isn’t it? In terms of constituent parts, granny is worth just over $160 (according to the DataGenetics blog, see http://www.datagenetics.com/blog/april12011/) but that’s not the same as qualitative ‘worth’. However, I’d argue that a lot of the outcomes that you say aren’t measurable, actually you can gauge through online measurement. Engagement, for example, is expressed in the Facebook dashboard and through sharing stats for Twitter (retweets, replies and so on). You can get an idea of awareness simply by how many conversations are out there regarding your company, your brands, your people and so on. Even perceptions and attitudes can be identified when you look at the totality of conversation and start pulling out topics and themes. I think perhaps a few years ago, this would be considered just the reactions of a subset of the whole audiences. But today, I’d say the online reaction pretty much mirrors the offline, so it’s valid to use online metrics as an indicator of PR success. And as we become more immersed in our online worlds, we’ll be able to get more insight.

Comments are closed.

728 Ad Block