September 18, 2018

Advancing communications measurement and evaluation

Craig Carroll, Ph.D. Interview: Greatness Does Not Occur Without a Measurement Mindset

Photo by Alannah Avelin

Craig Carroll, Ph.D. is the executive director of the OCR Network, and teaches in global corporate communications graduate programs around the world (Spain, Denmark, Singapore, New York). Courses include corporate communications research methods, CSR communications, and corporate reputation-building. He has previously taught at the University of Southern California’s Annenberg School and the University of North Carolina, and was selected as PR Week’s Educator of the Year in 2008. He has published three books on corporate reputation, including most recently the SAGE Encyclopedia of Corporate Reputation.

The Measurement Standard —Welcome to our Measurement Life Interview, Dr. Carroll! Let’s learn a little about you: tell us a bit about your path to where you are now, and your academic career. How did you become interested in measurement and evaluation, and what course of study did you follow?

I’m a product of the first generation of management books (In Search of Excellence, Leaders, and the various Best Places to Work). Yes, I read those in high school. For college, I sought out an undergraduate major where those were the types of ideas covered in the curriculum. I found that in the major of “organizational communication.” For my MA, I continued with organizational communication, with an emphasis in organizational development. I took an additional 18 graduate hours in business covering operations, accounting, strategy, and finance because I wanted to be prepared whichever direction I went for a PhD: business or communication.

I became interested in measurement and evaluation after my master’s degree, when I learned that having the right data + interpretation enabled me to participate in discussions and decisions, in the absence of the gray hair and 25 years of experience that everyone else in the room had.

My PhD is an interdisciplinary one from the University of Texas at Austin that included communications (journalism, organizational, political, rhetoric, advertising) and business, with 30 hours of coursework in research methods (quantitative, qualitative, historical, rhetorical & futures studies). A key aspect of my program was “intellectual entrepreneurship,” an agenda created at UT Austin focused on real-world application + service to society. Besides the interdisciplinary focus, what made my program different was my dissertation committee which consisted of *five* full professors (one was enough).

Now, I’m working with CCOs and EVPs of corporate affairs in developing talent development plans through futures work, building business cases for communications investments, getting more out of their agency relationships, auditing/evaluating company’s research, developing their own in-house tools and resources entrepreneurially, helping them to improve relationships with the C-suite, their boards, and each other. I also work with Marshall Goldsmith (#1 NYT and WSJ best-selling author and world’s #1 executive leadership coach, per Thinkers 50, Inc. and Fast Company) as a part of his “New York 50.”

TMS—What would you recommend for today’s students who are interested in communications? And, do you have any suggestions for ongoing ed for mid- to senior-career level practitioners?

For students interested in communications, don’t double major. Get your bachelor’s the simplest, cleanest, fastest way possible. Apply the extra hours to a graduate program. If you do an interdisciplinary program, pick a discipline as your home base and organize around it. If you have room for electives—go broad: psychology, sociology, economics, history, literature, rhetoric, art or music history, and a variety of writing courses—including those outside of comms. And most importantly, courses in research and entrepreneurship!

For mid-level, remember that what got you here won’t get you there! The higher you move up, the less personal/professional expertise matters, and the more talent development, listening, diplomacy, awareness (self-, organizational-, and environmental- awareness), and cross-functional collaboration matter.

For senior level, hire people better than you—and get out of the office. Engage. Do not become insular. Same thing: what got you here, won’t get you there.

In all of these areas, build and track your own checklists, metrics, and scorecards, and gamify the development of your own keystone habits + routines for professional growth.

TMS—What’s special about measurement and evaluation—why should communicators adopt a measurement mindset?

Visions are not materialized, strategies are not realized, goals are not accomplished, and greatness does not occur—without a measurement mindset.

Communicators should adopt a measurement mindset because it’s the only way to turn fuzzy, abstract ideas into concrete, observable, and mutually-agreed upon social/organizational realities—whether it is vision, a social purpose, corporate character, a desired legacy, commitment, loyalty, trust, or reputation building, rectification, or repair.

Visions are not materialized, strategies are not realized, goals are not accomplished, and greatness does not occur—without a measurement mindset. All of these are a matter of creating definitions, criteria, and measurements people accept, so that when you look at the same thing, you see the same thing. If you don’t know how to measure, track, and tweak performance, your days are numbered.

TMS—One of your areas of expertise is corporate reputation. Is this an easier or harder aspect of communications and public relations to measure? Why?

What makes corporate reputation a harder aspect to measure and evaluate is that people treat it as if it were a singular dimension (high/low, favorable/unfavorable). Reputation has multiple dimensions (prominence/top-of-mind awareness; public esteem (like, trust, respect, admiration); properties (leadership, workplace culture, products/services, and social, environmental, and financial performance). To this list, we can also add communications (truth in advertising or self-promotion). They also have what I call ‘plexes’: linkages to public issues, crises/scandals, rogue employees, “bad apples” in the industry, and their stances on social issues. I hope you appreciate the Ps to keep these memorable.

At the end of the day, what makes reputation difficult for communications is not the measurement, but getting the concurrence on what it is, why it matters (the precise causes and consequences for the company, how to measure it, and then how to link it to its causes and consequences). Companies not breaking reputation down into the above dimensions (the 4Ps), are getting insufficient value out of their reputation research.

Meaning, they’re not able to:

  1. Build a meaningful/valuable business case for investing in communications or reputation-building,
  2. Offer insights on whether to cut the red wire or green wire in real-time,
  3. Draw meaningful conclusions about causes and consequences of reputation change
  4. Facilitate post-crisis reflections and learning, or
  5. Properly value reputation on the books.

Too often, companies don’t find the effects on financial performance because they are looking at the total reputation score, when they should be looking at the isolated impacts of each of these 4Ps.

Then, it’s also difficult because useful data entails asking the right questions, of the right people, in the right way, to draw the right interpretations. And, doing so in an unobtrusive way so that the very act of engaging in reputation research does not change or alter your reputation!

TMS—Adoption of real measurement in public relations work is something that is often discussed, but companies still seem to struggle with implementing real and permanent ways to measure their communications efforts. Why do you think this is, and is it changing?

Companies are still struggling with measurement for several reasons, and it’s not changing for the better for several reasons.

First, companies face a constant tension between doing the work, and documenting, measuring, and accounting for the work—and they’re not given a plan of action to help them deal with this. The speed of change, the tyranny of the now, and the sense of urgency companies face means that communicators often choose the short-term win over the long-term gains generated by measurement. Communicators need to have a measurement strategy for overcoming these issues while enabling more effective execution in real time rather than after the game has been played.

The speed of change, the tyranny of the now, and the sense of urgency companies face means that communicators often choose the short-term win over the long-term gains generated by measurement.

Second, measurement is touted as the solution to everything. Measure this, measure that, measure everything. It’s often talked about unreflectively, in excess, and without a sufficient plan for connecting measurement to a strategy to support the business or practice, sustainably, in a way that generates ongoing value creation.

Third, communicators are not given a sufficient path for connecting all of the elements together that need to be measured. A company’s strategy needs to align and tie together the company’s mission, vision, current and desired performance, operating constraints, and resources and capabilities—and the measurement must also tie all these together. This is the only way that communicators are going to be able to guide and lead their teams and companies.

Fourth, too much of a focus is put on measurement after-the-fact rather than measurement and reflection on the measurements in real time.  Waiting until after a campaign is done to tout communication’s impact on sales or company behavior is no different than Monday morning quarterbacking. You might have all the statistics, but it’s just an argument, an opinion or an interpretation. Anyone with access to the numbers can come up with their own story.

The real value of measurement is for when you are on the field of play.  If you’re going to make the argument that you’re making an impact then you need to review the measurements in real-time, to learn, grow, tweak and make refinements.

Waiting until after a campaign is done to tout communication’s impact on sales or company behavior is no different than Monday morning quarterbacking.

If you review your game footage on a regular basis, then you can make the fine tuning in performance. If you’re not tracking and using the numbers in real-time to checklist your behaviors and activities, to improve your personal or team’s day-to-day performance and your and their (keystone) habits and routines, you’re just another armchair quarterback.

If you are not hitting your goal, you need to know whether it is a flaw in the strategic plan’s contents or in the execution. There’s a big difference between how you handle them. A breakdown in the plan is when the strategies and tactics are being implemented, but they are not effective. A breakdown in execution is when you fail to fully implement the plan tactics in the first place. Too often this is actually the case—the plan isn’t being implemented sufficiently, and there are no metrics or measurements to document the plan’s lack of follow through, so they change the plan!

This brings us to the last reason communicators struggle with measurement: fear! With measurement, there is nowhere to hide the lack of performance or sufficient follow through. Sometimes, we’re so focused on perfection that we don’t consider benefits of incremental progress.

TMS—Media relations work is changing as companies are finding success with shared and owned media outlets, rather than having such a heavy dependency on earned media. Is this good or bad for the practice of public relations?

Well, I guess it’s bad for public relations if you only define public relations in terms of earned media. My response is, “ehh, so what!” There are more aspects to public relations than just earned media. Public relations needs to be concerned with all the ways that organizations build, broker, and moderate relationships with their constituents, audiences, markets, publics, users, and stakeholders—and the different stages along the journey—from first awareness to interest to commitment to loyalty—even dissolution. In other words, public relations has much bigger concerns than (just) earned media.

TMS—What are your favorite measurement tools or projects?

My favorite measurement tools and projects are my own. First, I was a co-developer of the latest versions of DICTION software. DICTION analyzes language patterns to uncover persuasive or strategic intent. It can be used for to provide feedback on message strategy and desired/actual outcomes coming from these messages.

Then, second, I use Google worksheets and Microsoft Excel for setting up checklists, benchmarks and score cards for tracking my performance and that of my team on hourly, daily, weekly, monthly, and quarterly bases. The focus of these is to develop plans and pathways for the development of desired skills, keystone habits and routines, and eliminating counter-productive ones, that just aren’t going to occur without regular, intentional attention to the time put in.

TMS—Can you provide our readers with any examples you’ve seen of how a company used measurement or evaluation to significantly improve a program or to successfully demonstrate business results?

Sure, occasionally, I host webinars and conference calls for chief communication officers on topics such as this. Recently, I hosted with one Bowen Craggs out of London and Nestle. Bowen Craggs has an index called “Explain Yourself” which examines, rates and ranks companies for their explanations for who they are, what they do, and how they talk to the public about public crises or scandals. We (my research firm, “OCR” and Bowen Craggs) are working on a US-based version. People would have to join us on the conference calls to get the exact results.

TMS—Where are measurement and evaluation going? What great strides do you see in your crystal ball?

Hopefully, I hit on this. Not where measurement and evaluation is currently going, but where it needs to go.

Where I am interested in taking measurement and evaluation discussions is into how to use measurement and evaluation for advancing strategy, execution, and value creation. Figuring out from concept to execution what it is that a communication’s team wants to do to support and enable the business, tie all the pieces together (strategy, goals, mission, purpose, vision, constraints, resources, and current and desired future performance), develop the metrics and tracking mechanisms, build in a system for daily updating and reflection, and then score the teams daily, weekly, quarterly on the conduct of these activities, the relative value of these activities, and then grade them on their incremental progress in a fun and easy way. In other words, tying measurement into personal and team talent development through gamification!

TMS—If you could invent one magical measurement or evaluation tool to accomplish anything, what would it be?

Who’s to say it’s not already in development? Wink, wink.

How about two? See my response to the last question. That!

Second is a web-based application that publicly mines company promises out of their quarterly and annual reports and their CSR reports and produces a time-activated list that makes the assertion public when it is due to be accomplished.

Then, through crowd-sourcing, the public could document and verify through URL and other forms of certification (audio, video, multimedia testimonials, and public documents) whether the company actually did what they said they were going to do, and then grade the company on delivering on their promises.

Companies would be scored on the size of the aspiration, the length of time it took to accomplish, when they accomplished it (and whether they were on time, late, or simply abandoned it),  how important the activity is or was to society, and then what the impact was. Companies would of course be able to respond to any posts there.

Jennifer Zingsheim Phillips

Jennifer Zingsheim Phillips

Jennifer Zingsheim Phillips is the Director of Marketing Communications for CARMA. She is also the founder of 4L Strategies, and has worked in communications and public affairs for more than 20 years. Her background includes work in politics, government, lobbying, public affairs PR, content creation, digital and social communications, and media analysis.
Jennifer Zingsheim Phillips
728 Ad Block

Related posts

728 Ad Block