Advancing communications measurement and evaluation

A Trial Run of the New AMEC Integrated Evaluation Framework, and 7 Tips on How to Use It

AMEC-framework-question

I’ve just spent the better part of a day trying to navigate through the hottest thing in measurement this month: the new AMEC Integrated Evaluation Framework, unveiled with great fanfare at AMEC’s International Summit in London last month. It’s an interactive version of the original AMEC Valid Metrics Framework, designed to make it easier for clients to implement Barcelona Principles/standards-compliant measurement in their organizations.

The Framework at first glance

The new Framework has an online interface that features multi-colored squares. Each square requires you to provide information about your organization’s campaign or program. The 7 squares are:

  1. Objectives
  2. Inputs
  3. Activity
  4. Outputs
  5. Outtakes
  6. Outcomes
  7. Impact

When you click on each square it asks you questions like, “What are the broad objectives of your organization?” and then, “What are your communications objectives?”

“Ultimately, I love this framework, not because it is perfect, or even particularly easy to use, but because it poses the kind of questions that I’ve been answering for thirty years.”

Visually, it is certainly an enormous improvement over the old Framework (which, I will confess, I have never once been able to get a client to comprehend). And the interactive nature of this new Framework is a lot less daunting than the old PowerPoint version.

But in the end, it will be just as challenging to fill out as the earlier version, because all the problems inherent in the old Framework still exist, despite the sexy new front end. Sure, there’s a taxonomy that offers examples of the types of answers they’re looking for, but the confusion will persist.

This is because the very smart people who created the Framework actually live and breathe measurement every day of their working lives. Many if not most of the Framework’s creators primarily work with large organizations that do sophisticated PR and have at least some background in measuring results.

Unfortunately, your typical PR program is neither large nor sophisticated. Despite the impression that Samantha Jones (from Sex in the City) may have left on a generation, the vast majority of PR is for small to medium-sized businesses, restaurants, NGOs, and government agencies. And it is done by the people who show up at conferences, attend my workshops, and participate in webinars. These people, for the most part, ask questions and struggle with answers that are far more basic than those the Framework deals with. And so the new Framework is going to be difficult for many typical PR pros to use.

A test run of the new Framework

To try out the new Framework I completed it using as an example a PR campaign I recently helped a client measure. At first, I filled it out using the responses that members of the client’s PR team gave me at our first meeting (click the image below for a larger view):

PP-AMEC-Framework-Visual-1

As you can see, our initial meeting left large gaps between what the client’s team saw as organizational objectives and the goals of the particular event. There was considerable confusion between inputs, activities, and outputs, and no clear connection between the organization’s objectives and what the actual impact would be.

The good news is that, during the course of working with this client, we did eventually identify target audiences and objectives. We were able to connect the dots between the communications activity and the ultimate impact. Thanks to this and other work, that client now has a useful and convenient dashboard to update management on what is working and what is not working. It just took a bit of time to sort out the definition of “working” and “not working.”

So I went back to the Framework and filled it out again, this second time with the actual information I used to build the client’s dashboard.

7 tips for using the new AMEC Integrated Evaluation Framework

After a long day of checking the taxonomy and making sure I put everything in the right boxes, I’ve learned a lot about how to use the Framework. Here are seven tips for anyone trying to do the same:

#1. Before you go to Google and type in “AMEC Framework,” do your homework.

What all measurement requires, regardless of what framework, tool, or anything else you employ, is a thorough understanding of the business goals of the organization and how PR affects them, e.g., “What is the mission?” and “How does it make money?” and “What is the perceived role of PR in that process?”

For agency folks, this is your biggest weakness and one reason why PR gets no respect when budgets get tight. If you don’t have an agreed-upon definition of how PR contributes to the success of the organization, then you’ll never get past square one (literally).

So meet with your boss, your boss’s boss, or whoever is asking, “What have you done for me lately?” and make sure you have agreement on what the organizational goals are and how communications contributes to them. Click here for a recipe that will walk you through the process. When you are done, you should have a list of organizational goals, and a list of communications goals. (You are going to need them later.)

#2. Bake cookies to find your “Inputs”

Some of the answers to the Framework questions about target audiences and strategy may lie in other departments. Depending on the size of your organization, information on specific target audiences/personas or even overall strategy may well lie in sales, marketing, or customer intelligence. If you’re a nonprofit, answers may reside in membership or development. In government agencies, there may be a data center or a committee that has the answers.

Visit whatever departments hold the clues and bring treats. I’ve always gotten much more information with chocolate chip cookies than with an email. Depending on people’s stress level, a good Scotch whisky can also be an excellent persuader.

#3. Look at the communications budget for “Activities”

What you’re really doing with this Framework (and any measurement program, for that matter) is determining which efforts are worthwhile and which are not. The fundamental concept is “worth,” which implies a financial or resource commitment. So rather than just a laundry list of activities, which could quickly become a nightmare of random metrics, only list those activities that require either a significant amount of budget, time, or resources.  (You can get to the others later.)

To help you simplify the process: Since the Conclave on Social Media Measurement Standards has determined that you “earn” a share, I wouldn’t even bother with the “S” column; just include any shared data under “earned.” Also, note that “earned” doesn’t mean what you have already earned, but rather what you plan to do in terms of “earned” media, e.g., what you’re writing, the nature of the media outreach, speech writing, or anything that is going to require resources.

#4. Outputs are what you’ve checked off your to-do list

After you’ve listed all the activities, now you need to see what actually happened, i.e., Did any of that activity reach the agreed-upon target audiences? This is where you can “count” the number of media items that ran or that you “earned.” Tally up the paid media placements and anything that was shared. Add the data on clicks, time on site, or whatever metrics you’ve agreed are important from your web analytics platform. If you’re measuring events, count the number of attendees as well as anyone who used your hashtag. Whatever you do, try to avoid completely inaccurate definitions of “reach” and “impressions.”

#5. If you don’t have good survey or engagement data, skip the “Outtakes” section

Essentially, outtakes are what your target audience actually takes away from all the stuff you’ve listed in #4, above. In order to understand what a member of your audience actually “takes away,” you have to ask their opinion. In other words: Are they more aware or more likely to consider or prefer your brand? While engagement is not the same as awareness, it may be an acceptable proxy for evidence of attention on the part of your target audiences.

#6. “Outcomes” should be the same as your communications goals you listed in #1.

Go back to Section 1 and cut and paste your communications goals in the “Outcomes” section and change the tenses. For example, if the goal was “Increase preference in the new brand by 10%,” then the outcome should be “Increased preference in the new brand by 10%, as measured by pre/post testing.” If that didn’t happen, prepare a good explanation.

#7. “Impact” should be the same as the organizational goals you listed in #1.

Go back to Section 1 to copy and paste the business goals in the “Impact” section and change the tenses. For example, if your organizational goal is “Increase support for independence by 10% among Scottish women,” then your impact should be “Increased support for independence among Scottish women by 10% as measured by an average of public polls.”  If the impact is different, prepare a good explanation.

This Framework asks the right questions

Ultimately, I love this framework, not because it is perfect, or even particularly easy to use, but because it poses the kind of questions that I’ve been answering for thirty years. And I know all too well how hard many of them are to answer. If you get stuck and need a Measurement Sherpa to help you out, give me a call.

For those of you interested in frameworks I refer you to the work of Fraser Likely and the Task Force on Standardization of Communication Planning/Objective Setting and Evaluation/Measurement Models, a group that is looking at the dozens of measurement frameworks that currently exist.

(Author’s note: In the interest of transparency, I run a consulting organization that designs measurement programs, so I could conceivably profit from helping people fill out the new AMEC Integrated Evaluation Framework. I’ve also developed a different framework that I use in my work called the 6-Step System for Perfect Measurement.)

###

A version of this post originally appeared in Katie Paine’s Measurement Blog.

Katie Paine

Katie Paine

Katie Delahaye Paine, aka The Measurement Queen (@queenofmetrics), has been a pioneer in the field of measurement for three decades. She founded two measurement companies, KDPaine & Partners Inc., and The Delahaye Group. Her books, Measure What Matters (Wiley, March 2011) and Measuring Public Relationships (KDPaine & Partners, 2007) are considered must-reads for anyone tasked with measuring public relations and social media. Her latest book, written with Beth Kanter, Measuring the Networked Nonprofit: Using Data to Change the World, is the 2013 winner of the Terry McAdam Book Award.
Katie Paine
728 Ad Block

Related posts

728 Ad Block