To return to the current issue's contents page, click here.

To return to the contents page of the issue that this article appeared in, click here.

Comments Please!
Send us your thoughts on this article and we will post them in our Comments section.

Ask the experts ...

What Are the Most Common
Measurement Mistakes?

The Measurement Standard thanks the esteemed members of the IPR Commission on Measurement and Evaluation for their time and effort. (Read their in-depth profiles at www.instituteforpr.com. Katie Delahaye Paine, publisher of The Measurement Standard, is Chair of the IPR Commission on Measurement and Evaluation.)

We enjoy hearing from any readers who wish to provide their own answers or comment on the question at hand. Please write to The Editor.

The Question:

We all know that our measurement efforts are not perfect. It seems we rarely have just the right tools for the job, and we rarely get to use those tools to their full advantage. In fact, too often the techniques we use are only our best approximation of what they could and should be. Still, we do the best we can, and we improve where we can. One way to improve is to be sure we are not making any really big mistakes. And to help us out there, we Asked the Experts:

“In your experience, what types of research and/or evaluation are most frequently abused/misused? And how would you fix the problem?”

The Answers

From: Kathryn Collins
Director, Communications Research, General Motors Corporation

In my experience on the client side, I’d have to say the most frequent problem with measurement and evaluation is complexity. The more complex the statistical design, the higher the probability of misunderstanding and miscommunication downstream. Whenever possible, I’ve tried to simplify the analyses and provide recommended actions. The combination seems to overcome the confusion and potential misuse of the statistics later.

From: Jack Felton
IPR President and CEO

I think most professionals use quick surveys and informal research instead of doing basic PR research themselves or hiring someone else to do it. They rely on secondary research from marketing studies or publications they read. It has been my experience that few do their own surveys and those who do, do so infrequently. I’m afraid too many still try to solve their problems by the good old “seat of the pants” method. They can’t really evaluate programs, because they never bothered to develop specific objectives first.

From: Mark Weiner
CEO, Delahaye Medialink

I’ll share what I learned from a mentor, Kevin Clancy, who taught me that the most abused form of research is the focus group. People tend to find them attractive because they are relatively inexpensive, they are a useful diversion, and they seem to provide a genuine voice to the real feelings of real people.

These points may be true. But focus groups are only a first step in a serious research process, not the only step. Unfortunately and too often, focus group research is not married to the more quantifiable research tools that offer more demonstrable validity and reliability. The danger comes when people try to project focus group results to a larger population.

Focus groups are not comprised of either representative or projectable samples. That a company would make a serious commitment of marketing resources based on the opinions of a small group of people with time on their hands and who happen to be wandering through the mall on a particular day is insanity. Better to ask your mom: It’s cheaper than a focus group and at least you know who you’re talking to.

From: John Gilfeather
Vice Chairman, Roper Starch Worldwide

I agree with Mark (and our mutual friend Kevin Clancy) on this. Focus groups are great when they are used properly. Qualitative insights are sometimes more important than statistics. Having a feel for a market can be more important than an array of facts. But, having said that, focus groups are abused. There are too many times focus groups are done unnecessarily: the knowledge already exists, but nobody bothered to look for it; a bunch of people wanted to get out of the office; or people want to view real live customers, as if they were zoo attractions.

On this last point, I think it is frightening that the only way marketers see “real customers” is in focus groups. I have been in situations where clients were disappointed because groups were not “entertaining” or because alcohol was not available to the observers. I have also had to go behind the mirror to quiet down the party going on there.

Focus groups with clear and achievable objectives that are professionally moderated can be extremely valuable. Focus groups done otherwise are a waste of money.

From: David Michaelson
President, David Michaelson & Co., LLC

Content analysis is consistently one of the most abused of all the evaluation and measurement methods. It is abused in a number of different ways:

• Selective use of sources that limit the coverage included in the analysis
• Analysis that does not consider the intent of the program or the message strategy
• Reliance on column inches or other volumetrics as a measure, without consideration of the tonality or inclusion of key messages
• Use of advertising value equivalencies (AVEs), particularly those that rely on multiples and those that based AVE on published advertising rates
• Non-inclusion of the physical placement of the article (page, section, day of week, etc.) as a factor in the evaluation
• Use of a “clip book” as the extent of the analysis

These factors have hurt the acceptance of content analysis as a central measurement and evaluation tool. Content analysis can offer significant insights on performance and provide diagnostics that can improve program performance. However, it has to be done right.

From: Don W. Stacks
Program Director, Professor, Advertising and PR Department, University of Miami

From my experience, the most misused type of research is survey research that claims to have established a cause and effect relationship. Often the results are blatantly stated as “x caused y,” when we know that the only method that can establish causation is experimental methods, with all its caveats and limited generalizability.

What burns me more, however, is the argument that cross-lagged correlations can be used “to establish a causal relationship” between variables in a survey! (And I’ve seen some very well-respected research companies and public relations firms make such statistical statements.)

You know you need to measure your results, but chances are there’s never been enough money in your budget for evaluation. Until now.
KDPaine & Partners’ new Do-It-Yourself Dashboard system combines a Web-based application with professional consulting to enable PR professionals to customize their own PR dashboards. Look here for more information.

1. You’ll learn how to use hard numbers to prove the results of your PR efforts. (Plus, it's free.)

2. You’ll learn which are the right vendors for your measurement projects. (Yes, it's free.)

3. You’ll learn how to design your program right from the start to be easily measureable. (Plus, yes, it's free.)

Click here to
get your free
subscription now!

51c Durham Point Road, Durham, NH 03824
603-868-1550 fax: 603-868-3346 www.measuresofsuccess.com