Katie Paine’s All-Time Biggest
Measurement Mistakes
Part one of a two-part series.

by Katie Delahaye Paine

Can you top the major measurement muck-ups below? Then join our Biggest Measurement Mistake Contest! Send us your most interesting measurement mistakes and, if your misadventures are both truly instructive and entertainingly embarrassing, we’ll tell the world and send you a (not-so-standard) Measurement Standard mug.

One of our principal beliefs in the measurement industry is that you learn a lot more from measuring failure than you do from measuring success. Sure, it’s nice to have a way to show off your accomplishments, but isn’t it far more useful to identify programs that aren’t working and fix them or reallocate the resources?

And just because our Web site (www.measuresofsuccess.com) has the word “success” in it, doesn’t mean we haven’t had our share of failures. In fact, first at The Delahaye Group and now at KDPaine & Partners, we’ve been celebrating mistakes for nearly two decades. In the past, we awarded a premium parking spot to the person who made The Mistake of the Month: the error from which we could learn the most. The idea was that if we rewarded mistakes, people would be more likely to confess them—and everyone would learn from them.

(The tradition continues, but the reward has changed: As there is ample parking at our new galactic headquarters, we're thinking of changing the reward to first dibs on tying up at the dock.)

Needless to say, after some 200 Mistakes of the Month we’ve learned a few things. In the spirit of spreading the knowledge, we thought we’d share a few of our more enlightening measurement missteps with you. (Former clients and employees can stop feeling anxious—all names have been changed to protect the innocent and guilty alike.)

1. Business Objects objects
One of our early lessons in online searching taught us just how hard it is to get it right. Some ten years ago, Business Objects, a knowledge management software company in California, hired us to do a media analysis.

Our initial online searches pulled up a great many spurious references to “Small Business objects to such-and such legislation” and “Big Business objects to such-and-such.” In an attempt to make the search more accurate, we added “Inc.” to the search string, which narrowed the resulting clips down to a manageable number. There was only one petite problem: Business Objects is a French company and not an “Inc.” at all. As a result, we missed about half the articles, but didn’t find out until way too late...

There I was presenting the results when the representative from the PR agency mentioned the fact that certain key publications hadn’t even shown up on our list of media that covered Business Objects. Sacre bleu! Our search error was revealed before all, and our data and my presentation were instantly rendered meaningless...

2. A spoonful of insight makes the medicine go down
No one likes bad news, and no client likes only bad news. Some years ago, one of my clients had a particularly bad quarter, PR-wise. I wrote up the report from a highly negative perspective, leaving out the usual sweetening-of-the-bad-news-with-constructive-suggestions.

Unfortunately, my merciless version of the unvarnished truth went straight to my client’s boss. My client then spent an excruciating hour in the hot seat, after which she called to inform me that we'd lost the account. And she didn’t waste any effort sweetening the bad news...

3. Line up your (Peking) ducks in a row before you undertake your research
It was 1991 and the formation of the European Union was all over the news. We had recently hired several multilingual reader/analysts, and so naturally I decided that it would be incredibly timely and newsworthy to announce that we had “gone international” and could now offer our media analysis services in multiple languages. Well, quicker than you can say schadenfreude we had a major multi-national high-tech client sending us boxes and boxes of clips in half a dozen languages—the wrong languages...

We had French, German, Spanish, Arabic and Norwegian covered. But the assignment was to analyze the company’s image in China, Europe, and South and North America. In search of some depth for our international reader roster, we inquired of the local institutions of higher education, tried a few contacts in Boston and even cased a few ESL classes, but still came up empty-handed.

Finally, inspiration hit. We tried the local Chinese restaurant and found a bi-lingual technology expert in residence! Surprisingly, we couldn’t find a single Italian speaker working in any of the local dining establishments, whether they served osso bucco or not. We ended up giving those clips to the French and Spanish readers, praying that the god (dieu, dios, dio) of romance languages would smile on us.

We vowed that in the future we would be prepared to deliver on our promises, both by being sure we hadn’t painted too broad a stroke of what our capabilities were and also by making sure we could handle whatever came our way.

4. Let’s take another look at that forest…
Software enables you to track virtually any aspect of a PR program down to the tiniest detail of multiple products and multiple competitors. The data all goes into one database and—presto!—with a few keystrokes you’re buried in charts and graphs. But when you mix this technology with people who worry too much, they start thinking about too many possibilities.

I’ll never forget a fierce debate between our client, the account service folks and the production team about how we might categorize an article that mentioned multiple products and competitors, with different positioning on each product. The debate raged for some time and got more intense by the minute. Philosophical differences were threatening to end the meeting in fisticuffs.

I interrupted the brouhaha to ask a question, the answer to which I hoped would restore calm: “How often has this occurred in the past?” With all those databases at hand, we could get the answer pretty quickly: “Once every 537 articles.” There was a collective sigh and apologetic mutterings. Clearly, these were folks who had become lost in the saplings.

5. It doesn’t matter who’s to blame, it’s always your fault
We’d been doing press coverage analysis for Apple Computer for some time when, to speed up the process, we made a minor alteration in procedure. Instead of relying on “paper” copies of the articles, we took advantage of a new fax service one of the clipping firms was supplying. So, for all the major national publications (e.g., Wall Street Journal, New York Times, Washington Post), we began to analyze the daily faxed articles. This worked fine for several months, until...

I was presenting our results to the client and pointing to a chart that said “Business press dropped substantially this month.” This was big news. The client had, on average, received about 15 mentions a month in the Wall Street Journal, and our data showed precisely zero articles that month from the publication. The client’s PR agency representative stated that, as a matter of fact, there had been a number of articles that month in the Wall Street Journal!

Why the discrepancy? The clipping service had failed to fax us the clips. Gulp.

The lesson to be learned arrived like the thud of a mammoth clip book. The client pointed out—and I couldn’t argue with her—that, given our historical knowledge of their business, we should have known the data was too far off to be valid. Just goes to show: Never write the headline without double-checking the facts.

Stay tuned next month for more of my biggest measurement mistakes. And don’t forget to send us yours.

(Originally published July 30, 2003)

Katie Paine's
All-Time Biggest
Measurement Mistakes,
Part Two

by Katie Delahaye Paine

Can you top the major measurement muck-ups below or in part one of this article? Then join our Biggest Measurement Mistake Contest! Send us your most interesting measurement mistakes and, if your misadventures are both truly instructive and entertainingly embarrassing, we'll tell the world and send you a far-better-than-standard Measurement Standard mug.

In Part One of this article, I related five of my most awkward—but most educational—mistakes in the measurement business. Here are four more, and I've saved the best for last:

1. Message Mess
My common abbreviation for "messages" is "mess," and this was never more apropos than when we first started doing message analysis for a major computer company. We had agreed that telling our analysts up front what the messages were might bias their reading, so we told them to just write down whatever messages they thought the article contained.

The process might have worked if we'd had one reader instead of three. What came back were three lists of 20 messages each. We consolidated the 60 messages into 10 clearly dominant ones that all readers had identified. We threw the remainder into the "other" pile and proceeded with the tedious task of rereading all 500 articles. We had to determine whether the message identified by Reader #3 had appeared in any of the clips read by Readers #1 and #2, whether messages identified by Reader #1 were still appearing in the clips we had given to Readers #3 and #2... You get the idea. And we got an idea, too.

The next month, we had one analyst read all 500 clips. The only problem was that if she found a new, unique and relevant message in clip #278, that then showed up in clips #280-325, she had to reread clips #1-277 to see where else it had appeared.

That's why, today, we use computers.

2. Research to the Rescue
We had finally convinced a financial services firm to switch from their antiquated AVE measurement system to our more sophisticated system that defined success in terms of messages communicated.

What we hadn't taken into account, however, was that their AVE system had lulled them into a false sense of security; although the overall clip count was going up, their message communication had been steadily declining. Even worse, the PR employees' compensation was tied to performance, and one of the performance metrics was message communication. So our great work for them resulted in better measurement but less pay. Not really what you like to do for a client—or for your own reputation.

Research came to the rescue, however, as further analysis showed that the message decline was due to coverage of litigation. The firm was being sued and suing others at a remarkable rate, the coverage of which included few key messages. As the PR team could do very little about the lawsuits or their coverage, they quite justifiably felt that their compensation should not be affected. So, the PR team took their research results to the Executive Committee, which agreed with them and changed their compensation structure. Score another one for measurement.

3. KISS Me Even More, or,
Don't forget to keep it really, really simple.
I worked for five years on an international measurement program for a major multi-national package goods client. We began with a simple premise—create a series of metrics that would work across all countries and regions in which the company did business. I put together what I considered an absolutely brilliant plan that looked at three competitors per country, positive vs. negative coverage, and tracked a few key messages.

We presented to our immediate client contact, who loved it. She presented to her colleagues in Europe, who hated it. Back to the drawing board we went. Again and again and...

Five years and three different client contacts later, we finally had a system that was acceptable to everyone. The key criteria we decided upon were:

• Share of brand mentions
• Share of brand benefits
• Share of photographs
• Share of recommendations

We presented results and distributed reports. The request came back to make it simpler: Could we develop a score that would sum up the results? So we worked with the mathematicians and statistical wizards to come up with a verifiable, accurate score for each of the key criteria.

Then the request came back to make it more simple: Could we roll it up into one overall effectiveness score? Certainly. We then produced a report that graded the PR efforts on a scale of 1 to 100, by brand, in each country.

The request came back yet again—you got it—to simplify it even more: Could we get rid of the numbers and just tell them whether it was Excellent, Fair or Poor? Five years of work, hundreds of hours of analysis, testing and hand-wringing, and it all came down to three words!

4. The Naked Man Screensaver, or,
"I'm still getting used to the hardware."
In the middle of the five-year process described above, I was summoned to Europe to present results of the latest round of testing. As you might have gathered by now, this was not the most receptive audience—so I cringed when I found that my laptop was not compatible with the on-site projector. By a stroke of luck, my VP of sales' laptop was compatible. While the audience waited impatiently, we transferred files, switched outputs, and booted up my VP's laptop.

By now we had eaten up a quarter of the precious hour that we had flown 6,000 miles for, and I was as anxious and impatient as the audience. The client was in an even more disagreeable mood that usual and all my attempts at idle conversation fell flat.

Finally, the laptop booted up and the long-awaited presentation was just a few clicks away from the opening slide, when up popped the screensaver: An extremely handsome and naked man! Needless to say, my VP was used to working in a more private setting.

The moral of the story: Whomever opens unknown screensaver risks inserting foot into gaping mouth.

New! A State-of-the Art
PR Dashboard for $50 a Month
You know you need to measure your results, but chances are there’s never been enough money in your budget for evaluation. Until now.
KDPaine & Partners’ new Do-It-Yourself Dashboard system combines a Web-based application with professional consulting to enable PR professionals to customize their own PR dashboards. Look here for more information.

1. You’ll learn how to use hard numbers to prove the results of your PR efforts.

2. You’ll learn which are the right vendors for your measurement projects.

3. You’ll learn how to design your program right from the start to be easily measureable.

Click here to
subscribe now!
(It’s 100%
money-back guaranteed!)

51 Durham Point Road, Durham, NH 03824
603-868-1550 fax: 603-868-3346 www.measuresofsuccess.com