To return to the current issue's contents page, click here.

To return to the contents page of the issue that this article appeared in, click here.

Comments Please!
Send us your thoughts on this article and we will post them in our Comments section.

Katie Paine’s All-Time Biggest
Measurement Mistakes
Part one of a two-part series.

by Katie Delahaye Paine

This article first appeared in the June 2003 issue of The Measurement Standard. See Part 2 here.

One of our principal beliefs in the measurement industry is that you learn a lot more from measuring failure than you do from measuring success. Sure, it’s nice to have a way to show off your accomplishments, but isn’t it far more useful to identify programs that aren’t working and fix them or reallocate the resources?

And just because our Web site (www.measuresofsuccess.com) has the word “success” in it, doesn’t mean we haven’t had our share of failures. In fact, first at The Delahaye Group and now at KDPaine & Partners, we’ve been celebrating mistakes for nearly two decades. In the past, we awarded a premium parking spot to the person who made The Mistake of the Month: the error from which we could learn the most. The idea was that if we rewarded mistakes, people would be more likely to confess them—and everyone would learn from them.

(The tradition continues, but the reward has changed: As there is ample parking at our new galactic headquarters, we're thinking of changing the reward to first dibs on tying up at the dock.)

Needless to say, after some 200 Mistakes of the Month we’ve learned a few things. In the spirit of spreading the knowledge, we thought we’d share a few of our more enlightening measurement missteps with you. (Former clients and employees can stop feeling anxious—all names have been changed to protect the innocent and guilty alike.)

1. Business Objects objects
One of our early lessons in online searching taught us just how hard it is to get it right. Some ten years ago, Business Objects, a knowledge management software company in California, hired us to do a media analysis.

Our initial online searches pulled up a great many spurious references to “Small Business objects to such-and such legislation” and “Big Business objects to such-and-such.” In an attempt to make the search more accurate, we added “Inc.” to the search string, which narrowed the resulting clips down to a manageable number. There was only one petite problem: Business Objects is a French company and not an “Inc.” at all. As a result, we missed about half the articles, but didn’t find out until way too late...

There I was presenting the results when the representative from the PR agency mentioned the fact that certain key publications hadn’t even shown up on our list of media that covered Business Objects. Sacre bleu! Our search error was revealed before all, and our data and my presentation were instantly rendered meaningless...

2. A spoonful of insight makes the medicine go down
No one likes bad news, and no client likes only bad news. Some years ago, one of my clients had a particularly bad quarter, PR-wise. I wrote up the report from a highly negative perspective, leaving out the usual sweetening-of-the-bad-news-with-constructive-suggestions.

Unfortunately, my merciless version of the unvarnished truth went straight to my client’s boss. My client then spent an excruciating hour in the hot seat, after which she called to inform me that we'd lost the account. And she didn’t waste any effort sweetening the bad news...

3. Line up your (Peking) ducks in a row before you undertake your research
It was 1991 and the formation of the European Union was all over the news. We had recently hired several multilingual reader/analysts, and so naturally I decided that it would be incredibly timely and newsworthy to announce that we had “gone international” and could now offer our media analysis services in multiple languages. Well, quicker than you can say schadenfreude we had a major multi-national high-tech client sending us boxes and boxes of clips in half a dozen languages—the wrong languages...

We had French, German, Spanish, Arabic and Norwegian covered. But the assignment was to analyze the company’s image in China, Europe, and South and North America. In search of some depth for our international reader roster, we inquired of the local institutions of higher education, tried a few contacts in Boston and even cased a few ESL classes, but still came up empty-handed.

Finally, inspiration hit. We tried the local Chinese restaurant and found a bi-lingual technology expert in residence! Surprisingly, we couldn’t find a single Italian speaker working in any of the local dining establishments, whether they served osso bucco or not. We ended up giving those clips to the French and Spanish readers, praying that the god (dieu, dios, dio) of romance languages would smile on us.

We vowed that in the future we would be prepared to deliver on our promises, both by being sure we hadn’t painted too broad a stroke of what our capabilities were and also by making sure we could handle whatever came our way.

4. Let’s take another look at that forest…
Software enables you to track virtually any aspect of a PR program down to the tiniest detail of multiple products and multiple competitors. The data all goes into one database and—presto!—with a few keystrokes you’re buried in charts and graphs. But when you mix this technology with people who worry too much, they start thinking about too many possibilities.

I’ll never forget a fierce debate between our client, the account service folks and the production team about how we might categorize an article that mentioned multiple products and competitors, with different positioning on each product. The debate raged for some time and got more intense by the minute. Philosophical differences were threatening to end the meeting in fisticuffs.

I interrupted the brouhaha to ask a question, the answer to which I hoped would restore calm: “How often has this occurred in the past?” With all those databases at hand, we could get the answer pretty quickly: “Once every 537 articles.” There was a collective sigh and apologetic mutterings. Clearly, these were folks who had become lost in the saplings.

5. It doesn’t matter who’s to blame, it’s always your fault
We’d been doing press coverage analysis for Apple Computer for some time when, to speed up the process, we made a minor alteration in procedure. Instead of relying on “paper” copies of the articles, we took advantage of a new fax service one of the clipping firms was supplying. So, for all the major national publications (e.g., Wall Street Journal, New York Times, Washington Post), we began to analyze the daily faxed articles. This worked fine for several months, until...

I was presenting our results to the client and pointing to a chart that said “Business press dropped substantially this month.” This was big news. The client had, on average, received about 15 mentions a month in the Wall Street Journal, and our data showed precisely zero articles that month from the publication. The client’s PR agency representative stated that, as a matter of fact, there had been a number of articles that month in the Wall Street Journal!

Why the discrepancy? The clipping service had failed to fax us the clips. Gulp.

The lesson to be learned arrived like the thud of a mammoth clip book. The client pointed out—and I couldn’t argue with her—that, given our historical knowledge of their business, we should have known the data was too far off to be valid. Just goes to show: Never write the headline without double-checking the facts.

Stay tuned next month for more of my biggest measurement mistakes. And don’t forget to send us yours.

You know you need to measure your results, but chances are there’s never been enough money in your budget for evaluation. Until now.
KDPaine & Partners’ new Do-It-Yourself Dashboard system combines a Web-based application with professional consulting to enable PR professionals to customize their own PR dashboards. Look here for more information.

1. You’ll learn how to use hard numbers to prove the results of your PR efforts.

2. You’ll learn which are the right vendors for your measurement projects.

3. You’ll learn how to design your program right from the start to be easily measureable.

Click here to
subscribe now!
(It’s 100%
money-back guaranteed!)

51 Durham Point Road, Durham, NH 03824
603-868-1550 fax: 603-868-3346 www.measuresofsuccess.com