Evaluating the Effectiveness of Your Homeland Security Communications
How to put together a research and evaluation system that measures the success of your efforts.

by Katharine Delahaye Paine

Summary: The purpose and design of a communications dashboard is discussed, and a step-by-step procedure is given for determining the effectiveness of communication of Homeland Security messages.

Introduction:
How will you know if your Homeland Security messages are getting through?

In the wake of September 11th, Public Safety Departments, State Offices and other governmental agencies are working (and spending) feverishly to develop and communicate their Homeland Security plans. At some point, all of these plans need to be evaluated, because if you don’t have an evaluation system in place, you won’t know if people are hearing and understanding what you have to say. In this context, evaluating your communications’ effectiveness takes on a new urgency—for Homeland Security, it’s not just your budget or your job that’s on the line, it’s people’s lives.

To that end, we’ve developed these guidelines to help public information officers, public affairs directors and others involved in communicating Homeland Security measure the success of their efforts.

“Homeland Security.” Seldom has a single phrase gotten to be a buzzword in such a short period of time. Even Y2K had a few years to reach its ultimate pervasiveness. As with Y2K and the census, millions will be spent educating the public and communicating with constituencies about the issues and their solutions. The effectiveness of communications concerning Y2K and the census were ultimately measured in very concrete terms: “How many people complied with the census?” and “How many people became Y2K compliant?”

In the case of Homeland Security, measuring the effectiveness of the various safety-related messages involved is difficult, but not impossible. As with all measurement programs, the key is measuring results as you go along, rather than at the end, so that you can make adjustments along the way that will ensure your success.

Defining the dashboard that drives your program

When you’re driving a car, you look at your dashboard to learn if you have sufficient resources to get where you’re going, whether you’re headed in the right direction, and how fast you’re getting there. A perfect communications dashboard contains the key metrics that you need to know that your organization is headed in the right directions with sufficient resources, and at the right speed.

Each dashboard is unique to its organization, because every organization has its own goals and culture and channels of communication. Your dashboard must reflect the priorities of your organization, as well as the needs and responses of your target audiences.

That’s not to say that there aren’t standards within the profession as a whole. The Institute for PR has developed a series of guidelines for measuring PR that have been widely adopted by corporations, organizations, academics, agencies and research providers. It is important that your measurement plan adhere to those guidelines.

Example: This paper is a study of how Habitat for Humanity set up their dashboard to help monitor their reputation.

7 Steps to the Perfect Measurement System

There are seven basic steps to designing and implementing your Homeland Security communications measurement system.

Step 1: Understand the playing field

The first step in developing any research or evaluation program is to determine what you already know and don’t know. You need to have at your fingertips answers to the following questions:

• Who are your target audiences? What groups need to be part of your communications program: Hospitals? Schools? Public Safety Officers? Businesses? Make a list of every possible group that needs to hear and understand your messages.
• What are they seeing now about security, public safety issues and local government?
• Where do they go for trusted information about public safety efforts?
• What aspects of security and public safety are most important to them?
• What do they currently believe about your organization?
• What do you want them to believe?
• What do you need to do about it?

Many of these questions reflect public attitudes and opinions which you need to have a handle on before you begin any program.

Where to get the answers: You may need to do polls or surveys. Frequently you can add questions to existing surveys (see the article on Omnibus Studies in this issue) or take advantage of research being conducted by local ad agencies or media outlets. Other questions relative to your messages and what audiences you are targeting need to be answered and prioritized by the leaders of your organization. Chances are you’ll need to get them in a room together and hammer out the answers.

Step 2. Set measurable objectives

Set objectives your management will buy into. You and your management must agree on specific, concrete goals if your communications program is to enjoy success. Unless your management buys into the objectives you are measuring against, they are likely to consider any measurement program a waste of time.

You are what you measure. You also need to remember that it is human nature to become what you measure. If your success is measured by the number of articles you generate, you will naturally do all in your power to generate articles. Most people often neglect to tie their measures of success back to the ultimate objectives of the entire organization.

Example: Suppose you are in charge of communications for a police department, and you contribute to the success of your organization by generating trust and cooperation between your department and its audiences. If the media and local residents feel trust and confidence in the organization, they will be more likely to cooperate in a crisis or whenever cooperation is needed.

How does one evaluate that trust and confidence? A stack of clips a mile high doesn’t indicate anyone’s likelihood to cooperate. Clips can certainly tell you whether the media is buying into your messages, and to an extent they can tell you the degree to which the media trusts your spokespeople (as opposed to going somewhere else for information), but clips alone are not a measure of your contribution to the organization’s success.

At the very least you need to determine:

• What messages people saw
  
• How many people saw them
  

Ideally, you’d do this by having your clips read by members of the target audience to see what their takeaway was and factor that into your dashboard. But message communications is only one part of your dashboard. You need to test your messages to make sure that your audiences are in fact believing what they read or hear.

Step 3: Defining your criteria

Once you’ve reached basic agreement on what you will be accountable for, the next step is to translate those objectives into specific criteria that will end up on your dashboard. Criteria tend to start with either dollar signs or percentage – i.e. “percent of the population that has seen my message,” or “cost per message communicated.” Frequently used criteria include:

• % of exposure containing key messages
• Cost per message communicated
• % of positive vs. negative articles
• % of audience hearing key messages
• % of audience believing key messages
• % of audiences more likely to act
• Cost per percentage increase among those believing key messages

This is by no means a comprehensive list, but can serve as a guideline for developing your program. Good criteria readily translate to something that is directly relevant to the goals of your organization and inherently meaningful to your management.

Example: The person in charge of Distance Learning for a state school was struggling for meaningful measures of success. He could readily count the number of people enrolled in distance learning programs in the state, but that wasn’t convincing the legislature or even his managers that the program was really paying off. However, after probing a bit deeper, he found that 75% of those enrolled went into higher paying high-tech jobs. Clearly the program was turning out better educated, better trained potential employees. In this case, the state office of economic development knew that the presence of an educated workforce was the most important factor for companies considering relocating to the state. And, states typically allocated up to $10,000 per job in tax breaks to entice companies to relocate to the state. Using this information, the distance learning program compared its cost against the cost of tax incentives to demonstrate that it was a far better use of state funds. That was a convincing argument.

Step 4: Deciding on a benchmark:
What do you compare your results against?

Measurement is, above all, a comparison tool. You want to compare your results to another department or to another state or to your own results over time. Ideally you would do all three. Comparing results to another state or town gives you some perspective and appeals to people’s natural competitiveness. (Nothing like showing that some local rival, that may be able to trounce you in high school football, has a less effective—better still, less cost effective—program than yours.)

Comparisons between communities has its downside. No two towns are alike; tax bases, educational levels and property tax values may all vary enough to distort results. Comparing results between departments is also tricky. While such comparisons do in fact drive individuals to excel, you can exacerbate animosities between departments unless such comparisons are carefully handled.

Of course the most common benchmark is to look at your results over time. This should always be done, because it allows you to map results against other events, e.g., increases or decreases in budgets, changes in administration or changes in policy. Media tracking should be done on a monthly basis. Public opinion should be polled as often as your budget allows; it is better to do a smaller sample more often so you get more current data than a huge comprehensive survey every two years. Regular, ongoing sampling yields by far the best and most useful data.

Step 5: Select your measurement tool

You must select a tool based on objectives. If your stated objectives are to increase awareness, you need to conduct a survey to test awareness. If your stated objectives are to elevate the trust and credibility of your department, you need to measure trust and credibility. If your stated objectives are to communicate messages, you should be tracking messages in all relevant media.

Another factor to consider when selecting your measurement tool is timing. Media content analysis can be turned around in a couple of weeks, surveys typically take a month or more to complete. Work backwards from when you need data (for a meeting, a presentation or to make strategic planning decisions from) and that may determine what tools are available to you.

Step 6: Analyze results and draw actionable conclusions

Research is just trivia without insight; it’s like having a dashboard but no wheels and no destination. Look at the data and draw actionable conclusions. Your dashboard should reveal the strength of your relationships with each audience, enabling you to address any weakness immediately.

Step 7: Take action and measure again

If your results show that your messages aren’t being seen or believed, change tactics to make sure that they are. If a segment of your constituencies aren’t getting the message via PR, consider other ways to communicate: underwriting of public radio, sponsoring an educational event, setting up a booth at a mall. If the WRONG message is getting out there, reevaluate what you’re saying. Are you trying to communicate conflicting messages, or too many messages at once? Are you using the appropriate media? Are your spokespeople properly trained? Are they credible? Does the media seek them out, or are they constantly trying to find someone else to talk to? Research can provide the answers, but you need to provide the next steps and changes that will improve the program and ensure your success.

Other considerations

Public relations and public affairs officers frequently want to know how they can isolate their results from the results of other activities. The key is in the timing. If you roll out a program step-by-step and measure results after each step, you can document the additive effect of each element.

Example: Southwest Airlines measures success by the number of tickets sold. They roll out a PR campaign first and measure seats sold. Then they add in advertising and again measure seats sold, then add in direct mail, etc., etc. Similar techniques can be used to measure Web traffic, as well as attitude and awareness.

Whatever you do, make sure you post your dashboard in a place where everyone can use it and learn from it. Whether it’s on a bulletin board or on everyone’s computer screen, it’s no good to anyone who can’t see it. Remember, a dashboard is a tool; the more it gets used, the better your results will be.