Advancing communications measurement and evaluation

Give Me Qualitative Insights Over Numbers And Creativity

"The way I feel is hard to quantify." "How hard, on a scale from one to ten?"

PR Measurement Has Got It Back to Front

This is the third of a series of articles by Jim Macnamara on the limitations of quantitative metrics in public relations measurement and evaluation. Read the first two: Why Metrics Don’t Add Up To Evaluation For Public Relations,” and PR Measurement and Evaluation Needs More Than a One Trick Survey Monkey.

When you get to be an aging contrarian like me, you will have read a half-century or more of well-intentioned advice and tips on how to create effective public relations and communication campaigns, how to get into the boardroom, endless tirades about the need for numbers, and many passionate pleas about the value of creativity.

Well, let me throw another spanner into the workings of PR orthodoxy and dominant ideology – and I will back it up with a contemporary case study that has just saved a government department millions of dollars and is likely to save hundreds of lives.

This is not seizing a chance to blow my own trumpet about one of my own campaigns, as I was only one of a large multidisciplinary team of health professionals, community leaders, clinicians, communication professionals, and researchers. In fact, I was only one of three professional researchers involved in the project – although I have had the pleasure and learning by being what we call the CI (Chief Investigator).

Here’s the case study.  I will let it tell a story that has several significant implications for PR and communication practitioners.

Case Study: Increasing Rates of Breast Cancer Screening Among Certain Ethnic Groups

OK male readers, I recognize this is about a problem that directly affects women. But we all need to listen up for at least three reasons:

  1. The unnecessary death of women has life-changing consequences for husbands, male partners, brothers, sons, and other male family members as well;
  2. Men can play a key role in solving the problems described, as you will see; and
  3. At a professional level the learnings of this project apply to all PR and communication campaigns.

The problem

The problem is that immigrant women of Indian and Sri Lankan descent, who comprise substantial ethnic groups in Australia, are resistant to and often refuse to have a mammogram (commonly known as a breast screen). Why? And how to increase the rate, because research has shown that early detection is critical to reducing deaths from this devastating disease?

The old approach, Part 1: Over the years I have seen numerous PR managers and consultancies leap in with creative solutions – often off the top of their heads. Blind faith in their own intuitive creativity and instincts has seen millions of dollars poured into campaigns, and even some awards won. But those unknowns – why and how to create change –are what researchers call the research questions, because they require investigation, as opposed to creative PR concepts.

The old approach, Part 2: The next part of the problem was that some researchers a few years before had done a survey which reported that women in these ethnic groups did not understand modern Western medicine and just needed education, were scared, or were just too busy. These findings proved next to useless, as subsequent campaigns were no more effective than previous ones. A number of health officials were suspicious that the survey results were not the real answers to the research questions (due, perhaps, to the problem of self-reporting in surveys).

So in 2013, as part of its ongoing anti-cancer campaigns, the Cancer Institute in the state of New South Wales (NSW) in Australia awarded funding for a project to identify ways to increase the rate of breast screening by women in these groups.

A new approach

A five month project was undertaken by the successful tenderers, a triumvirate comprised of the Multicultural Health Communication Service (MHCS) of the NSW Department of Health, the Refugee Health Service (RHS), and academics from the University of Technology Sydney (UTS) – sorry, I can’t help at least one plug in the story.

In a nutshell, here’s what the 2014 campaign was comprised of:

  1. Community consultations with Indian and Sri Lankan groups such as community leaders, clubs, women’s groups in those cultures, church groups, and so on, to discuss the problem and women’s attitudes, perceptions, and concerns;
  1. A permanent reference group, established with representatives of the communities to meet regularly and work with health professionals, communicators, and researchers planning the campaign;
  1. An evaluation model, designed in the early weeks of the project (significantly, the funding body stipulated that 15% of the budget had to be assigned to evaluation). This included setting up data collection mechanisms for each step of the project, ranging from recording, producing transcripts, and analyzing content of consultative meetings and focus groups, to capturing baseline demographic and statistical data (inputs), though tracking of outputs, outtakes, and finally outcomes (in line with best practice PR evaluation as well as health communication research);
  1. Intercept interviews at ethnic festivals, which were short and largely based on structured questionnaires, but they broadened the base of data beyond women who attended community consultation meetings;
  1. Focus groups with Indian and Sri Lankan women in the key susceptible age groups (59–74) who community leaders and groups helped recruit and coax into attending (with full confidentiality, no real names, etc.);
  1. Simultaneously, a global literature review was conducted of published research in relation to intimate body interventions such as breast and cervical cancer tests among Indian, Sri Lankan, Arabic, and women of other non-Western cultures. This included academic research as well as studies commissioned by government departments and health authorities. It uncovered numerous detailed studies worldwide with relevant and insightful findings and evaluation methods. It thus proved to be very informative, and demonstrated the value of academic research integrated with practice;
  1. Then we assembled baseline data on the number of Indian and Sri Lankan women having mammograms each year, the number seeking information about mammograms, and the number attending education talks or seminars in consultation with the Department of Health, clinics, and other health organizations;
  1. Finally, drawing on all this information (formative research) we collectively designed a campaign called PINK SARI. The name was created by the community groups – not some creative agency. The graphics were described as ‘uncool’ by several young self-declared ‘design thinkers’ and specialists in ‘creative intelligence’ wearing all black and half a pound of metal in various appendages. But the symbols, language, and metaphors came from the target audience, so they were meaningful to them.

Insights from the research

The various events, information materials, and other campaign activities produced were informed by the findings about the real barriers preventing women in these groups from having breast screens (which I’m sure you’re keen to know). These were:

  1. Religious beliefs (e.g., fatalistic beliefs that if you get cancer it is the will of God);
  2. High levels of modesty among these cultural groups in relation to exposing themselves to strangers (including opposition to breast screens by their husbands);
  3. Other cultural factors, such as bringing shame on one’s family if you were found to be ‘diseased’ and even reducing the marriage prospects of daughters;
  4. Literal translations of health information from English into Tamil, Hindi, and Singhalese did not make sense in many cases; and
  5. A lack of ‘cultural competency’ among health workers doing breast screening (i.e., lack of sensitivity to the women’s beliefs, fears, and concerns, and doing things like having men at the reception of health clinics, even though female clinicians did the mammograms).

New approaches as a result

The campaign fundamentally changed prevailing approaches to the problem of increasing breast cancer screenings. For instance, it:

  • Scrapped TV commercials;
  • Pulped all translated brochures;
  • Instigated training courses for all breast screen clinic staff;
  • Provided a 100% guarantee of privacy and all-women staff in breast screen clinics;
  • Rewrote all information ‘in language’ by working with community groups and volunteers (not translations); and
  • Called on the resources of local community leaders in each area to act as champions and advocates, including signing ‘pledges’.

Media publicity was found to have supplementary awareness-raising benefits, but was not a primary tactic, contrary to many PR campaigns. Social media were found to be largely irrelevant to women in the target age and ethnic groups – although future campaigns targeting younger women are planning to experiment with social media.

5 research lessons to learn

What’s the point or points? Well, this case study demonstrates at least five key points that I will briefly summarize:

  1. The campaign incorporated quantitative data. We know that politicians and senior managers want to see numbers, and the ultimate proof of effectiveness of this campaign will be increased numbers of women in the target groups having breast screens. So some numerical data is essential.
  2. The campaign was based on substantial formative research. In fact, points 1–7 of the campaign outline above were formative research undertaken before the communication activities were designed. This formative research was partly quantitative, but included considerable qualitative research. The qualitative research yielded deep insights that, I think anyone has to agree, provided the real intelligence and understandings that put this campaign on track.
  3. In-depth qualitative research and collaborative research and planning (some of the research could be considered action research) achieved community engagement – something many PR and marketing practitioners talk about but try to drive top down. This engagement resulted in the campaign being designed from the ground up and pre-tested at every stage. In short, target audience ‘buy in’ was ensured before it even started.
  4. Evaluation was planned and data collection mechanisms put in place from the outset – not added post-hoc. A significant increase in the number of Indian and Sri Lankan women undertaking mammograms is the final outcome metric required to demonstrate success. That might take one or several years to establish. But the campaign has (1) accurate baseline data for comparison, and has put in place mechanisms to progressively track (2) feedback from community health centres and medical clinics on any changes in attitudes and perceptions within the target audience; (3) attendance at breast screening information talks and seminars by women in the target audience; (4) changes in inquiry rates about breast screening; and (5) intention to have a breast screen (based on six-monthly surveys in sample communities). Early indications of (2), (3), and (4) above are all positive.
  5. Finally, and interestingly, the campaign cost less than the previous TV advertising campaign that achieved none of the desired outcomes.

The instigators of this campaign are not struggling for recognition by senior management; they are asked to come and brief heads of a number of health organizations and even the office of the Minister of Health is taking a keen interest.

Give me deep insights and understanding any time over hour-long PowerPoint presentations of statistics and the well-intentioned but uninformed creativity of PR or advertising executives.

### Thanks to From Poverty to Power for the image.

Jim Macnamara

Jim Macnamara

Jim Macnamara PhD, FAMI, CPM, FAMEC, FPRIA is Professor of Public Communication at the University of Technology Sydney, a position he took up in 2007 after a 30-year career working in journalism, PR and media research which culminated in selling the CARMA Asia Pacific franchise that he founded to iSentia (formerly Media Monitors) in 2006. He is the author of 15 books, including his latest, Organizational Listening: The Missing Essential in Public Communication (Peter Lang, 2015), as well as Public Relations Theories, Practices, Critiques (Pearson, 2012); The 21st Century Media (R)evolution: Emergent Communication Practices (Peter Lang, New York, 2010, 2014); and Journalism and PR: Unpacking ‘Spin’, Stereotypes and Media Myths (Peter Lang, New York, 2014).
Jim Macnamara
728 Ad Block

Related posts

728 Ad Block