Measuring the Impact of Media

10 Nov

Measuring the impact of media accurately is challenging. Findings of minimal effects abound when intuition tells us that an activity that an average American engages in over forty hours a week is likely to have a larger impact. These insignificant findings have been typically attributed to the frailty of survey self-reports of media exposure, though debilitating error in dependent variables has also been noted as a culprit. Others have noted weaknesses in research design, inadequate awareness of analytic techniques that allow one to compensate for error in measures, etc. as stumbling blocks.

Here are a few of the methods that have been used to overcome some of the problems in media research, along with some modest new proposals of my own:

  • Measurement
    Since measures are error-prone, one strategy has been to combine multiple measures. Multiple measures of a single latent concept can be combined using latent variable models, factor analysis, or even simple averaging. Precaution must be taken to check that errors across measures aren’t heavily correlated, for under such conditions improvements from combining multiple measures are likely to be weak or non-existent. In fact, deleterious effects are possible.

    Another point of worry is that measurement error can be correlated with irrelevant respondent characteristics. For instance, women guess less than men on knowledge questions. Hence responses to knowledge questions are a function of ability and propensity to guess when one doesn’t know (tallied here by gender). By conditioning on gender, we can recover better estimates of ability. Another application would be in handling satisficing.

  • Measurement of exposure
    Rather than use self-assessments of exposure, which have been shown to be correlated to confounding variables, one may want to track incidental consequences of exposure as a measure of exposure. For example, knowledge of words of a campaign jingle, attributes of a character in a campaign commercial, source (~channel) on which the campaign was shown, program, etc. These measures factor in attention, in addition to exposure, which is useful. Unobtrusive monitoring of consumption is, of course, likely to be even more effective.

  • Measurement of Impact
    1. Increased exposure to positive images ought to change procedural memory and implicit associations. One can use IAT or AMP to assess the effect.
    2. Tracking Twitter and Facebook feeds for relevant information. These measures can be calibrated to opinion poll data to get a sense of what they mean.
  • Data Collection
    1. Data collection efforts need to reflect half-life of the effect. Recent research indicates that some of the impact of the media may be short-lived. Short-term effects may be increasingly consequential as people increasingly have the ability to act on their impulses – be it buying something, or donating to a campaign, or finding more information about the product. Behavioral measures (e.g. website hits) corresponding to ads may thus be one way to track impact.
    2. Future ‘panels’ may contain solely passive monitoring of media use (both input and output) and consumption behavior.
  • Estimating recipient characteristics via secondary data
    1. Geocoded IP addresses can be used to harvest secondary demographic data (race, income, etc.) from census
    2. Para-data like what browser and operating system the customer uses etc. are reasonable indicators of tech. savvy. And these data are readily harvested.
    3. Datasets can be merged via matching or by exploiting correlation across items and by calibrating.

Sharing Information about Sharing Misinformation

16 May

The Internet has revolutionized the dissemination of misinformation. Easy availability of incorrect information, gullible and eager masses, and ease of sharing has created fertile conditions for misinformation epidemics.

While a fair proportion of misinformation is likely created deliberately, it may well spread inadvertently. Misinformation that people carry is often no different than fact to them. People are likely to share misinformation with the same enthusiasm as they would fact.

Attitude congenial misinformation is more likely to be known (and accepted as fact), and more likely to be enthusiastically shared with someone who shares the same attitude (for social, and personal rewards). Misinformation considered useful is also more likely to be shared, e.g. (mis)-information about health-related topics.

The chance of acceptance of misinformation may be greater still if people know little about the topic, or if they have no reason to think that the information is motivated. Lastly, these epidemics are more likely to take place among those less familiar with technology.

Idealog: Internet Panel + Media Monitoring

4 Jan

Media scholars have for long complained about the lack of good measures of media use. Survey self-reports have been shown to be notoriously unreliable, especially for news, where there is significant over-reporting, and without good measures, research lags. The same is true for most research in marketing.

Until recently, the state of the art aggregate media use measures were Nielsen ratings, which put a `meter’ in a few households, or asked people to keep a diary of what they saw. In short, the aggregate measures were pretty bad as well. Digital media, which allows for effortless tracking, and the rise of Internet polling however for the first time provides an opportunity to create `panels’ of respondents for whom we have near perfect measures of media use. The proposal is quite simple: create a hybrid of Nielsen on steroids and YouGov/Polimetrix or Knowledge Network kind of recruiting of individuals.

Logistics: Give people free cable and Internet (~ 80/month) in return for 2 hours of their time per month and monitoring of media consumption. Pay people who already have cable (~100/month) for installing a device and software. Recording channel information is enough for TV, but Internet equivalent of a channel—domain—clearly isn’t, as people can self-select within websites. So we only need to monitor the channel for TV but more for the Internet.

While the number of devices on which people browse the Internet, and watch TV has multiplied, there generally remains only one `pipe’ per house. We can install a monitoring device at the central hub for cable, and automatically install software for anyone who connects to the Internet router or do passive monitoring on the router. Monitoring can also be done through applications on mobile devices.

Monetizability: Consumer companies (say Kellog’s, Ford), Communication researchers, Political hacks (e.g. how many watched campaign ads) will all pay for it. The crucial innovation (modest) is the addition of the possibility to survey people on a broad range of topics, in addition to getting great media use measures.

Addressing privacy concerns:

  1. Limit recording information to certain channels/websites, ones on which customers advertise, etc. This changing list can be made subject to approval by the individual.
  2. Provide for a web-interface where people can look/suppress the data before it is sent out. Of course, reconfirm that all data is anonymous to deter such censoring.

Ensuring privacy may lead to some data censoring and we can try to prorate the data we get it a couple of ways –

  • Survey people on media use
  • Use Television Rating Points (TRP) by sociodemographics to weight data.

Reliving some of the high points of the 2008 presidential campaign

13 Nov

November 10, 2007: One of the first scandals to break out during the campaign was about planted questions in Hillary’s townhall meetings. “They asked me if I would ask the senator a question. I said, ‘Sure, you know,'” Gallo-Chasanoff told CNN. “He showed me in his binder, he had a piece of paper that had typed out questions on it. And the top one was planned specifically for a college student. It said ‘college student.'” ‘A video on MSNBC shows Gallo-Chasanoff reading the question word for word, and then winking when she was done.’ ABC News

November 10, 2007: “I love my wife and my five sons and their five wives. Wait a second. Let me clarify that. They each have one.” Mitt Romney (Economist gave this quip the title – Best Freudian slip; ABCNews.com)

December 12, 2007: In kindergarten, Senator Obama wrote an essay titled ‘I Want to Become President.’ “Iis Darmawan, 63, Senator Obama’s kindergarten teacher, remembers him as an exceptionally tall and curly haired child who quickly picked up the local language and had sharp math skills. He wrote an essay titled, ‘I Want To Become President,’ the teacher said.”
From: Clinton campaign’s press-release.

December 13, 2007: “It’ll be, ‘When was the last time? Did you ever give drugs to anyone? Did you sell them to anyone?'” Shaheen on Obama
Bill Shaheen (husband of NH Senator-elect Jeanne Shaheen; national co-chairman of Clinton’s campaign at that point)

February 24, 2008: Bill Clinton speaking about Hillary’s inability to win caucus states – “the caucuses aren’t good for her. They disproportionately favor upper-income voters who, who, don’t really need a president but feel like they need a change.” Audacity of Hopelessness by Frank Rich

March 8, 2008: “She is a monster, too – that is off the record – she is stooping to anything,” Samantha Power; Obama’s foreign policy adviser.

March 10, 2008: Hillary Clinton chief spokesman Howard Wolfson declared Monday that Clinton does not consider Obama qualified to be vice president.

March 11, 2008: “I will not be discriminated against because I’m white. Geraldine Ferraro

“If we can’t trust Mitt Romney on Ronald Reagan, how can we trust him to lead America?”
From John McCain’s attack ad on Romney

“The Clintons will be there when they need you,” said a Carter friend. (Maureen Dowd, NY Times)

May 3, 2008: When asked, at the Republican presidential primary debate at Simi Valley, whether any of the candidates did not believe in evolution , three candidates – Tancredo, Brownback, and Huckabee – raised their hands.

May 9, 2008: “Senator Obama’s support among working, hard-working Americans, white Americans, is weakening again.” (Hillary Clinton, Interview with USA Today)

Google News - Clinton Accuses Obama
Google News Archives timeline graph of citations of 'Clinton Accuses Obama' between August 2007 and August 2008

August 21, 2008: “I think – I’ll have my staff get to you. It’s condominiums where – I’ll have them get to you.” (John McCain unsure about the number of houses he owns.)

A special tribute to Palin:

September 24, 2008: “As Putin rears his head and comes into the air space of the United States of America, where– where do they go? It’s Alaska. It’s just right over the border. (Interview with Katie Couric, CBS News)

In defense of Palin, she never said that she could see Russia from her house. (Time)

September 25, 2008: Couric: And when it comes to establishing your worldview, I was curious, what newspapers and magazines did you regularly read before you were tapped for this to stay informed and to understand the world?
Palin: I’ve read most of them, again with a great appreciation for the press, for the media.
Couric: What, specifically?
Palin: Um, all of them, any of them that have been in front of me all these years.
Couric: Can you name a few?
Palin: I have a vast variety of sources where we get our news, too.
CBS News

October 1, 2008: “Well, let’s see. There’s — of course — in the great history of America rulings there have been rulings.” Sarah Palin (When asked by Couric to name a Supreme Court decision, other than Roe vs. Wade, that she disagreed with; CBS News)

News coverage of Sarah Palin

12 Nov
News Coverage of Sarah Palin and Joseph Biden
News Coverage of Sarah Palin and Joseph Biden
Ratio of News stories by day covering Sarah Palin and Joseph Biden
Ratio of News stories by day covering Sarah Palin and Joseph Biden
Coverage of Sarah Palin's interview with Gibson, Couric, and Tina Fey's impersonation on SNL
Coverage of Sarah Palin's interview with Gibson, Couric, and Tina Fey's impersonation on SNL
Ratio of stories citing John McCain that also cited Sarah Palin vis-a-vis Ratio of stories citing Barack Obama that also mentioned Joe Biden
Ratio of stories citing John McCain that also cited Sarah Palin vis-a-vis Ratio of stories citing Barack Obama that also mentioned Joe Biden
Palin's coverage: MSNBC and Fox
Palin's coverage: MSNBC and Fox

Difference in Coverage of Clinton and Obama

12 Mar

The following tables tally up the articles that mention “Barack Obama” or “Hillary Clinton” in their body or title. The results show that both New York Times and Washington Post cover Obama at much lower rates than average rate of coverage in “US Newspaper and Wires.”

The differences are particularly significant given that most articles follow the “horse race” format, and hence mention both Obama and Clinton.

  WP Clinton* WP Obama* NYT Clinton* NYT Obama* LN Obama* LN Clinton*
Apr-07 85 81 98 113 3479 2324
May-07 110 87 110 93 3309 2373
Jun-07 122 101 112 78 3425 2874
Jul-07 142 104 104 82 3820 3276
Aug-07 119 109 103 76 4070 3201
Sep-07 152 115 160 94 4088 3649
Oct-07 171 108 159 95 4813 4742
Nov-07 174 111 164 108 4391 4445
Dec-07 195 169 194 164 6134 4774
Jan-08 342 354 357 335 15276 10540
Feb-08 315 330 409 436 18857 11658
  LN Ratio WP Ratio NYT Ratio
Apr-07 1.50 0.95 1.15
May-07 1.39 0.79 0.85
Jun-07 1.19 0.83 0.70
Jul-07 1.17 0.73 0.79
Aug-07 1.27 0.92 0.74
Sep-07 1.12 0.76 0.59
Oct-07 1.01 0.63 0.60
Nov-07 0.99 0.64 0.66
Dec-07 1.28 0.87 0.85
Jan-08 1.45 1.04 0.94
Feb-08 1.62 1.05 1.07
Avg. 1.27 0.81 0.78

*Article count from LexisNexis Power Search with search term “Barack Obama” and “Hillary Clinton” respectively. The source field was constrained to “New York Times”, “Washington Post”, and “US Newspaper and Wires” respectively.

The hubris of journalists

18 Jan

Nick Bryant, a correspondent covering Australian elections for the BBC, wrote the following in one of his blog posts,

“My name is Nick and I fear I am in danger of becoming an Australian political junkie. I find myself boring friends with the swings needed to win obscure marginals, which, up until six weeks ago, I never knew existed. My mind is cluttered with useless information, like how the South Australian seat of Makin is named after a post-war Australian ambassador to Washington.

Had you asked me 18 months ago, I would have hazarded a guess that Eden-Monaro was a type of Dutch cheese. Now I can quote the land mass of this all-important bellwether seat.”

While Nick Bryant did – on average – a reasonable job of reporting on the Australian elections, it seems debatable whether journalists can start filing in-depth analytically rich reports on a country days after landing in a country about which they know next to nothing.

Another reporter (Kevin Connolly) from the BBC – this time covering the US elections– wrote,

” On your first days in a new assignment as a reporter, you work hard – sometimes a little too hard – to look for clues that will help you to decode life in your new adopted home.

When we changed planes in Chicago midway through my never-ending New Year’s Eve, I found myself lingering in the self-help section of the bookstore, puzzled by the sort of advice for which Americans are prepared to pay. I now own copies of God Wants You To Be Rich and You’re Broke Because You Want to Be. “

There is a danger that journalists new to the country will weigh idiosyncratic details about the country they notice disproportionately in their analysis and reporting.

Good reporting – as is understood – is seen to a large extent as a tough-minded commitment to pursuing the ‘truth’. It is seen as a skill that surpasses bounds of geography and culture. And certainly, there are elements of it that remain constant throughout. However, lack of a deeper understanding of the country, and culture can severely jeopardize not only “ability to contextualize events and issues, but also “objective” elements of reporting. The ability to contextualize is of particular importance for the apathetic ill-informed home country readership.

The foreign reporting standards have dropped precipitously as the length of foreign tours has dropped precipitously over the last many years to now average between one and three years. Reports from foreign journalists nowadays often take the quality of a tourist blog with substandard reports about preconceived notions that need validation. In this age of the Internet, I am in fact unsure of the need for foreign journalists. Liaisons with prominent news organizations within the country should be pursued to produce reports.

While the problem of under-qualified reporters is the most prominent in foreign reporting, it is not limited to it. Greenhorns reporting on politics often times carry the open-eyed celebrity wonderment about the political figures they report on.

And the news is…An ‘out of control’ General, freedom, democracy, and strawberry tarts

4 Nov

On a slow news day, Western journalists took to telling the world about how ‘Pakistan’s Musharraf’—it was nice of them to put the possessive noun as it is not always clear—had declared an emergency. In doing so, they paid as much attention to a third-world dictator enacting Martial law, as they have in some time. The calamitous event was reported with the usual combination of scant detail, high impact graphics, half-baked, and sometimes skipping the oven altogether, analysis, all topped with alarmist rhetorical flourishes.

The news day actually started a day earlier with reporting about speculation, and government warnings against speculated action. The US, proudly and staunchly backing Musharraf at least since the day after that calamitous day in 2001, warned the dictator ‘against Martial law’ (San Jose Mercury News). When the threat of declaring martial law grew, the ‘world grew concerned’ (Guardian) simultaneously. And when a ‘desperate Musharraf’ (Trend Information, Azerbaijan!) declared an emergency, it left the US in a ‘tizzy’ (Calcutta Telegraph), as it ‘reel[ed]’ (AFP) under the blow. The US immediately took umbrage and issued a ‘condemnation’. The ‘world’, not to be left behind, ’roundly condemned’ (Deutsche Welle, Germany), ‘slammed’ (Earthtimes, UK), and ‘rapped’ (The Province, Canada) General Sahib. Rice went further, for she wanted ‘Pakistan to evolve as a democracy’, and appealed for a return to the ‘constitutional course’ (Bloomberg) even as the phrase “cruel and unusual punishment” in the US constitution was being debated on to see if ‘coercively inducing a drowning sensation’ met that criteria. All this condemnation must have left Musharraf chagrined.

It was a ‘Sad day for Pakistan’ (The Daily Star, Bangladesh) when an ‘out of control’ (Nation Multimedia, Thailand) Musharraf chose to launch a ‘coup within a coup’ and put Pakistan under his ‘iron fist’ (The Standard, Hong Kong). The wordsmiths at Hindustan Times online division, which doubles as a soft porn website, found time to craft the smart-aleck headline “Under General Anesthesia” to describe the events of ‘Black Saturday’ (Malaysia Sun).

On that ‘black’ day, Western journalist’s thoughts didn’t stay long with people in Pakistan, as reporting on martial law gave way to more pertinent matters like ‘threat of nuclear weapons falling into wrong hands’, and ‘War on Terror.’

After all, the concerns of the media are solely determined by what (and how) they can best pander conditional on what is available. The Pavlovian reactions to international crises, the cues media uses to determine when to cry fire and when to cheer, are all rather simplistic—Democracy is good, autocrats are bad. Forget then that sometimes ‘enlightened moderation’ is the best alternative. Ignore too that nothing has changed substantively in Pakistani politics—control still rests with the General for it was only a ‘coup within a coup.’ Banish any thoughts that Benazir ran a regime true to her name, only if in levels of corruption.

I misstate my point for people who know little don’t need to deliberately ignore. They simply write. Till a new story appears and fuels a new news cycle, provides more cause for alarm, and more time to run ads.

It appears that Pakistan has had its day in the sun. The New York Times is not waiting with baited breath. There is a cell phone jammer in the market that can stop the person sitting next to you from yammering while you try to read about Paris Hilton. It is the Number 1 story on the New York Times website. Yeah, she is back.