Measuring the Impact of Media

10 Nov

Measuring the impact of media accurately is challenging. Findings of minimal effects abound when intuition tells us that an activity that an average American engages in over forty hours a week is likely to have a larger impact. These insignificant findings have been typically attributed to the frailty of survey self-reports of media exposure, though debilitating error in dependent variables has also been noted as a culprit. Others have noted weaknesses in research design, inadequate awareness of analytic techniques that allow one to compensate for the error in measures, etc. as stumbling blocks.

Here are a few of the methods that have been used to overcome some of the problems in media research, along with some modest new proposals of my own:

  • Measurement
    Since measures are error-prone, one strategy has been to combine multiple measures. Multiple measures of a single latent concept can be combined using latent variable models, factor analysis, or even simple averaging. Precaution must be taken to check that errors across measures aren’t heavily correlated, for under such conditions improvements from combining multiple measures are likely to be weak or non-existent. In fact, deleterious effects are possible.

    Another point of worry is that measurement error can be correlated with irrelevant respondent characteristics. For instance, women guess less than men on knowledge questions. Hence responses to knowledge questions are a function of ability and propensity to guess when one doesn’t know (tallied here by gender). By conditioning on gender, we can recover better estimates of ability. Another application would be in handling satisficing.

  • Measurement of exposure
    Rather than use self-assessments of exposure, which have been shown to be correlated to confounding variables, one may want to track incidental consequences of exposure as a measure of exposure. For example, knowledge of words of a campaign jingle, attributes of a character in a campaign commercial, source (~channel) on which the campaign was shown, program, etc. These measures factor in attention, in addition to exposure, which is useful. Unobtrusive monitoring of consumption is, of course, likely to be even more effective.

  • Measurement of Impact
    1. Increased exposure to positive images ought to change procedural memory and implicit associations. One can use IAT or AMP to assess the effect.
    2. Tracking Twitter and Facebook feeds for relevant information. These measures can be calibrated to opinion poll data to get a sense of what they mean.
  • Data Collection
    1. Data collection efforts need to reflect half-life of the effect. Recent research indicates that some of the impacts of the media may be short-lived. Short-term effects may be increasingly consequential as people increasingly have the ability to act on their impulses – be it buying something, or donating to a campaign, or finding more information about the product. Behavioral measures (e.g. website hits) corresponding to ads may thus be one way to track impact.
    2. Future ‘panels’ may contain solely passive monitoring of media use (both input and output) and consumption behavior.
  • Estimating recipient characteristics via secondary data
    1. Geocoded IP addresses can be used to harvest secondary demographic data (race, income, etc.) from census
    2. Para-data like what browser and operating system the customer uses etc. are reasonable indicators of tech. savvy. And these data are readily harvested.
    3. Datasets can be merged via matching or by exploiting correlation across items and by calibrating.

Sharing Information about Sharing Misinformation

16 May

The Internet has revolutionized the dissemination of misinformation. Easy availability of incorrect information, gullible and eager masses, and ease of sharing has created fertile conditions for misinformation epidemics.

While a fair proportion of misinformation is likely created deliberately, it may well spread inadvertently. Misinformation that people carry is often no different than fact to them. People are likely to share misinformation with the same enthusiasm as they would fact.

Attitude congenial misinformation is more likely to be known (and accepted as fact), and more likely to be enthusiastically shared with someone who shares the same attitude (for social, and personal rewards). Misinformation considered useful is also more likely to be shared, e.g. (mis)-information about health-related topics.

The chance of acceptance of misinformation may be greater still if people know little about the topic, or if they have no reason to think that the information is motivated. Lastly, these epidemics are more likely to take place among those less familiar with technology.

Idealog: Internet Panel + Media Monitoring

4 Jan

Media scholars have for long complained about the lack of good measures of media use. Survey self-reports have been shown to be notoriously unreliable, especially for news, where there is significant over-reporting, and without good measures, research lags. The same is true for most research in marketing.

Until recently, the state of the art aggregate media use measures were Nielsen ratings, which put a `meter’ in a few households, or asked people to keep a diary of what they saw. In short, the aggregate measures were pretty bad as well. Digital media, which allows for effortless tracking, and the rise of Internet polling however for the first time provides an opportunity to create `panels’ of respondents for whom we have near perfect measures of media use. The proposal is quite simple: create a hybrid of Nielsen on steroids and YouGov/Polimetrix or Knowledge Network kind of recruiting of individuals.

Logistics: Give people free cable and Internet (~ 80/month) in return for 2 hours of their time per month and monitoring of media consumption. Pay people who already have cable (~100/month) for installing a device and software. Recording channel information is enough for TV, but Internet equivalent of a channel—domain—clearly isn’t, as people can self-select within websites. So we only need to monitor the channel for TV but more for the Internet.

While the number of devices on which people browse the Internet, and watch TV has multiplied, there generally remains only one `pipe’ per house. We can install a monitoring device at the central hub for cable, and automatically install software for anyone who connects to the Internet router or do passive monitoring on the router. Monitoring can also be done through applications on mobile devices.

Monetizability: Consumer companies (say Kellog’s, Ford), Communication researchers, Political hacks (e.g. how many watched campaign ads) will all pay for it. The crucial innovation (modest) is the addition of the possibility to survey people on a broad range of topics, in addition to getting great media use measures.

Addressing privacy concerns:

  1. Limit recording information to certain channels/websites, ones on which customers advertise, etc. This changing list can be made subject to approval by the individual.
  2. Provide for a web-interface where people can look/suppress the data before it is sent out. Of course, reconfirm that all data is anonymous to deter such censoring.

Ensuring privacy may lead to some data censoring and we can try to prorate the data we get it a couple of ways –

  • Survey people on media use
  • Use Television Rating Points (TRP) by sociodemographics to weight data.

The 2008 Presidential Campaign in Brief

13 Nov

November 10, 2007: One of the first scandals to break out during the campaign was about planted questions in Hillary’s town hall meetings. “They asked me if I would ask the senator a question. I said, ‘Sure, you know,'” Gallo-Chasanoff told CNN. “He showed me in his binder, he had a piece of paper that had typed out questions on it. And the top one was planned specifically for a college student. It said ‘college student.'” ‘A video on MSNBC shows Gallo-Chasanoff reading the question word for word, and then winking when she was done.’ ABC News

November 10, 2007: “I love my wife and my five sons and their five wives. Wait a second. Let me clarify that. They each have one.” Mitt Romney (Economist gave this quip the title – Best Freudian slip; ABCNews.com)

December 12, 2007: In kindergarten, Senator Obama wrote an essay titled ‘I Want to Become President.’ “Iis Darmawan, 63, Senator Obama’s kindergarten teacher, remembers him as an exceptionally tall and curly haired child who quickly picked up the local language and had sharp math skills. He wrote an essay titled, ‘I Want To Become President,’ the teacher said.”
From: Clinton campaign’s press release.

December 13, 2007: “It’ll be, ‘When was the last time? Did you ever give drugs to anyone? Did you sell them to anyone?'” Shaheen on Obama
Bill Shaheen (husband of NH Senator-elect Jeanne Shaheen; national co-chairman of Clinton’s campaign at that point)

February 24, 2008: Bill Clinton speaking about Hillary’s inability to win caucus states – “the caucuses aren’t good for her. They disproportionately favor upper-income voters who, who, don’t really need a president but feel like they need a change.” Audacity of Hopelessness by Frank Rich

March 8, 2008: “She is a monster, too – that is off the record – she is stooping to anything,” Samantha Power; Obama’s foreign policy adviser.

March 10, 2008: Hillary Clinton chief spokesman Howard Wolfson declared Monday that Clinton does not consider Obama qualified to be vice president.

March 11, 2008: “I will not be discriminated against because I’m white.” Geraldine Ferraro

“If we can’t trust Mitt Romney on Ronald Reagan, how can we trust him to lead America?”
From John McCain’s attack ad on Romney

“The Clintons will be there when they need you,” said a Carter friend. (Maureen Dowd, NY Times)

May 3, 2008: When asked, at the Republican presidential primary debate at Simi Valley, whether any of the candidates did not believe in evolution, three candidates – Tancredo, Brownback, and Huckabee – raised their hands.

May 9, 2008: “Senator Obama’s support among working, hard-working Americans, white Americans, is weakening again.” (Hillary Clinton, Interview with USA Today)

Google News - Clinton Accuses Obama
Google News Archives timeline graph of citations of 'Clinton Accuses Obama' between August 2007 and August 2008

August 21, 2008: “I think – I’ll have my staff get to you. It’s condominiums where – I’ll have them get to you.” (John McCain unsure about the number of houses he owns.)

A special tribute to Palin:

September 24, 2008: “As Putin rears his head and comes into the airspace of the United States of America, where– where do they go? It’s Alaska. It’s just right over the border. (Interview with Katie Couric, CBS News)

In defense of Palin, she never said that she could see Russia from her house. (Time)

September 25, 2008: Couric: And when it comes to establishing your worldview, I was curious, what newspapers and magazines did you regularly read before you were tapped for this to stay informed and to understand the world?
Palin: I’ve read most of them, again with a great appreciation for the press, for the media.
Couric: What, specifically?
Palin: Um, all of them, any of them that have been in front of me all these years.
Couric: Can you name a few?
Palin: I have a vast variety of sources where we get our news, too.
CBS News

October 1, 2008: “Well, let’s see. There’s — of course — in the great history of America rulings there have been rulings.” Sarah Palin (When asked by Couric to name a Supreme Court decision, other than Roe vs. Wade, that she disagreed with; CBS News)

News Coverage of Sarah Palin

12 Nov
News Coverage of Sarah Palin and Joseph Biden
News Coverage of Sarah Palin and Joseph Biden
Ratio of News stories by day covering Sarah Palin and Joseph Biden
Ratio of News stories by day covering Sarah Palin and Joseph Biden
Coverage of Sarah Palin's interview with Gibson, Couric, and Tina Fey's impersonation on SNL
Coverage of Sarah Palin's interview with Gibson, Couric, and Tina Fey's impersonation on SNL
Ratio of stories citing John McCain that also cited Sarah Palin vis-a-vis Ratio of stories citing Barack Obama that also mentioned Joe Biden
Ratio of stories citing John McCain that also cited Sarah Palin vis-a-vis Ratio of stories citing Barack Obama that also mentioned Joe Biden
Palin's coverage: MSNBC and Fox
Palin's coverage: MSNBC and Fox

Difference in Coverage of Clinton and Obama

12 Mar

The following tables tally up the articles that mention “Barack Obama” or “Hillary Clinton” in their body or title. The results show that both New York Times and Washington Post cover Obama at much lower rates than “US Newspaper and Wires.”

The differences are particularly significant given that most articles follow the “horse race” format, and hence mention both Obama and Clinton.

  WP Clinton* WP Obama* NYT Clinton* NYT Obama* LN Obama* LN Clinton*
Apr-07 85 81 98 113 3479 2324
May-07 110 87 110 93 3309 2373
Jun-07 122 101 112 78 3425 2874
Jul-07 142 104 104 82 3820 3276
Aug-07 119 109 103 76 4070 3201
Sep-07 152 115 160 94 4088 3649
Oct-07 171 108 159 95 4813 4742
Nov-07 174 111 164 108 4391 4445
Dec-07 195 169 194 164 6134 4774
Jan-08 342 354 357 335 15276 10540
Feb-08 315 330 409 436 18857 11658
  LN Ratio WP Ratio NYT Ratio
Apr-07 1.50 0.95 1.15
May-07 1.39 0.79 0.85
Jun-07 1.19 0.83 0.70
Jul-07 1.17 0.73 0.79
Aug-07 1.27 0.92 0.74
Sep-07 1.12 0.76 0.59
Oct-07 1.01 0.63 0.60
Nov-07 0.99 0.64 0.66
Dec-07 1.28 0.87 0.85
Jan-08 1.45 1.04 0.94
Feb-08 1.62 1.05 1.07
Avg. 1.27 0.81 0.78

*Article count from LexisNexis Power Search with search term “Barack Obama” and “Hillary Clinton” respectively. The source field was constrained to “New York Times”, “Washington Post”, and “US Newspaper and Wires” respectively.

The Hubris of Journalists

18 Jan

Nick Bryant, a correspondent covering Australian elections for the BBC, wrote the following in one of his blog posts,

“My name is Nick and I fear I am in danger of becoming an Australian political junkie. I find myself boring friends with the swings needed to win obscure marginals, which, up until six weeks ago, I never knew existed. My mind is cluttered with useless information, like how the South Australian seat of Makin is named after a post-war Australian ambassador to Washington.

Had you asked me 18 months ago, I would have hazarded a guess that Eden-Monaro was a type of Dutch cheese. Now I can quote the land mass of this all-important bellwether seat.”

While Nick Bryant did a reasonable job of reporting on the Australian elections, it is debatable whether journalists can start filing in-depth analytically rich reports on a country days after landing in a country about which they know next to nothing.

Another reporter, Kevin Connolly, from the BBC – this time covering the US elections wrote,

“On your first days in a new assignment as a reporter, you work hard – sometimes a little too hard – to look for clues that will help you to decode life in your new adopted home.

When we changed planes in Chicago midway through my never-ending New Year’s Eve, I found myself lingering in the self-help section of the bookstore, puzzled by the sort of advice for which Americans are prepared to pay. I now own copies of God Wants You To Be Rich and You’re Broke Because You Want to Be. “

There is a danger that journalists new to the country will weigh idiosyncratic details about the country they notice disproportionately in their analysis and reporting.

Good reporting is seen as a tough-minded commitment to pursuing the truth. It is seen as a skill that surpasses bounds of geography and culture. And certainly, there are elements of it that remain constant throughout. However, lack of a deeper understanding of the country, and culture can severely jeopardize not only “ability to contextualize events and issues, but also “objective” elements of reporting. The ability to contextualize is of particular importance for the apathetic ill-informed home country readership.

The foreign reporting standards have dropped precipitously as the length of foreign tours has dropped precipitously over the last many years to now average between one and three years. Reports from foreign journalists nowadays often take the quality of a tourist blog with substandard reports about preconceived notions that need validation. In this age of the Internet, I am in fact unsure of the need for foreign journalists. Liaisons with prominent news organizations within the country should be pursued to produce reports.

While the problem of under-qualified reporters is the most prominent in foreign reporting, it is not limited to it. Greenhorns reporting on politics often times carry the open-eyed celebrity wonderment about the political figures they report on.