Sample this

1 Aug

What do single shot evaluations of MT (replace it with anything else) samples (vis-a-vis census figures) tell us? I am afraid very little. Inference rests upon knowledge of the data (here – respondent) generating process. Without a model of the data generating process, all such research reverts to modest tautology – sample A was closer to census figures than sample B on parameters X,Y, and Z. This kind of comparison has a limited utility: as a companion for substantive research. However, it is not particularly useful if we want to understand the characteristics of the data generating process. For even if respondent generation process is random, any one draw (here – sample) can be far from the true population parameter(s).

Even with lots of samples (respondents), we may not be able to say much if the data generation process is variable. Where there is little expectation that the data generation process will be constant, and it is hard to understand why MT respondent generation process for political surveys will be a constant one (it likely depends on the pool of respondents, which in turn perhaps depends on the economy etc., the incentives offered, the changing lure of incentives, the content of the survey, etc.), we cannot generalize. Of course one way to correct for all of that is to model this variation in the data generating process, but that will require longer observational spans, and more attention to potential sources of variation etc.

Moving further away from the out-party

1 Jun

Two things are often stated about American politics (most vociferously by Mo Fiorina): political elites are increasingly polarized, and that the issue positions of the masses haven’t budged much. Assuming such to be the case, one expects the average distance between where partisans place themselves and where they place the ‘in-party’ (or the ‘out-party’) to increase. However it appears that the distance to the in-party has remained roughly constant (and very small), while distance to the out-party has grown, in line with what one expects from the theory of ‘affective polarization’ and group-based perception.

‘Representativeness heuristic’, base rates, and Bayes

23 Apr

From the Introduction of their edited volume:
Tversky and Kahneman used the following experiment for testing ‘representativeness heuristic’ –

Subjects are shown a brief personality description of several individuals, sampled at random from 100 professionals – engineers and lawyers.
Subjects are asked to assess whether description is of an engineer or a lawyer.
In one condition, subjects are told group = 70 engineers/30 lawyers. Another the reverse = 70 lawyers/30 engineers.

Results –
Both conditions produced same mean probability judgments.

Discussion –
Tversky and Kahneman call this result a ‘sharp violation’ of Bayes Rule.

Counter Point –
I am not sure the experiment shows any such thing. Mathematical formulation of the objection is simple and boring so an example. Imagine, there are red and black balls in an urn. Subjects are asked if the ball is black or red under two alternate descriptions of the urn composition. When people are completely sure of the color, the urn composition obviously should have no effect. Just because there is one black ball in the urn (out of say a 100), it doesn’t mean that the person will start thinking that the black ball in her hand is actually red. So on and so forth. One wants to apply Bayes by accounting for uncertainty. People are typically more certain (lots of evidence it seems – even in their edited volume) so that automatically discounts urn composition. People may not be violating Bayes Rule. They may just be feeding the formula incorrect data.

Interviewer Assesments of Respondent’s Level of Political Information

15 Mar

In the National Election Studies (NES), interviewers have been asked to rate respondent’s level of political information – “Respondent’s general level of information about politics and public affairs seemed – Very high, Fairly high, Average, Fairly low, Very low.” John Zaller, among others, have argued that these ratings measure political knowledge reasonably well. However there is some evidence that challenges the claim. For instance, there is considerable unexplained inter and intra-interviewer heterogeneity in ratings – people with similar levels of knowledge (as measured via closed-ended items) are rated very differently (Levendusky and Jackman 2003 (pdf)). It also appears that mean interviewer ratings have been rising over the years, compared to the relatively flat trend observed in more traditional measures (see Delli Carpini, and Keeter 1996 and Gilens, Vavreck, and Cohen 2004, etc).

Part of the increase is explained by higher ratings of respondents with less than a college degree; ratings of respondents with BS or more have remained somewhat more flat. As a result, difference in ratings of people with a Bachelor’s Degree or more and those with less than a college degree is decreasing over time. Correlation between interviewer ratings and other criteria like political interest are also trending downward (though decline is less sharp). This conflicts with evidence for increasing ‘knowledge gap’ (Prior 2005).

The other notable trend is the sharp negative correlation (over .85) between intercept and slope of within-year regressions of interviewer ratings and political interest, education, etc. This sharp negative correlation hints at possible ceiling effects. And indeed there is some evidence for that.

Interviewer Measure – The measure is sometimes from the pre-election wave only, other times in the post-election wave only, and still other times in both waves. Where both pre and post measures were available, they were averaged. The correlation between pre-election and post-election rating was .69. The average post-election ratings are lower than pre-election ratings.

Comparing datasets, reporting only non-duplicated rows

27 Feb

The following is in response to question on the R-Help list.

Consider two datasets –

reported <-
structure(list(Product = structure(c(1L, 1L, 1L, 1L, 2L, 2L,
3L, 4L, 5L, 5L), .Label = c(“Cocoa”, “Coffee C”, “GC”, “Sugar No 11″,
“ZS”), class = “factor”), Price = c(2331, 2356, 2440, 2450, 204.55,
205.45, 17792, 24.81, 1273.5, 1276.25), Nbr.Lots = c(-61L, -61L,
5L, 1L, 40L, 40L, -1L, -1L, -1L, 1L)), .Names = c(“Product”,
“Price”, “Nbr.Lots”), row.names = c(1L, 2L, 3L, 4L, 6L, 7L, 5L,
10L, 8L, 9L), class = “data.frame”)

exportfile <-
structure(list(Product = c(“Cocoa”, “Cocoa”, “Cocoa”, “Coffee C”,
“Coffee C”, “GC”, “Sugar No 11″, “ZS”, “ZS”), Price = c(2331,
2356, 2440, 204.55, 205.45, 17792, 24.81, 1273.5, 1276.25), Nbr.Lots = c(-61,
-61, 6, 40, 40, -1, -1, -1, 1)), .Names = c(“Product”, “Price”,
“Nbr.Lots”), row.names = c(NA, 9L), class = “data.frame”)

Two possible solutions –
A.
m <- rbind(reported, exportfile)
m1 <- m[duplicated(m),]
res <- m[is.na(match(m$key, m1$key)),]

B.

exportfile$key <- do.call(paste, exportfile)
reported$key <- do.call(paste, reported)
a <- reported[is.na(match(reported$key, exportfile$key)),]
b <- exportfile[is.na(match(exportfile$key, reported$key)),]
res <- rbind(a, b)

Correcting for Differential Measurement Error in Experiments

14 Feb

Differential measurement error across control and treatment groups or in a within-subjects experiment, pre and post-treatment measurement waves, can vitiate estimates of treatment effect. One reason for differential measurement error in surveys is differential motivation. For instance, if participants in the control group (pre-treatment survey) are less motivated to respond accurately than participants in the treatment group (post-treatment survey), the difference in means estimator will be a biased estimator of the treatment effect. For example, in Deliberative Polls, participants acquiesce more during the pre-treatment survey than the post-treatment survey (Weiksner, 2008). To correct for it, one may want to replace agree/disagree responses with construct specific questions (Weiksner, 2008). Perhaps a better solution would be to incentivize all (or a random subset of) responses to the pre-treatment survey. Possible incentives include – monetary rewards, adding a preface to the screens telling people how important accurate responses are to research, etc. This is the same strategy that I advocate for dealing with satisficing more generally (see here) – which translates to minimizing errors, than the more common, more suboptimal strategy of “balancing errors” by randomizing the response order.

Against Proxy Variables

23 Dec

Lacking direct measures of the theoretical variable of interest, some rely on “proxy variables.” For instance, some have used years of education as a proxy for cognitive ability. However, using “proxy variables” can be problematic for the following reasons — (1) proxy variables may not track the theoretical variable of interest very well, (2) they may track other confounding variables, outside the theoretical variable of interest. For instance, in the case of years of education as a proxy for cognitive ability, the concerns manifest themselves as follows —

1) Cognitive ability causes, and is a consequence of, what courses you take, and what school you go to, in addition to of course, years of education. GSS for instance contains more granular measures of education – for instance did the respondent take science course in college. And nearly always the variable proves significant when predicting knowledge, etc. This all is somewhat surmountable as it can be seen as measurement error.

2) More problematically, years of education may tally other confounding variables – diligence, education of parents, economic strata, etc. And then education endows people with more than cognitive ability; it also causes potentially confounding variables such as civic engagement, knowledge, etc.

Conservatively we can only attribute the effect of the variable to the variable itself. That is – we only have variables we enter. If one does rely on proxy variables then one may want to address the two points mentioned above.

Education and Economic Inequality

7 Dec

People seem to believe that increasing levels of education will reduce economic inequality. However, it isn’t clear if the policy is empirically supported. Here are some potential ways increasing levels of education can impact economic inequality –

  1. As Grusky argues, the current high wage earners whose high wages depend on education and lack of competition from similarly educated men and women (High Education Low Competition or HELCO) from similarly highly educated will start earning a lower wage because of increased competition (thereby reducing inequality). This is assuming that HELCO won’t respond by trying to burnish their education credentials, etc. This is also assuming that HELCO exists as a large class. What likely exists is success attributable to networks, etc. That kind of advantage cannot be blunted by increasing education of those not in the network.
  2. Another possibility is that education increases the number of high paying jobs available in the economy and it raises the boats of non-HELCO more than HELCO.
  3. Another plausible scenario is that additional education produces only a modest effect with non-HELCO still mostly doing low paying jobs. This may due to only a modest increase in overall availability of ‘good jobs.’ Already easy access to education has meant that many a janitor and store clerk walk around with college degrees (see Why Did 17 Million Students Go to College?, and The Underemployed College Graduate).

Without an increase in ‘good jobs,’ the result of increase in education is an increased heterogeneity in who succeeds (random draw at the extreme) but no change in proportion of successful people. Or, increasing equality of opportunity (a commendable goal) but not reduction in economic inequality (though in a multi-generation game, it may even out). Increasing access to education also has the positive externality of producing a more educated society, another worthy goal.

How plentiful the ‘good jobs’ are depends partly on how the economic activity is constructed. For instance, there may have once have been a case for only hiring one ‘super-talented person’ (say ‘superstar’) for a top shelf job (say CEO). Now we have systems that can harness the wisdom of many. It is also plausible that that wisdom is greater than that of the superstar. It reasons then that the superstar be replaced; economic activity will be more efficient. Or else let other smart people who can contribute equally (if educated) be recompensed alternately for doing work that is ‘beneath them’.

R – Recoding variables reliably and systematically

12 Nov

Survey datasets typically require a fair bit of repetitive recoding of variables. Reducing errors in recoding can be done by writing functions carefully (see some tips here) and automating and systematizing naming, and application of the recode function (which can be custom) –

fromlist <- c(“var1″, “var2″, “var3″, “var4″, “var5″)
tolist <- paste(c(“var1″, “var2″, “var3″, “var4″, “var5″), “recoded”, sep=””)
data[,tolist] <- sapply(data[,fromlist], function(x) car::recode(x , “recode.directions”))

Simple functions can also be directly applied to each column of a data.frame. For instance,
data[,tolist] <- !is.na(data[,fromlist])
data[,tolist] <- abs(data[,fromlist] – .5)