Epistemology

King et al. (1994: 75, note 1): ‘[a]t its core, real explanation is always based on causal inferences’.

Limiting Discussion to positivist Qualitative Methods

Qualitative Methods can be roughly divided into positivist (case-studies etc.) and interpretive. I will limit my comments to positivist Qualitative Methods. The differences between positivist qualitative and quantitative methods ‘are only stylistic and are methodologically and substantively unimportant’ (King et al., 1994:4). Both methods share ‘an epistemological logic of inference: they all agree on the importance of testing theories empirically, generating an inclusive list of alternative explanations and their observable implications, and specifying what evidence might infirm or affirm a theory’ (King et al. 1994: 3).

Causal Inference in Empirical Data

To impute causality, science relies either on evidence based process knowledge, or clever experiment design that obviates (perhaps more correctly, mitigates) the need to know the process – though researchers often are encouraged to have a story to explain the process, and test variables implicated in the ’story’.

Experimentation provides one of the best ways to reliably impute causality. However for experiments to have value outside (the labs), the ‘treatment’ must be ‘ecological’ as in reflect the typical values that the variables takes in the world –for example, effect of news is best measured with either real life news clips or something similar, and the findings must hold up in the field (through surveys/field experiments). The problem is that most problems in Social Science cannot be studied experimentally. Brady et al. (2001:8) write: ‘A central reason why both qualitative and quantitative research are hard to do well is that any study based on observational (i.e., non-experimental) data faces the fundamental inferential challenge of eliminating rival explanations’. While most Social Science papers (and certainly practitioners) talk about ‘eliminating’ rival explanations, one doesn’t quiet have to do that. Social Scientists often times include variables reflecting ‘rival explanations’ as ’straw variables’ (to refer to the straw man they will blow at the end) in a regression equation, to show how much variance is ‘explained’ by their variable of choice as compared to the ’straw variables’.

In Quantitative Methods, to impute causality, one makes a variety of assumptions including: a ‘ceteris paribus’ (all other things being equal – which may mean assigning away everything ‘else’ to randomization) clause, error term (non-systematic) part is not correlated with other independent variables, to infer the correlation between an explanatory variable x and dependent variable y can only be explained as x’s effect on y. In analyzing survey data, one ‘controls’ for variables using regression, or some other similar way. There are a variety of assumptions in regression models and penalty for violation of each of these assumptions. In Qualitative Methods, one can either analytically (or where possible empirically) control for variables, or trace the process.

Qualitative Methods: some handles on generalizability

Traditional probability sampling theories are built on the highly conservative assumption that we that we know nothing about the world. And the only systematic way to go about knowing it is through ‘random sampling’, a process that delivers ‘representative data’ on average. Newer sampling theories, however, acknowledge what we know about the world by selectively over-sampling things (or people) we are truly clueless about, and under-sampling where we have a good idea. For example, polling organizations under-sample self described partisans, and over-sample non-partisans. This provides a window for positivist qualitative methods to make generalizable claims. Qualitative methods can overcome their limitations and make legitimate generalizable claims if their sampling reflects the extent of prior knowledge about the world.

Other than sampling, there are other analytical ways of getting a handle on the variables that ‘moderate’ the effect of a particular variable that we may be interested in studying. For example, we can analytically think through how education will affect (or not affect) racist attitudes. Analytical claims are based on deductive logic and a priori assumptions or knowledge. Hence the success of analytical claims is contingent upon the accuracy of the knowledge, and the correctness of the logic.

One of the problems that have been repeatedly pointed out about Qualitative research is its propensity to select on the dependent variable. Selection on dependent variable deviously leaves out cases where for example the dependent variable doesn’t take extreme values. Selection bias can not only lead to misleading conclusions about causal effects but also about causal processes. It is important hence not to use a truncated dependent variable to do one’s analysis. One of the ways one can systematically drill down to causal processes in qualitative research is by starting off with the broadest palette, either in prior research or elsewhere, to grasp the macro-processes, and other variables that may affect the case. Then cognizant of the particularistic aspects of a particular case, analyze the microfoundations or microprocesses present in the system.

Positivism and Empiricism

Scientific method roughly refers to systematic analysis of empirical data. The quality and the strength of the ’system’ – forever open to challenge – determine any claims of epistemic superiority that the ’scientific method’ may make over other competing claims of gleaning ‘knowledge’ from data. The extent to which claims are solely arbitrated on scientific merit is limited by a variety of factors, as outlined by Lakatos, Kuhn, and Feyerabend, resulting in at best an inefficient process and at worst something far more pernicious. I however ignore such issues and focus narrowly on methodological questions around causality and generalizability in qualitative methods.

In science, inquiry into generalizable causal processes is greatly privileged, and for good reason – causality and generalizability taken together can provide basis for policy action, for mounting intervention, among other things. One can also think of knowledge of causal processes as providing predictive power. However, not all kinds of data make themselves readily accessible to imputing causality, or even to making generalizable descriptive statements. For example, causal inference in most historical research remains out of bounds. AddKeeping this in mind, I analyze how qualitative methods within Social Sciences (can) interrogate causality and generalizability.

Causality:

Hume felt that there was no place for causality within empiricism. He argued that the most we can find is that “the one [event] does actually, in fact, follow the other”. More broadly, causality is nothing but an illusion occasioned when events follow each other with regularity. That formulation however didn’t prevent Hume from believing in scientific theories for he felt that regularly occurring constant conjunctions were sufficient basis for scientific laws. Theoretical advances in the 200 or so years since Hume have been able to provide a deeper understanding of causality, including a process based understanding and an experimental understanding.

Donald Rubin, a Professor of Statistics at Harvard, defines causal effect as, “Intuitively, the causal effect of one treatment, E, over another, C, for a particular unit and an interval of time from t1 to t2 is the difference between what would have happened at time t2 if the unit had been exposed to E initiated at t1 and what would have happened at t2 if the unit had been exposed to C initiated at t1: ‘If an hour ago I had taken two aspirins instead of just a glass of water, my headache would now be gone,’ or because an hour ago I took two aspirins instead of just a glass of water, my headache is now gone.’ Our definition of the causal effect of the E versus C treatment will reflect this intuitive meaning.”

It is important to note that RCM as presented above depicts an elementary causal connection between two Boolean variables: one level of explanatory variable (two aspirins) with single effect (headache is gone). Often times, the variables take multiple values and to study that, we need to mount a series of experiments. Similarly one may want to analyze the effect of a variable in different subgroups of populations, for example, effect of aspirin on women as compared to men. All the above two scenarios do is highlight the problems in coming up with a robust understanding of causal processes between (here) two variables. Fundamentally though, our RCM understanding of causal effect remains unchanged.

Rubin Causal Model provides a counterfactual deterministic understanding of causality that is firmly based in the logic of experiment design. RCM formulation can be expanded to include a probabilistic understanding of causal effect. Just as a note: A probabilistic understanding of causality implicitly accepts that certain parts of the explanation are still missing, and hence is absent of a necessary and sufficient condition though attempts have been made to include necessary and sufficient clauses in probabilistic statements. David Papineau (Probabilities and Causes, 1985, Journal of Philosophy) writes, “Factor A is a cause of some B just in case it is one of a set of conditions that are jointly and minimally sufficient for B. In such a case we can write A&X ->B. In general there will also be other sets of conditions minimally sufficient for B. Suppose we write their disjunction as Y. If now we suppose further that B is always determined when it occurs, that it never occurs unless one of these sufficients sets (let’s call them B’s full causes) occurs first, then we have, A and X condition conjugated with Y is equivalent with B. Given this equivalence, it is not difficult to see why A’s causing B should be related to A’s being correlated with B. If A is indeed a cause of B, then there is a natural inference to Prob(B/A) > Prob(B/-A): for, given A, one will have B if either X or Y occurs, whereas without A one will get B only with Y. And conversely it seems that if we do find that Prob(B/A) > Prob(B/-A), then we can conclude that A is a cause of B: for if A didn’t appear in the disjunction of full causes which are necessary and sufficient for B, then it wouldn’t affect the chance of B occurring.”

Papineau’s definition is a bit archaic, and doesn’t quite cover the set of cases we define as probabilistically causal. John Gerring (Social Science Methodology: A Criterial Framework, 2001: 127,138; emphasis in original), provides a definition of probabilistic causality: ‘[c]auses are factors that raise the (prior) probabilities of an event occurring. (…) [Hence] a sensible and minimal definition: X may be considered a cause of Y if (and only if) it raises the probability of Y occurring.’

A still more ’sensible’ and still ‘minimal’ definition of causality, can be found in Gary King et al. (Designing Social Inquiry: Scientific Inference in Qualitative Research, 1994: 81-82), ‘the causal effect is the difference between the systematic component of observations made when the explanatory variable takes one value and the systematic component of comparable observations when the explanatory variable takes on another value.’

Brief discussion on Causal Inference in Qualitative and Quantitative Methods

While the above formulations of causality – Rubin Causal Model, Gerring, and King – seem more quantitative, they can be applied equally to qualitative methods. A parallel understanding of causality, used much more often in qualitative social science, is a process based understanding of causality wherein you trace the causal process to construct a theory. Simplistically speaking, in Quantitative methods in Social Sciences, one often times deduces the causal process, while in Qualitative methods the understanding of the causal process is induced from deep and close interaction with ‘data’.

Both deduction and induction processes, however, are rife with problems. Deduction privileges formal rules – statistics – that straightjacket the systematic deductive process so that the deductions are systematic and cognizant of explicit assumptions (like normal distribution of data, linearity of the effect, lack of measurement error, etc.). The formal deductive process bestows a host of appealing qualities like generalizability (when an adequate random sample of population is taken) or even systematic handle on causal inference. In quantitative methods, the methodological assumptions for deduction are cleanly separated from data. The same separation – between the formal deductive process with a rather arbitrarily chosen statistical model and data – however makes the discovery process less than optimal, and sometimes deeply problematic. Recent research by Ho and King (Matching as Nonparametric Preprocessing for Reducing Model Dependence in Parametric Causal Inference, Political Analysis, 2007), and methods like Bayesian Model Averaging by Volinsky, have gone some ways in providing ways to mitigate problems with model selection.

No such sharp delineation between method and data exists in qualitative research, where data is collected iteratively – [in studies using iterative abstraction (Sayer 1981, 1992; Lawson 1989, 1995) or grounded theory (Glaser 1978; Strauss 1987; Strauss and Corbin 1990)] – till it explains the phenomenon singled out for explanation. Grounded data driven Qualitative methods often run the risk of modeling in particularistic aspects of data, which impedes the reliability with which they can come up with a generalizable causal model. This is indeed only one kind of qualitative research for there are others who do qualitative analysis in the vein of experiments – for example with a 2×2 model, and yet others who will test apriori assumptions by analytically controlling for variables in a verbal regression equation to get at the systematic effect of explanatory variable on the explanandum. Perhaps more than grounded theory method, the pseudo-quantitative style qualitative analysis runs the risk of coming to deeply problematic conclusions based on the cases used.

Epistemology of Causality

How do we know that something is the ’cause’ of something and how do we impute ‘causality’ through data?

To impute causality in quantitative models, we rely on the argument that it is unlikely that the change in Y could be explained by anything else other than X since we have ’statistically controlled for other variables’. We ‘control’ for variables via experiments or we can do it via regression equations. This allows us to isolate the effect of say variable x on y. There are of course some caveats and some assumptions that go along with using these methods but robust experimental designs still allow us to impute causality in a fairly robust way. Generally the causal claim is buffeted with description of a plausible causal pathway. All of the analysis and the resulting benefits of reliably imputing causality are predicated on our ability to ‘correctly’ assign numbers to ‘constructs’ (the real variables of interest).

Let’s analyze now how qualitative methods can impute causality. While it seems reasonable to assume that ’systematic’ ‘qualitative’ analysis of a problem can provide us with a variety of causal explanations and under most circumstances provide us with a reasonably good idea of how much each of the explanatory variables affects the dependent variable, there are crucial problems and limitations that may induce bias in the analyses. Additionally, we must define what constitutes as ’systematic’ analysis.

Another thing to keep in mind is that ethics and rigor are not enough to impute causality. What one needs are the right epistemic tools.

A lot of qualitative research is marred by the fact that it ’selects on the dependent variable’. In other words it sees a dependent variable and then goes sleuthing for the possible causal mechanisms. It is hard in that case to impute wider causality between variables because the relationship hasn’t been tested for varying levels of X and Y. It is useful to keep in mind that sometimes it is all that we can hope to achieve. Additional problems can emerge from things like “selection bias” and logical fallacies like “Post hoc ergo propter hoc”. Partly the way qualitative research is written can also impose its own demands and biases including demands for narrative consistency.

It is unclear to me whether a system exists to impute causality reliably using qualitative methods. There are however some techniques that qualitative methods can borrow from quantitative methods to improve any causal claims that they may be inclined to make – one is to use a representative set of variables, the other is to look for ‘natural experiments’, and pay attention to larger sociological issues and iterate through why alternative explanations don’t apply as well here – a sort of a verbal regression equation.

There are of course instances where deeper more in depth analysis of few cases allows one to get a deeper understanding of the issue but that shouldn’t be mistaken as coming up with causes.

Epistemology of generalization in empirical methods

There is very little space that we get edge ways when we think about a systematic theory of generalization for empirical theories unless. To generalize we must either ‘know’ fundamental causal mechanisms and how they work under a variety of contextual factors or use probability sampling. Probability sampling theories are built on the belief that we know nothing about the world. Hence we need to take care to collect data (which ideally transposes to the constructs) in a way that makes it generalizable to the entire population of interest.

Causal arguments in Qualitative research

For making ‘well grounded’ causal arguments in qualitative research – say with a small n – the case must be made for generalizability of the selected cases, use deduction to articulate possible causal pathways, and then bring them together in a ‘verbal regression equation’ and analyze which of the causal pathways are important – as in likely or have a large effect size- and which are not.

Epistemic standards in interpretation and methodology

Quantitative methods share a broad repertoire of skills that is shared across the disciplines while comparatively no such common epistemic standards exist across variety of qualitative sub-streams that differ radically in terms of what data to look at and how to interpret the data. Common epistemic standards allow for research to be challenged in a variety of ways. From Gay and Lesbian studies to Feminist Scholarship to others – there is little in common in terms of epistemic standards and how best to interpret things. What we then have is merely incommensurability. Partly of course different questions are being asked but even when same questions are being asked – there appears to be little consensus as to what explanation is preferred over the other. While each new way to “interpret” facts in some ways does expand our understanding of the social phenomena, given the incommensurability in epistemic standards –we cannot bring all of them to a qualitative ‘verbal regression equation’ (my term) through which we can reliably infer the size of the effect of each.

Caveat Lector
The above article deals with the debate between qualitative methods and quantitative methods on a small select sample of issues – generalizability and causality – that are explicitly more tractable through quantitative models. It would be unwise to construe larger points about relevance of qualitative methods from the article.

Statistics are only meaningful to the extent that people can identify the phenomenon being measured, come up with a sensible measurement scales to measure primary or secondary observable phenomena and then interpret the results and display them in a lucid fashion. Often times that’s too much to ask and our world is now crumbling under the load of heaps of pointless incomprehensible statistics.

Increasingly, we are trying to understand the world around us via numbers. To this end, a host of research centers and organizations now annually release rankings on issues ranging from corruption to democracy to freedom of press. These rankings are then featured on prime real estate across media and used in homilies, laudatory notes and everything in-between; to buttress indefensible claims; and to bring a sense of “objectivity” to a media saturated with rants of crazed morons.

“Lost in translation” are subtleties of data, methods of data collection and of analysis, and the caveats. What remains, often times, are savaged numbers that peddle whatever theory that you want them to hawk.

Understanding with numbers

The field of social science has been revolutionized in the recent decades with “positivist” approaches using statistics dominating the field. The rise in importance of “numbers” in research is not incidental for numbers provide powerful new ways, particularly statistics, to analyze concepts. Today numbers are used to understand everything from democracy to emotions. But how do we go about measuring things and assigning number to thing which we haven’t yet even been able to define, much less explain?

Let me narrow my focus to creation, interpretation and usage of rankings to substantiate the problems with using statistics.

More Specifically, Rankings

Reporters Sans Frontiers (Reporters without Borders and henceforth called RSF) came out with its annual “Worldwide Press Freedom Rankings”. The latest rankings place USA at 53, along with Botswana and Tonga, India at 105 while Jamaica and Liberia are ranked 26 and 83 respectively. The top ranked South Asian country in the rankings is Bhutan at 98. Intuitively, the rankings don’t make any sense and a little digging into RSF’s methodology for compiling these rankings explains why.

Media’s fascination with rankings

The rankings received wide attention and made it to the front pages of countless newspapers. There is a reason why rankings are the choice nourishment of media starved of any “real information”. Numbers capture, or so is thought, a piece of “objective” information about the “reality”. Their usage is buoyed by the fact that rankings are seductively simple and easy to interpret. Everyone seems to intuitively know the difference between first and second. All that needs to be done is present the fluff, the requisite shock and horror and the article is written.

On to the problems with rankings or the “rank smell”

How can you measure objectively when you need a subjective criterion to come up with a scale?

This is something I raise earlier when I talk about how we can understand concepts like democracy or say emotions using numbers. Researchers do it by assigning number or related phenomenon – in the case of emotions it may be checking the heart rate or doing a brain scan or counting the number of times you use certain words, while in the case of democracy it can be how frequently the elites change, or how many people vote in the elections. But still numerous problems remain especially when we try to order these relatively hazily defined concepts. Say for example the elite turnover in US Congress has of late been fairly close to 2% and that doesn’t seem fairly democratic to me and how does it compare with somewhere like India, where elite turnover may be higher but where members of one family have held key positions in India politics since inception.

Relativity
To rank something means to determine the relative position of something. Rankings NEVER tell one about absolute position of something unless of course they are an incidental result of a score on a shared scale. For instance – RSF’s ranking of USA at 53 in the worldwide press freedom rankings doesn’t tell one whether USA’s press enjoys freedom over say a particular bare threshold below which a functioning press can’t be legitimately said to exist. A lot of people have misinterpreted India’s slide from 80, in 2002, to 105. They believe that it is a slide in absolute terms but the rankings only tell us of a slide in relative terms. There may be an argument to made that India is doing better than it is doing in 2002 in absolute terms but not in relative terms to say other countries. In other words, the press freedom in India may have improved since 2002 but as compared to other countries, India’s press is less free today.

The scale of things

To rank something, one has to use a common scale. Generally a scale, especially one measuring a complex concept like democracy, would be a composite scale of a variety of variables. One now needs to think of a couple of things. How does one weight the variables in the scale between time periods and between countries? For example how do you account for higher usage rates of media in one country (and possibly associated higher level of censorship) to say a country with low media usage and possibly lower total censorship? One may also argue that the media penetration is lower by deliberate action (as in limitation for foreign content owners to broadcast) or other factors (poverty). One must also tackle the problem of assigning “weight” to each facet.

Methodology

RSF ranking are based on a non-representative survey of pre-chosen experts. Hence it is more of a poorly conducted opinion poll rather than a scientific survey. Statistics gets its power of generalizability from the concept of randomization. RSF methodology is more akin to conducting a poll of television pundits on who will win the elections and I am fairly sure that the results would be more often wrong than right.

Secondly, questionnaire includes questions about topics like Internet censorship. No explicit mechanism has been detailed where we know that these scores are weighted based on say Internet penetration in each country. If no cases of Internet censorship was reported in Ghana, and it consequently gets a higher ranking as compared to a country Y whose press is freer but did report one case of Internet censorship – it implies the system is flawed. Let me give you another example. India has the largest number of newspapers in the world and there is a good chance that the total number of journalists harassed may well exceed that of Eritrea. It doesn’t automatically flow that Eritrean press is freer. One may need to account for not only the number of journalists (for more journalists per capita may mean a freer press) but also crime against journalists per capita. In the same vein we may need to account for countries which in general have a high crime rate and where journalists by pure chance, rather than say a government witch hunt, may have a higher chance of dying.

One also need to account for the fact that statistics on these crimes are hard to come by especially in poor countries with barebones media and there is a good chance that they are under-reported there.

On the positive side
Rankings do give one some estimate about the relative freedom of a country. Proximity to Saudi Arabia in the ranking does give us an idea about the relative media freedom.

A lot of the criticism lobbed against India’s low placement in the RSF rankings has been prompted by people’s perception of India as a functioning democracy with a relatively free press. What go unmentioned are episodes like Tehelka and the one faced by Rajdeep Sardesai of recently. India’s press, especially in small towns, is constantly under pressure from the local politicians who monitor aggressively.

What can RSF do?
I would like to see a more detailed report on each country especially marking areas where India is lagging behind. Release more data. Aside from protecting sources, there should be no concerns regarding release of more data. Release it to the world so policy makers and citizens can better understand where improvements need be made.

Release a composite score index that is comparable across time rather than countries. There are far too many problems comparing countries. Controlling for major variables like economic growth etc., we can get a fairly good estimate of how things changed in the course of a set of years.

Conclusion

Whenever we do use numbers to understand concepts, we sacrifice something in what we understand or our conceptual understanding. Some numbers like demographics are relatively non-debatable. Even there debates have arisen in defining who are say Caucasians etc? More debatable are how numbers are used in say the realm of content analysis. What does it mean when a person says a particular word in a sentence? Does it mean that somebody who uses the word “evil” twice in describing Bush hates him twice as much as the person who only uses it once? The understanding and “counting” of words has largely been limited to simple linear additions. We haven’t yet tried understanding strength of words as an equation of countless variables or more importantly learnt how to work with that much data so we use shortcuts in our understanding.

Numbers can give one a sense of false objectivity. The ways numbers are trimmed and chopped to support a particular point of view leave them meaningless, yet powerful.

The problems that I describe above are twin fold – errors in coming up with rankings and errors in reporting the rankings. In all we need to be careful about the numbers we see and use. It doesn’t mean that we need to distrust all the statistics that we see and burrow our nose but we can do well by being careful and honest.

Closing Thought

According to UNECA, Ethiopia “counted 75 000 computers in 2001 and 367 000 television sets in 2000. Only 2.8 % of the total number of households of the country had access to television and approximately 18.4 % of people had a radio station in 1999 and 2000.” These numbers do inform. They talk about poverty. For the West, obsessed with issues of liberty and running from its own increasingly authoritarian regimes, press freedom is “the” issue. In the hustle they miss some of the more important numbers coming from other countries that tell different stories.