Qualitative Vs. Quantitative Methods

9 Jun

Epistemology of Causality

How do we know that something is the ’cause’ of something and how do we impute ‘causality’ through data?

To impute causality in quantitative models, we rely on the argument that it is unlikely that the change in Y could be explained by anything else other than X since we have ‘statistically controlled for other variables’. We ‘control’ for variables via experiments or we can do it via regression equations. This allows us to isolate the effect of say variable x on y. There are of course some caveats and some assumptions that go along with using these methods but robust experimental designs still allow us to impute causality in a fairly robust way. Generally, the causal claim is buffeted with a description of a plausible causal pathway. All of the analysis and the resulting benefits of reliably imputing causality are predicated on our ability to ‘correctly’ assign numbers to ‘constructs’ (the real variables of interest).

Let’s analyze now how qualitative methods can impute causality. While it seems reasonable to assume that ‘systematic’ ‘qualitative’ analysis of a problem can provide us with a variety of causal explanations and under most circumstances provide us with a reasonably good idea of how much each of the explanatory variables affects the dependent variable, there are crucial problems and limitations that may induce bias in the analyses. Additionally, we must define what constitutes as ‘systematic’ analysis.

Another thing to keep in mind is that ethics and rigor are not enough to impute causality. What one needs are the right epistemic tools.

A lot of qualitative research is marred by the fact that it ‘selects on the dependent variable’. In other words, it sees a dependent variable and then goes sleuthing for the possible causal mechanisms. It is hard in that case to impute wider causality between variables because the relationship hasn’t been tested for varying levels of X and Y. It is useful to keep in mind that sometimes it is all that we can hope to achieve. Additional problems can emerge from things like “selection bias” and logical fallacies like “Post hoc ergo propter hoc”. Partly the way qualitative research is written can also impose its own demands and biases including demands for narrative consistency.

It is unclear to me whether a system exists to impute causality reliably using qualitative methods. There are however some techniques that qualitative methods can borrow from quantitative methods to improve any causal claims that they may be inclined to make – one is to use a representative set of variables, the other is to look for ‘natural experiments’, and pay attention to larger sociological issues and iterate through why alternative explanations don’t apply as well here – a sort of a verbal regression equation.

There are of course instances where deeper more in-depth analysis of few cases allows one to get a deeper understanding of the issue but that shouldn’t be mistaken as coming up with causes.

Epistemology of generalization in empirical methods

There is very little space that we get edgeways when we think about a systematic theory of generalization for empirical theories unless. To generalize we must either ‘know’ fundamental causal mechanisms and how they work under a variety of contextual factors or use probability sampling. Probability sampling theories are built on the belief that we know nothing about the world. Hence we need to take care to collect data (which ideally transposes to the constructs) in a way that makes it generalizable to the entire population of interest.

Causal arguments in Qualitative research

For making ‘well grounded’ causal arguments in qualitative research – say with a small n – the case must be made for generalizability of the selected cases, use deduction to articulate possible causal pathways, and then bring them together in a ‘verbal regression equation’ and analyze which of the causal pathways are important – as in likely or have a large effect size- and which are not.

Epistemic standards in interpretation and methodology

Quantitative methods share a broad repertoire of skills that is shared across the disciplines while comparatively no such common epistemic standards exist across a variety of qualitative sub-streams that differ radically in terms of what data to look at and how to interpret the data. Common epistemic standards allow for research to be challenged in a variety of ways. From Gay and Lesbian studies to Feminist Scholarship to others – there is little in common in terms of epistemic standards and how best to interpret things. What we then have is merely incommensurability. Partly, of course, different questions are being asked but even when same questions are being asked – there appears to be little consensus as to what explanation is preferred over the other. While each new way to “interpret” facts in some ways does expand our understanding of the social phenomena, given the incommensurability in epistemic standards –we cannot bring all of them to a qualitative ‘verbal regression equation’ (my term) through which we can reliably infer the size of the effect of each.

Caveat Lector
The above article deals with the debate between qualitative methods and quantitative methods on a small select sample of issues – generalizability and causality – that are explicitly more tractable through quantitative models. It would be unwise to construe larger points about the relevance of qualitative methods from the article.

Social Science and the Theory of All

22 Apr

Social phenomenon, unlike natural phenomenon, is bound and morphed not only by nature (evolution, etc.) but also history, institutions (religious, governance, etc.), and technology, among others. Before I go any further, I would like to issue a caveat: the categories that I mention above are not orthogonal and in fact, do trespass into each other regularly. We can study particular social phenomena in aggregate through disciplines like political science, which study everything from study of psychology to institutions to history, or study them by focusing on one particular aspect – psychology or genetics – and investigating how each effect multiple social phenomena like politics, communication, etc.

Given the disparate range of fields that try to understand the social phenomenon, often the field is straddled with multiple competing paradigms and multiple theories within or across those paradigms with little or no objective criteria on which the theories can be judged. This is not to say that theories are always mutually irreconcilable for often they are not (though they may be seen as such – which is an artifact of how they are sold), or that favoring one theory automatically implies rejecting others. The success of a theory, hence, often depend on how well it is sold and the historical proclivities of the age.

Proclivities of an age; theories of an age

Popular paradigms emerge over time and then are discarded for entirely new ones. It is not that the old don’t hold but just that the new ones hold the imagination of the age. Take for example variables that people have chosen to describe culture over the ages – Weber argued religion was culture, Marx argued that political economy was culture, Freud proposed a psycho-analytical take on culture (puritan, liberated, etc.), Carey proposed communication as culture, political theorists have argued institutions as culture, bio-evolutionists argue that cognition and bio-rootedness are primary determinants of culture, Tech. evangelists have argued technology is culture, while others have argued that infrastructure dictates culture.

It is useful to acknowledge that the popularity of the paradigms that were used to define culture had something to do with the most important forces shaping culture at that particular time. For example, it is quite reasonable to imagine that Marx’s paradigm was a useful one for explaining the industrial society (in fact it continues to be useful), while Carey’s paradigm was useful to explain the results of rapid multiplication (and accessibility) of communication (mass-) media. I would like to reissue this caveat that adopting new paradigms doesn’t automatically imply rejecting the prior ones. In fact intersection of old and new paradigms provide fecund breeding grounds for interesting arguments and theories – for example political economy of mass media and its impact. Let me illuminate the point with another example from Political Science which a decade or so ago saw a resurgence of cultural theory at the back of Huntington’s theory of ‘Clash of civilizations’. Huntington’s theory didn’t mean an end to traditional paradigms like economic competition; it just postulated that there was another significant variable that needed to be factored in the discourse.

The structure of scientific revolutions

Drawing extensively from historical evidence from the natural sciences, Thomas Kuhn, a Harvard physicist, argued in his seminal book, The Structure of scientific revolutions, that science progressed through “paradigm shifts.” While natural sciences paid scant attention to the book, the book provoked an existentialist crisis within the social sciences. To arrive at that crisis point, social scientists made a number of significant leaps (not empirically based) from what Kuhn said – they argued that growth of social science was anarchic, its judgments historically situated and never objective, and hence the social sciences were pointless – or more correctly had a point but were misguided. This self-flagellation is typical in social sciences that have always been more introspective about their role and value in society as compared to the natural sciences, which have always proceeded with the implicit assumption that ‘progress’ cannot be checked and eventually what they produce are merely tools in service of humanity. Of course, that is quite bunk and has been exposed as such without making even the slightest dent in the research in science and technology. Criticizing natural sciences, especially the majority of it that is in service of ‘value-free’ economics, doesn’t take away from the questions that Kuhn posed for the social sciences. Social scientists, in my estimation, put disproportionate emphasis on Kuhn’s work. Social science is admittedly much behind in terms of coming up with generalizable theories, but they have been quite successful in identifying macro-variables and phenomena.

The most intractable problem that social scientists need to deal with is answering what is the purpose of their discipline. Is it to describe reality or to critique it or engineer alternative realities? If indeed it is all of above, and I believe it is, then social science must think about melding its often disparate traditions – theory and practice.

Rorty and the structure of philosophical revolutions

Richard Rorty in his book, Philosophy and the Mirror of Nature, launches a devastating attack on philosophy – especially its claims to any foundational insights. Rorty traces the history of philosophy and finds that the discipline is embedded, much more deeply than social science, in the milieu of paradigm shifts – philosophers from different ages not only offer different “foundational” insights but often deal with different problems altogether.

Battling at the margins

Those who argue that the singular purpose of social science should be to normatively critique it and offer alternative paradigms are delusional. Understanding how a society works (or how institutions work, people work) is important to craft interventions – be it drug policy or engineering new governance systems. Normative debates often times are nothing but frivolous debates at the margins. The broad overarching problems of today don’t need normative theorists devoted to analysis – though I don’t dispute their contribution – they are evident and abundantly clear. When we take out the vast middle of what needs to be decided, normative theory becomes a battle at the margins.

Post-positivist theorizing; and the sociology of research

The most significant challenges for social science as discipline lie within the realm of how the discipline aggregates research and moves forward and how that process is muzzled by a variety of factors.
Imre Lakatos sees “history of science in terms of a continuous competition between alternative research programs rather than of successive conjectures and refutations on the one hand, or of total paradigm-switches on the other.” Lakatos argues that any research program possess a kernel of theoretical principles which are taken as fixed and hence create a ‘negative heuristic’ that forbids release of anomalous results, and instead scientists are directed to create a “protective belt” of auxiliary assumptions intended to secure correctness of theoretical principles at the core. Finally, ‘positive heuristic’ is at work to “Defend and extend!” (Little, 1981)

Post-positivist scientific philosophy, like the ones forwarded by Kuhn and Lakatos, raise larger questions about the nature (and viability) of the scientific enterprise. While we may have a firmer grasp of what we mean by a good scientific theory, we are still floundering when it comes to creating an ecosystem that foments good social science and creates a rational and progressive research agenda. (Little, 1981) We must analyze the sociology, and political economy of journal publication as the whole venture is increasingly institutionalized and as careerism, etc. become more pronounced.

Social Science, Epistemology, and Future Directions in Research

4 Oct

There is a schism that runs right in the middle of social science divvying up the field between the critical theorists and the positivists. Positivists aspire to model the success in natural science and hence tend to focus on the causality and statistical proof, according to Dr. Tang at the National Taiwan University. Critical school, on the other hand, starts from some starts from some philosophical axioms, for example, “social good” or “justice for all” or “maximizing social good.”

The approach by critical theorists is riven with difficulties due to the multiplicity of the philosophical starting points around which one can build theories. As one would expect, critical theory today has myriad “schools,” each based on different philosophical assumptions and each with, if one may say, their own geometry and calculus which works only in their own universe. Positivists circumvent the epistemological and other philosophical issues that dog the critical theorists by relying on the natural science model of doing research that stresses on coming up with a falsifiable hypothesis that can then tested either via experimentation or observation (within a representative randomized sample). Given the difficulty of defining and measuring useful variables in social science, positivists rely upon their own methodological axioms though a lot of research is currently underway to help refine the methodology.

The dichotomy in the field brings one to question how social science should ideally be conducted. The answer depends on what one expects from social science. One may argue that social science has ceded its primary responsibility of trying to resolve the dispute between different philosophical paradigms and wrestling with issues pertaining to the nature and future of society. Without the moral or philosophical grounding, a lot of research may seem like monkey work – repetitive and commercial applications aside utterly aimless. On the other hand, one may argue that quantitative work often illuminates how humans and society works and it is first important to understand both of them before we move on to the task of circumscribing their behavior in philosophy. I would argue that social science’s aims need to be a hybrid of both of the strands. Social science needs to continue to grasp with the important epistemological and philosophical questions that underpin our existence and provide direction in a way to where research is headed. At the same time, social science needs to be more pro-active in understanding humans and society.