Deliberative Poll proceeds as follows – Respondents are surveyed, provided â€˜balancedâ€™ briefing materials, randomly assigned to moderated small group discussions, allowed the opportunity to quiz experts or politicians in plenary sessions, and re-interviewed at the end. The “effect” is conceptualized as average Post-Pre across all participants.
The effect of the Deliberative Poll is contingent upon a particular random assignment to small groups. This isn’t an issue if small group composition doesn’t matter. If it does, then the counter-factual imagination of the â€˜informed publicâ€™ is somewhat particularistic. Under those circumstances, one way want to come up with a distribution of what opinion change may look like if assignment of participants to small groups was different. One can do this by estimating impact of small group composition on the dependent variable of interest, and then predicting the dependent variable of interest under simulated alternate assignments.
See also: Adjusting for covariate imbalance in experiments with SUTVA violations
Consider the following scenario: control group is 50% female while the participant sample is 60% female. Also assume that this discrepancy is solely a matter of chance, and that the effect of the experiment varies by gender. To estimate the effect of the experiment, one needs to adjust for the discrepancy, which can be done via matching, regression, etc.
If the effect of the experiment depends on the nature of the participant pool, such adjustments wonâ€™t be enough. Part of the effect of Deliberative Polls is a consequence of the pool of respondents. It is expected that the pool matters only in small group deliberation. Given people are randomly assigned to small groups, one can exploit the natural variation across groups to estimate how say proportion females in a group impacts attitudes (dependent variable of interest). If that relationship is minimal, no adjustments outside the usual are needed. If however there is a strong relationship, one may want to adjust as follows: predict attitudes under simulated groups from a weighted sample, with probability of selection proportional to the weight. This will give us a distribution â€“ which is correct â€“ as women may be allocated in a variety of ways to small groups.
There are many caveats, beginning with limitations of data in estimating impact of group characteristics on individual attitudes, especially if effects are heterogeneous. Where proportions of subgroups are somewhat small, inadequate variation across small groups can result.
This procedure can be generalized to a variety of cases where effect is determined by the participant pool except where each participant interacts with the entire sample (or a large proportion of it). Reliability of the generalization will depend on getting good estimates.