Monthly Archives: December 2014

Recommended: Report on Survey Participation Refusals

The American Association for Public Opinion Research (AAPOR) convened a task force to address the increasing tendency of people to refuse to respond (as is their right) to a survey. This group prepared a report, published in September 2014, documentation what is a refusal and characterizes who refuses to participate in surveys. The report also discusses efforts to persuade initially reluctant individuals to participate (refusal conversion) and how that effort might infringe on someone’s privacy. If you are conducting almost any type of survey, you will have to confront participation refusals, and this document can serve as a starting point for handling the conflicting demands of scientific integrity and an individual’s right to be left alone. Continue reading

Recommended: In search of justification for the unpredictability paradox

This is a commentary on a 2011 Cochrane Review that found substantial differences between studies that were adequately randomized and those that were not adequately randomized. The direction of the difference was not predictable, however, meaning that there was not a consistent bias on average towards overstating the treatment effect or a consistent bias on average towards understating the treatment effect. This leads the authors of the Cochrane review to conclude that “the unpredictability of random allocation is the best protection against the unpredictability of the extent to which non-randomised studies may be biased.” The authors of the commentary provide a critique of this conclusion on several grounds. Continue reading

Recommended: Requiring fuel gauges. A pitch for justifying impact evaluation sample size assumptions

This blog entry from the International Initiative for Impact Evaluation talks about the deficiency in many research proposals sent to that organization. They rely too much on standardized effect sizes, which are impossible to interpret and often misleading. The authors also criticize the Intraclass Correlation Coefficients (ICCs) that are included in the sample size justification for many cluster based or hierarchical research designs. The ICCs, they say, often seem to be pulled out of thin air. It is a hard number to get sometimes and they suggest that you consider a range of ICCs in your calculations or that you run a pilot study. Continue reading