The International Association of Survey Statisticians publishes a newsletter every six months that covers general information about surveys, announcements about meetings, and other activities of the association.

International Association of Survey Statisticians. The Survey Statistician newsletter. Available at http://isi-iass.org/home/services/the-survey-statistician/.

Let’s be honest here. Your choice of which three questions to use was arbitrary. You didn’t run a formal data analysis that showed that a three item scale has just as good psychometric properties as a five item scale.

So just say in the methods section that you needed to shorten the survey and the choices as to what items to remove was made based on the subjective opinion of a subject matter expert (yourself).

Then when the excessively picky reviewer (in my opinion) rejects your paper, send it to a different journal.  In general, it’s a bad idea to change scales because a reviewer is likely to raise concerns. But I think that the practicalities of research sometimes require you to shorten a survey, because a shorter survey with uncertain psychometric properties might still be preferred to a longer survey that half of your volunteers refuse to complete.

Research is often a series of difficult compromises between theoretical ideals and practical realities. Shortening your scale is limitation that you do need to acknowledge, but having a limitation like this should not disqualify you from publishing your results. If we only published research with no limitations, the research journals would be a lot thinner.

The one thing that you emphasize is that the decision to shorten to three items rather than five items was made PRIOR TO THE COLLECTION OF ANY DATA. You can prove this, if the peer-reviewer doesn’t trust you, by offering to share the protocol that you send to and that was approved by the IRB. It’s still an arbitrary choice, but it is not one that would be biased by the data analysis. It’s not like you ran ten different analyses and then chose the one with the smallest p-value.

International Association of Survey Statisticians. The Survey Statistician. Available at: http://isi-iass.org/home/services/the-survey-statistician/.

Qualtrics is a private company located in Provo, Utah, that provides proprietary software for conducting dynamic surveys online. Their target markets is businesses interested in consumer research and employee attitudes, websites interested in visitor feedback, and academic researchers. It has security levels that are sufficiently strong by the University of Missouri system to meet their DCL3 security standard, their second highest standard.

Even though UMKC has an agreement with Qualtrics, you need to pay additional funds out of your own budget to use it (here are the details).

There is an open source product, REDCap, that also conducts surveys, among other things. You may want to investigate whether REDCap meets your needs before investing in Qualtrics. I’m guessing that Qualtrics might have an edge on ease of use or ability to modify surveys dynamically, but I haven’t researched either product in great detail.

There’s a third product that UMKC researchers have used in the past, Survey Monkey. It’s a nice product, but doesn’t have any real advantages over REDCap.

You need to be a bit careful here. An observational study means (more or less) that the patients get to select the treatment they are getting rather than having their choice be dictated by the flip of a coin. Any time the patients get to choose, you have the risk of selection bias, even if 100% percent of the patients participate in the observational study.

Now I know there are some observational studies where patients don’t get to choose, such as a study comparing BRCA1 to BRCA2 breast cancer patients, but the overall point is still valid.

Problems when patients fail to respond to a call to participate in a survey, for example, might be better characterized as nonresponse bias. There is no consensus in the research community as to how low a participation rate would cause concerns about nonresponse bias. In some settings, you have basic demographic information about the nonresponders, and you can mitigate concerns about nonresponse if you can show that the demographic profile of the nonresponders is comparable to the demographic profile of the responders.

I personally do not worry about nonreponse if the response rate is greater than 90%, I start to worry if the response rate drops below 70% and I get very nervous if it drops below 50%. But this is a very arbitrary choice on my part. You can argue some of these numbers based on a sensitivity analysis, perhaps.

I have not seen any published cutpoints, arbitrary or not, in the peer-reviewed literature. If there are any, I’d love to see them.

Your worries are on target. To assess the appropriateness of any sampling procedure, you need to ask yourself “Who was left out?” In this sampling procedure,  the cars left out are the cars that neither passed you nor were passed by you, because they were travelling at exactly the same speed as you.

Suppose that 90% of the cars drive at 55mph and 9% drive faster than 55mph and 1% drive slower than 55mph, then 9 cars will pass you for every 1 that you pass. But 91% of the cars obey the speed limit. You never see most of them because if they are 1,000 feet in front of you or behind you when you start your data collection, those cars will remain 1,000 feet away when you are done with your data collection. Now, realistically, I don’t believe that 90% of the cars drive at exactly 55mph, but the data collection described above is clearly biased because it is unable to sample the fraction who travel exactly at 55mph and it undersamples those cars that drive only a tiny bit faster or slower than 55 mph.

What’s worse is that the cars that drive slower are very unlikely to drive a lot slower than 55 mph. It would suicide to drive more than 5 mph below the speed limit. But among those who drive faster than the speed limit, the sky’s the limit. Some may only drive 2 mph above the speed limit and some may drive only 5 mph above the speed limit, but there are still more than enough drivers out there who are 10 to 20 mph above the speed limit and these are the drivers who are most likely to appear in your sample.

So, it’s clearly a biased sample. It would be better to sit by the side of the road at 0 mph and point your radar gun at every car that passes by. There may still be some biases, especially if there is a relationship between the speed at which you drive and the frequency and duration of your trips. But if there are biases, they would be a lot less than the biases inherent in the reporter’s sampling method.