I am drafting up a policy on statistical support for research at my part-time job at UMKC. It is loosely based on standards at the University of California, Davis and Kansas University Medical Center. An early draft appears below. I’ve gotten some suggestions that setting a minimum percentage effort is a bad idea. What do you think?

Statistical support for your research grant

If you are considering including the support of a statistician from the Department of Biomedical and Health Informatics on your research grant, you’re making a smart move. It not self-serving for us to point this out; it is a conclusion that many researchers have already made on their own.

The money you spend on statistical support will provide you with a far greater return through improved research efficiency. Statisticians have the tools to insure that your sample size is large enough to provide the precision that you need, but not so large as to be wasteful. We are aware of many alternative research designs that can, at times, provide efficiencies large enough to allow you to reduce your sampling needs.

The money you spend on statistical support will also greatly improve your prospects of getting funded. Reviewers expect to see a strong research team and this includes a statistician with graduate level training and substantial data analysis experience. You may be more than capable of conducting your own data analysis, but without the credentials that we already have, you will have a hard time convincing a reviewer of your skills.

The contributions that we offer extend well beyond a sample size justification and a data analysis plan. We have many years of experience in providing consultation across a very broad range of science and medicine. We know about research innovations in other fields that may be less well known in your discipline. We have many contacts in the research community and can help add find expertise for your research team in areas beyond Statistics.

If you want to work with us, we need to set up some ground rules in advance to avoid any misunderstandings.

Planning time frame

You cannot use our name in your grant without having us participate fully in the development of your grant. If we haven’t been active participants in the planning, we have no way to vouch for the integrity of the study design and data analysis plan. Generally, you need to include us early in the development of the research grant, typically two months or more prior to the submission date. Under no circumstances will we consider working on a research grant if we just learn about it less than one month from the submission date.

Data management budget

We will not work on any grant that does not have funding in the budget for a data manager. A data manager is critical to guarantee data integrity and research efficiency. We cannot do our jobs well without this guarantee. If you do not have data management support in house, we can provide assistance through the Center for Health Insights.

Appropriate FTE support levels

The normal FTE effort on a research grant is 20%, and this level needs to be maintained throughout the duration of the research grant. You cannot collect the data on your own and then dump it in our lap during the last year of the grant. We must be active participants during the data collection phase in order to do our job well.

Some grants will require more than 20% effort. These include research studies with

• a large number of outcome variables,
• a longitudinal design,
• multilevel sampling,
• interim analyses,
• an adaptive design,
• an expectation of high dropout rates/many missing values.

Some studies may require less than 20% effort. Generally, these would be studies where you conduct your own data analyses, but we provide oversight and review. The minimum, even in these cases, is 10% effort. The only exception to this rule is a research grant that does not ask for any salary support for anyone on the research team.

Once we agree on a level of support appropriate for your grant, you cannot change that level of support, even if the research methods change markedly, without discussing this with us first.

These are not optional recommendations

The recommendations in this document are mandated for all research. If you do not wish to abide by these recommendations on review times and funding levels, please find your statistical support elsewhere. There may be rare exceptions to these requirements, but these need to be cleared first by the department chair.

Acknowledgements

This document was developed using guidelines at the University of California, Davis1 and Kansas University Medical Center2. We encourage you to review these resources to better understand the justification for our requirements.

1 http://ucdmc.ucdavis.edu/ctsc/area/biostatistics/financialConsiderations.html

2 http://www.kumc.edu/school-of-medicine/department-of-biostatistics/grant-preparation.html

I have to help write NIH grants from time to time, and I need to always keep front and center the criteria that NIH peer reviewers use when they evaluate grants. They look at five broad areas: significance, investigators, innovation, approach, and environment. This document explains what each of these five broad areas means. 

National Institutes of Health. Definitions of Criteria and Considerations for Research Project Grant (RPG/X01/R01/R03/R21/R33/R34) Critiques. Updated March 8, 2018. Available at https://grants.nih.gov/grants/peer/critiques/rpg_D.htm.

What percentage effort is reasonable for Biostatistics support on a research grant? The UC Davis Biostatistics Group says 10% as a bare minimum, 35-60% for straightforward projects with uncomplicated analyses, and 50-100%+ for large or complex projects. They give examples of large and complex projects: interim analysis, multi-site projects, development of novel statistical methods, and assembly of data from large, complex, or poorly documented administrative or survey data sets.

They also describe how to split the effort between a PhD Biostatistician, who supervises the overall effort, and a MS Biostatistician, who does most of the data management and statistical analysis.

Another point worth noting is that any grant listing less than 10% effort for a Biostatistician requires a special sign off.

UC Davis Biostatistics Group. Guidelines for estimating biostatistician effort and resources on grants. Available at http://ucdmc.ucdavis.edu/ctsc/area/biostatistics/Documents/UCD_Biostat_Effort_Guidelines.pdf or http://ucdmc.ucdavis.edu/ctsc/area/biostatistics/financialConsiderations.html.

If you are writing a research grant, there are a lot of statistical issues that you need to consider. This guide, prepared by the American Statistical Association, highlights three areas: framing the problem, designing the study, and specifying the data analysis plan. It doesn’t talk enough about data management, but otherwise it is an excellent resource.

Ming-Wen An, William Christiansen, Debashis Ghosh, Sujit Ghosh, Scott Holan, Leslie McClure, Karen Messer, Cynthia Rudin. Statistical Issues Seen in Non-Statistics Proposals A Guide to Improving the Science in Non-Statistics Proposals for Funding from the American Statistical Association Committee on Funded Research. Available at http://ww2.amstat.org/misc/StatisticalIssuesInProposals.pdf.

The problem with incentives has been long documented in the literature. Alfie Kohn summarized much of this in a 1995 book, Punished by Rewards: The Trouble with Gold Stars, Incentive Plans, A’s, Praise and Other Bribes. There’s a newer book out (2011) that I have not read yet that has much the same message: Daniel Pink’s Drive: The Surprising Truth About What Motivates Us.

The problem with incentives and other rewards is not that they never work. They do work in some settings, and getting the pay structure wrong is a motivational disaster. The problem with incentives is that they don’t seem to help much with the complex tasks associated today’s work, especially grants. In fact, sometimes the incentives can serve as a distraction and cause a decline in productivity.

So what do you do if incentives don’t work. Dr. Kohn suggests that the three keys are collaboration, content, and choice. People enjoy work that is collaborative, so try to work on encouraging teamwork rather than rewarding individual performance. People also enjoy work that they perceive as meaningful, so make sure they understand how the type of work they do relates to making the organization better. Finally, give people, to the greatest extent possible, the ability to choose how they approach their work.

Daniel Pink has a similar message: you should stress

“autonomy-the desire to direct our own lives, mastery-the urge to get better and better at something that matters, purpose-the yearning to do what we do in the service of something larger than ourselves.”

I hesitate to summarize more of Mr. Pink’s book based on just a few excerpts and an interview on NPR, but it sure sounds a lot like what Dr. Kohn is saying.

I’d like to add one more thing to the list. Often the problems in a workplace is that there are too few incentives. For grant writing, there are already lots of incentives in place. The problem is that there are too many disincentives. If you are in charge of an organization, look carefully at all the roadblocks, hurdles, and impediments that keeping people from writing more grants. Do your best to remove all these disincentives before you plan to add new incentives.

So what are the disincentives to writing research grants? It depends a lot on the organization, but many of us do not know how to write well. So access to training is important.

Another problem is the administrative barriers to cross administrative lines. I’ve done a lot of research with people from Kansas University Medical Center, which is just a short drive from the University of Missouri-Kansas City, but sometimes it feels like the two institutions are worlds apart. Part of it is the state line the you have to cross to get from one place to the other, but part of it is just the tendency of any organization to think about issues from an institutional perspective rather than a broader perspective.

Finally, there are lots of rules and regulations that make little sense other than to serve as barriers to conducting research. One regulation that I find troubling is that many of the grant opportunities are unavailable to part-time employees. You can try to make a rationalization for this restriction, but it really just is a way of punishing those of us who need the flexibility of part-time work because of child care issues.

Now, the problems with disincentives to producing research are not just at my organization. It’s a problem at every place I’ve ever worked and I’m guessing it is more or less a universal problem. But if someone at UMKC asked me “How can we get more research done here?” my response would be “Stop punishing your successful researchers.” It’s a bit harsh but it would get the people in power, I hope, to start thinking about all the disincentives that are already in place.

One issue that Mr. Pink brought out in his NPR interview is that organizations look at incentive programs as a way to solve their problems because they are easy to implement. It’s the easiest thing in the world to throw money at a problem to try to solve it. But giving people autonomy is hard work. Showing that the work that your employees do has a bigger purpose is even more difficult. But if you want to be successful in getting your organization to do something important, you have to do the hard work.

Michael Lauer. “A Grant Submission New Year’s Resolution” published December 20, 2015 on the Open Mike blog. Available at http://nexus.od.nih.gov/all/2015/12/30/grant-submission-new-years-resolution/.