StATS: My second grant, part 2 (created 2006-09-13)

I took a three day workshop on grant writing and prepared a draft grant as part of the student exercises in that class. It's not in the format that I need to use, but it outlines most of the goals and efforts of my proposed work. Here's what I wrote:

As the research biostatistician at Children's Mercy, Dr. Simon has supported the development of literally hundreds of clinical trials across just about every branch of pediatric medicine. In his experience and in the experience of many of the people he works with, the greatest practical problem in conducting medical research is slow accrual of patients. Clinical trials invariably have problems recruiting the planned number of patients in a reasonable time frame. According to one source1, more than 80% of all clinical trials fall short of their accrual goals. The net result is too many research studies that appear with inadequate sample sizes and confidence intervals that are so wide that they are effectively uninterpretable.

The problem of inadequate accrual is a serious issue for Institutional Review Boards (IRBs). If a study has an inadequate accrual rate, that changes the balance of risks and benefits. The patients that are enrolled in the study experience inconvenience and possibly pain, and may even suffer increased risk but do not provide information with sufficient precision to produce scientifically valid results. The NIH has recognized this as well and has documented the need for better tools in an Program Announcement which calls in particular for

"Development of appropriate outcomes measures and quality indicators for the IRB review process for measurement of adequate protection of human subjects."

Foundations and government agencies that support medical research directly bear the costs of a study with poor accrual rates. When they invest their limited funds in such a project, this represents money taken away from a different research project that would have met its accrual goals and would have provided scientifically useful output. The amount of research money tied up in these underperforming clinical trials represents a huge lost opportunity cost that dilutes the effectiveness of the agency's grant program.

Although slow accrual rates are a serious threat to the overall research endeavor, they are treated like the crazy aunt in the attic. No one talks about slow accrual rates, and no one does anything about them, and everyone wishes they would just go away.

The goal (long range objective) of this research program is to increase the awareness in the research community that accrual problems are a serious threat to the research endeavor and to understand the factors that cause problems with patient accrual. This will serve as preliminary data in the background and significance section of an NIH R03 or R21 grant to produce quantitative tools for planning and monitoring accrual rates in clinical trials.

Specific Aim #1. Estimate the proportion of studies approved by the Children's Mercy IRB between 2001 and 2004 that have experienced delays in the completion of the studies because of accrual rates problems or which ended without recruiting the original planned number patients.

During months 1 and 2, identify a sampling frame of IRB studies. At this point, the sampling frame will be broad and inclusive. Any prospective study that uses human subjects will be entered into a database.

During months 3 and 4, obtain and review the research protocol submitted to the IRB and scan it for the following eligibility requirements:

  1. Both a target sample size and a time frame for completion of the study are clearly stated in the research protocol.
  2. The specified time frame must be long term (at least 12 months) but short enough so that the estimated completion date would occur before December 2005.
  3. A study that is still open is eligible for review as long as the target completion date would have been prior to December 2005.

An important statistic computed at this point is the proportion of all prospective studies that fail to specify a completion date. These represent studies where an important component of the research plan is left out.

During months 4-6, obtain all interim reports and final reports submitted to the IRB for studies identified in the previous step. Estimate the accrual rate at each reporting point and calculate whether it is behind schedule, on time, or ahead of schedule. For those studies with a final report, calculate the proportion that finished with fewer than the originally planned number of subjects and the average percentage shortfall in these studies.

Specific Aim #2. Evaluate the reasons (and excuses) that are offered in interim and final reports in those studies with identified accrual problems.

During months 7-9, among those studies identified as being behind schedule at an interim report, identify any text in that report that offers an explanation for the delay or which offers a proposed change to help get the study back on track. Perform the same task for any study with a final report that ended later than planned.

Also during months 7-9, among those studies that finished with fewer than the planned number of subjects, identify any text in that report that offers an explanation for the shortfall.

During months 10-12, an independent reviewer will read the textual information collected and develop a list of categories that could represent a small and manageable set of discrete responses that captures the full range of textual responses. Two reviewers will then take that list of categories and independently assign individual text items to one or more of these categories. The degree of agreement among the reviewers will be estimated and the reviewers will meet to resolve any discordance between the two classifications.

Evaluation. The results of these quantitative and qualitative analyses will be shown to members of the IRB to get their feedback about the nature and frequency of responses offered when a clinical trial fall behind schedule. These IRB members will be asked if this is consistent with their experience and if any important issues have been left unstated. They will also be asked if the information provided here is useful and if it gives them ideas on how to change and improve their review process.

Executive summary. As the research biostatistician at Children's Mercy, Dr. Simon has supported the development of literally hundreds of clinical trials across just about every branch of pediatric medicine. In his experience and in the experience of many of the people he works with, the greatest practical problem in conducting medical research is slow accrual of patients. It is not unusual to see a research project that has gone a full year past its completion date and yet the researcher still has less than half of the subjects originally planned for. Slow accrual causes serious difficulty for researchers, for groups that fund research, and for groups that review research, such as Institutional Review Boards (IRBs) and Data and Safety Monitoring Committees.

There is little appreciation, however, in the research community for the problem caused by slow patient accrual. This lack of concern is caused, in large part, by the lack of hard data on scope and magnitude of the problem with slow patient accrual. This research proposal asks for $15,000 for a pilot study to evaluate a random sample of 100 studies approved by the Children's Mercy IRB. This sample will produce an estimate of the proportion of studies that fall behind schedule, the proportion that end with fewer patients than originally planned, and the average shortfall in those studies. A qualitative analysis of the interim and final reports will help identify the major factors that cause slow patient accrual. The results of this research will serve as preliminary data for an R03 or R21 grant to produce quantitative tools for planning and monitoring accrual rates in clinical trials. These tools will help researchers plan appropriate and realistic accrual rates and monitor them regularly, so that corrective action can be taken early to get a study back on track.

The teacher laughed at the "crazy aunt in the attic" analogy, but suggested that I would be better off leaving out that analogy.

My first grant, by the way, went in on schedule and I should hear back in October. Here is the final draft in PDF format.

Some of the earlier weblog entries about my grant

This page was written by Steve Simon while working at Children's Mercy Hospital. Although I do not hold the copyright for this material, I am reproducing it here as a service, as it is no longer available on the Children's Mercy Hospital website. Need more information? I have a page with general help resources. You can also browse for pages similar to this one at Category: Accrual problems in clinical trials.