StATS: Initial work on the KCALSI grant (created 2006-07-17)

I am submitting a grant in response to a KCALSI RFP (note that this link is to a Microsoft Word document). According to the RFP, the general structure of the grant should follow the structure used by NIH. Here is a review of the structure of a typical NIH grant.

The NIH recommends that you develop a solid hypothesis

Many top-notch NIH grant applications are driven by strong hypotheses rather than advances in technology. Think of your hypothesis as the foundation of your application -- the conceptual underpinning on which the entire structure rests. www.niaid.nih.gov/ncn/grants/plan/plan_c1.htm

I believe it is a mistake to force all research into a hypothesis-driven mode, and the NIH acknowledges this on the same web page

However, an ongoing trend has moved us toward more applied research to discover basic biology or develop or use a new technology, especially in key areas such as organisms used for bioterrorism. If your application is not hypothesis-based, state this in your cover letter and give the reasons why the work is important.

If we had to come up with a hypothesis, it would be:

Researchers and IRBs who have good statistical tools for the initial planning and the ongoing monitoring of clinical trials would produce better research. They would construct realistic targets for their sample sizes rather than promising a sample size that could not be delivered in a reasonable time frame and budget. They would also be able to detect early when accrual rates are suffering and take appropriate corrective action before too much harm was done. This could include early termination of trials that are floundering and where any additional time and money would represent throwing good money after bad.

The same page warns against overly broad hypotheses and points out that

If you have more than one hypothesis, choose the better one.

I wonder if the focus on both planning and monitoring is too broad or represents two separate hypotheses.

The review criteria, according to the KCALSI RFP, is the same as those of NIH, meaning:

1. Significance. Does this study address an important problem? If the aims are achieved, how will scientific knowledge or clinical practice be advanced? What will be the effect of these studies on the concepts, methods, technologies, treatments, services, or preventions that drive this field?

2. Approach. Are the conceptual or clinical framework, design, methods, and analyses adequately developed, well integrated, well reasoned, and appropriate to the aims of the project? Does the applicant acknowledge potential problem areas and consider alternative tactics?

3. Innovation. Is the project original and innovative? For example: Does it challenge existing paradigms or clinical practice or address an innovative hypothesis or critical barrier to progress in the field? Does the project develop or use novel concepts, approaches, methods, tools, or technologies?

4. Investigators. Are the investigators appropriately trained and well suited to carry out this work? Is the work proposed appropriate to the experience level of the principal investigator and other researchers? Does the investigative team bring complementary and integrated expertise to the project (if applicable)?

5. Environment. Does the scientific environment contribute to the probability of success? Do the studies benefit from unique features of the scientific environment or subject populations or use useful collaborative arrangements? Is there evidence of institutional support? www.niaid.nih.gov/ncn/grants/write/write_c1.htm

but the RFP also mentions that

Criteria include scientific and technical significance, originality and likelihood of success of achieving external funding.

I'm not sure what the difference is between "scientific" and "technical" but both probably represent the NIH "Significance." The term "originality" is probably comparable to "innovation." The last criteria, though, "likelihood of success of achieving external funding" is not in the NIH guidelines. What this term means is probably articulated better in an earlier bullet of the KCALSI RFP

Proposals should describe how the proposed grant support will facilitate the development and submission of a proposal(s) seeking external funding from government and/or private agencies. The long-term expectations regarding the institutions extended research programs and potential for expanded research in related areas of research should also be articulated.

Here's how I plan to address each of the five NIH criteria.

Significance. A good literature review will help here. I want to point out the problems that most study suffer from in inadequate sample sizes. I'm going to get a few letters of support and I hope that those letters point out that (a) good planning and monitoring of accrual rates is important, and (b) right now the research community doesn't do a good enough job of monitoring accrual rates.

Approach. We are looking at a combination of Bayesian predictive distributions and statistical process control charts. Both of these approaches have a long history of providing successful solutions to difficult problems. Scott Berry has a good quote that I only vaguely remember, but it is something like "Bayesian analysis is hard in that thinking about research is hard."

Innovation. Most innovative ideas do not represent the creation of something totally new but rather applying an idea proven in one area to an entirely different arena. The innovation here is not the Bayesian analysis per se or the control charts per se. It is the application of these methods not to the final data analysis but to the initial planning and on-going monitoring.

Investigators. We should highlight our respective strengths in Bayesian data analysis and in Quality Control. We should also point out that Statisticians are in a position to see a lot of research so we know the research process better than anyone else. Finally, I need to point out my work with IRBs and DSMBs.

Environment. The KCALSI should already be familiar with the quality of research done at CMH and KUMC. I also should point out the collaboration between CMH and HeartLab. Some simple statistics on the number of clinical trials or the number of protocols reviewed by the IRB would be helpful here.

The general structure of an NIH grant along with specific instructions appears in the following bullet list.

Face Page [PDF] The RFP points out that this page "should identify only the lead institution and Principal Investigator in addition to other required information about the lead institution." That's me!

Description, Performance Sites, Key Personnel, Other Significant Contributors, and Human Embryonic Stem Cells [PDF] The RFP points out that this page "should clearly detail collaborating investigators and institutions." That's Byron Gajewski.

Table of Contents [PDF]

Detailed Budget for Initial Budget Period (or Modular Budget) [PDF] The RFP has specific instructions here:
"i. Requests may be up to $25,000 for one year
ii. Salary and fringe benefits may be requested and must be in accord with set institutional guidelines.
iii. Indirect cost rates may not exceed 10%. Indirect costs must be included within the $25,000 limit.
iv. Equipment requests may not exceed 20% of the total request.
v. The budget and budget justification should only represent funds requested of the KCALSI."

Budget for Entire Proposed Period of Support (not applicable with Modular Budget). Although this form is found in most NIH grants, the RFP here specifically mentions to NOT include this page.

Budgets Pertaining to Consortium/Contractual Arrangements (not applicable with Modular Budget) Although the RFP does not specifically mention this, I do not believe this form is needed.

Biographical Sketch – Principal Investigator/Program Director (Not to exceed four pages) [PDF] and Other Biographical Sketches (Not to exceed four pages for each – See instructions) The RFP limits these to THREE PAGES. and asks for the PI's biosketch first followed by all other investigators in alphabetical order. The biosketches belong in the Appendix.

Resources Research Plan [PDF] The RFP

Introduction to Revised/Resubmission Application (Not to exceed 3 pages.) Although the RFP does not specifically mention this, I do not believe this form is needed.

Introduction to Supplemental/Revision Application (Not to exceed one page.) Although the RFP does not specifically mention this, I do not believe this form is needed.

Research plan. The RFP asks that this section "must be no more than 10 single-spaced pages at 12 pt Arial or Times New Roman font with one inch margins throughout". The RFP also asks for items A-D of the standard NIH research plan:

A. Specific Aims

B. Background and Significance

C. Preliminary Studies/Progress Report

D. Research Design and Methods

Items in the NIH research plan not asked for are:

E. Human Subjects Research Protection of Human Subjects

Data and Safety Monitoring Plan

Inclusion of Women and Minorities

Targeted/Planned Enrollment Table

Inclusion of Children

F. Vertebrate Animals

G. Select Agent Research

H. Literature Cited

I. Multiple PI Leadership Plan

J. Consortium/Contractual Arrangements

K. Resource Sharing

L. Letters of Support (e.g., Consultants)

I suspect that the RFP will still want sections H and L.

Checklist. Although this form is found in most NIH grants, the RFP here specifically mentions to NOT include this page.

Personal Data Form Page. Although this form is found in most NIH grants, the RFP here specifically mentions to NOT include this page.

Appendix. Although you can include all sorts of interesting things in the appendix, I am planning to only include the two things that this RFP calls for.

Here's how I see the 10 pages of the research plan being used.

Abstract. Half of the first page.

Specific Aims. The bottom half of the first page. Here is some guidance from NIH.

Your specific aims are the objectives of your research project, what you want to accomplish, and your project milestones. Write this section for audiences, primary reviewers and other reviewers, since they'll all read it. Choose aims reviewers can easily assess. Your aims are the accomplishments by which the success of your project is measured. Recommended length of this section is one page. A common mistake new applicants make is being too ambitious. You should probably limit your proposal to three to four specific aims. www.niaid.nih.gov/ncn/grants/write/write_j1.htm

I think we have three specific aims for this grant.

Background and Significance. Three pages. Here is some guidance from NIH.

In the Background and Significance section, reveal you are aware of opportunities, gaps, and roadblocks in your field. Show reviewers your intimate familiarity with the field and knowledge about research being done, referring to all relevant scientific literature. If you leave out an important work, reviewers will assume you're not aware of it. Use this section to convey the breadth of your knowledge of your field and highlight why you are uniquely qualified to do the research. While you can refer to unpublished work, including information learned through personal contacts, make sure the literature you note here is also in your Literature Cited section. www.niaid.nih.gov/ncn/grants/write/write_k1.htm

The key points I want to identify is that accrual problems are important and that there are no good quantitative tools for evaluating accrual. Then I want to point out how statistical control charts are a logical data presentation format. I want to introduce the date-gap concept. Then I need Dr. Gajewski to provide about a page on why Bayesian methods and in particular, predictive distributions, have been used in medical research.

Preliminary Studies/Progress Report. Two pages. Here is some guidance from NIH.

By providing preliminary data, you build reviewer confidence you can handle the technologies, understand the methods, and interpret results. Preliminary data will help show you have the expertise to do the job.Interpret preliminary results critically. Give alternative meanings to the data to show you've thought the problem through and will be able to meet future challenges. If you don't do this, the reviewers will. www.niaid.nih.gov/ncn/grants/write/write_l1.htm

 I want to outline some simple approaches used with the sample data described in Stats: Monitoring accrual rates (May 30, 2006, Model, Quality). I want to contrast the control charts using yearly, quarterly, and monthly rates with control charts using date gaps. Dr. Gajewski will show some histograms on the distribution of estimated finish times using an optimistic, flat, and pessimistic prior.

Research Design and Methods.  Three pages. Some guidance from the NIH is helpful here.

When reviewers judge your application your Research Design and Methods section has the most weight. This section describes the experimental design and procedures -- how you will perform the research. Think carefully about how to organize it. You may want to divide your Research Design and Methods section into a description of your research and your methods, placing the methods section last. Another option is to organize this section by the five NIH review criteria. There is no recommended page length to this section, though you must stay within the 25-page limit of the sections listed on Develop Your Research Plan. It's helpful to create a graphical timetable showing how and when you will accomplish your aims, including any overlap of experiments and alternative paths. Use flow charts and decision trees to show paths of experiments and how they progress, including paths that show alternatives -- what you will do if you get negative results. You can use the same graphics you created to plan the project to present it in the application. www.niaid.nih.gov/ncn/grants/write/write_m1.htm

An additional point remembering is that this is the only spot in the grant where we can get technical.

One way to organize technical and nontechnical information is to keep the parts of the application most reviewers will likely read -- abstract, significance, and specific aims -- simple and nontechnical, and get technical and detailed only in the methods section. Your methods section will need to spell all your experiments out in fine detail. www.niaid.nih.gov/ncn/grants/write/write_d3.htm

Each of the three pages will describe work on each of the three specific aims. Dr. Gajewski will write the methods used on the first aim, we will split the work on the methods used on the second aim, and I will write the methods used on the third aim.

Literature cited. As far as I can tell, this does not count against the ten page limit. We should still try to keep this to a single page.

Letters of Support. One page per letter. I have several people I want to approach about this.

Most important of all: the deadline. The deadline for this grant is August 2, and I will prepare additional material on this weblog as I write this grant.

This page was written by Steve Simon while working at Children's Mercy Hospital. Although I do not hold the copyright for this material, I am reproducing it here as a service, as it is no longer available on the Children's Mercy Hospital website. Need more information? I have a page with general help resources. You can also browse for pages similar to this one at Category: Accrual problems in clinical trials.