P.Mean: Interpreting p-values in a published abstract, part 1 (created 2010-04-14). |
In one of my recent webinars, I asked people to read the following abstract and interpret the p-values presented within.
The Outcome of Extubation Failure in a Community Hospital Intensive Care Unit: A Cohort Study. Seymour CW, Martinez A, Christie JD, Fuchs BD. Critical Care 2004, 8:R322-R327 (20 July 2004) Introduction: Extubation failure has been associated with poor intensive care unit (ICU) and hospital outcomes in tertiary care medical centers. Given the large proportion of critical care delivered in the community setting, our purpose was to determine the impact of extubation failure on patient outcomes in a community hospital ICU. Methods: A retrospective cohort study was performed using data gathered in a 16-bed medical/surgical ICU in a community hospital. During 30 months, all patients with acute respiratory failure admitted to the ICU were included in the source population if they were mechanically ventilated by endotracheal tube for more than 12 hours. Extubation failure was defined as reinstitution of mechanical ventilation within 72 hours (n = 60), and the control cohort included patients who were successfully extubated at 72 hours (n = 93). Results: The primary outcome was total ICU length of stay after the initial extubation. Secondary outcomes were total hospital length of stay after the initial extubation, ICU mortality, hospital mortality, and total hospital cost. Patient groups were similar in terms of age, sex, and severity of illness, as assessed using admission Acute Physiology and Chronic Health Evaluation II score (P > 0.05). Both ICU (1.0 versus 10 days; P < 0.01) and hospital length of stay (6.0 versus 17 days; P < 0.01) after initial extubation were significantly longer in reintubated patients. ICU mortality was significantly higher in patients who failed extubation (odds ratio = 12.2, 95% confidence interval [CI] = 1.5–101; P < 0.05), but there was no significant difference in hospital mortality (odds ratio = 2.1, 95% CI = 0.8–5.4; P < 0.15). Total hospital costs (estimated from direct and indirect charges) were significantly increased by a mean of US$33,926 (95% CI = US$22,573–45,280; P < 0.01). Conclusion: Extubation failure in a community hospital is univariately associated with prolonged inpatient care and significantly increased cost. Corroborating data from tertiary care centers, these adverse outcomes highlight the importance of accurate predictors of extubation outcome.
It is a bit dangerous to read only the abstract, of course, but this was intended for a general illustration. When you read an abstract like this, a good first step is to identify the four components of the research hypothesis, which is commonly labeled with the acronym PICO.
Not every research hypothesis will have all four components, of course, but for most papers, it is useful to try to identify all four elements. For some studies, there is no comparison group, and for some studies, there is an exposure rather than an intervention.
Let's try to identify PICO for this study.
Reading the abstract, it looks like P = patients in a community hospital ICU is a reasonable choice. The intervention (actually an exposure) is I = experiencing an extubation failure. The comparison group is C = patients in a community hospital ICU who experienced a successful extubation, but actually there is comparison within this group. So P = patients experiencing an extubation failure and C = patients with successful extubation. There are several outcomes:
What do the p-values tell us?
The first set of p-values, which are only described as being large (P > 0.05) show that patient groups were similar with respect to several important demographic variables. These are not outcome measures, but rather measures intended to see if the patient group and the comparison groups deviated markedly on an important covariate. Thankfully, the answer to this is no.
The next two p-values are only described as being small (P < 0.01). These p-values are associated with ICU and hospital length of stay. This tells you that there is strong evidence against the null hypothesis that the average values are the same between extubation failures and successful extubations. By looking at the summary statistics provided we can conclude that there is evidence that extubation failures have longer lengths of stay. There is also support for higher ICU mortality among extubation failures (P < 0.05).
But there is insufficient evidence to support a higher hospital mortality rate. The odds ratio is 2.1, which implies an increased risk, but there is a lot of sampling error in this estimate, so the result could be due to sampling error. Note that this paper uses ambiguous mathematical notation throughout but here it is especially confusing. The notation of P < 0.15 could, in theory represent a small p-value (0.02 is certainly less than 0.15). But having read enough of these abstracts, I know that P < 0.15 really means something like 0.10 < P < 0.15. It would be much better to present an exact p-value.
Total hospital costs has a small p-value (P < 0.01). Looking at the summary statistics, it is obvious that there is support for the hypothesis that total hospital costs are significantly higher for extubation failure patients.
Now none of these results should be too surprising. If you can't get the tube out, the patient has to stay longer, pay more money, and (at least in the ICU) suffer a greater mortality risk. The value of this paper is not in proving the obvious, but rather in quantifying it. The true value of this paper is in the confidence intervals that provide a solid measure of the true costs, both financial and physical, that a patient has to endure if he/she is an extubation failure.
So how would you interpret the confidence intervals? This was also a question I posed during the webinar.
The abstract does not provide a confidence interval for the ICU or hospital length of stay. Shame on them!
The confidence interval for the odds ratio of 12.1, associated with the risk of ICU mortality goes from 1.5 to 101. Wow, that is a very wide interval. It does exclude 1, so there is a statistically significant increase in the risk of mortality in the ICU for patients with extubation failures. We can say further that we are confident that the increase in risk is at least 50%, even after allowing for sampling error.
The confidence interval for the odds ratio of 2.1, associated with the risk of hospital mortality goes from 0.8 to 5.4. This interval includes the value of 1, so there is no statistically significant difference in hospital mortality between the two groups. The risk of hospital mortality, however, could plausibly be as large as a five fold increase in risk. I'd complain richly about such a wide interval in other contexts, but here I'm willing to give a pass. Mortality is not a primary endpoint, so the study was not designed to estimate mortality rates precisely.
As an aside, I should mention that in most research settings, mortality is notoriously difficult to estimate precisely, because you need a large number of events (deaths) in order to have reasonable power and precision. A rough rule of thumb is that you should try to get 25 to 50 events in each group. You have to read the full paper, but the total number of hospital deaths are 6 and 17 in the successful extubation and extubation failure groups, respectively. It looks like you'd need about a three or fourfold increase in sample size (3*17 is approximately 50, 4*6 is approximately 25) to be able to handle an event like hospital mortality successfully.
Finally, the increase in total hospital costs $34,000 (I'm rounding here) has an associated confidence interval of $23,000 to $45,000. Since this interval excludes the value of zero, you can conclude that there is a statistically significant increase in costs for extubation failures. Even after allowing for sampling error, the increase in costs is at least $23,000 and possibly as much as $45,000.
This work is licensed under a Creative Commons Attribution 3.0 United States License. This page was written by Steve Simon and was last modified on 2010-04-15. Need more information? I have a page with general help resources. You can also browse for pages similar to this one at Category: P-values.