Stroke Severity and Hospital Performance Ranking

Results of a study that used a risk model to compare hospital performance in care for acute ischemic stroke [JAMA. 2012;308(3):257-264] showed that adding stroke severity to the risk model was associated with a considerable improvement in the model’s ability to discriminate and change mortality performance rankings for a larger proportion of the hospitals.

Reporting risk-standardized outcomes for Medicare beneficiaries hospitalized with acute ischemic stroke is part of an increasing effort in the United States to use outcome measures to better define quality and value of healthcare in the United States. Use of adequate risk-adjusted outcome measures is essential for accurately assessing the quality of hospitals and ensuring the appropriate performance ranking and eligibility for financial incentives.

Current risk models for acute ischemic stroke do not include adjustment for stroke severity. This study was conducted to evaluate if the inclusion of initial stroke severity, based on National Institutes of Health Stroke Scale (NIHSS) scores, in a claims-based risk model for hospital 30-day mortality for acute ischemic stroke would alter hospital outcome ratings. The NIHSS is a 15-item neurological examination scale with scores between 0 and 42; higher scores indicate more severe stroke.

The study included data on 127,950 fee-for-service Medicare beneficiaries with ischemic stroke from 782 Get with the Guidelines^®-Stroke participating hospitals between April 2003 and December 2009. Performance of claims-based hospital mortality risk models with and without inclusion of NIHSS scores for 30-day mortality was assessed. Hospital rankings from both models were compared to evaluate model discrimination, hospital 30-day mortality outcome rankings among the participating hospitals, and value-based purchasing financial incentive categories.

The study found that the mean (standard deviation [SD]) NIHSS score was 8.23 (8.11) (median, 5; interquartile range, 2-12) across the entire study population. Within the first 30 days of hospitalization, there were 18,186 deaths (14.5%), which included 7,430 deaths (5.8%) during the index hospitalization.

When comparing hospital mortality models with NIHSS scores and those without NIHSS scores, the study found that the models with NIHSS scores had significantly better discrimination than models without (C statistic, 0.864 [95% confidence interval (CI), 0.861-0.867] vs 0.772 [95% CI, 0.769-0.776]; P<.001).

Overall, 26.3% of hospitals that were ranked in the top 20% or bottom 20% of performers by the claims model without NIHSS scores were ranked differently by the model with NIHSS scores. When using the model with NIHSS scores, 57.7% of hospitals initially classified as having “worse than expected” mortality were reclassified to “as expected.”

Significant improvement of model performance after the addition of NIHSS was demonstrated by both the significant net reclassification improvement (93.1% [95% CI, 91.6%-94.6%]; P<.001) and significant integrated discrimination improvement (15.0% [95% CI, 14.6%-15.3%]; P<.001) indexes.

Based on these results, which show substantially worse discrimination in a hospital risk model based on claims data alone without adjustment for stroke severity compared with a model that adjusts for stroke severity using the NIHSS, the authors suggest that “it may be critical to collect and include stroke severity for optimal hospital risk adjustment of 30-day mortality for Medicare beneficiaries with acute ischemic stroke.”

Limitations of the study included a patient population that may not be representative of all patients hospitalized with acute ischemic stroke, inclusion of fee-for-service Medicare patients only, use of variables taken from claims data, lack of adjustment for therapies provided in the risk models, and potential differences in methodologies used to rank hospitals compared to that used by CMS.