Abstract: A multidisciplinary team at ChristianaCare, headquartered in Wilmington, DE, designed a comprehensive pathway utilizing an internally developed predictive model to proactively reduce oncology readmissions. The study was applied at the time of clinical decision making on an oncology inpatient unit to patients discharged to home receiving care at one of 3 oncology practices. The developed pathway supported enhanced discharge planning for patients identified as high-risk by targeting opportunities in prioritizing patients, communication between the care team, medication reconciliation, and patient expectations. These interventions included day of discharge visit, follow-up calls, and follow-up visits. The readmission rate declined from 34% to 24% for the total population, and from 45% to 29% for the high-risk population. The team overcame challenges with data available at the time of clinical decision making, data fluidity, and Health Insurance Portability and Accountability Act (HIPAA) compliance.
Readmissions to the hospital for oncology patients discharged within the past 30 days is not only costly to the health system but also psychologically, emotionally, and physically taxing for the oncology patient and their families. Reducing readmission risk is vital to improving patient outcomes and maintaining financial stability.1 It is important to work with oncology patients to make the transition from inpatient to outpatient setting as smooth as possible while ensuring the patient remains in the most appropriate clinical setting. Enhanced discharge planning with a focus on patients who have an increased risk of readmissions may decrease this burden.
Intervention strategies focusing on high-risk patients have been shown to be effective in preventing readmissions.2 An important component of preventing readmissions is prioritizing limited health care resources to the right patients at the right time. With electronic health records (EHRs), computer models can collect patient information and forecast patients with an increased risk of readmission. This novel approach offers the ability to decrease costs and improve the quality of care.3 Proven interventions include 48-hour nurse callbacks and 5-day follow-up appointments, which reduced unplanned oncology hospital readmissions in nonsurgical patients.4 At present, most predictive models are implemented upon discharge, however, recent literature shows that predicting risk scores during hospitalization is more beneficial in preventing readmissions.5 The LACE model (Length of stay, Acute admission status, Charlson Comorbidity index, Emergency department visits in the past year) and HOSPITAL (Hemoglobin at discharge, discharged from Oncology unit, Sodium level at discharged, Procedures during hospitalization, Index hospitalization Type, number of Admission the past year, and Length of stay) are two recent models that produced a significant decrease in readmissions by providing a risk score during the hospital stay.6
ChristianaCare brought together oncology clinicians and performance improvement experts with industrial engineering expertise to design a comprehensive pathway that utilized an internally developed predictive model to reduce readmissions. The model enabled the team to focus on patients who were predicted to be high risk and then to target interventions for that group to prevent readmissions.
Setting and Identification of Predictors
ChristianaCare is a private non-profit academic medical center with campus locations, in Newark, DE, Wilmington, DE, and Elkton, MD, with more than 1200 inpatient beds, a Home Care company, and a variety of ambulatory services located across the region. This study was carried out on an inpatient oncology unit focusing on patients discharged to home who had established oncology care at one of three outpatient oncology practices located on the Newark campus of ChristianaCare. The project was separated into four phases: baseline, testing, pilot, and sustain.
The data collection processes started by utilizing the clinical team and published literature to identify potential information that might be useful as features. The collected data consisted of multiple variable types including patient demographics, clinical (lab test results, vitals, etc), medical history, socioeconomic, and financial. The total number of features collected was 142, describing 6336 ChristianCare Hospital oncology visits from 2013 to 2017.
The descriptive statistics were then shared with the team to evaluate the accuracy of these potential features. Additionally, data pre-processing was an essential step in building a machine learning model, it modifies the data to an appropriate shape to be readable by data mining algorithms. Data pre-processing includes cleaning, normalization, categorization, and imputation. One of the challenges with developing a predictive tool that could run during the patient’s stay was final clinical and operational information was unavailable in the electronic medical record. To mitigate this, required values coded post-discharge were pulled from previous visits or eliminated. Missing data were evaluated for the type of missingness to determine the best method for data pre-processing or if it should be removed. Missingness types included missing completely at random, missing at random, and missing not at random.7
Model Development and Data Insights
During the model development process, insights were gleaned regarding the oncology patient population, which aided in developing effective interventions. The model development is a cyclical process where multiple feature selection and models are applied, and the optimal model is selected based on accuracy metrics and end-user feedback. The optimal analysis for feature selection was recursive feature elimination, which reduced the important variables to 15 (Table 1). These features were used in training a random forest with cross-validation to achieve an accuracy of 77.4% and area under the ROC curve (AUC) of 0.59. The team approved of this accuracy because if 50% of correctly predicted readmissions were prevented, the overall readmission rate would drop by 5%.
The list of variables was challenging for the team to interpret due to the complex nature of data mining models. This led to further analysis through using association rules, which is a data mining technique used to show hidden relationships between variables within a large data set. The key finding was that home health patients have a higher risk of readmission; a higher proportion of the readmitted patients (40%) were discharged to home health than the non-readmitted (20%).
Operationalizing the Model Into a Pathway
To intervene on the high-risk patients in the most effective way, the team utilized the data analysis results and created broad interventions for all high-risk patients as well as a targeted approach for home health patients. The operational process of distributing and using the model was tested for 3 months. Then a pilot was created to verify the effectiveness in partnership with three different oncology practices and our home health organization for 7 months. The pathway created targeted opportunities for improvement including prioritizing patients, communication between roles, medication reconciliation, and patient expectations. Figure 1 outlines the pathway, and bolded items illustrate new processes implemented.
By implementing a daily list of high-risk patients, there was a clear and manageable target population to prioritize. A report with these high-risk patients was autogenerated and emailed to participating clinicians at 7:30am each morning. Prior to the creation of the pathway, the patient care facilitator (PCF) and outpatient case manager (OCM) were performing follow-up calls. However, based on running the model on historical information, the PCF was only able to complete 20% and OCM only completed 56% of the calls to those identified as high risk. By trying to contact all patients, many high-risk patients who would have the greatest benefit were never contacted.
Communication was enhanced by sharing the responsibility of follow-up calls between the roles involved in care post-discharge and providing a method of shared documentation. Previously, this work was done in silos where the PCF and OCM would document their calls in their preferred documentation tool and would not coordinate timing. In the meantime, the provider practice did not have clear expectations for a follow-up visit, and they may not be aware of pertinent information from the follow-up calls. With the new process, the provider practice became included in the follow-up call process and targeted having a follow-up visit within 10 days of discharge. In situations where the patient was found to not be doing well, follow-up office visits were scheduled immediately. Additional support for this intervention was found in the baseline data: 43% (24/56) of readmissions occurred within the first 10 days post-discharge (Figure 2).
Medication reconciliation is often complex for oncology patients. The smaller targeted population allowed the home care agency to apply additional resources to high-risk patients. The inpatient case manager (ICM) communicated to the homecare agency through the referral, and then the homecare agency provided the patient with a medication bag and would communicate with the practice if any issues were found at the first home visit. A pharmacy review of medications on the day of discharge was also proposed, but limited resources made this intervention unfeasible.
By communicating with the high-risk patients during their acute episode, the clinical team was striving to set clear expectations for post-discharge care. The care team was made aware of the high-risk during rounds, and there was an emphasis for the care manager to have a day-of-discharge visit to be able to answer any patient questions. Also, the follow-up phone calls and quick turnaround for an office visit provided an additional opportunity for a patient to address any concerns.
Pathway Evaluation and Sustainability
To ensure a feedback loop for those involved in the pilot, a process to measure outcomes and compliance was created (Figure 3). The feedback loop provided transparency and the ability to learn and act quickly if metrics did not achieve their target. For weekly data, follow-up call completion by role was dispersed. Monthly data included compliance information for case management visit on day of discharge, and follow-up visit compliance, model performance, and outcomes including the readmission rate, length of stay (LOS), and emergency department (ED) visits. Additionally, chart reviews were completed on all readmissions monthly to evaluate if any data was missing from those not predicted as well as determine if the readmission could be prevented.
Outcomes
The readmission rate saw a steady decline during the implementation of the model. For the total population, the readmission rate started at 34% (56/165), during the pilot declined to 24% (103/432), and maintained a rate of 25% (52/203) through sustainment. For the high-risk populations, the readmission rate started at 49% (31/63), during the pilot declined to 29% (38/131), and maintained a rate of 31% (19/61) through sustainment (Table 2).
Secondary metrics for this project included ensuring the average LOS and percent of patients with ED visits within 30 days. For the baseline data, 6.5 days was identified as the average LOS and the goal was to keep LOS below the baseline average. During the pilot phase for the total population, the average LOS decreased to 5.9 days and the number of ED visits dropped by 3%. There was also a decline in the high-risk population for both metrics (Table 2).
Metrics for compliance of the project interventions for high-risk patients included ICM visit on the day of discharge, percentage of with follow-up call from each area (PCF, OCM, and provider practice), and percentage of patients with follow-up visits complete. As seen in Table 3, targets were met for each metric except for ICM visit on the day of discharge, which was below goal by 16%. This was attributed to weekend staffing constraints.
The model performance during the project was continuously reviewed to ensure accuracy. The AUC for this predictive model averaged at 0.54, which identified the model performing better than random chance. The decrease in the AUC value was expected as the interventions on high-risk patients prevented some readmissions, so they appeared as false positives in the monthly model evaluation.
Discussion
Utilizing predictive modeling to identify those at risk for readmission supports the ability to prioritize health care resources to meet patient care needs. The transition of care from acute care to the ambulatory setting can be challenging. Through this project, the clinical care needs of patients and the risk for readmission were communicated to the patient and various members of the care team. This allowed for conversations with patients to consider additional services post-discharge to support a safe transition home.
Multiple outreach opportunities were achieved through the care team communication with the patient. During these outreach opportunities, the care team confirmed appropriate follow-up appointments were established and addressed any discrepancies in the discharge plan including but not limited to review of medications, ability to obtain prescribed discharge medications, connection with home services or equipment, and review of knowledge regarding instructions for when to call a physician.
Challenges and Limitations
While this study reduced readmissions for oncology patients, it was not without limitations and challenges. Of primary importance was the challenge in applying theoretical concepts and models to a real-time clinical setting. Theories of predictive modeling are well studied and proven in academic research, however, impactful application in real-time clinical settings has many challenges.
The most significant challenge was developing an accurate model using available data in the EHR at the time of clinical decision-making. Many models in the literature study and evaluate care processes based on retrospective data in health care. Retrospective data provide additional sources for data elements that are often incomplete or missing at the time of clinical decision-making within the EHR. This is mainly due to the delay in clinical documentation and coding. To ensure adoption by the clinical team, the availability to model at the time of decision-making was key, and these data issues were resolved with modification of data elements and thorough data analysis during model development.
Secondly, the data was fluid, ie, as the patient care progressed in the hospital, the data elements used in the model changed leading to a new risk score for the patient. Forty-Five percent of patients did not have a risk level change during their hospital stay. The team agreed that providing interventions to patients who ultimately were not high risk was acceptable, and the higher priority was to ensure those who did not become high risk till the day of discharge still received the post-discharge interventions. The process was improved by including on the daily risk score report those who were discharged prior day to ensure no high-risk patients were missed.
To help improve the clinician team’s confidence and understanding of a black box data mining model, it was attempted to use model-agnostic methodologies for explaining the data elements driving a specific risk score like Surrogate trees and LIME. Unfortunately, these methods performed poorly and could not explain the main driving factors of patient scores. By not having this level of detail, tailored interventions were more difficult to apply.
In addition to the data challenges, the team faced process implementation challenges. First, there was a lack of available resources to perform the processes 7 days a week due to lower staffing coverage often seen in acute and ambulatory settings. The team made the decision to proceed despite this challenge with plans to justify additional resources based on anticipated study results. Another challenge to overcome involved the ability to share and communicate patient-specific data across the care team due to differences in hospital-based and ambulatory services and private physician offices. The Health Insurance Portability and Accountability Act prevents sharing patient information across private practices. This made it difficult to coordinate and create transparency for effective interventions. To manage this, triaging to the correct practice was done manually by the health system staff for risk notification and reporting, which reduces efficiency and makes it more difficult for care teams to quickly target patient interventions.
Future Work
The lessons learned and outcomes of this study have wide applicability to similar care teams and patient populations, and additionally, there are areas for further study. The first area of need is better understanding the impact of the process on patient experience. At the outset, a potential benefit was identified in measuring and understanding patient’s satisfaction or dissatisfaction with the process. The initial hypothesis was that, as a result of our coordinated processes, communications, and follow-up, patients would experience the interventions as largely positive relative to those who did not go through the process. However, due to limitations in the number of surveys received, the assumption was unable to be tested.
Another area for further study would be to evaluate the impact of using lower thresholds of readmission risk to include more patients in the process. By doing so, it is hypothesized more readmissions would be prevented. Additionally, the prediction model can be retrained because a considerable number of high-risk patients are prevented from being readmitted, the model training data can be modified to include recent records and analyzed to derive new insights about important features and enhancing accuracy. Finally, having proven results from this work, justification is available for additional resources to expand this process to more practices within the Oncology Service.
Conclusion
A coordinated and collaborative approach for the transition of care occurred with the application of proactive knowledge of predicting readmission risk. As seen through the results, this collective process had a positive impact and decreased the readmission rate for this oncology inpatient unit.
References
1. Maali Y, Perez-Concha O, Coiera E, Roffe D, Day RO, Gallego B. Predicting 7-day, 30-day and 60-day all-cause unplanned readmission: a case study of a Sydney hospital. BMC Med Inform Decis Mak. 2018;18(1):1. doi:10.1186/s12911-017-0580-8
2. James TA, Kasumova G, Alapati A, Mamtani A. Unplanned readmissions following breast cancer surgery. Am J Surg. 2019;218(5):988-992. doi:10.1016/j.amjsurg.2019.06.017
3. Manning AM, Casper KA, Peter KS, Wilson KM, Mark JR, Collar RM. Can predictive modeling identify head and neck oncology patients at risk for readmission? Otolaryngol Head Neck Surg. 2018;159(4):669-674. doi:10.1177/0194599818775938
4. Montero AJ, Stevenson J, Guthrie AE, et al. Reducing unplanned medical oncology readmissions by improving outpatient care Transitions: a process improvement project at the Cleveland Clinic. J Oncol Pract. 2016;12(5):e594-e602. doi:10.1200/JOP.2015.007880
5. Flaks-Manov N, Topaz M, Hoshen M, Balicer R, Shadmi E. Identifying patients at highest-risk: the best timing to apply a readmission predictive model. Int J Integrated Care. 2019;19(4):331. doi:10.5334/ijic.s3331
6. Nguyen OK, Makam AN, Clark C, et al. Predicting all-cause readmissions using electronic health record data from the entire hospitalization: model development and comparison. J Hosp Med. 2016;11(7):473-480. doi:10.1002/jhm.2568
7. Rubin DB. Inference and missing data. Biometrika. 1976;63(3):581-592.
