Barriers to Lymphoma Clinical Practice Guideline Adherence: A Pilot Mixed-Methods Research Study

Abstract: Little is published about adherence barriers to lymphoma clinical practice guidelines (CPGs). This article examines the most frequent barriers to adherence to lymphoma CPGs. An explanatory sequential mixed-methods study with quantitative and qualitative parts was conducted. In the quantitative part, 52 surveys were completed by lymphoma physicians. There were adherence barriers in the social/environmental, organizational, and professional attitudes domains. Additionally, survey results were supplemented by interviews with 12 of the 52 survey respondents in the qualitative part. Three themes emerged from the interviews in the interpretation of the results related to barriers: patient comorbidities, inadequate use of technology, and medical insurance. Physicians in academic practices reported more difficulty in adhering to lymphoma CPGs in all domains than did physicians in nonacademic practices. Older, more experienced physicians reported less difficulty adhering to the lymphoma CPGs in organizational and professional attitude domains than the younger physicians. Lymphoma doctors wish to have data on how the adherence to CPGs translates to positive patient outcomes and a decrease in health care costs. They expect to be able to make decisions that are exceptions to CPGs when patients have comorbidities and to have technology be utilized more efficiently.

Professional organizations have developed clinical practice guidelines (CPGs) for almost every type of tumor. The CPGs are developed to reduce variability in clinical practice and are based on the best available evidence by following the Institute of Medicine’s (IOM) standards.¹ These evidence-based best practices are meant to ensure quality treatment for all patients. Due to increasing pressure to practice evidence-based medicine and adhere to standards of care, CPGs have become increasingly significant for all clinicians, insurance companies, and physicians.^2,3

However, there is nonuniform adherence to CPGs across cancer types in the United States. Several authors have postulated that a lack of adherence to CPGs by oncologists may be related to a lack of sufficient quality in guideline development, financial conflicts among the authors designing the CPGs, and additional internal and external barriers for physicians.^2-4 Lack of adherence to CPGs may affect patient care both favorably—eg, not administering a highly toxic chemotherapy regimen to a patient with poor performance status—and unfavorably—eg, not administering an appropriate standard of care to a patient who can tolerate it.^1,3,4 There are no reports on the most frequent barriers to adherence to lymphoma CPGs and how the barriers affect lymphoma doctors. 

The aim of this study was to understand the factors contributing to lymphoma CPG adherence in a range of domains—social/environmental, organizational, and professional—to gain more insight into potential improvements needed in CPGs and initiatives to encourage adherence.

Methods

Institutional Review Board approval was obtained from Drexel University prior to starting the study. The study took place from approximately March 2017 to April 2018, including the data analysis and reporting. The survey data and semi-structured interview content were de-identified, and confidentiality was maintained. 

Participants, Sample, and Recruitment

The survey was deployed to about 700 lymphoma doctors as a nonrandom sample via Qualtrics software. The participation was on a voluntary basis and based on electronic consent. The lymphoma doctors surveyed were required to be practicing, licensed hematologists or oncologists in the United States, either as lymphoma specialists or as general oncologists treating lymphoma patients.  

Research Design and Rationale

The study used an explanatory sequential mixed-methods methodology, which included quantitative (QUANT) and qualitative (qual) parts (the uppercase letters indicate more weight, whereas lower-case letters indicate lower weight). In the QUANT part, a survey instrument was adapted specifically for lymphoma CPGs and deployed to lymphoma doctors. In the qual part, 12 follow-up interviews were conducted with participants who also responded to the survey (nested sample). 

The QUANT survey results were integrated with the experiences and themes of lymphoma doctors from the qual part. Using individual qualitative and quantitative methods only provides strong views and has natural limitations.⁵ This design contextualizes and deeply combines the understanding of the complexity of the study problem vs data indicating that each research phase on its own might not have succeeded as well.^5-8

Conceptual Framework

The first framework used in the interview creation is the seminal work provided by Cabana et al,⁹ in which a few barriers impede adherence to guidelines: behaviors, emotions, and attitudes of doctors, including a lack of agreement about CPGs, and a lack of motivation to follow the CPGs consistently. The second framework as a theory categorized these types of barriers and others as organizational, social or environmental, and professional.¹⁰  There is some overlap, since none of the frameworks could stand on its own for the development of the instrument and study in lymphoma CPGs, and to meet IOM’s standards for development and maintenance of CPGs. Consequently, barriers were organized in the survey as social (Q29 battery of questions/barriers), organizational (Q13 domain), and professional (Q30 domain). All were aligned to the IOM’s vision on the standards of CPGs.

Instrument

A nonvalidated survey was revised significantly to match the constructs according to the conceptual framework and answer the research questions.¹¹ The survey instrument included Likert scales and was first piloted, as recommended by Johnson and Christensen.¹² The barriers and lymphoma doctors’ attitudes were divided into three domains.

The social/environmental (Q29) domain included questions such as the scientific development and methodology of lymphoma CPGs, on tools (eg, AGREE II) used for developing lymphoma CPGs, on the utilization of lymphoma CPGs and improvement in quality of cancer treatment, on the utilization of lymphoma CPGs and positive outcomes for patients (eg, longer survival), and on the utilization of lymphoma CPGs for diagnosis and management of lymphoma.

The organizational (Q13) domain included questions on barriers such as those experienced with medical and administrative resources, frequency of updates for lymphoma CPGs, abundance of CPGs from medical oncology organizations, navigation barriers such as technology and access, available clinical pathways at the workplace, and the flexibility of CPGs.  

Lastly, the professional (Q30) domain included questions on doctors’ attitudes in not following lymphoma CPGs consistently. Examples included professional attitude questions on doctors’ autonomy when they follow lymphoma CPGs, the effect of seniority in the field of lymphoma (eg, number of years of experience in the field) and doctors’ utilization of CPGs, their attitudes towards other treatment options outside lymphoma CPGs, lymphoma doctors’ attitudes toward health care costs, and quality of care when using CPGs.  

Data Analysis

Survey answers were imported into the Statistical Package for Social Sciences version 24 (courtesy of Drexel University). Linear regression and logistical regression were used as analysis models to examine associations between practicing CPGs and each of the three domains and covariates. Not all associations studied are included here. The covariates or independent variables used in the statistical models were sex, age group, region of practice, location/type of medical practice, and duration of medical practice. A significance level of 0.3 was required to allow a variable into the model, and a significance level of 0.35 was required for a variable to stay in the model. 

The total score is the sum of all the response scores for each respondent per domain. For Q29, a higher score indicates that it is easier to implement lymphoma CPGs. For Q13, a higher score indicates that it is more difficult to implement lymphoma CPGs. For Q30, a higher score indicates that it is more difficult to implement lymphoma CPGs. Descriptive and in vivo manual coding in two recursive cycles were used to develop and analyze themes and concepts from the qualitative interviews and transcripts.¹³ 

Results

Despite strong efforts of a twice-weekly follow up, only 52 lymphoma doctors completed the survey with a poor return rate of 7.4%. Respondents included 34 males (65%) and 18 females (35%). The respondents were 61% lymphoma hematologists, 14% general oncologists who treat lymphoma patients, and 25% were licensed as both hematologists and oncologists. Thirty-one of the respondents (59%) practiced medicine in academic hospitals, 7 (14%) practiced in community hospitals, and 11 (19%) worked in private practice. Table 1 shows the full demographics and professional characteristics of the respondents.

Most respondents (88%) who answered the survey use National Comprehensive Cancer Network (NCCN) lymphoma CPGs. Six respondents (11.5%) do not use NCCN lymphoma CPGs and instead use others, including institutions’ guidelines and a combination of American Society of Hematology/European Hematology Association/European Society of Medical Oncology guidelines. 

In the Q29 domain (social/environmental), 69% of respondents found lymphoma CPGs easy to understand and fully utilized them. Ninety-two percent of the respondents believed the CPGs are evidence based, and 94% found that the methodology used to develop and update lymphoma CPGs reasonable. Seventy-nine percent of respondents believed that the methodology of CPG development is not affected by conflicts of interest of those constructing CPGs. Table 2 has the 3 most frequently experienced barriers for Q29 domain.

In the organizational domain Q13, 69% of respondents found lymphoma CPGs convenient to use because they are easy to find in a typical day. Eighty percent and 76% of respondents found that medical and administrative resources, respectively, were not significant barriers to implementation of CPGs. Eighty percent of respondents felt that frequency with which CPGs are updated was adequate to facilitate their implementation. Seventy-three percent of respondents did not find the number of medical oncology organizations that develop lymphoma CPGs to be a barrier. 

Only 47% of the respondents agreed that they would utilize lymphoma CPGs more often if lymphoma CPGs were available online and easy to navigate, while 37% were neutral. This question was the only one that had a mode of zero, and it had the highest number of respondents in the neutral category. It was not clear how having updated technology related to the notion of having lymphoma CPGs online and using them more often. The lack of using technology more efficiently at work was identified as a barrier. It was uncertain whether easily available technology would be desirable to have at work to make the adherence to lymphoma CPGs more frequent and efficient. Consequently, technology as a question, format, and its usefulness toward the adherence of lymphoma of CPGs was explored in qualitative interviews (the qual part).

In the professional attitude domain (Q30), respondents’ attitudes in not consistently following lymphoma CPGs were examined. Seniority expressed in years of experience in the field was not a strong factor that prevented doctors from following lymphoma CPGs (61% of the respondents agreed). Fifty-one percent of respondents agreed that using CPGs increases the quality of care, regardless of the owner or developer of the CPGs (27% were neutral, while 22% disagreed). Fifty-one percent of respondents believe they still have autonomy when following lymphoma CPGs; of the remaining 49%, 29% believe that they lack autonomy when following CPGs, and the other 20% are neutral. The most frequently experienced barriers to adherence to CPGs in the Q30 professional domain are shown in Table 3. 

Associations

For the Q29 (social/environmental) domain, respondents from academic practices had lower total scores than respondents from nonacademic practices (point estimate -6, P=.0087), suggesting that physicians who practice in academic teaching hospitals are less likely than those in private practice or community hospitals to implement lymphoma CPGs. 

For the Q13 domain, age group (eg, 37-54, 55-63, and 64-81 years) and the location of the medical practice were the two influential covariates (P=.0283 and P=.0202, respectively). The total score of organizational attitudes was about 4 points higher for the location of medical practice (the point estimate was positive 3.59). This could represent more difficulty in practicing lymphoma CPGs due to barriers arising from the organizational Q13 domain among responders from academic teaching hospitals compared with those from other types of practices. The total score of Q13 was 2 points lower for every increase between the three age categories. The point estimate was negative 1.92. Older respondents were less likely to have a higher total score in the Q13 domain. 

For the Q30 domain, between the total Q30 score and covariates, there were three influential covariates: age group (P=.0035), location of medical practice (P=.0003), and the South region (P=.0287). The total score of this domain was about 4 points lower for every increase in age category. Older respondents were less likely to have a higher total Q30 score. As the age group category increased, the total Q30 score decreased by four points, as demonstrated by a negative point estimate of 3.91. This negative slope would mean that as age increases, it is less difficult to implement lymphoma CPGs. On the other hand, the total score of professional attitudes was about 9 points higher for the respondents from the academic teaching hospitals and 6 points higher for the respondents from the South region compared with those from other regions. The clinical implications of these associations are in the Discussion section.

Theme and Subthemes From the Qual Part

The QUANT results were followed up by and integrated with interviews discussing the experiences of the lymphoma doctors. The themes that emerged from the additional 12 interviewees were (1) barriers to the adherence to lymphoma CPGs from the perspective of patients’ factors and comorbidities; (2) technology as an encompassing theme; and (3) medical insurance as a potential barrier to fully implementing lymphoma CPGs (Supplemental Table 1).

Discussion

In this small study in the field of lymphoma, physicians in academic practices reported more difficulty in adhering to lymphoma CPGs in all domains than did physicians in nonacademic practices. Older, more experienced physicians reported less difficulty adhering to the lymphoma CPGs in organizational and professional attitude domains than the younger physicians. Doctors believe lymphoma CPGs such as NCCN CPGs follow IOM standards for development and are evidence-based. Lymphoma CPGs, however, do not always consider individual patient factors or the heterogeneity of the disease itself. Quality of lymphoma CPGs was found to be high and was not a barrier in adhering to lymphoma CPGs. Likewise, lack of motivation, lack of awareness, and having negative attitudes toward lymphoma CPGs were not found as barriers to the adherence of lymphoma CPGs.  

While the lymphoma doctors found lymphoma CPGs evidence-based they wanted to have compelling evidence published on how adherence translate to a decrease in health care costs.  Patient outcomes and impact on health care costs by adhering to lymphoma CPGs are not referenced fully in the lymphoma field and are expected, as noted in this study. Further the expectations are to be allowed to make exceptions to the adherence of lymphoma CPGs, due to patients’ comorbidities, variability, and tolerability without being called noncompliant. This exception should not be a reflection on a doctor’s inability to adhere to lymphoma CPGs but rather a warranted documentation and as an option to electronic medical records (EMRs) with the help of technology.  

Having frequent updates made to lymphoma CPGs to be in sync with most research known, while very well liked as per the results of the study, is not enough to be made available on the internet. There is a demand for technology to be flexible and to be utilized more efficiently in the implementation of lymphoma CPGs and clinical pathways and their welcomed updates. Additionally, insurance plans should provide uniformly a much better conduit and coverage for patients’ variability, patients’ drug tolerability, and for lowering patients’ deductible costs regardless of the geographical region.

Below is a detailed comparative discussion of the above summary of findings. 

Q29 and Q13 (Type of Medical Practice) 

The statistically significant associations of the total score of individual Q29 and Q13 domains and the location/type of the medical practice suggest that specialists such as hematologists who usually practice in academic teaching hospitals or oncology doctors who practice in academic teaching hospitals vs private practice or community hospitals may differ in implementing lymphoma CPGs. Due to the different directions of point estimates (negative 6 in Q29 and positive 3.59 in Q13) and the meaning of the total score (refer to Data Analysis), lymphoma doctors may have more difficulty implementing lymphoma CPGs in the Q13 domain or are less likely to implement lymphoma CPGs in Q29 if they are from academic hospitals. 

One possible explanation for this finding might be that academic physicians are more specialized in their practices than nonacademic physicians; perhaps this specialization leads academic physicians to feel more comfortable deviating from CPGs, or just not using them at all, than nonacademic physicians. Perhaps nonspecialist, nonacademic physicians feel the need to refer to CPGs more frequently to make sure they are providing the best level of care. A second explanation for this finding might be differences in insurance company treatment of academic vs nonacademic sites or differences in general.^14,15 For example, perhaps insurance companies permit deviation from CPGs at large academic sites more than at nonacademic sites, which was mentioned in the qual part. A third explanation, which was referred to in the qual part as well, might include differences in patient characteristics at academic and nonacademic sites. Perhaps academic sites see referrals of more heavily pretreated patients who have additional comorbidities resulting from prior therapies; this, in theory, could lead to a greater need to deviate from CPGs for complicated patient cases.^16,17

Q13 and Q30 (Age Group) 

The association between the total score and the age group is indicative that older, more experienced lymphoma doctors may adhere more to CPGs than younger doctors. This finding has not been reproduced in other papers in the literature.^18-20 In those papers, outside the lymphoma field and based on these 2 domains, more senior doctors have reported difficulties in implementing new and updated CPGs, as they believe their established ways work well.^18-20 Perhaps our findings represent changing attitudes among physicians of different age groups toward CPGs. For example, as lymphoma CPGs have become more complicated, incorporating novel agents, it may be that older, more experienced physicians need to refer to them more commonly to provide standard of care, whereas younger physicians closer to their training may refer to CPGs less.

Q30 (South Region) 

Our study found that respondents in the South region of the United States reported more barriers to adherence in the professional domain than did respondents from other regions. Several factors may have contributed to this finding including differences in treatment care costs, differences in insurance coverage, differences in patient comorbidities (eg, more obesity, more diabetes, more neuropathy), and differences in access to health care facilities affecting treatment recommendations. The regional variations are also confounded by the hospital types, being allowed to order more tests for referrals to medical specialists or for decision making in gray areas of CPGs, and they may even relate to the amount of money dedicated to health care provided to the underinsured and noninsured.^21-24 Another factor may simply be statistical mischance, as the number of respondents from each of the four regions of the country was low. These are speculations, and it was beyond the scope of this study to determine whether adherence to lymphoma CPGs affects patient outcomes.

Mixed-Methods Integration of Results in the Qual Part

Interviewed participants clarified that technology is well-liked but not well-utilized in medicine. Lymphoma CPGs are not always part of EMRs at all institutions and private offices in the United States, and, when they are, they are not updated frequently at the same rate that the NCCN lymphoma CPGs are on the NCCN website. Participants explained that the high rate of neutral answers from the survey is an issue of a lack or limitation of time and that the survey question was not very precise. The reason was due to the fact that “online,” a term used in the survey technology question, has many meanings. Even if CPGs are available online, they are not necessarily embedded in the EMR (Supplemental Table 1). 

Limited treatment options for patient care was clarified as a barrier to fully following the lymphoma CPGs in the qual part (Supplemental Table 1). Many disagreed that lymphoma CPGs limit treatment options, but, in the qual part, it was clarified that lymphoma patients who relapse multiple times differ in the extent and tolerance of treatment. Lymphoma CPGs are less helpful when a patient needs treatment beyond the second line of lymphoma relapse or are not always helpful for patients with comorbidities. This interpretation is also found in the literature, but not necessarily fully documented in the lymphoma field, for older adults with more comorbid conditions, who may need more treatment options for the same health quality and outcomes.^25-28 

Lymphoma doctors agreed that lymphoma CPGs cannot be implemented equally or consistently, as best treatment options for patients who have comorbidities, advanced age, and financial factors secondary to insurance coverage (Supplemental Table 1). These situations make it more difficult to comply with the lymphoma CPGs. Financial factors such as high out-of-pocket costs are unfortunate and should be fixed. There were abundant answers with the following sentiment: “Out-of-pocket costs should be reviewed more uniformly across the country as they are not right for patients.” The financial implications of different treatment decisions within all guidelines can vary significantly as found in the literature. Patients’ out-of-pocket payments for various treatment regimens may also vary significantly.²⁹

All participants agreed they were not aware of published lymphoma studies showing an association of consistently using lymphoma CPGs with positive patient outcomes, so they could not answer the related survey question fully. The participants recommended also that the question be changed to increase its clarity for future studies. They requested that a more specific and detailed question be developed or divided into two questions, when discussing “positive patient outcomes.” More precisely, the respondents said that, when it comes to outcome, not every patient will respond to antilymphoma treatment well, nor all the time, to treatment recommended from lymphoma CPGs, especially when one is discussing outcome for a patient who is newly diagnosed (eg, first line of treatment recommended from the lymphoma CPGs) vs for a patient who is relapsed.  

Some participants added that, while the topic is complex, there are available data on guideline and clinical pathway adherence for other types of tumors, such as colon cancer or lung cancer.^30-32 For example, Hoverman et al researched 2 distinct databases to evaluate survival according to pathway status in patients with colon cancer.³³ Results from these two distinct databases suggested that treatment of patients with colon cancer on-pathway costs less; use of these pathways demonstrated clinical outcomes consistent with published evidence.³³ Furthermore, most participants shared that they were not aware of literature that health care costs were reduced with the adherence specifically to lymphoma CPGs (Supplemental Table 1). While this is concerning in a community where health care costs are high, it is not totally inconceivable. From a payer’s perspective, the American Society of Clinical Oncology recommendations should include a statement in their policy definitively urging clinicians to use clinical pathways (created based on CPGs) in every oncology practice to improve patient care in their practices.³⁴ 

Overall exceptions not to fully use the lymphoma CPGs are due to factors such as patients’ comorbidities, age, and financial factors secondary to lack of consistent insurance coverage.  

Doctors expressed and emphasized that CPGs are guidelines; eg, physicians should be allowed to follow their own clinical judgement when the CPGs do not meet patients’ needs. Lymphoma CPGs are developed based on clinical research trials, and they do not always represent the general population. Specifically, doctors stated that while lymphoma CPGs are very helpful, they are still just guides, and physicians would not like to be called noncompliant in cases when they cannot follow CPGs. The participants expressed that “they [the doctors] may disagree at times with lymphoma CPGs because they do not meet patients’ needs or patient preferences.” At those times, nonadherence is not the correct term.

Nonadherence was also mentioned when doctors clarified their results about insurance coverage in the qual part. Participants who helped interpret the QUANT results believed that, in the end, the difference in insurance coverage hurts the patient and takes away from doctors’ time. This amount of time doubles because lymphoma is not one disease but many subtypes, which adds to the degree of negativism or “bad side” of insurances from doctors’ experiences. Lymphoma doctors have to fight much more for lymphoma patients due to the many subtypes of lymphoma.³⁵ Patients diagnosed with lymphoma and patients who relapse multiple times differ in how they tolerate treatment, especially in older adults and patients with more comorbid conditions who may need more treatment options for same health quality and outcomes.^25-28 

Additionally, the doctors believed that one gap in following fully the lymphoma CPGs are treatment options beyond second line of lymphoma treatment. This finding was confounded by other factors, and it was not necessarily due to lymphoma CPGs. The development and updates of lymphoma CPGs are based on strict clinical research trials.³⁷ Clinical research results are limited because of firm inclusion and exclusion criteria for patients who entered clinical research trials. But patients who are not eligible to enter the rigid clinical research trials are abundant in real life. Pharmaceutical companies and academic research institutions should apply, based on rationale, the new Food and Drug Administration initiative of including patients who are more representative of real-life scenarios who present with many comorbidities.^36,37 Payers should be very cognizant of this initiative and understand from which patient population the study results came about (eg, the inclusion/exclusion criteria of the trial that led to the approval of a drug) to support uniformly approved treatment for patients across the United States regardless of the type of regional insurance coverage.

Limitations and Main Recommendations for Future Research

The small study size dictates caution to be exercised against the results and mandates more research to take place. The low response to the survey threatens the generalizability of the results. However, using a mixed-methods design built on the strength of each data collection part (QUANT and qual) to minimize the weaknesses of a single approach. This type of design is known to increase both the validity and reliability of data.⁷ 

Further, participants interviewed were from the QUANT part of the study. A nested sample was desired to help with internal validity.^5,6 This nested approach for the qual part resulted in clarified answers from the survey and added current clinical practice insights that could not be explained via the survey; if these participants would not have participated in the survey, input would not have been as relevant. Also the participants from survey (QUANT part) and the interviews (qual part) were spread relatively equally from the 4 geographical regions of United States. 

In preparation of a small study sample, mitigations were built in before the study started. For example, the survey was piloted first with lymphoma experts to improve content validity. While the survey instrument is not validated, as its validation was outside the scope of this study, some of the constructs of the survey, including emotional and behavioral contents, have been cited and examined before.¹¹ In summary, while the sample size was, in the end, small, the authors had planned prospectively for some mitigations to minimize bias. The type of study design and the process of piloting the survey instrument were important mitigations used a priori to improve at least the internal validity of the study.

It is not fully known why the response rate was poor in the QUANT part, but it has been stipulated, and confirmed with a few doctors from the qualitative interviews, that these respondents get 4 to 5 surveys a day, and the main researcher was not known to them. As stated, the purpose of the study was to find new content constructs and provide results that could be investigated in larger studies in the field of lymphoma. 

Additionally, main recommendations for future research in the lymphoma field include the need to conduct large studies to further investigate factors of nonadherence, such as insurance barriers due to high deductibles and the incorporation of cost information into the development of guidelines. Studying the association between adherence to lymphoma CPGs and patient outcomes, and adherence to CPGs and cost of care, would be useful in demonstrating the value of lymphoma CPG adherence to practicing physicians.

Conclusion

This preliminary study provides unique insights into the field of lymphoma CPGs, how physicians from multiple settings and groups perceive and use them, and what barriers exist to consistent CPG adherence by lymphoma doctors in United States. Doctors believe lymphoma CPGs, such as NCCN CPGs, follow IOM standards for development and are evidence-based. Lymphoma CPGs, however, do not always consider patients’ factors or the heterogeneity of the disease itself. While doctors found lymphoma CPGs to be evidence-based, they want to have compelling evidence published on how adherence translates to a decrease in health care costs. There is a demand for technology to be flexible and to be utilized more efficiently in the implementation of lymphoma CPGs as well as pathways and their welcomed updates. Additionally, insurance plans should uniformly better account for patients’ variability, patients’ drug tolerability, and aim to lower patients’ deductible costs regardless of the geographical region.

Supplementary materials can be found in the PDF of this article.

References

1. Graham R, Mancher M, Miller Wolman D, Greenfield S, Steinberg, E, eds; Institute of Medicine. Clinical Practice Guidelines We Can Trust. Washington, DC: The National Academies Press; 2011.

2. Abdelsattar ZM, Reames BN, Regenbogen SE, Hendren S, Wong SL. Critical evaluation of the scientific content in clinical practice guidelines. Cancer. 2015;121(5):783-789. doi:10.1002/cncr.29124

3. Lenzer J, Hoffman JR, Furberg CD, Ioannidis JP; Guideline Panel Review Working Group. Ensuring the integrity of clinical practice guidelines: a tool for protecting patients. BMJ. 2013;347:f5535. doi:10.1136/bmj.f5535

4. Reames BN, Krell RW, Ponto SN, Wong SL. Critical evaluation of oncology clinical practice guidelines. J Clin Oncol. 2013;31(20):2563-2568. doi:10.1200/JCO.2012.46.8371

5. Creswell JW. Educational Research: Planning, Conducting, and Evaluating Quantitative and Qualitative Research. 5th ed. Upper Saddle River, NJ: Pearson; 2015.

6. Onwuegbuzie AJ, Collins KMT. A typology of mixed methods sampling designs in social science research. The Qualitative Report. 2007;12(2):281-316.

7. Creswell JW. Research Design. Qualitative, Quantitative, and Mixed Methods
Approaches. 4th ed. Thousand Oaks, CA: Sage Publications Inc; 2014.

8. Tashakkori A, Teddlie C. Mixed Methodology: Combining Qualitative and Quantitative Approaches. Thousand Oaks, CA: Sage; 1998.

9. Cabana MD, Rand CS, Powe NR, et al. Why don’t physicians follow clinical practice guidelines? A framework for improvement. JAMA. 1999;282(15):1458-1465.

10. Oxman A, Flottorp S. An overview of strategies to promote implementation of evidence-based health care. In: Silagy C and Haines A, eds. Evidence-Based Practice in Primary Care. 2nd ed. London, England: BMJ Books; 2001:101-119.

11. Goldberg SL, Akard LP, Dugan MJ, Faderl S, Pecora AL. Barriers to physician adherence to evidence-based monitoring guidelines in chronic myelogenous leukemia.
J Oncol Pract. 2015;11(3):e398-e404. doi:10.1200/JOP.2014.001099

12. Johnson BR, Christensen L. Educational Research: Quantitative, Qualitative, and Mixed Approaches. 5th ed. Thousand Oaks, CA: Sage Publications; 2014.

13. Saldana J. The Coding Manual for Qualitative Researchers. 2nd ed. Los Angeles, CA: Sage Publications; 2013.

14. Yu PP. Oncology clinical pathways: A form of governance? J Oncol Pract. 2018;14(3):144-146. doi:10.1200/JOP.17.00084

15. Kirkwood MK, Hanley A, Bruinooge SS, et al. The State of Oncology Practice in America, 2018: Results of the ASCO Practice Census Survey. J Oncol Pract. 2018;14(7):e412-e420. doi:10.1200/JOP.18.00149

16. Leukemia and Lymphoma Society, LLS. Choosing a Blood Cancer Specialist or Treatment Center. https://www.lls.org/sites/default/files/FS6_ChoosingABCSOTC%20Facts%20FINAL.pdf. Revised March 2015. Accessed March 27, 2019.

17. National Comprehensive Cancer Network. NCCN 2017. B-cell Lymphomas NCCN Clinical Practice Guidelines, version 7. December 5, 2017. nccn.org website. https://www.nccn.org/professionals/physician_gls/default.aspx. Accessed March 27, 2019.

18. Falandry C, Campone M, Cartron G, Guerin D, Freyer G. Trends in G CSF use in 990 patients after EORTC and ASCO guidelines. Eu J Cancer. 2010;46(13):2389-2398. doi:10.1016/j.ejca.2010.04.031

19. Link H, Nietsch J, Kerkmann M, Ortner P; Supportive Care Group (ASORS) of the German Cancer Society (DKG). Adherence to granulocyte-colony stimulating factor (G-CSF) guidelines to reduce the incidence of febrile neutropenia after chemotherapy—a representative sample survey in Germany. Support Care Cancer. 2016;24(1):367-376. doi:10.1007/s00520-015-2779-5

20. Munteanu MC, Jordan JC. A view into clinical practice guidelines: who uses them, who doesn’t and possibly, why. Qual Res Med Healthcare. 2017;1(2):44-50. doi:10.4081/qrmh.2017.6544

21. Graboyes EM, Mayer E, Sharma AK, Lentsch EJ, Day TA. Adherence to National Comprehensive Cancer Network Guidelines for time to initiation of postoperative radiation therapy for patients with head and neck cancer. Cancer. 2017;123(14):2651-2660. doi:10.1002/cncr.30651

22. Monson JR, Probst CP, Wexner SD, et al; and Consortium for Optimizing the Treatment of Rectal Cancer (OSTRiCh). Failure of evidence-based cancer care in the United States: the association between rectal cancer treatment, cancer center volume, and geography. Ann Surg. 2014;260(4):625-632. doi:10.1097/SLA.0000000000000928

23. Fisher ES, Bynum JP, Skinner JS. Slowing the growth of health care costs-lessons from regional variation. N Engl J Med. 2009;360(9):849-852. doi:10.1056/NEJMp0809794

24. Lindskog EB, Gunnarsdóttir KÁ, Derwinger K, Wettergren Y, Glimelius B, Kodeda K. A population-based cohort study on adherence to practice guidelines for adjuvant chemotherapy in colorectal cancer. BMC Cancer. 2014;14:948. doi:10.1186/1471-2407-14-948

25. Chagpar R, Xiang Y, Chiang Y, et al. Adherence to stage-specific treatment guidelines for patient with colon cancer. J Clin Oncol. 2012;30(9):972-979. doi:10.1200/JCO.2011.39.6937

26. Francke AL, Smit MC, de Veer AJE, Mistiaen P. Factors influencing the implementation of clinical guidelines for health care professionals: A systematic meta-review. BMC Med Inform Decis Mak. 2008;8:38. doi:10.1186/1472-6947-8-38

27. Hodeib M, Chang J, Liu F, et al. Socioeconomic status as a predictor of adherence to treatment guidelines for early-stage ovarian cancer. Gynecol Oncol. 2015;138(1):121-127. doi:10.1016/j.ygyno.2015.04.011

28. Stienen JC, Hermens R, Wennekes L, et al; for the PEARL study group. Variation in guideline adherence in non-Hodgkin’s lymphoma care: impact of patient and hospital characteristics. BMC Cancer. 2015;15:578. doi:10.1186/s12885-015-1547-8

29. Wharam JF, Zhang F, Lu CY, et al. Breast cancer diagnosis and treatment after high-deductible insurance enrollment. J Clin Oncol. 2018;36(11):1-7. doi:10.1200/JCO.2017.75.2501

30. Boland GM, Chang GJ, Haynes AB, et al. Association between adherence to national comprehensive cancer network treatment guidelines and improved survival in patients with colon cancer. Cancer. 2013;119(8):1593-1601. doi:10.1002/cncr.27935

31. Bristow RE, Chang J, Ziogas A, Anton-Culver H, Vieira VM. Spatial analysis of adherence to treatment guidelines for advanced-stage ovarian cancer and the impact of race and socioeconomic status. Gynecol Oncol. 2014;134(1):60-67. doi:10.1016/j.ygyno.2014.03.561

32. Jackson GL, Zullig LL, Zafar Y, et al. Using NCCN clinical practice guidelines in oncology to measure the quality of colorectal cancer in the veterans’ health administration. J Natl Compr Canc Netw. 2013;11(4):431-441.

33. Hoverman JR, Cartwright TH, Patt DA, et al. Pathways, outcomes, and costs in colon cancer: Retrospective evaluations in two distinct databases. J Oncol Pract. 2011;7(suppl 3):52s-59s. doi:10.1200/JOP.2011.000318

34. Newcomer LN, Malin J. Payer view of high-quality clinical pathways for cancer.
J Oncol Pract. 2017;13(3):148-150. doi:10.1200/JOP.2016.020503

35. Swerdlow SH, Campo E, Pileri SA, et al. The 2016 revision of the World Health Organization classification of lymphoid neoplasms. Blood. 2016;127(20):2375-2390. doi:10.1182/blood-2016-01-643569

36. Kim ES, Bruinooge SS, Roberts S, et al. Broadening eligibility criteria to make clinical trials more representative: American Society of Clinical Oncology and Friends of Cancer Research joint research statement. J Clin Oncol. 2017;35(33):3737-3744. doi:10.1200/JCO.2017.73.7916

37. Food and Drug Administration. Considerations for the Inclusion of Adolescent Patients in Adult Oncology Clinical Trials Guidance for Industry. https://www.fda.gov/downloads/Drugs/GuidanceComplianceRegulatoryInformation/Guidances/UCM609513.pdf. Published March 2019. Accessed March 27, 2019.