Assessing Patient Satisfaction in Behavioral Health: The Influence of Anonymity

Collecting a patient satisfaction survey after an episode of care in a behavioral health hospital setting is increasingly an industry standard (Boyer et al., 2009), which allows direct feedback to be provided to a facility about the quality of the care received (Kelstrup et al., 1993). Despite the proliferation of patient satisfaction data collection, there is a lack of “best practices” for how such collection should be performed. One potential question is whether the responses a patient provides should be anonymized and not connected to other identifying patient information.

In academic clinical research, anonymity is often a condition of IRB approval for study design. Such studies typically allow for greater information to be gathered about a participant than would be gained on a single satisfaction survey, however. By contrast, in a practice setting, identifying information is stored by necessity as part of a patient’s medical record. Separating satisfaction from this data leaves a large gap in the knowledge collected, and specific plans of action for quality improvement would be harder to implement. However, the academic standard exists to address concerns that identifying information could introduce bias into participants’ responses. In practice, could non-anonymity bias patient responses, inflating satisfaction scores?

Reviews of satisfaction scales for use in psychiatric care often fail to address anonymity as an influence of data collection at all (Miglietta et al., 2018, Boyer et al., 2009), and other reviews of influences on patient satisfaction also do not mention anonymity as a factor (Williams & Wilkinson, 1995; Thornton et al., 2017; Priebe & Miglietta, 2019). What little research has investigated differences in patient satisfaction based on anonymity has not found any significant differences in satisfaction scores, reliability, or validity (Leonhard et al., 1997; Zimmerman et al., 2017).

So, is there a “correct” approach? What differences exist between anonymous and non-anonymous patient satisfaction responses in psychiatric settings? As a vendor for psychiatric patient outcomes reporting, we had the unique position to assess this question directly.

Methods

Our dataset contains satisfaction data collected by 212 facilities offering psychiatric inpatient (e.g., acute and residential) and outpatient (e.g., partial hospitalization groups) services in 39 states across the United States and territories, between the years 2015 to 2021. Facilities issued satisfaction surveys to patients at the end of treatment, 1-3 days prior to discharge, and patient completion was voluntary.

Prior to 2018, satisfaction surveys did not include identifiable patient information, and results were reported only at the facility level. New surveys with identifiable patient information, specifically patient account number, were introduced in April 2018, creating a brief transition period where surveys were collected both with and without patient identifiers. This overlapping time is marked in our results.

Patient satisfaction data were collected on one-page survey forms with 14-16 items rated on a scale from 1 (Strongly Disagree) to 5 (Strongly Agree). Survey items differed slightly by treatment setting (inpatient vs outpatient), and individual items were further altered between the anonymous and identified versions of the surveys. However, an average satisfaction score was calculated on the same scale for all survey varieties, ranging from 1 to 5 (with 5 being the highest possible satisfaction average).

Examination of different survey types (anonymous vs. identified) included all data available from 2015 to 2021, and compared average satisfaction scores, as well as selected specific items frequently reported that are shared by both inpatient and outpatient surveys.

Results

Inpatients contributed 1,918,566 surveys and outpatients contributed 271,497. Patients from all age groups and severity levels were included in analyses, though patient demographics could not be analyzed for anonymous surveys given the nature of those surveys.

When comparing outpatient and inpatient data across all years, the conversion to identified satisfaction surveys did not meaningfully alter score patterns. In fact, the only statistical change was a slight lowering of average satisfaction after the conversion (p < .001, η2 < .001).

>> VIEW Table 1: Average Patient Satisfaction Score

A further examination of individual items across satisfaction surveys showed similar results: If any change occurred when surveys became identified, it was a slight lowering of scores, again with limited practical importance.

>> VIEW Table 2: Patient Satisfaction Item Scores

Conclusions

In literature and our data, it appears there is no meaningful impact of anonymity on satisfaction survey data collection in a behavioral health setting. Results from our analyses indicated patients are willing to share their attitudes and feedback with their care providers, regardless of the presence of an account number. The addition of a patient account number to satisfaction surveys (making them non- anonymous) led to a slight lowering of individual item and average satisfaction scores. Overall, the differences observed were statistically significant but lacking in practical impact, suggesting the presence of identifying information does not contribute any additional pressure of socially desirable responses when patients complete the survey.

By connecting patient satisfaction to a patient’s record, an organization is better able to pinpoint where treatment/services may be excelling or falling short of meeting patient needs, and thus better able to respond and improve their quality of care (Hackman et al., 2007; Hudak et al., 2004). The decision to collect anonymous or identified patient satisfaction surveys remains an organizational choice, but our results suggest there is no “correct” answer from a purely data quality perspective. While anonymity may be a guideline for survey research broadly speaking, as long as patients are reasonably assured their responses are kept private and pertain to their own treatment, they seem willing to share their true thoughts even if they could be identified.

Rachel B. Nowlin, MS, and Sarah K. Brown, DrPH are data analysts with Mental Health Outcomes.

References

Boyer L, Baumstarck-Barrau K, Cano N, Zendjidjian X, Belzeaux R, Limousin S, Magalon D, Samuelian J-C, Lancon C, & Auquier P. Assessment of psychiatric inpatient satisfaction: A systematic review of self-reported instruments. European Psychiatry. 2009; 24: 540-549.

Hackman A, Brown C, Yang Y, et al. Consumer Satisfaction with Inpatient Psychiatric Treatment Among Persons with Severe Mental Illness. Community mental health journal. 2007;43:551-564.

Hudak PL, Hogg-Johnson S, Bombardier C, McKeever PD, Wright JG. Testing a New Theory of Patient Satisfaction with Treatment Outcome. Medical care. 2004;42:726-739.

Leonhard C, Gastfriend DR, Tuffy LJ, Neill J, & Ploug A. The effect of anonymous vs. nonanonymous rating conditions on patient satisfaction and motivation ratings in a population of substance abuse patients. Alcoholism: Clinical and Experimental Research. 1997;21(4), 627-630.

Miglietta E, Belessiotis-Richards C, Ruggeri M, Priebe S. Scales for assessing patient satisfaction with mental health care: A systematic review. Journal of psychiatric research. 2018;100:33-46.

Priebe S, Miglietta E. Assessment and determinants of patient satisfaction with mental health care. World Psychiatry. 2019;18(1):30-31. doi:10.1002/wps.20586

Thornton RD, Nurse N, Snavely L, Hackett-Zahler S, Frank K, & DiTomasso RA. Influences on patient satisfaction in healthcare centers: a semi-quantitative study over 5 years. BMC Health Services Research. 2017;17:361. DOI 10.1186/s12913-017-2307-z

Williams B, & Wilkinson G. Patient satisfaction in mental health care: Evaluating an evaluative method. British Journal of Psychiatry. 1995;166(5), 559-562. doi:10.1192/bjp.166.5.559

Zimmerman M, Holst CG, Mehring LB, Moon S, Berges A. The impact of anonymity on psychiatric patients' ratings of satisfaction with the initial evaluation. Ann Clin Psychiatry. 2017 Nov;29(4):220-226. PMID: 29069106.

The views expressed in Perspectives are solely those of the author and do not necessarily reflect the views of Behavioral Healthcare Executive, the Psychiatry & Behavioral Health Learning Network, or other Network authors. Perspectives entries are not medical advice.