Know Your Numbers: We Must Protect Integrity of Patients` Self-Reporting

I recently purchased a new computer, and at the end of the process I was given a document and an ingratiating speech asking me to provide an excellent rating (“perfect 10 experience”) on the satisfaction survey that would soon be arriving via email. This was not my first experience with this, and so I thought little of it at the time. However, I had serious misgivings in subsequent days when I viewed a story detailing Russian interference in the 2016 election, followed by my completion of an article on the importance of tracking clinical outcomes in behavioral healthcare by using patient self-report measures.

We have entered a time where one must question the source and the validity of opinions and ratings. Are customer service ratings of 10 being politely coerced? Are political opinions being manipulated by false information on the internet? Are patient self-report ratings of symptoms, functioning and quality of life any more reliable than a consumer satisfaction rating?

I believe there are two issues that need to be teased out to answer these questions. One of them is longstanding, the other scientific. There are many ways of influencing the beliefs and attitudes of others, ranging from subtle forms of persuasion to brutal forms of brainwashing. Asking someone a leading question might get the answer you prefer, as would various forms of incentives or threats. We have long been aware of these processes, and it must be said that modern-day satisfaction surveys have crossed a line. They have lost all dignity and believability by pleading for top ratings.

However, satisfaction ratings are quite different from scientifically-designed questionnaires. It has long been known that many patients give high satisfaction ratings for their doctors because they like them and don’t want to offend them, even though they have experienced negative clinical outcomes. Questionnaires or self-report measures have been developed to avoid this bias. While they may not be immune to people exaggerating their health in a positive or negative direction, they have been developed based on scientific principles, and data analytics can help identify exaggerated surveys.

Patient self-report measures

Researchers develop clinical questionnaires by testing how the complete tool and each of its items succeeds in quantifying some targeted clinical factor, as compared with other established ways of measuring it. Standards must be met for survey instruments to be considered reliable and valid as measures. This is quite different from asking people for a single satisfaction rating, say from 1 to 10, or asking for a rating of your mood at the moment based on one item.

It is important for the public to begin to understand these distinctions, and it is critical for behavioral healthcare executives to be fluent in discussing these distinctions. If you want to promote your program as better than others, there must be a rating system that people understand and trust. Unfortunately, most programs and professionals have rarely used valid scientific measures for differentiating their services. We urgently need to promote the validity of our measures and the relative superiority of our clinical results.

Behavioral healthcare is threatened from two directions. We have no biologically based lab tests for measuring what we treat, and yet we have an actual gold standard, patient self-report measures, that a naïve person might view as inadequate and unscientific. Behavioral healthcare industry leaders need to seize whatever megaphone they can find to loudly proclaim the value of self-report measures and the level of clinical improvement their services have achieved.

Know your numbers

Health and wellness coaches have encouraged people struggling with chronic medical conditions to “know your numbers.” We have measures for cholesterol, blood pressure and blood sugar that increasing numbers of people are grasping, but we have no such number for behavioral healthcare. We cannot wait for a validated lab measure to be developed, and so we must choose from our available measures today.

The crucial concept in clinical care is improvement. For example, is the patient’s blood pressure higher or lower? We can assess improvement in the behavioral healthcare field with existing patient self-report measures as well. Each measuring tool has a total score and a way of benchmarking change. We can determine how significantly people are changing on that measure, and the typical statistic used for capturing clinical change is called effect size.

Effect size can have a wide range of values, but the relevant clinical values begin at 0 and hit a maximum at 1. These numbers, essentially fractions or proportions of one, constitute the range for how people change during treatment. Measurement with this statistic is dynamic, always comparing first to most recent contact. People can achieve a small, medium, or large amount of change. Those descriptors correspond to threshold effect sizes of 0.2, 0.5, and 0.8, according to the classic textbook written by Jacob Cohen.

An interesting side note here is that we can agree on a statistical measurement without needing to agree on the measurement tool. There are many valid measuring tools and we would have great difficulty getting everyone to agree on which one to use. The effect size statistic can incorporate the statistical properties of each tool being used, and then produce a number that is valid for comparisons across tools. I am proposing that we add behavioral healthcare’s effect size to the “know your numbers” campaign.

Why, when and how

I realize many clinicians and executives will react to my proposal with indifference due to their need to confront other more pressing issues. It may well be that adopting my proposal is not urgent, but it is important. Acknowledgement as a legitimate scientific enterprise can fluctuate in imperceptible ways over time. The behavioral healthcare industry provides a wide range of services for mental health and substance use disorders. These services are viewed with skepticism in many quarters, and my proposal is intended to improve the work we do, generate increasing respect, and ultimately improve funding.

My proposal to join the “know your numbers” movement will take time to get right, and so we should start working on this idea now. The final product may look quite different from my brief sketch here. We need to start by assembling a working group consisting of a broad range of stakeholders within the behavioral healthcare industry along with experts in research and statistics. I would argue that the final product we want to produce should be easily understood by the public and scientifically bulletproof.

We must push back on the current tide that is undermining rating scales. Once people lose all trust in satisfaction ratings due to blatant pressure and manipulation, the contamination of all self-report measures is sure to follow. This means that one of our best ways for understanding how people respond to care will be denigrated. The responsibility to confront this potential adverse event falls to the current generation of leaders within behavioral healthcare.

Ed Jones, PhD, is senior vice president for the Institute for Health and Productivity Management.