New Adaptive Suicide Risk Assessment Tool Among Veterans

The US Department of Veterans Affairs’ (VA) 2021 National Veteran Suicide Prevention Annual Report states that in 2019 there was an average of 17.2 veteran suicides per day, with 6.8 among those with Veterans Health Administration (VHA) encounters in 2018 or 2019 and 10.4 without. Robert Gibbons, PhD, is working to advance mental health research and utilize the computerized adaptive test suicide scale for the United States veteran population.

Please tell us a little bit about yourself.

My name is Robert Gibbons, and I am the Blum-Riese Professor of Biostatistics at the University of Chicago. The previous recipient of the Blum-Riese professorship was Janet Rowley who discovered the molecular basis from leukemia for which she received the Presidential Medal of Freedom from President Barack Obama, so I have very big shoes to fill.

I’ve had a lifelong interest in the development of statistical methods, specifically for the advancement of mental health research. From my days as a graduate student at the University of Chicago, I developed advanced statistical methods for applications in psychology and in psychiatry.

We are going to be discussing our recent paper, published in PLOS One on validation of our computerized adaptive suicide scale, the CAT-SS, in our nation’s veteran population. This is joint work with Dr Lisa Brenner, PhD, and her team in the Veterans Administration (VA).

Thank you, Professor Gibbons. Could you tell us what the computerized adaptive test suicide scale is and how it differs from other tools that are currently used for veterans?

In order to understand the computerized adaptive suicide scale or CAT-SS, it is first important for you to understand how we have advanced the field of mental health measurement through the development of computerized adaptive tests or CATs based on multidimensional item response theory. Traditional mental health measures use a small, fixed set of symptoms that are identical for all people regardless of the severity of their underlying mental health disorder, for example, depression, anxiety, suicidality, or substance misuse.

Now, for any given person, there is little information in those items targeted to their specific level of severity. The PHQ-9, as an example, has 9 fixed items. Everyone receives those same 9 items and maybe 1 or 2 of them kick in at the level of severity of a given individual. What these tests do is sacrifice the precision of measurement for the speed of measurement.

By contrast, adaptive tests quickly learn the severity level of a person based on their responses to questions during the assessment and then target the items from a much larger bank of items, maybe 500 to 1000 items, to the specific severity level of that person. We do not initially know that specific severity level, but we learn it through the adaptive process of the interview and we learn it quickly.

As such, CATs maximize the precision of measurement while completely eliminating clinician burden and minimizing patient burden because the same items are not asked repeatedly. There is no response bias for the person recalling their previous answers and getting stuck in a rut. These tests can be repeatedly administered at any interval.

We can test depression severity and suicide risk hourly during ketamine infusions or daily during novel treatments, in trials over long periods of time such as a year or more. Thesetests are based on self-reports and take advantage of cloud computing, so they can be administered anywhere in the planet. In fact, we test people all over the world in or out of the clinic on any internet-capable device. Since the CAT-SS is the only adaptive suicide test based on multidimensional item response theory, we can develop a crosswalk between symptoms of depression, anxiety, and posttraumatic stress disorder (PTSD) with traditional suicidality questions.

A person who may have suicidal precursors such as hopelessness, helplessness, anhedonia, and social isolation but has not yet experienced suicidal ideation or behavior will still receive an appreciable score on the CAT-SS. However, a traditional scale like the Columbia-Suicide Severity Rating Scale, or C-SSRS, will score the subject as having zero suicide risk because they have not yet experienced suicidal thinking. This is a major advance over traditional approaches and allows us to measure change much more precisely in suicidality over time and in response to treatment interventions.

Can you elaborate on how this tool demonstrated added value compared with some of these other current suicide risk prediction practices?

There now have been several scientific studies that demonstrate the added value of the CAT-SS over traditional measures of suicidality. The current VA study, for example, demonstrated that for every 10-point change in the CAT-SS score, which is measured on a 100-point scale, there was a 50% increase in active suicidal ideation with plan or intent and a 77% increase in suicide attempts. This represents an almost eightfold increase in suicide attempt risk across the range of the scale.

Adjusting for suicide attempt in the past year, which is traditionally the gold standard of predicting future suicidal behavior, it had very little effect on the predictive accuracy, indicating that the CAT-SS is uniquely predictive of future suicidal ideation and attempt. Relative to the golden standard of suicide attempt in the prior year, the CAT-SS dramatically increased the predictive accuracy for all the suicidal outcomes that we considered in our study, including future suicide attempt.

Other investigators like Brian Mustanski, PhD and Johnny Berona, PhD at Northwestern University have now published 2 separate studies finding the same effects of the CAT-SS, significantly increasing predictive accuracy of suicidal outcomes over traditional approaches. This adds even further confirmation and independent validation of the effects that we found in the present study.

Study findings suggest that the CAT-SS would rapidly facilitate precise and personal screening and assessment of suicide risk severity if implemented into the electronic medical record. Could you expand on this?

Yes, this is absolutely true. The CAT-SS and all of the other CAT mental health measures are now available in the VA Mental Health Assistant for routine screening and measurement. This means all of our veterans have access to these new tools directly through the VA. Now, the precision of the CAT-SS for personalized screening and assessment of suicide risk has also been clearly demonstrated.

As I mentioned before, in the ketamine randomized clinical trials at Columbia University conducted by Michael Grunebaum, MD and John Mann, MD, the CAT-SS was compared to a much longer clinician-administered Beck SSI. The CAT-SS identified a much larger treatment effect of ketamine relative to the comparator midazolam relative to the traditional clinician-rated assessment.

What’s important about this comparison is the clinician assessment required 20-30 minutes to administer and the CAT-SS only requires 110 seconds. The traditional clinician-administered scale requires a clinician to administer and we do not have enough clinicians to do that. The CAT-SS does not require a clinician, so the ability to bring these kinds of high-quality measurements into the field is so important.

Similarly, in a large department of defense study, the Wingman-Connect study, Peter Wyman, PhD, and colleagues from Rutgers University developed a group level suicide prevention intervention and conducted a cluster randomized study, which showed that it significantly decreased suicidality as measured by longitudinal CAT-SS assessments.

The CAT-Mental Health (CAT-MH), including the CAT-SS are now integrated into several electronic health record systems including Epic in several large healthcare systems here at the University of Chicago as an example. A study using the CAT-MH Depression Measures showed significant increases in annual depression screening in primary care when implemented into the Epic EHR and administered via the MyChart patient portal relative to just doing traditional clinic-based screening only.

And how will this tool be implemented in the clinical settings?

The VA is now using these tools for the prediction of suicide attempts and completion and the screening and measurement of PTSD, another project in CAT that we developed with our collaborators, Dr Brenner and her team at the VA. As I mentioned before, the VA clinics now have access to the CAT-MH directly through the VA Mental Health Assistant.

The VA already has passive suicide risk prediction algorithms such as REACH VET that are capable of risk stratification based on feature extraction from the electronic medical record. A natural application for the CAT-SS would be to follow up this risk stratification in all veterans who screened positive for suicide risk, immediately after they are identified in real time, and then longitudinally over time, both in and out of the clinic to determine whether or not suicide risk is increasing. This will save lives.

Is there anything else you would like to add to the conversation today that we didn’t touch upon?

Always. There are numerous exciting applications of this technology that, we and others, have already been exploring. Screening and measurement in primary care, in emergency departments are critical applications, but let me give you a few specific and exciting examples. Working with our collaborators at UCLA, we have offered screening and measurement-based care to over 85,000 undergraduate students over the past 4 years based on our adaptive tests for depression, anxiety, and suicidality.

Then, based on the scores, we have triaged them to either an internet-based cognitive behavior therapy and peer counseling or to clinic or emergency department services for those that are more severe or who might have immediate suicide risk. This is all virtual. We are screening college students for suicide risk remotely at 3 o’clock in the morning with a safety net that is developed so if they screen positive in real time, they are contacted proactively by a suicide hotline and then referred for treatment as necessary.

The project has been so successful that we are now rolling this out to the 2.1 million students in the State of California community colleges, supported by an ALACRITY Center grant from the National Institute of Mental Health to University of California, Los Angeles under the direction of Michelle Craske, PhD.

As another very cool application, the Substance Abuse and Mental Health Services Administration and Research Triangle Institute are conducting a $30 million national prevalence study of mental health and substance use disorders, and they are using the CAT-MH as a first stage screener to determine who will receive a full structured clinical interview lasting 1-2 hours with a trained clinician. Rather than training all the clinicians in the world to go out and assess using the structured clinical interview for DSM disorders (SCID), everybody included in the sample were able to do this in 2-stage screening; identify those who we think would meet criteria using our tools and then validate or verify that using trained clinicians.

Finally, I want to acknowledge the over 15 years of continuous funding from the National Institute of Mental Health that made all of this possible and my clinical coinvestigators, doctors David Kupfer, MD, Ellen Frank, PhD, David Brent, MD, MSHyg, Paul Pilkonis, PhD, Ben Lahey, PhD, and our statistical collaborator, David Weiss, PhD. This work represents the future of mental health measurement, and it is available for large scale use today. Thank you for your attention and for your interest in our work.