Should Response Time Be a Performance Indicator?

For decades many EMS systems have gauged performance by measuring how fast they respond to emergencies. With citizens expecting a quick response and limited data on other aspects of performance, response time became one of the few ways to assess the quality of an EMS system.

In recent years, however, many industry leaders have begun to question the benefit of measuring and evaluating systems based only on response times. Shaving seconds or even minutes off response times seems to benefit only a small subset of clinical conditions. At the same time, meeting response-time standards can cost communities hundreds of thousands of dollars. Yet community officials and the public often still expect rapid responses to every call—and the media frequently points out when those expectations are not met.

With payers looking for ways to tie reimbursement to quality, measuring the effectiveness and efficacy of EMS will soon be more critical than ever. At this year’s Pinnacle EMS Leadership Forum, leaders of the EMS industry will gather to discuss the most critical issues they face today, including the importance of performance improvement and the EMS Compass initiative to develop a system of performance measures. The one performance measure that gets the most attention—good and bad—is response time. We asked several Pinnacle faculty members to address these questions:

Are response times a good measure of an EMS system’s performance? Should EMS agencies continue to design systems around response-time goals?

Here (edited for length and clarity) are their answers.

Brian LaCroix, President, Allina Health EMS

No. But ignore them at your peril.

If EMS is the practice of medicine, measuring the value of a system based on how fast you drive seems archaic and overly simplistic. However, it wasn’t all that long ago when all most ambulance agencies had to offer was a quick response.

Prior to the 1960s, most well-intended “ambulance drivers” had two skills: comfort around chaos and driving fast. This rapid-response model was rooted in the experience of our of police and fire colleagues.

But the majority of EMS calls are not related to time-sensitive problems. There are certain calls when response time is important, but that list is a small one, and it’s shrinking over time (think how the proliferation of AEDs has shifted the importance of getting an ALS rig on scene). I concede there is also a huge issue of perception. In an emergency there is often a high degree of anxiety, and the sooner someone shows up to help, the sooner that anxiety might be relieved.

But speed is dangerous. It is well documented that the higher the collision speed, the more serious the consequences in terms of injury and material damage. So it seems reasonable to take a hard look at the value of fast driving before we put our staff, patients and the public at risk. Most studies demonstrate that the time saved driving with lights and siren is modest at best. So if we decide to drive fast to save time, we ought to be sure we need to perform some sort of lifesaving intervention that is time-sensitive. Otherwise it does not make sense.

I opened these comments about response times by saying “Ignore them at your peril.” This is an acknowledgement that even though there are plenty of reasons to slow down, responses times are still the common currency by which the general public judges quality in EMS. For now we can’t ignore that. But it’s incumbent on EMS leaders to continue to educate the public that EMS is indeed the practice of medicine, and driving fast is not a key indicator of quality.

Steve Knight, PhD, Senior Associate, Fitch & Associates

Response time has historically been used as a surrogate measure for system effectiveness. The assumption was that faster is better: The quicker the system responded, the higher the quality of clinical outcomes. Recent evidence-based research has allowed us to better understand the relationship between clinical outcomes and response times. This current research has suggested that our response times have little impact on clinical outcomes outside a small subset of call types. From this perspective, response time is not the best measure of system performance, and greater emphasis on developing evidence-based measures of clinical quality is required.

However, we shouldn’t abandon response-time goals entirely. Establishing a system’s response-time performance is still a good method of articulating service levels to the community and stakeholders and for holding the system accountable for performance. I suggest this should continue until such time that we have a robust set of evidence-based clinical measures that are readily available and easily accessible. Through this lens, the fact that the desired service level is established and reported, and the system is held accountable, may have greater value than the response-time measure itself.

Bruce Moeller, PhD, Assistant County Administrator, Pinellas County, Fla.

You must look at this from two perspectives: Does a better response time provide greater benefit to the patient? And can we “measure” it with a degree of accuracy and consistency? Surprisingly, the answer is: not really.

The literature shows that response time has a positive impact on patient outcome in only a small number of cases. And therein lies the problem: We expend significant energy and resources to address a relatively small number of incidents. It is understood that if your family member is in distress, you want an almost-instant response from EMS. But is this the best use of limited public resources? What should the response-time criteria be? Historically EMS response-time criteria were developed for cardiac arrests—about 1% of cases. And in cases of trauma, many agencies struggle to get patients off the scene quickly and to definitive care, thereby mitigating the impact of a rapid response time.

Time is easy to measure—response times are not. We still have no universally accepted definition of response time that utilizes the same “start-the-clock” and “stop-the-clock” criteria. Research found EMS agencies in Florida used nine different definitions of response time. More important, these agencies had a bias to use a definition that made them look better. When a single performance metric is used so frequently and so publicly, there is motivation for some EMS managers to worry about public perception rather than focusing on patient outcomes.

In Pinellas County we overcame ambiguity by using a single response-time definition, from agency dispatch until arrival on scene, for all 19 providers in our system. And since we have all the data in a regional computer-aided dispatch (CAD) system, ensuring a first responder response time of less than 7:30 for 90% of incidents is relatively easy.

Response time can be a valuable performance metric for EMS agency design. However, until we use it thoughtfully and honestly, it will continue to have limited utility.

Norman Seals, Assistant Chief, Dallas Fire-Rescue

As an industry we have historically touted the importance of response times, and our various oversight bodies have been trained to expect that measure in our regular reports. We have also done a very good job of training our customers to expect us to arrive at their “emergency” within an average of six minutes. However, I believe we need to begin reeducating those groups to understand that, with the exception of cardiac arrest and a few other critical conditions, response times do not have significant impact on patient outcomes. 

I also believe that we, as EMS leaders and administrators, need to shift our strategic planning processes away from response-time metrics to the value-based metrics that appear to be looming. Within my agency, we report annually on average response times for our EMS units. We have begun the process of educating city management and city council on the changing face of EMS and the projected impacts of healthcare reform on our EMS operations. Additionally, we are working with a consultant to develop plans for moving forward with realigning operations to meet the coming changes.

Scott Matin, MBA, Vice President, Clinical, Education and Business Services, MONOC

While the medical community has believed for decades that it practiced medicine based on strict research and exhaustive investigation, the truth is that much of what healthcare providers have done for years is the result of gut feelings, theory and “because we’ve always done it this way.”

Fortunately EMS has come a long way. Along with the general medical community, EMS has adopted the strict premise of evidence-based medicine. This doesn’t just mean giving the right medications or performing the right procedures, but in the case of EMS it also refers to the best utilization of resources.

Over the last decade we have learned that this obsession we’ve had for years with response time as a measure of performance is grossly inaccurate. Research has shown that with the exception of high-acuity calls such as cardiac arrest, myocardial infarction, stroke, airway obstruction and severe trauma, the majority of calls to which EMS is dispatched aren’t as time-sensitive and don’t show better outcomes with a more rapid response. Additionally, emphasis on response time may have unintended consequences such as more motor vehicle crashes and skill degradation when increasing the number of paramedics answering a finite number of calls.

EMS agencies need to stop designing systems around response-time goals. While EMS systems need to meet public expectations, these expectations should be weighed against available resources and the ultimate good of the community. For EMS agencies to effectively convey the message that faster doesn’t always mean better, they must open an ongoing dialogue with their communities. Agencies need to communicate what quality is in EMS and how it can be obtained. Programs that can lead to better outcomes include a robust first responder system utilizing current police and fire resources; public education on how to identify strokes and heart attacks and when to call 9-1-1; and community-wide education and buy-in to develop a public-access CPR and AED program.

Todd Stout, Founder and President, FirstWatch

Response times are one of many potentially important measures of an EMS system’s performance. While the current discussions seem to focus on the fact that they are clinically significant in only a very small percentage of our calls, I believe that for the foreseeable future, response times will remain emotionally significant for the patient, their family, bystanders, etc. So if we believe the patient experience is important, or if reimbursement is based partially on patient satisfaction at some point, response times will remain relevant.

Another factor that can’t be dismissed is that there is a decent percentage of calls where a delayed response time may not affect the patient or family, but would leave another responding public safety agency on scene waiting for an ambulance to arrive. 

Third, for at least some calls, fast response times really do matter clinically, so for those calls, monitoring response-time compliance is still important and should remain so. I believe it’s just as wrong to say “response times don’t matter” as it is to say “all response times matter”; perhaps we just need to examine the evidence to determine which calls they matter for and what those times should be—and even stratify response-time requirements based on call types.

Finally, response times were an accepted measure of quality in EMS for many years. And the variables and inputs that need to be adjusted to improve response times are actually fairly straightforward. So while the importance of response times may be changing, if a system has struggled to have good response times, it’s likely to struggle just as much with meeting other, more difficult-to-address quality measures.

For the reasons above, I believe response-time goals (perhaps with better stratification) should still be part of good system design. EMS systems are complex in any situation, so I should be clear that no single aspect of a system should be the entire focus of system design. 

Greg Mears, MD, Medical Director, ZOLL

Systems of care evaluate the outcomes of patients with time-dependent illnesses and injuries from first healthcare provider contact through definitive care. As a result, EMS is challenged to provide a timely response to a subset of patients that is small compared to the overall number of EMS responses. EMS should continue to build systems to quickly identify and ensure timely lifesaving care for cardiac arrest, trauma, STEMI, stroke, and acute airway compromise. Other EMS responses can be managed using time intervals that are acceptable to the EMS system’s community and customer expectations.

Chris Cebollero, MS, Senior Partner, Cebollero & Associates

This is such a long-standing debate. In EMS we seem to live and die by the clock. There has been no real proof that a response-time standard truly makes a difference. Ever since the late ’70s, when a report written by Dr. Mickey Eisenberg stated that quick initiation of CPR and prehospital medical treatment were serious elements in the survival of sudden cardiac arrest patients, it has been falsely quoted as setting a response standard of 8:59. 

In today’s EMS field, first responder agencies arrive on scene within five minutes for most calls. Patients with life-threatening conditions now have certified responders able to place rescue airways and use AEDs long before a transport unit arrives.

As we move into the future, more concentration needs to be placed on patient outcomes, patient satisfaction and transporting patients where they need to go for the best care. With that said, the clock is always going to be a factor in the EMS field. What finally needs to be done is factually coming up with what that response-time standard needs to be.

Skip Kirkwood, MS, JD, Director, Durham County (NC) EMS

Response times are one measure of an EMS system’s performance. Prompt response is important to citizens, and timely response must continue to be one element of an EMS system’s performance measures.

It is fashionable to say, “The evidence shows response performance doesn’t matter.” That is not a complete or correct statement of the science. There is little to show that the ambulance response interval, by itself, improves clinical outcomes. However, that’s the beginning of the discussion, not the end. Most of these studies focus on whether the patients on one side of the line do better than those on the other. Since response performance is part of the total time from event to definitive care, prompt response is important to a patient suffering from stroke, STEMI, trauma or other time-sensitive infirmity. Some time-sensitive interventions (like defibrillation in cardiac arrest) do not require an ambulance, so we need to look at EMS systems, not just ambulance response performance. There is much work yet to be done on this topic.

Measuring an EMS system’s performance requires far more than a single performance measure. If you ask customers, prompt response matters. How prompt? Who knows? It’s probably never been studied from that perspective. But other things matter also. The quality of clinical medicine matters, although patients and their families are probably ill-equipped to judge that. Good, meaningful internal quality improvement programs are essential to defining a “good” EMS system. Competent employees matter, and competence should improve with experience, so employee turnover, satisfaction and engagement are important measures. And good stewardship of the public’s money also matters, so a variety of financial performance measures are also important.

We need to first agree on a set of benchmark performance measures we can use fairly and equitably across all the business models in EMS. Then we need to use those to educate our communities as to precisely what makes a “good” EMS system. Only when we can do that will we be able to push the “speed demon” out of first place as the measure of an EMS system’s performance.

Jay Fitch, PhD, Founding Partner and President, Fitch & Associates

Response times are only one component of how a system’s performance should be measured. There are a variety of other measures to consider that include clinical outcomes and fiscal effectiveness.

Years ago EMS systems were designed with a one-response-time-fits-all approach to 9-1-1 calls. More contemporary research has demonstrated that extraordinarily short response times are only clinically required in a small number of call types.

Most would agree that solid self-help instructions at dispatch and a quick response by uniformed personnel (fire, law enforcement or volunteer first responders) provide a stabilizing influence for the incident. These may be more important to clinical outcomes on some types of calls than an 8:59 response by the ambulance. Response times have to be engineered, measured and managed to balance the risk to the patient, the caregiver and the community. Not every 9-1-1 request merits a “hot” response from the system.

That said, customer satisfaction is part of one of the three elements of the Triple Aim. Patients and families want their EMS system, including the transport component, to flow smoothly and quickly. In today’s instant society, a long wait at any point in the service cycle will be perceived negatively. Some communities have been willing to pay the additional costs of having short response times but now feel compelled to reassess the implications of those decisions in light of other financial priorities. Stakeholders must be educated that sending the correct type of help, which is able to manage the patient’s needs within a reasonable time frame, may be more important than considering response times as the primary measure of performance.

2015 Pinnacle EMS Leadership Forum

The Pinnacle EMS Leadership Forum is the premier event for EMS leaders from all service models, for every size of service. Sponsored by Fitch & Associates, it is now in its 10th year. Pinnacle 2015 will be held at the Omni Amelia Island Plantation Resort, near Jacksonville, FL, August 4–6, 2015. For more see pinnacle-ems.com.