The Power of Theory in an Empirical World

I have a confession to make before getting down to business. I have always loved theory. When I was in high school, I read an article on existentialism. I don’t know why, and I’m not sure that I understood much of the article, but I found its profound questions and expansive scope to be thrilling. This was a preview to what I would enjoy in college and beyond, from Sartre to Freud to Kohut. My doctoral dissertation was a theoretical critique of the work of the psychoanalyst Heinz Kohut.

I return to this topic in response to a quote in the Sept. 5 edition of The New York Times. Jennifer Szalai wrote, “[T]heory can provide a handy framework, transforming the messy welter of experience into something more legible, but it can also impose a narrative that’s awkward, warped or even damaging.” The sophistication of this sentence exceeds any of my thoughts during my years of study. I would expand, or possibly trivialize, this judgment into a tripartite model: theory can illuminate, obscure or mesmerize.

Theory can illuminate

We need a theoretical paradigm for understanding the psychological complexities of people. Freud’s paradigm of unconscious emotional conflicts dominated for many decades. Freud was long on theory and short on empirical validation. As clinicians like Aaron Beck tried to validate aspects of Freud’s theory, confirmation was not forthcoming and other theories were stimulated instead. Beck became one of the creators of the “cognitive revolution” in theory and therapy.

We tend to think of cognitive-behavioral therapy as the quintessential evidence-based practice, but its origins lay in Beck’s theoretical musings upon finding that his research did not confirm Freud’s psychodynamic theory of depression. He constructed an alternative paradigm built around the primacy of thoughts in the development of psychological disorders. Research validated his theory.

In the world of business, there are important theories as well. The young entrepreneurs of Silicon Valley encouraged businesspeople to move quickly, without concern about breaking things, and to find innovative ways of transforming existing industries and creating new ones. In an earlier generation, Jack Welch became a successful executive at GE with legions of followers by focusing painstakingly on reducing errors, streamlining operations, and increasing efficiency.

Some of these success stories started with a few cases or projects and progressed to a general theory, while others began with a new theory that generated many supporting demonstrations of the theory. In either case, the theory is performing an important function. As organizing concept or as heuristic engine, a theory can shine light on important phenomena. People and businesses can be transformed as a result of practices that arise from theory.

Theory can obscure

Theories can stop us from advancing knowledge if they lack an empirical foundation or contradict known facts. For example, dreams have been studied extensively through the years and there is no evidence that they illuminate the psychology of the individual. This lack of evidence is ever more alarming as it stands next to the burgeoning fields of brain and sleep study. Nonetheless, psychoanalysis is still in business and its dream theories provide a modern affront to psychological knowledge.

While a focus on efficiency generally makes good business sense, it may obscure the needs of certain businesses. As noted, efficiency studies may not be a fertile breeding ground for new, innovative products, and a focus on efficiency might also fail to reveal how existing operations need a fundamental overhaul. The costs of lost opportunities will mount, often silently, as a focus on existing operations blocks the exploration of new processes and product ideas.

Theories may also be applied inappropriately based on a zealous adherence. For example, trauma impacts psychological development, and our comprehensive theory of trauma describes both common symptoms and how victims may not understand how their symptoms derive from trauma. However, a clinician who views every instance of these symptoms as attributable to an underlying trauma might make clinical errors. That clinician could do harm by promoting the overzealous application of the theory. The notorious McMartin child sexual abuse trial in the 1980s is an extraordinary example of the damage that can be created by turning a theory into a weapon.

Theory can mesmerize

Ideas can be so grand in scope, so clever, so inspiring in their ability to connect separate findings that they mesmerize us. Existentialism wins awards for grand scope in its attempts to provide the meaning of existence (or the lack thereof). Strategic planning is a type of business theorizing energizing many corporations, and the “blue ocean” model stands out as a clever, fruitful approach. Cognitive behavioral theory is so common as to not require further praise, but its ability to link thoughts, feelings and behaviors together in meaningful ways remains impressive.

Mesmerizing can be good or bad, depending on the empirical evidence behind the theoretical trance. Grand theories by their very nature exceed the confines of evidence. For example, the writings of BF Skinner on radical behaviorism were thought-provoking, but the grand goal of finding a single key to human psychology should be rejected on its face. Freud’s writing was brilliant and his conceptual schemas bold, but his single case studies provided weak evidence upon which to assemble such ingenuity. Theories that provide a system for organizing disparate facts and ideas are compelling, but they may simply be expedient since they often can’t be proven on their own.

Theory forces us to think big—and small

The reader may at this point conclude that theory is a tedious, sometimes harmful waste of time. I offer this defense: There is value in trying to stretch our understanding of the world beyond the facts. We cannot abandon the facts in the process, nor get too attached to what is purely conceptual. But we cannot pretend knowledge progresses by mining one fact after another.

We need to think big and small, both conceptually and empirically, with tension rather than a strict hierarchy between these activities, pushing us forward toward greater knowledge. Sometimes we advance our knowledge with surprising new facts, and sometimes with a new theory that challenges our prevailing understanding.

We cannot make the “evidence-based,” “empirically-supported” mantra of today a reality without robust theory. The facts don’t arrange themselves. We need to make sense of them and establish a way of organizing them.

Let me close with a clinical example. I thought the emergence of EMDR several years ago was an embarrassment to the field. I viewed it as wild theory and bizarre practice without an empirical basis. The data subsequently came in supporting EMDR as a clinical practice.

Some experts now surmise that EMDR is another form of the exposure model, an anxiety-reduction technique comparable to other types of CBT. Others suspect EMDR fits better within the literature on information processing. In any case, while research studies continue, experts will debate the best theoretical framework for encompassing our findings to date.

Strange as it may seem, empirical battles sometimes get resolved on the field of theory. Alternatively, as Beck demonstrated in his early work, esteemed theories can be discarded on the field of research. We can have breakthrough studies and breakthrough theories. Ideally, they inform one another in an iterative process we describe as scientific knowledge.

Ed Jones, PhD, is senior vice president of the Institute for Health and Productivity.