In Katrina`s Wake

Climbing up 42 flights of stairs is no picnic at the best of times. But imagine doing so in the middle of a hot, humid Louisiana evening with nothing but a flashlight beam to guide you.

     Welcome to the world of Captain Stephen Gordon, the New Orleans Police Department's commander of communications. Not long after Hurricane Katrina roared through the Gulf area, the city's 24-channel M/A-COM trunked 800 MHz radio system unexpectedly failed, leaving police, fire and EMS officers with nothing more than portable-to-portable talk-around capability. Since the main generator was located on top of the 42-story Energy Centre in downtown New Orleans-and the equipment room that powered it was on the 34th floor-there was nothing for Captain Gordon and technicians from Goff Communications to do besides trudge up the fire escape to get to it.

     "We had to climb up 42 floors to find out what had gone wrong," Gordon says. "Then we had to come back down 42 floors to get what we needed to fix it, and then go back up 42 floors to get back there."

     Considering that New Orleans' $26-million public-safety radio network was designed with double redundancy by engineering consultant Dominic Tusa, the main transmitter site's failure was a surprise. It was also a tremendous blow to public-safety coordination in the flood-ravaged city. With local telephone and cell phone service also down, New Orleans' police, fire and EMS officers found themselves on their own.

     Eventually, the NOPD discovered that it wasn't the M/A-COM network that failed, but rather the main transmitter site's backup generator. It was a casualty of the storm, having been machine-gunned by window glass blown out of the nearby Hyatt Hotel. The story of this and other Katrina calamities, the ways in which city and M/A-COM employees struggled to repair the damage, and the lessons that can be learned from the hurricane debacle are what this special report is all about.

New Orleans' Public-Safety Radio System
     Dominic Tusa is president of Tusa Consulting Services in New Orleans. He designs public-safety radio systems for a living. With a mandate to serve more than 3,000 police, fire and EMS users with both voice and data, Tusa had his work cut out for him. That's why he erred on the side of caution in his design. For instance, to achieve 97% in-car radio coverage, Tusa built four transmission sites around the city, with the main simulcast transmitter located safely atop the Energy Centre.

     "This site was selected for its excellent wide-area coverage potential, the ability to place the radio equipment within the structure's hardened interior core and because the standby electric power generator could be installed on the rooftop, protecting it from flooding and low-level airborne debris," he says.

     "A second simulcast transmit/receive site is located in the New Orleans East neighborhood and shares a tower facility owned by Cox Communications," Tusa adds. "The equipment contained here is identical to that of the Energy Centre site, except that the standby generator is supported by both natural gas and liquefied petroleum gas (LPG) sources. Permission to install the secondary LPG fuel source for the Energy Centre site was denied by the original building owners."

     The other two smaller sites are located at Louis Armstrong International Airport and the Irish Bayou. All were interconnected by microwave antennas housed within protective radomes, and with extra-strong bracing to resist wind shifting.

     The entire network is connected by microwave to a hardened control point housed at police headquarters. "That's where simulcast timing, receiver voter-selectors, a 24-channel receiver site, console/network controllers, telephone interconnect, mobile data interfaces and other related equipment are housed," Tusa says.

     Besides ensuring that its two main sites had redundant power supplies, Tusa's radio system was designed to send alarms to the separate dispatch centers maintained at NOPD headquarters and New Orleans Fire Department/EMS headquarters. As soon as the main power failed, both centers would be warned automatically. Should something go wrong with the natural gas generator, a second alarm would be sent to both sites, warning that the transmitter was drawing directly from its batteries.

     As an added safeguard, the two dispatch centers were connected, so that if one was knocked out of service, the other could be used jointly by police, fire and EMS.

     Installed in 1995 and upgraded in 1999, 2003 and 2004, Tusa's radio system was designed to be robust and doubly redundant. So what happened?

Katrina's Contribution
     Let's get one thing straight: New Orleans' public-safety radio system "continued to work during the heavy winds of Katrina. It was the lack of power that caused us to have problems," says Gordon. "Approximately 80 hours after the storm was over, our radio system was back working again, with excellent coverage from the Energy Centre site."

     "The system worked throughout the hurricane, and continued to work after it was over," echoes David Cerqua, M/A-COM's sales director for the Southeast region. The only apparent exception: The microwave link to the New Orleans East site was knocked out of alignment by Katrina, effectively isolating this site from the rest of the network.

     However, Katrina did some unnoticed damage that led to the network's subsequent failure. "What no one knew at the time was that when the windows exploded out of the Hyatt Hotel, some of the glass penetrated and holed the generator's radiator," Cerqua says. "What this meant was, once the commercial electricity failed and the transmitter switched to the natural gas generator, coolant started leaking out of the radiator. It was this leak that led to its shutting down and switching to battery backup. Since this backup was never meant to be more than a stopgap, the radiator had to be fixed so that the generator could be restored and service maintained."

Enter the Flood
     Had the levees held, the Energy Centre transmitter site would have never lost power. But the levees broke, water got into the subterranean power conduits and electric service failed.

     The water also flooded the ground floor of the NOPD's headquarters, forcing dispatch and all other personnel to evacuate. "Actually, the NOPD's dispatch is on the second floor, so it wasn't affected at all," says Tusa. "But due to the ground-floor flood, they had to leave."

     At the fire/EMS dispatch center, the flood damage was nowhere near as severe: only 6" to 8" of water covered the ground-floor carpets. Unfortunately, with all hell breaking loose in the Big Easy at the time, there was no way to know how far the floodwaters would rise. As a result, this dispatch center was abandoned as well. Afterward, it was learned that the second-floor NOFD dispatch room survived unscathed, but that radio equipment in the ground-floor EMS room was damaged.

     Unfortunately, it was only after both centers had been abandoned that the Energy Centre generator failed. The M/A-COM system sent an alert to both dispatches as per procedure, but since there was no one in either facility to receive it, the alarm went unnoticed. Hours later, when the batteries ran down, the second alarm was also sent to the unmanned rooms. This is why the subsequent radio network failure caught everyone by surprise.

Fighting Back
     When the network failed, Gordon and his staff knew they had to act. Fighting their way through the floodwaters just hours before dawn, they began the long trek up the blacked-out Energy Centre. "We had to haul tools up 42 flights without air conditioning, and with nothing but flashlights for illumination," Gordon says. "It was brutal."

     The NOPD wasn't alone: Although M/A-COM technicians had been blocked from entering the city by overzealous police, M/A-COM managed to contact specialists from Goff Communications for help. Goff personnel with expertise in cellular towers were already in town restoring service for Nextel.

     "I pressed M/A-COM to attempt to locate a service provider, possibly helping the cellular industry, that might have been trapped by Katrina," says Tusa. "Goff was there to support Nextel, and fortunately M/A-COM found them and had communications with them via satellite telephone."

     Due to flooding around the Energy Centre, it wasn't until September 1 that "all equipment, boats and personnel were in place," says Tusa. Only after they climbed to the roof did Gordon, his officers and Goff technicians discover the radiator damage. Remarkably, "all other radio infrastructure equipment, including the site's rooftop-mounted antenna systems, had survived without any apparent damage."

     After the generator radiator was repaired, the unit was restarted. "Once standby electrical power was restored and considered to be stable, the infrastructure equipment initiated a final self-restart and placed itself back online at approximately 4:30 on the morning of September 2," Tusa says. "At this point, and to our knowledge, the only public-safety trunked radio communication network operable within the Orleans, Jefferson, St. Bernard and Plaquemines areas then was the New Orleans 800MHz radio network."

     As a precaution, the NOPD decided to have a backup diesel generator flown to the roof by National Guard helicopter. Once it was in place, the next problem was fuel, which was also brought in by air. "The trouble is that, under federal guidelines, National Guard choppers can only airlift fuel in small five-gallon tanks," says Gordon. "So we had to have pallets of them delivered by air, then fill the generator tank by hand." To make matters worse, it turned out that someone had mistakenly put jet fuel (kerosene) in the tanks rather than diesel. No matter: The NOPD learned that by adding a quart of oil to each five-gallon tank, the resulting mix "would burn like diesel," says Cerqua.

Other Casualties
     As for the other sites? The New Orleans East site eventually failed after its natural gas supply was disrupted and its LPG supply ran out. Once FEMA got a diesel generator on site, service was restored. The airport site stayed up throughout, except for a short gap between the loss of commercial power and the startup of the generators. Unfortunately, the Irish Bayou site ended up under 18 inches of water, despite being raised above previous flood levels reached by Hurricanes Betsy and Camille.

     The NOPD's control point survived the storm, but its microwave antenna had to be re-aimed. As well, the diesel generator wouldn't start due to fuel problems-troubles that required electricity to get the fuel pumping properly, so that the generator could make more electricity. Due to problems with natural gas supply and a continuing lack of electricity, the NOPD control point was still being powered by diesel generators as of mid December 2005.

     This notwithstanding, New Orleans' police, fire and EMS were able to talk to each other by radio as of 4:30 a.m., September 2. They've been on the air ever since.

James Careless is a freelance journalist with extensive experience covering public-safety communications issues.