Hitting the Books: Voice-controlled AI copilots could lead to safer flights

Siri and Alexa were only the beginning. As voice recognition and speech synthesis technologies continue to mature, the days of typing on keyboards to interact with the digital world around us could be coming to an end — and sooner than many of us anticipated. Where today's virtual assistants exist on our mobile devices and desktops to provide scripted answers to specific questions, the LLM-powered generative AI copilots of tomorrow will be there, and everywhere else too. This is the "voice-first" future Tobias Dengel envisions in his new book, The Sound of the Future: The Coming Age of Voice Technology.

Using a wide-ranging set of examples, and applications in everything from marketing, sales and customer service to manufacturing and logistics, Dengel walks the reader through how voice technologies can revolutionize the ways in which we interact with the digital world. In the excerpt below, Dengel discusses voice technology might expand its role in the aviation industry, even after the disastrous outcome of its early use in the Boeing 737 MAX.

Excerpted from THE SOUND OF THE FUTURE: The Coming Age of Voice Technology by Tobias Dengel with Karl Weber. Copyright © 2023. Available from PublicAffairs, an imprint of Hachette Book Group, Inc.

REDUCING THE BIGGEST RISKS: MAKING FLYING SAFER

Some workplaces involve greater risks than others. Today’s technology-driven society sometimes multiplies the risks we face by giving ordinary people control over once-incredible amounts of power, in forms that range from tractor trailers to jet airplanes. People carrying out professional occupations that involve significant risks on a daily basis will also benefit from the safety edge that voice provides — as will the society that depends on these well-trained, highly skilled yet imperfect human beings.

When the Boeing 737 MAX airliner was rolled out in 2015, it featured a number of innovations, including distinctive split-tip winglets and airframe modifications that affected the jumbo jet’s aerodynamic characteristics. A critical launch goal for Boeing was to enable commercial pilots to fly the new plane without needing new certifications, since retraining pilots is very expensive for airlines. To achieve that goal, the airliner’s software included an array of ambitious new features, including many intended to increase safety by taking over control from the crew in certain situations. These included something called the Maneuvering Characteristics Augmentation System (MCAS), which was supposed to compensate for an excessive nose-up angle by adjusting the horizontal stabilizer to keep the aircraft from stalling— a complicated technical “hack” implemented by Boeing to avoid the larger cost involved in rewriting the program from the ground up.

The 737 MAX was a top seller right out of the gate. But what Boeing and its airline customers hadn’t realized was that the software was being asked to do things the pilots didn’t fully understand. As a result, pilots found themselves unable to interface in a timely fashion with the complex system in front of them. The ultimate result was two tragic crashes with 346 fatalities, forcing the grounding of the 737 MAX fleet and a fraud settlement that cost Boeing some $2.5 billion. Additional losses from cancelled aircraft orders, lowered stock value, and other damages have been estimated at up to $60 billion.

These needless losses — financial and human — were caused, in large part, by small yet fatal failures of cockpit communication between people and machines. The pilots could tell that something serious was wrong, but the existing controls made it difficult for them to figure out what that was and to work with the system to correct the problem. As a result, in the words of investigative reporter Peter Robison, “the pilots were trying to retake control of the plane, so that the plane was pitching up and down over several minutes.” Based on his re-creation of what happened, Robison concludes, “it would have been terrifying for the people on the planes.”

When voice becomes a major interface in airliner cockpits, a new tool for preventing such disasters will be available. In traditional aviation, pilots receive commands like “Cleared Direct Casanova VOR” or “Intercept the ILS 3” via radio from dispatchers at air traffic control. After the pilots get this information, they must use their eyes and hands to locate and press a series of buttons to transmit the same commands to the aircraft. In a voice-driven world, that time-wasting, error-prone step will be eliminated. In the first stage of voice adoption, pilots will simply be able to say a few words without moving their eyes from the controls around them, and the plane will respond. According to Geoff Shapiro, a human factors engineer at the former Rockwell Collins Advanced Technology Center, this shift trims the time spent when entering simple navigational commands from half a minute to eight seconds — a huge improvement in circumstances when a few moments can be critical. In the second stage, once veteran pilots have recognized and accepted the power of voice, the plane will automatically follow the spoken instructions from air traffic control, merely asking the pilot to confirm them.

A voice-interface solution integrating the latest capabilities of voice-driven artificial intelligence can improve airline safety in several ways. It gives the system self-awareness and the ability to proactively communicate its state and status to pilots, thereby alerting them to problems even at moments when they might otherwise be distracted or inattentive. Using increasingly powerful voice-technology tools like automatic speech recognition and natural language understanding, it also allows the airplane’s control systems to process and act on conversational speech, making the implementation of pilot commands faster and more accurate than ever. It facilitates real-time communications linking the cockpit, air traffic control, the airline carrier, and maintenance engineers to remove inconsistencies in communication due to human indecision or misjudgment. In the near future, it may even be able to use emerging voice-tech tools such as voice biometrics and real-time sentiment analysis to determine stress levels being experienced by pilots —information that could be used to transmit emergency alerts to air traffic controllers and others on the ground.

Voice technology won’t eliminate all the traditional activities pilots are trained to perform. But in critical moments when the speed of response to messages from a control tower may spell the difference between survival and disaster, the use of a voice interface will prevent crashes and save lives. This is not a fantasy about the remote future. Today’s planes have all the electronics needed to make it possible.

One field of aviation in which safety risks are especially intense is military flying. It’s also an arena in which voice-enabled aviation is being avidly pursued. Alabama-based Dynetics has received $12.3 million from DARPA, the Pentagon’s storied defense-technology division, to develop the use of AI in “high-intensity air conflicts.” The third phrase of the current three-phase research/implementation program involves a “realistic, manned-flight environment involving complex human-machine collaboration” — including voice communication.

The US Air Force is not alone in pursuing this technological advantage. The next generation of the MiG-35, the highly advanced Russian fighter jet, will apparently feature a voice assistant to offer advice in high-pressure situations. Test pilot Dmitry Selivanov says, “We call her Rita, the voice communicant. Her voice remains pleasant and calm even if fire hits the engine. She does not talk all the time, she just makes recommendations if the plane approaches some restrictions. Hints are also provided during combat usage.”

Voice-controlled flying is also in development for civilian aircraft. Companies like Honeywell and Rockwell are designing voice interfaces for aviation, with an initial focus on reducing pilot workload around tedious tasks involving basic, repetitive commands like “Give me the weather at LAX and any critical weather en route.” More extensive and sophisticated use cases for voice tech in aviation are steadily emerging. Vipul Gupta is general manager of Honeywell Aerospace Avionics. He and his team are deeply focused on perfecting the technology of the voice cockpit, especially its response speed, which is a crucial safety feature. Their engineers have reduced the voice system’s average response time to 250 milliseconds, which means, in effect, that the system can react more quickly than a human pilot can.

Over time, voice-controlled aircraft systems will become commonplace in most forms of aviation. But in the short term, the most important use cases will be in general aviation, where single-pilot operators are notoriously overloaded, especially when operating in bad weather or congested areas. Having a “voice copilot” will ease those burdens and make the flying experience safer for pilot and passengers alike.

Voice-controlled aircraft are also likely to dominate the emerging field of urban air mobility, which involves the use of small aircraft for purposes ranging from cargo deliveries to sightseeing tours within city and suburban airspaces. New types of aircraft, such as electric vertical takeoff and landing aircraft (eVTOLs) are likely to dominate this domain, with the marketplace for eVTOLs expected to explode from nothing in 2022 to $1.75 billion in 2028. As this new domain of flight expands, experienced pilots will be in short supply, so the industry is now designing simplified cockpit systems, controlled by voice, that trained “operators” will be able to manage.

Vipul Gupta is bullish about the future of the voice-powered cockpit. “Eventually,” he says, “we’ll have a voice assistant where you will just sit in [the aircraft] and the passenger will say, ‘Hey, fly me there, take me there. And then the system does it.’”

As a licensed pilot with significant personal experience in the cock- pit, I suspect he will be right —eventually. As with most innovations, I believe it will take longer than the early adopters and enthusiasts believe. This is especially likely in a critical field like aviation, in which human trust issues and regulatory hurdles can take years to overcome. But the fact is that the challenges of voice-powered flight are actually simpler in many ways than those faced by other technologies, such as autonomous automobiles. For example, a plane cruising at 20,000 feet doesn’t have to deal with red lights, kids dashing into the street, or other drivers tailgating.

For this reason, I concur with the experts who say that we will have safe, effective voice-controlled planes sooner than autonomous cars. And once the technology is fully developed, the safety advantages of a system that can respond to spoken commands almost instantly in an emergency will be too powerful for the aviation industry to forgo.