How 'Microsoft Flight Simulator' became a 'living game' with Azure AI

Microsoft Flight Simulator is a triumph, one that fully captures the meditative experience of soaring through the clouds. But to bring the game to life, Microsoft and developer Asobo Studio needed more than an upgraded graphics engine to make its planes look more realistic. They needed a way to let you believably fly anywhere on the planet, with true-to-life topography and 3D models for almost everything you see, something that's especially difficult in dense cities.

A task like that would be practically impossible to accomplish by hand. But it's the sort of large-scale data processing that Microsoft's Azure AI was built for. The company was able to push 2.5 petabytes worth of Bing Maps satellite photo data through Azure machine learning to construct the virtual world of Flight Simulator. You could say it's really the cloud that brings the game to life. Azure also helps to model real-time weather. (That's how some players were able to chase recent hurricanes.)

The franchise's hardcore fans were eager to more realism in a new title,according to Jorg Neumann, Microsoft's head of Flight Simulator. Specifically, they asked for visual flight rules (VFR). "It basically means the pilot can orient themselves by just looking out the window," Neumann said. "And in order to do that, the planet below needs to look extraordinarily close to reality. So that was the mission."

After a bit of investigation, Neumann realized that Bing Maps' data set essentially covered the entire planet. The only problem? It was all in 2D. After using some of that data to build a flyable 3D version of Seattle, Neumann turned to the Azure team to craft a machine learning method for converting the entire planet into a giant 3D model.

"AI has just tremendously grown in the last few years," said Eric Boyd, CVP of Azure AI, in an interview. "It's really driven by the massive amounts of data that are now available, combined with the massive amounts of compute that exist in the cloud ... The results you can see are really pretty spectacular where you can come up with algorithms that now look at literally every square kilometer of the planet to identify the individual trees, grass and water, and then use that to build 3D models."

Azure's integration goes beyond the shape of the world. It also powers the flight controller voices using AI Speech Generation technology, which sound almost indistinguishable from humans. It's so natural that many players may think Microsoft is relying solely on voice actors.

Since the company began exploring ways to bring Azure AI into the game in 2016, the capabilities of machine learning have also evolved dramatically, according to Boyd. ”The AI algorithm space has really grown in the last several years,” he said. “And so vision algorithms, which is what's heavily used to identify all these different trees and buildings and classify them exactly, those have come a tremendous way."

Since it leans so heavily on the cloud, Flight Simulator is a "living game" in the truest sense, Neumann said. All of the machine learning algorithms the game relies on will steadily improve over time, as the company irons out bugs and optimizes the engine. (And perhaps becomes more aware of potential issues, like the typo that created a 212-story tower in Melbourne.)  But he points out the algorithms can only be as good as the source data, so Microsoft is working harder to refine that as well.

"Right now we have a bunch of planes flying overhead on the Northern Hemisphere because there's no clouds, so we're giving we're getting new satellite and aerial data," Neumann said. "We're going to process all that data with machine learning, and we'll have a 'world update.' We're going to have world updates every two months or so, is the plan. We're picking a region of Earth and putting some focus on it."

The first of those updates is aimed at Japan and will launch on September 28th, but Microsoft is also planning to look at areas of the world where private pilots aren't nearly as prevalent, like South America and Africa. Neumann hopes that exploring those untapped areas may make more people interested in flight simulation in general, and perhaps even spark a love for real-world aviation.

That's partially why he's so focused on capturing realism wherever possible. Take the weather: The game breaks the planet's atmosphere into 250 million boxes, where it can track things like temperature and wind direction in real-time. That means you're guaranteed to have a different flight experience every time you play. Neumann is particularly excited to see how the game will change during winter, when there's snow in the sky and entirely new types of weather patterns.

Flight Simulator's reliance on Azure will only grow stronger, especially if Microsoft stars bringing in more data from sources like satellites that track wildfires, or planes monitoring wind turbulence. "Yes we can have that data, but how do we use it?" Neumann said. "How do we get it to people? That's why this whole stack for me is fascinating and it enables experiences like this, but it's really just the beginning of what we're seeing."