From Sharktopus to Na’vi: It’s Time to Inject “Reality” Into Augmented Reality

Why is augmented reality so terrible? Better yet, why isn’t AR awesome!? It’s a question we’ve been asking ourselves for the past few months, and we’ve found one consistent answer: it’s about seeing AR in the right light.

Any CG artist that has attempted to put a 3D model into a live shot knows how important lighting is. One misshaped shadow or reflection and it looks fake. Like some of the low budget films you see on the SyFy channel, the CG looks so fake it’s hilarious. On the other hand, correct the lighting and suddenly it looks like a bunch of transformers are actually walking through the streets of New York. So, to sum it up, when it comes to CG in movies, fake is funny (and not good funny) and realistic is awesome. Augmented reality is exactly the same way.

The first time you see an AR mobile app it’s pretty cool, but the novelty wears off. Right now AR is like low-rent CG, and it’s completely unbelievable. It’s also fighting an uphill battle thanks to Hollywood. In order for AR to create realistic lighting, it has to do all the calculations in real-time, on a mobile device. To put this in perspective, a movie like Avatar uses render farms (about 1,000,000,000 times as powerful as your mobile device) to generate every frame, and that takes between 50-100 hours. That means Hollywood has over one trillion times the power and capability to create realistic graphics. To the average person, none of that matters. What does matter is how good it looks.  And when you can’t make it, you fake it.

Faking lighting in real-time is actually not that difficult thanks to the progress made in video games over the years.  One way to fake lighting is with a technique called environment mapping, more specifically, sphere mapping. With the help of a reflective “gazing ball” and an HDR camera, it’s possible to gather all the lighting information for a static scene. For a technical look at how this works, you can watch this video from Google, but put simply, a few shots of the ball with the correct exposures, combined with some vector geometry make it possible to map a 3D object with realistic lighting. The end result starts to actually put the “reality” into AR. It’s no Avatar, but it’s getting there at a rapid pace.

The most exciting part about this field is that hardware has steadily been doubling in speed and power about every 18 months.  Soon the device in your pocket will have more power than the best desktop computer out today. That means the ability to do more calculations, and create even more realistic 3D models to be placed in the real-world and not just augment your reality, but enhance it.

In the short term, this meas that AR can become a more realistic option for advertising and marketing. A car company doesn’t want to use an AR advertisement that’s going to make the chrome on their $70,000 car look like chintzy plastic, and a record label doesn’t want the lead singer of the band to look like a mannequin. But in the long term, quality AR could give us a virtual walk through of a building before it’s built, or lay a vivid 3D game layer on the world around us. The film and gaming industries have blazed the trails for us, it’s just a matter of time before our mobile hardware is able to take us there.