… Immersive Visualization
This article is part of a series, The Seven Innovations That Will Change the World.
We are currently experiencing an interesting divergence. On one hand, our interactions with the digital world are intermediated less and less by physical screens – the televisions, computer monitors, smart phones, and tablets that we traditional associate with the digital. These days, more and more, we can simply talk to our devices in order to interact with them. Amazon Echo and its disembodied persona, Alexa, is a good example, but I’m equally astounded by my new noise-cancelling headphones (my mom bullied me into buying them after a recent bad experience on a flight had me praying for an Exorcist to be on the plane). I can control a lot of my phone’s functionality by speaking to the headphones or even just touching them. Sonos, the wireless speaker company, even has a name for this concept: sonic internet or sonic culture.
On the other hand, our consumption of media and entertainment, particularly television and film, is increasing, and these media are necessarily consumed through glass. American adults watched, on average, 4 hours and 46 minutes of television (that’s up 21 minutes from six months ago) and 25 minutes of video on phones, tablets, and computers (Wired, Nielsen). These figures do not include movies, streaming services like Netflix, or other non-conventional sources of media (like watching news or entertainment videos over social media sites).
What’s curious about this divergence is that even though our digital lives are being disintermediated from the screen, the media and entertainment that we consume is becoming more and more digital! Streaming services are becoming more popular while many households are cutting the chord and not bothering to pay for their age-old analog television service anymore. Quality and quantity of programming has increased, and we have more choice over how we acquire this content. Many people are finding that they do not have to be beholden to traditional television service providers in order to see great programming.
The digitization of entertainment goes beyond streaming services, Chromecast dongles, and Rokus. Gaming, which we typically associate with adolescent boys in their basements, is becoming a more pervasive form of entertainment. Even more so, e-sports have made gaming a spectator experience, with the associated apps and streaming services to watch them online. Game streaming generated $4.6 billion in revenue in 2017 (Motley Fool). Amazon’s service, Twitch, dominates the market with 665 million viewers, which are more than HBO, Netflix, and ESPN, combined. In 2014, Twitch was the fourth largest source of internet traffic, behind Netflix, Google, and Apple (Wall Street Journal). However, Google, Twitter, Microsoft, and Facebook are all nipping at Amazon’s heels and offering their own services to gamers and e-sports viewers. Facebook sells the virtual reality (VR) headset, Oculus, and hopes that 360° e-sports streaming will give it an edge over the competition and promote sales for Oculus (Wired).
Gaming itself has come a long way from the era of the console wars between Nintendo, Sony, and later Microsoft. Gamers can now play on computers, smartphones, and tablets. The most popular game in the world right now, Fortnite Battle Royale, with more than 125 million players since it was released in 2017, is available on PlayStation 4, Xbox One, Windows, Macs, Nintendo Switch, as well as iOS and Android. However, more and more games, usually on computers or smartphones, are immersing players themselves in the gameplay. Pokémon Go is a prime example, where the setting of gameplay is the users’ surroundings. Minecraft, another game of massive popularity, while not an augmented reality (AR) game like Pokémon Go, also has a mind-body connective therapeutic quality that few other games have been able to replicate.
As artificial intelligence, voice recognition, and haptic and gestures technologies advance, technology product managers will bundle these innovations into more and more useful applications or “skills” for Alexa. The glass in our homes and in our pockets will become less and less cluttered by the minutiae of our digital lives. Not so long-ago Facebook released a digital assistant known as “M,” but they folded it shortly afterwards. It seems that they are gearing up for a second act called Aloha (I had previously recommended the name “Geoffrey”) and a home video calling device called Portal to accompany Aloha (Techcrunch). Alexa is a window into our digital retail lives and often gives us access to music as well. Google Assistant allows us to control various appliances in our homes. Now our social lives will be voice activated as well. Einstein, from Salesforce, is a little mentioned, but powerful AI assistant for businesses. Sonos is working on devices that will integrate all of these voice-enabled digital assistants into one seamless user experience. With our digital lives conveniently more and more sonic, as Sonos would say, there will be less encumbering us from consuming entertainment on our leftover screens, whether its television shows and movies, live news, sports, or e-sports, games that we are playing, or even games that we are directly immersed in.
Rather than remaining passive consumers of entertainment, we are going to increasingly interact with our media. Pokémon Go was pivotal in launching this new genre of immersive entertainment to the masses and making AR go mainstream, but there are many other examples of immersion, interaction, and play appealing to people. One of the most important developments in the success of the NFL was the growth of fantasy football. By being able to play the game in a new way and interact with the league, engagement and viewership increased. Webkinz was an early example of children’s toys being mixed with the digital, by giving stuffed animals interactive lives online. Lego has dived head first into this trend, and is developing powerful AR apps that allow users to bring their Legoscapes, creations, and characters to life on a smartphone or tablet.
This trend, or more broadly, these innovations, I encompass under the term immersive visualization. Immersive visualization uses visual technologies to allow viewers to more closely see, interact with, and participate in the viewing experience. AR is only one technology bundled into this innovation. The defining characteristic of immersive visualization is dynamic interaction. VR is also an immersive technology, as is heads-up display (HUD), holograms, and I would even argue that alternative reality games (ARG) and gamification are immersive (although they do not necessarily have to be digital).
Beyond the purely entertaining, there is a growing body of research that shows that turning any task or activity into a game makes it more captivating and better for learning and knowledge acquisition and retention. This trend is called gamification, and the resulting games or quests are known as serious games. Companies like Lego are using gamification to drive engagement and sell more product. However, the education industry is developing quest-based learning based on the principles of serious games to improve learning outcomes. City managers and planners are seeing how they can incorporate whimsical games into the mundane details of cities to improve such important factors such as safety and traffic flow. The next time you see some interesting riddles or music making instruments protruding from your local bus stop, consider that it may have been put there to make it seem like the time you spend waiting for the bus is shorter, so you are more likely to ride the bus more often.
While AR is the superimposition of the digital over the physical world, virtual reality (VR) represents an immersion into a completely digital world. Via a headset, users can only see what is projected into the headset. These are often 3D environments, and the headsets relative position and orientation determines what parts of the environment are seen or not seen. By blocking out the actual physical world, VR has a powerful psychological effect on people. People begin to think that they are actually inside the VR environ and begin to act accordingly. Just like in the real world, people are apprehensive to jump out of VR airplanes and skydive because their minds begin to think that they are actually in an airplane and thus jumping implies mortal peril. VR has been known to magnify emotional responses far more than traditional cinema. Imagine being able to walk through and around the pivotal scene of a movie rather than watching it through a screen.
This psychological response makes VR another powerful tool for education, learning, and development, as well as other fields, such as therapy. The entertainment and media industry, travel industry, health care, and art and design industries are all developing applications for VR. Retailers are also clamoring for AR/VR applications to their businesses. There are now AR apps to visualize what furniture will look like if you purchased it for your home, how you would look in new apparel, or what cosmetics will look like when applied to your face. When shopping for a new home you can now skip the open houses and just take VR tours of the houses. The American Academy of Optometry is devoting an entire special edition of its Optometry & Vision Science journal to “Assistive Technology in Vision Impairment,” with a special emphasis on immersive visualization technologies. Face swaps and puppy dog faces were only the beginning of the AR revolution!
Barbie has recently seen a resurgence in popularity by pairing the doll with a video blog that is generated through motion-capture acting (Vice News). Netflix is beginning to dip its toes into immersion by developing interactive television shows where viewers will make choices that effect the plot progress and ending of the shows. Netflix is also releasing a show based on Minecraft and has plans to integrate more programming adapted from gaming into its platform (Bloomberg).
Immersion may also be a lifesaver for commercial real estate developers that have over-invested in retail space. Before Sears declared bankruptcy, the mall vacancy rate was at 8.6% (Reis, WSJ). VR arcades and other immersive experience centers are popping-up around the United States, and new companies such as Two Bit Circus and Meow Wolf are designing experience centers that perfectly fit within the floorplan of vacant department stores. They are anchoring more and more shopping centers that have lost department stores or other attractive retailers. Given the recent popularity of escape rooms and pop-up experiences designed for Instagrammers, these new immersive experience centers certainly have legs to run on. Greenlight Insights, a VR research firm, reports that immersive entertainment will be an $8 billion business by 2023 (Wired).
One of my favorite memories with my friends Luke and Catherine is when we were staying at a house on the Tamar River in northern Tasmania. One night we walked out to the riverbank and used Google Sky to explore the southern heavens. As natives of the Northern Hemisphere, the app was superimposing over the night sky stars we had never heard of and drawing constellations that we had never seen before. I had been gifted so many Aussie and Kiwi trinkets with the Southern Cross and seen so many friends with the constellation tattooed somewhere on their bodies, but never before had I seen those four stars with my own eyes until Google Sky showed me where to gaze.
This looming trend towards the immersive and visual is digitally driven, so computer engineers and software developers will continue to be in greater and greater demand. However, these technologies are visual and artistic. Someone at Google has to draw those constellations. Those Pokémon do not render themselves. If you want to see what buildings and neighborhoods used to look like by just walking around with your phone, someone is going to need to meticulously create the 3D models that materialize on your screen. Graphic designers, animators, artists, cartographers, photographers, architects, optical experts, mathematicians, and their ilk will be the indispensable partners of the coders. Facebook has developed a 3D and 360-degree camera that resembles a device out of Star Wars. The device must have taken thousands of hours of work from cinematographers, digital photography engineers, designers, and software engineers. I believe that we are on the cusp of the largest hiring spree of these “creative types” that for so long we have relegated to freelance work and starving artist status.
From the Bureau of Labor Statistics:
Employment of arts and design occupations is projected to grow 4 percent from 2016 to 2026, slower than the average for all occupations, adding about 33,700 new jobs. More workers will be needed to meet the growing demand for animation and visual effects in video games, movies, television, and on smartphones, as well as to help create visually appealing and effective layouts of websites and other media platforms. Other arts and design workers are employed in industries that are projected to decline, however, including publishing, manufacturing, and floral shops.
The median annual wage for arts and design occupations was $45,250 in May 2017, which was higher than the median annual wage for all occupations of $37,690.
The BLS is greatly underestimating the importance that these occupations will play in the future of our economy. Immersion is more enthralling. Immersion is more effective. Immersion, when implemented properly, can be more profitable. Consumers, as they become more acquainted with immersive experiences, will demand more and more of them. Media, entertainment, and more of our visual world will continue to march towards the immersive and interactive, just as digital is becoming more sonic.