Multisensory Experiences in Virtual Reality and Augmented Reality Interaction Paradigms

Introduction

Smart systems incorporate functions of sensing, actuation and control to describe and analyze a given situation, in order to make smart decisions. Interactive technologies are key in smart systems. They involve different paradigms and a variety of sensors and displays that make it possible for a user to have enriched experiences through his/her senses. Nowadays, we are experiencing an evolution of interactive technologies that are even embedded in small devices, such as tablets and smartphones, reaching wide audiences. Most interactive systems make use of sight (e.g. 3D models) and hearing (e.g. music, narrator, sonic effects), but also other senses can take part of these systems, including touch (e.g. feeling the shape and texture of virtual objects), smell (e.g. the aroma of food) and vestibular (e.g. travelling inside a computer-generated world while feeling the movements and vibrations). Of course, the last cases are less spread over consumer-grade devices, on the one hand, because of the increased costs and on the other hand, because of the needed hardware. However, multisensory experiences have been researched for many years. One of the earliest immersive, multisensory machines is the Sensorama, that was patented back in 1962 by Morton Heilig (United States Patent No. US3050870A, 1962). The technology integrated in the Sensorama allowed a single person to see a stereoscopic film enhanced with seat motion, vibration, stereo sound, wind and aromas, which were triggered during the film. This was a visionary system, and has been referred to as “the cinema of the future” (Heilig, 1992; Robinett, 1994), although at the time it was invented, it did not attract wide audiences, maybe because the technology was too incipient to be widely exploited.

In the last few years, the rapid technological advancements have allowed the development of commercial solutions that integrate a variety of multimodal displays in movie theatres, such as the 4DX (“4DX | Absolute Cinema Experience,” n.d.) or the Pix 5D cinema (“Pix 5D Cinema,” n.d.). Some claim that this technology shifts the cinema experience from “watching the movie to almost living it” (Yecies, 2016), also enhancing the cinematic experience while creating a new and contemporary version of storytelling, which can be conceptualised as a “reboot cinema” (Tryon, 2013).

Multisensory experiences can be integrated in systems that make use of different interaction paradigms, such as Virtual Reality (VR), Augmented Reality (AR) or, in a wider sense, Mixed Reality (MR). Examples of these paradigms to create cinema-related experiences can be found in (Portalés, Viñals, Alonso-Monasterio, & Morant, 2010; Vosmeer & Schouten, 2014), to name some. While VR substitutes the real world by a synthetic one, AR enriches the real world by means of virtual stimuli, without (completely) replacing the real world. On the other hand, MR is understood as the result of merging the real and virtual worlds at some point along the “real virtuality continuum”, which connects completely real and virtual environments. Although the MR concept was defined for the first time by Milgram (Milgram & Kishino, 1994), it has become very popular in the last years because of the emergence of cutting-edge technologies in this field, such as more sophisticated headsets and glasses among others.