To Immerse or Not Immerse: Designing VR

(Photo: Brainstorming VR models for “Shockwaves” with illustrator/game designer Matt Moores and NYU videogame designer Clara Fernandez, NYC Interactive Fiction at ITP/NYU. May 8, 2015) 

May 10, 2015: Virtual reality (“VR”) as we know it today first emerged in the 1950s and ‘60s. For an engaging oral history of VR, visit “Voices from a Virtual Past” published by The Verge. In the 1990s, we studied VR at the Interactive Telecommunications Program (ITP) at New York University. Coding in VRML and Java 3D – skills now deemed “useless” by VR pioneer Jaron Lanier – graduate students donned clunky Darth Vader headsets and fretted about roll, pitch and yaw – i.e. how things fly.

Fifteen years later and you can create VR with a smart phone, free software and a cardboard headset. As VR hardware comes of age, VR gear makers face the pressing issue of content. Where is it? What is it?

Meanwhile content creators grapple with the language of a new medium. Producing cinematic VR involves a workflow similar to that of a film, albeit you shoot with multiple cameras to capture a 360-degree panorama, and then stitch the footage. While this process can be complicated, what happens when you add narrative/story and interactivity?

How to design VR content?

We pose this as a question because no one knows the answers yet. It’s still too early in the development of this new medium to be rubberstamping a process that requires discovery, experimentation and a boatload of failures. However, we can tap on other mature disciplines, such as games, interactive theatre, performance and digital art, film, and even architecture, to hone in on the design of a VR piece.

From a personal standpoint, I’ve written a stage play, “Shockwaves,” about military interrogators in Iraq. I’d like to turn some or all of the content into a VR piece. How do I do it? I’ll walk readers through the initial phases of my journey – the early stages of a design process that has yet to be nailed down or executed. Some of these design considerations may be helpful to future VR content creators.

If you have a piece of content and are eyeing VR as a tool for execution, two types of questions – macro and micro – arise. Helicopter up 60 feet and ask the macro question: what’s the point of using VR? Once you’ve determined that VR will bring your content to life, ask the micro questions. For example, how do I design a blueprint for VR? Then, what do I need to execute?

Macro Questions

1. Is VR the right medium for your content? Do you even need VR?

Before the Rift hits the shelves, the general consensus on VR: it’s so cool! If you’re thinking about creating a VR piece or translating legacy content into VR, think again. Some types of content are better suited for other mediums. A tightly woven film requiring fast cuts may lose its momentum – if not logic – in a virtual reality environment.

“It’s not that you have empathy, it’s that you are actually having the emotions.” VR pioneer Mark Bolas on the Future of Narrative

In theatre, the dictum “form follows story/content” guides a writer to structure a play according to the story being told. The same rule of thumb applies to film. For example, “Pulp Fiction” and “The Joy Luck Club” use episodic screenplay structure. If these ensemble pieces had been squeezed into three-act screenplay structure – the hero’s journey – the creators may have done a disservice to the content.

As a medium, does VR serve a purpose with regard to your content? Why tell a story or make a game or experience in VR? What’s the point of ushering a user/player/viewer into a 360-degree simulated world?

In an interview on Upload VR, Mark Bolas, associate professor at USC’s School of Cinematic Arts and director of the Mixed Reality Lab and Studio at USC’s Institute for Creative Technologies, emphasizes one of the most compelling reasons to use VR. It doesn’t involve jaw-dropping spectacle but tapping into our emotions.

“One of the things to think about is that [VR] is always a first person perspective, and at first you think of that as being, well of course, first person like a camera view point, but in fact what it means is first person from a person’s perspective. So you are there, you have agency, everything is from your perspective. And as result, there is an entire range of motions that we can give people that are first person emotions.For example, guilt. You don’t really feel guilty if you watch a movie, personally guilty. But you can personally feel guilty if you shoot someone in a virtual world. So there’s this new range of emotions that we can start authoring for, and I think that’s one of the highlights of it as a new medium.”

In “Shockwaves,” we want users to step into the shoes of both an interrogator and a detainee. In addition, what does it feel like to come face-to-face with a suicide bomber? What does it feel like to wear the bomb and self-destruct? Some of these scenarios may be too potent, terrifying or traumatizing for users. Where do we draw the line?

2. How Much Immersion?

What degree of immersion best serves your content? Different tools can create varying levels of immersion. Do you use virtual reality (VR), augmented reality (AR) or projection mapping?

Virtual reality entails complete immersion in a simulated reality. Once a user puts on the headset, he/she is transported into another reality – one that is divorced from the real world. At this stage in the development of VR, it’s not the social or shared experience that can be achieved in gaming.

Augmented reality and projection mapping can simulate or overlay digital objects within the real world. A user experiences partial immersion. Using the reality as a stage, you can dress this stage with digital images. Question: what is the reality? Four blank walls? Subway station? Auditorium? Zoo?

AR technology, such as Microsoft’s HoloLens, is still in development. When talking to developers, they’re more at ease with VR than AR. Word in the trenches is that VR is easier to execute than AR. At present, if you’re planning to use immersive technology, the recommendation is to go with VR. Given the evolutionary speed of these new tools, this recommendation may be obsolete in months. Again, honor your content first and foremost. What tools will empower you to realize your vision?

In “Shockwaves,” if we want users to feel as if they’ve been plunked into the middle of an American base in Iraq, why not use projection mapping to create the base? Not only do you eliminate headsets but also the experience becomes more social and easily shared. However, a VR headset can promote the feeling of isolation that a detainee feels in an interrogation booth.

Can you mix and match tools to achieve different levels of immersion and achieve a coherent and naturally flowing user experience?

3. Interactivity: Tug-a-war between story and game

Do you want to make your VR piece interactive or not? Linear VR in which a user simply floats through a stunningly rendered VR world can be engaging.

For the filmmaker or playwright, do you produce your content for the stage or a movie theatre and then bolt on a component using immersive technology? For example, do we stage “Shockwaves” as a traditional theatre piece and then add a VR installation that enables users to experience an interrogation? Or do we integrate immersive technology into the play itself? If so, how do we do it?

Consider that the minute you introduce interactivity and grant a user any degree of agency, execution can grow infinitely more complicated.

If you begin with a linear story, you have to determine the plot points at which you can introduce branching structure. Once you use branching structure and allow a user to make choices, you then have to figure out how these choices impact the narrative flow of your story (or game). How many choices and where each choice leads can become a confusing nest of storylines if you don’t limit user choices. If any of your threads trickle to a slow death and don’t reintegrate into the story spine, you can lose your audience in a blink. The lessons of interactive fiction – sometimes painfully learned over the past two decades – apply to content in an immersive environment.

However, in an immersive environment, you may not be able to rely on text links or buttons to trigger user action. How do you make a VR narrative interactive, engaging users to even greater degree?

Storytellers from theatre and film have much to gain from brainstorming their concept with game designers. Instead of presenting a boring PPT presentation at NYC Interactive Fiction, I decided to put “ShockWaves” to the test with videogame and game designers at NYU. Happily, my doors were blown off.

Once you figure out how users will navigate through a VR world, the question becomes how do users interact with objects in this environment? What types of cues – visual, sound or tactile or a combination thereof – will signal a user to make a choice or take an action? Feedback from game designers may prove invaluable to storytellers. (Note: We will cover more ground on this area in Part II, drilling down to micro questions on execution.)

The design of VR involves asking high-level questions before you nosedive into the creative soup. While immersive technology provides a new tool set for content creators, the purpose of these new tools is not to wow an audience for a few brief minutes. These tools serve content, breathing life into a story, game or experience. They can provide a fertile breeding ground for new forms of art and entertainment.

– J Dakota Powell