GMU:Immersive Essays/Kristin Jakubek

VR project title: THE SCENT OF A FLOWER WE HAVE NOT FOUND

Intention

I enrolled in this course because I am especially interested in the idea of the video essay as a means to manipulate already existing video material (primarily found on the digital archives of Youtube) to identify visual similarities, create a new statement, impression or course of events.

The task to translate screen-based, flat, 2D image and video material into an immersive VR environment in a way that feels appropriate, relevant and warrants putting on the VR glasses is challenging.

For the video-essay approach to VR, I considered topics that would full-fill following criteria: (1) Be researchable; (2) Room for subjective view / personal opinion / personal comment; (3) Opportunities for diverse visualisation through different media ( video, still image, audio, interview, 3D models ); (4) Opportunity for audio immersion.

Resulting from a conversation with a friend who works in psychotherapy I settled on the topic of ‚Ketamine therapy‘, a potentially controversial topic, which is currently gaining acceptance as a new way to treat treatment resistant depression. Ketamine has been abused and consequently stigmatised as a recreational party drug. I find the friction between societal condemnation of drug-use, lacking understanding of depression and mental health issues and personal authority concerning drug-use, interesting and something I wanted to research and deal with through video inside a VR experience.

The title ‚THE SCENT OF A FLOWER WE HAVE NOT FOUND‘ comes from a book titled ‚signs of life‘ and is a sensually metaphorical description of desire. This felt like a compelling title first because this half-sentence conjures up a mental image and synaesthesia (e.i. the mix-up of sensory perception possible while on Ketamine) of scent, which inspired the red-thread of flowers -as memory fragments- that shows up throughout the experience. Secondly, as the idea of searching for something connects back to the hoped for relief a patient longs for when undergoing such pioneering therapy approach.

Overall I see my role for this project as a digital curator who‘s task it is to find visual and audio material, as well as thematic connections and storytelling methods to translate selected artworks into new forms for an immersive VR environment. (Re-inventing them in a new virtual context, while keeping their integrity.)

Overall the VR experience is my subjective interpretation of a Ketamine induced therapeutic trip, an immersive essay on the mental state someone might experience when absolving such an experimental drug trial.

What is immersive space?

An immersive VR experience has the opportunity to go beyond the purely digital by also including further senses. My intention for an exhibition scenario of this VR experience, would be to create a space that includes a chair with a blanket similar to the chair the visitor will be on in the digital experience. In this way, the chair would connect physical and virtual reality. (Preferably inside a clinical space enclosed by curtains - Inspiration: https://tankmagazine.com/tank/2018/03/stine-exhibition/ ). I would also control the smell in the room through a soothing scent, both to draw a link to the project title and provoke a more immersive experience. During my research I discovered that a Ketamine-infusion therapy shows some remarkable similarities to a VR set-up that could be emphasised in such an installation. Namely, patients are seated in a chair, given an eye-mask as well as headphones with soothing music and a blanket. It can be said that the Ketamine-infusion therapy is in itself an immersive experience, as patients receive their infusion and are left to experience an altered world of perception without ever leaving the room.

Self-evaluation

+/- I expected that it would be (more easily) possible to create smooth transition that could morph videos, shapes, 3D models, terrain and scenes. Throughout the process my vision for the experience continued to change as I realised the limitations of the program and my amateur know-how.

+ I struggled a lot with creating a visual style that feels connected to my personal vision and is visually exciting. Being completely new to Unity it remains a dialogue to understand what is possible and allow that to inform the design process.

- I expected that colour, materiality and shapes (of the selected media and their presentation surface) will play a key role to create a haptic experience. But I initially underestimated the power of the digital 3D space e.g. the skybox to ground the viewer and give them a point of reference.

- I ‚lost‘ a good deal of time with 3D collage experiments inside Unity, which I ultimately dropped. This process showed me in what ways the design approach for VR differs from film or photography.

- I got so busy with the execution inside Unity and acquiring basic technical know-how, that the comprehensible translation from research made on the topic, to the finished project ist not as in-depth and traceable as I‘d like it to be. (The focus on learning technical basics lead to intuitive approach in actualisation of research.)

+ Was able to adapt and learn on the go.

- The visual translation process should have been more methodic. Instead the media selection process was highly intuitive and could have lead to more unexpected connections by going more in-depth and searching for additional media beyond works I already knew.

- The finished experience has a specific focus on trying to visualise Ketamine as a felt experience, however, it includes no comment or draws focus to the multifaceted controversial aspects related to Ketamine use and mental health, which initially drew me to the subject.

- I didn‘t find a way to utilise the different ideas for typography and poem by Shannon May Powell and more inventive transitions between objects and scenes I had in mind.

Background Info/ Subject Intro

„I am going to die in this dentist’s chair.

My eyes are closed, but I can still see skulls outlined with white against a black background. I have an epiphany: God is death. I’m in the midst of a real-life version of the hallucinogenic ride in Willy Wonka and the Chocolate Factory, all in my own mind. A monitor emits a steady beep, and for a second, I think I’m flatlining. But no: I’ve just completed my first infusion of ketamine."'

- Excerpt from a first-person essay on Ketamine therapy by Alice Levitt for Vox (source: https://www.vox.com/first-person/2018/7/24/17603616/depression-treatment-severe-ketamine-special-k)

Ketamine is primarily used as an aesthetic but doctors had long been curious about the euphoric effects of Ketamine, because a lot of people who received it when undergoing surgery would start smiling and laughing. „That‘s the reason that many people before have said: 'Oh, maybe it can be used for depression'.“ - 'I tried ketamine to treat my depression. Within a day, I felt relief.', Alice Levitt for Vox

Scene 1

To translate the practice of the video essay into VR the careful selection of visual and audio material felt key to creating the experience. The challenge was taking existing 2D (art)works and adapting them to a complimentary 3D space.

During the process the necessity to create a black space became quickly apparent, because some of the found videos have the best effect when their outlines melt with a black backdrop, which allows the video content to appear as if floating in space. This creates a more immersive experience, which I exaggerated further in scene 1 by surrounding the viewer with video-planes from all sides.

Both inspired by and using actual footage from the anime ‚Mind Game‘ the scene opens in - what could be understood as - an ‚out-of-body experience'. The opening scene the VR visitor sees is of a small, confused, figure, alone, in a vast black space - One might think: 'Am I seeing myself from outside my body?' Then suddenly a screen appears, revealing that this figure is also inside a fully digital space.

The second featured artwork is Sarah Schönfeld‘s work ‚All You Can Feel‘, which shows photographs of liquified drugs and other neurotransmitters - including Ketamine - exposed on photonegatives. The work shows these substances as round molecules, or orbs/planets. Hereby, casting them in an unusually artistic light that focuses more on structures and colours as opposed to the bodily effect/high. In this way the video is both an introduction into the bio-chemical first phase of a Ketamine therapy, when the liquified drug is introduced to the body, but also my personal alternate view on a medical topic (& here biological process).

The treatment chair became the anchoring point for the viewer. I modified a 3D model of a dentist chair from Sketchfab inside Cinema4D by adding a blanket with a print, which make it tie-in with the overall style of the experience.

The decision was made to anchor the viewer on the treatment chair and allow the action to unwind around the user while seated. From a technical point of view this makes sense as it will decrease the risk of nausea, from being move from point A to point B, or accidents when users walk around with VR glasses. Conceptually this also aligns with the actual Ketamine-infusion-therapy experience.

Additionally audio plays a crucial role both for a video essay - as these videos are usually narrated with personal commentary or the author‘s line of argumentation/statement - and, because one of the effects of Ketamine are the increased stimulating effect of any noise.

One patient said her experience was guided by the soothing music that was played from headphones during her treatment, the sound shaped the visual and bodily experience.

„A beautiful soundscapes that drove how I felt throughout that whole experience. It just felt like I was floating in this beautiful, enchanting, musical world.“ - Bridget, writer and producer at ‚What‘s Working‘ - source: ‚Ketamine Treatment for Depression: What It‘s Actually Like [Episode 2]‘) This is why all following media in the scene is accompanied by Ana Roxanne's track - 'Immortality'.

Scene 2

As I imagine a therapy patient goes through several phases during his infusion therapy I created a second, more light filled, airy and less claustrophobic phase.

In both the first and second scene of the VR experience I used a scene from the anime 'Mind Game' by Masaaki Yuasa. In the anime the main character Nishi, dies only to wake up in a black space where he sees his death as a video-recording from enormous screens that surround him from all sides (Inspiration scene 1). Nishi then enters into a completely white space (Inspiration scene 2) where he encounters a figure who keeps changing it‘s appearance. The shape-shifting figure is god, and because Nishi has so many associations of what god might look like the figure remains in flux. Exactly this visual stream of fluctuating association I interpret as a visual representation of Ketamine. In my research I found that one patient described his state as: 'one borderless perceptual hallucination'. This might be because Ketamine breaks open connections in the brain, which leads the brain to form new neural-pathways. This biochemical process plays with memory and can trigger hallucinations. Additionally scientists found that Ketamine triggers reactions in your cortex that enable brain connections to regrow. However, it is still not fully understood how the drug works to alleviate symptoms of depression.

The treatment space, consisting of a door, a wall (that has turned soft)and an IV monitor, which remain as remnants of reality. The actions that unfold around the viewer in the form of videos and 3D surrounding symbolise a possible therapeutic conversation the patient is having with themselves.

"And then I was gone, down the rabbit hole of hallucination. My mind skipped though grid-style maps of city parks. I occasionally took a deep breath or wiggled my fingers just to remind myself I still could. I later learned that what I was experiencing is known as a "K-hole," which is rare at the low does I took." - 'I tried ketamine to treat my depression. Within a day, I felt relief.', Alice Levitt for Vox

3D shapes and adapted models

One primary aim for the project was to create a visual style that feels artistic and personal. I wanted to get away from a general ‚computer game style‘ as the ‚default‘ look of downloadable models (from Sketchfab) that seen inside Unity felt like they had nothing to do with me. To achieve a step in the right direction I modified textures, created shapes inside Cinema4D and re-composed models before placing them inside the Unity Editor. The look inside C4D vs. Unity is visibly different and signifies that I still have much to learn in terms of shaders and controlling light in a scene.

Nudibranch are sea slugs, that are small and squishy in extravagant colours and patterns. Originally I wanted to insert one as a kind of ‚spirit guide‘ into the VR experience. Eventually this adapted to creating abstract shapes that float around the viewer in the final scene. The shapes are created inside C4D and had to be reduced to low poly meshes with a backed texture to run correctly inside Unity. I animated these shapes to float with a script that allows me to modify their coordinates in the inspector window.

Scene 2.1

The final sequence borders on a sensory overload as the plane on which an edited video sequence is playing, is attached to the user‘s point-of view, meaning wherever the viewer looks the plane is at the centre of vision. After struggling to find other virtual forms for video presentation, with this solution I found a compromise between the established, conventional, flat format for playing videos and an immersive virtual space. Moreover, because of this, by looking around the viewer can influence the interplay between the playing video and his/her surroundings opening up the space for a personalised view while inside the experience. As every Ketamine-therapy trip is highly individualistic this was an additional aspect I‘d hoped to incorporate.

Here, I find the contrast between video footage depictions from ‚real-life‘ placed inside a virtual environment particularly exciting. To draw from the video essay approach, the video is using moving image as a compilation and assembly in the pursuit of creating something new from previously existing media found in digital archives, like Youtube, hosted online.

Scripts used

Making sculptural shapes float:

SimpleFloating.cs

Crossfading between Unity scenes:

SceneLoader.cs

t.b.a

Video Documentation

[1]