GMU:Immersive Essays/Elliy (Yee-Shiuan Peng)

In the context of “immersive essays”, I have been curious to experiment with 360 photos and footage in VR, altering images in different ways to bring distortions to our normal perception of space and time. With these media I have been interested in developing personal experiences of documenting and sharing personal narratives, as well as possible platforms for an interpersonal exchange of experiences across spacetime.

The work is still very much in progress, continuously evolving, and experimental. I will document my experiences from the semester in the following order: 1) Concepts, ideas, background, 2) Technical aspects 3) Examples of experiments and development, 4) Future directions.

1) Concepts, ideas, background

◉ Exploring our perception of space and time:

An inquiry into our arbitrary experience of spacetime – the idea is to awaken the realization that our ordinary perception and experience of events in spacetime is actually completely arbitrary in a fundamental sense, but that we are completely aware of this fact due to the consistent way we experience the sequence, order, and distribution of objects and events in spacetime. Consistency does not undermine the arbitrariness of the set of physical laws that happen to underlie our physical experience in our particular universe.

By becoming aware of the fundamental arbitrariness of our experiences, the point is to extend that realization into an inquiry and realization of the arbitrary nature of all of the values that we have lived by in our existence and perhaps never questioned. Altered space times sequences will seem very absurd to the kind that is used to perceiving events in spacetime in a specific order, but actually the one we are used to is no more “absurd” than any other random or nonrandom pattern of physical existence.

In realizing the absurdity or the arbitrary that govern our reactions and interpretations in daily life, are we able to reevaluate and reinvent a narrative of existence independently of our physical existence and the arbitrary laws that they follow in this physical universe? These are all very abstract conceptual proposals, but the goal is to enter into a world of abstraction by dissecting and decomposing the concrete elements of experience that we take for granted.

◉ Exploring and exploiting out ordinary experience of spacetime – using exaggerated visual elements to bring to awareness our normal way of perceiving the world

Take normal films (or images) from a 360 camera and vary some aspects of the time and the space of the images to bring to attention the “normal way” that we perceive in daily life, such as the single direction forward flow of time (and entropy), the normal proportions of things in space – we are all living in the “arbitrary bubble” of perception that we have grown so used to, that we never even reach the point to questioning.

◉ VR + 360 camera as a way of bridging people across space and time:

Immersive video environments immediately lead me to think about the possibility of connecting the experiences of people who otherwise could not join each other physically in the same space and time. This idea was already present, and the motivation strengthened with the corona pandemic. These ideas may or may not be directly related to the concept above about the inquiry into the arbitrary experience of spacetime. The above is primarily a personal inquiry into the “normality” or perception, whereas the latter is more about using the media of VR and 360 filming as a way of allowing people to experience virtually different environments, with the possibility of immersively experiencing the perspective of someone from a different place, culture, background.

The motivation stems also from a similar fashion as the former in the sense of questioning “the world we are used to seeing and experiencing”, except the perception in question here is on a more concrete, less metaphysical level, in the sense of physical environments, living situations, cultures, social situations, etc. Most people living in one country cannot easily imagine life in a different environment, a different society, different levels of comfort, etc, and thus it is very difficult for people to actually relate to the reality of life of people under different living circumstances, making it very difficult to take the crisis situation of others as seriously and urgently as they would seem to merit.

Namely, people in the first world are living inside a bubble of their own living conditions. The corona pandemic serves as an illustration of this – for once in human history, it seems that all people around the world, regardless of country or culture, can actually relate to the urgency of a “crisis”, but even then, what we currently agreed to as calling “crisis” is nothing more than an agreement among the countries in power as to what is not in their “norm of living comfort” – what was, and still is, much more urgent and compelling in other countries that cannot be compared to “our” current state “crisis” is still not even really recognized as “crisis”. This is the reality of the global state in the present moment.

Can these problems be improved if we introduce more immersive ways of experiencing the environment of different people in different places. The goal there is to be able to reach more people, and bridge a connection between more people who normally would be living in “separate universes” on the same planet (or in some cases, even in the same city, or right next to each other), and so the goal would be to create something that is as easily accessible as possible, i.e., something accessible from the mobile phone + google cardboard (as opposed to developing a fancy immersive game that can only be accessed by people privileged enough to own a high tech VR device.

◉ Summary of concepts:

Moreover, the common goal of the ideas, is to use immersive media and narratives to allow people to briefly, if at all, get outside of their own “bubble universes” that they are used to living in – or perhaps just to even first become aware of the fact that we are, in any cause, always experiencing our existence through the lenses of an arbitrary perception.

The “Virtual Reality” is just another concrete metaphor of the “Reality”, which in effect is always like VR glasses, except the mind that we think and perceive through cannot be so easily “taken off” as physical VR glasses – or can they?

Can we, through these immersive media, inspire people to briefly remove the “VR glasses of their mind” that they have been wearing their whole lives without ever realizing? Can we, through exploring what is “virtual”, deepening our awareness into what we think and accept as “real”? Those are essentially the underlying questions to be explored in the different possible projects surrounding VR +360 camera.

2) Technical aspects of development

◉ Interactivity – While immersive imagery is already powerful in its own sense (much more than 2D anyways), it would still be nice to introduce aspects of interactivity into the scenario: Gaze control/activation of sound/stories/field recordings in specific elements in the scene Home scene – boxes containing “memories/homes” Immersive first person narration – ex: when you look at certain points and objects in the scene, certain sounds or narratives are activated through gaze control Brainwaves Live 360 streaming and exchange – interaction across different spacetimes AR combined with VR

◉ Interpersonal exchange Across spacetime Across cultures Across different perspectives of people in the same spacetime Starting as a movement of a series of exchange of videos making use of similar kinds of experimentation. Ex: an exchange project among different schools

◉ VR as a tool for exchange among people – using google cardboard, mobile phones, things that most people have access to, and then simply giving direction to an (artistic – read, existential) exchange Documentation – exchange of personal “essays” (experiences) in different spacetimes through some sort of platform that makes these “first hand experiences” accessible/sharable to others – sharing is not only about physical things, but also (and more importantly), intangible things, such as personal experiences, emotions, personal anecdotes

◉ Photogrammetry – digitalizing 3D environments and preserving ephemeral objects in the language of information → Specific idea of “immersive essays” platforms: “Virtual home exchange” – people give a tour/introduction of their own “homes”/living spaces with very specific and personal field recordings of specific things, and a chance for others to experience; all the places I’ve ever called “home” Travel anecdote exchange – people who are unable to travel (for whatever reasons, can experience personal anecdotes of other/serves as immersive way of personal documentation Culture anecdote exchange A platform with a globe that lands you at a random place or by change; all user provided content (can this type of thing be funded by the government as a social project, etc?)

◉ Platforms: Possible platforms that can be easily accessible by people across different devices Aframe Three.js Phone application

3) Examples of experiments and development

◉ Manipulating imagery from 360 camera – using footage taken from the 360 camera, experimentation in the following manner to explore spacetime:

Reversed time: Filming one person walking backwards and completing a series of actions while in backwards motion, and then reversing everything so that the rest of the world looks like it’s going in reverse time Especially interesting when you include elements of irreversible entropy, such as breaking egg shells, etc Contrast in passing times – time lapse vs slow motion, ex: filming backwards while at the same time moving very slowly, so that the rest of the world looks like it’s moving very fast

Reversed space: The phenomenon of reversed direction in our retina (that things are actually existing in the opposite senses than in the directions that our brain process) Making everything upside down and walking through (or just experiencing through) the world upside down Warping of spacetime – start with normal videos, then all of a sudden start warping different areas in the video (or specifically the direction/object that the person is looking at, activated by gaze control)

Multiple of the same person: Replacing other people’s faces (or another chosen/detected object) with: one’s own face; a fish’s head; a cloud, etc

Collage elements: Combining the real with the surreal/artificial Freeze, then taking apart of one (360) scene to reveal another scene that lies beneath (the never ending layers of spacetime) (ex: along the lines of landscape elements, such as mountain, tree, building, etc etc, then the objects/elements start floating on top of the scene below until they eventually flow out of the screen

◉ Immersive essays from google maps and the concept of “home”:

In working with a 360 camera and reflecting about the idea of bridging people from different environments, it also came to mind that not many people actually have access to 360 cameras, even if it comes to organizations, institutions, or schools, so it is not necessarily the most accessible device to be sharing context from (although surely it is good for production)

So I started exploring already existing 360 immersive image content from different places, and naturally the first place one would think of is google street view. While trying to figure out how to access the images from google street view and use them in some sort of exchange platform, I also realized that it was possible to create 360 images from a regular (2D) phone camera using the google street view app, which inspired me to think more about a platform for an exchange of immersive narratives from different places and people. To set up the first examples, I reflected upon the theme of “home” which is at once a universal one, and one that is (perhaps not so evidently) very abstract. The theme of “home” is an especially conceptual one for me personally, as I have lived in so many different places, and through time and space, “home” has become very clearly an “abstract state of mind”, that moves throughout different spaces and times, and yet each “place” that I have experiencing life in still stands out as a different “home” in its own way.

I did a virtual spacetime travel of looking up all the locations that I have ever lived in (“lived” being very loosely defined as somewhere I have spent time to any extent to which I did not feel like I was a "traveler" or simply passing by, whether it is ranging from a few weeks to a few years (which of course is an arbitrary definition)) in google street view and downloading their corresponding 360 images. I added all of these images to a scenario that one travels through, and these images in themselves became already an “immersive essay” to me – an essay with no words – perhaps that is the strongest?

I realized very strongly that visual impression, particularly an immersive, 360 surround image, even though one has very clearly in one’s mind that it is anything but “real” (and even the large pixels speak for themselves), but there is something very delicate about the immersive environment in triggering deep memories inside one’s mind that one knows one has, but has rarely tapped into so profoundly – that is one of the powers of immersive media.

In looking into the virtual poor resolution rendering of one of my “homes” I couldn’t help but want to look into the window to see whether someone was home. And yet in my mind it was so clear and not clear at all at the same time, that no matter how much I wanted, I would never be able to “push open” that “door” that lay between “me” and “my home in that point in space time”. It was a very powerful feeling, and it inspired me to want to build a platform where people share their “homes” with their own narratives (which could be anything from words to poetry to music to sound recordings, abstract sounds, or anything that is possible of being shared with the technology.

I had already thought about the power of bringing the “reality” of different living environments to different people, and the element of “home” (just as one arbitrary example) would bring another level of “personal aspect”, something that drives an even deeper level of “empathy” than mere “virtual physical immersion in vision and hearing” – emotions, and how to add personal emotions into immersive “narratives” shared from different people in different times and places

The project is far from complete, but I am very much interested in building specific opportunities for people to share highly personal experiences through the use of immersive media, for the goal of raising the degree of “empathy” that can be experienced among people…

◉ Technical elements: Gaze control – using photogrammetry, I scanned two main “characters” that would serve as “guides” to accompany people in their journey through my different “homes” – these are the characters “Bibi” and “Mimi” – using gaze control, one communicates with the guides of the scene and teleports to different places in space and time.

Using google cardboard SDK, the scenarios were exported as an iPhone and android app ready for use with any cellphone VR device with control from the gaze control.

4) Future directions/possibilities/continuations to other projects

◉ Implications to other projects:

Dome films: Projection in a dome is an alternative to the use of VR – the possibility of producing short pieces for the dome environment with elements from these projects: “Lost in spacetime” Reversed time and entropy Multiple of the same person Flying situations (on a drone) – falling up, etc Underwater Build a whole underwater scenario, special underwater costumes, choreography, etc → new field of underwater theater (!) Conceptual questions: What does the world look like to a photon (whose “time” is frozen and simultaneously happening all at once)?

◉ Interactive dome game – interactivity is also still possible in a dome environment, and with different possibilities of interactivity Simply apply all the “abnormal spacetime/metaphysical elements” as elements of the “chaotic/uncertain” perception, and apply concentration in order to make the “world” more “stable” or vice versa

Question: can “creativity” be measured/detected? ( → maybe we have to see that with people’s personal labelings of moments where they are (feeling) “creative”) What do brainwave signals look like when a person is engaged in especially creative thinking, action, etc?