375
edits
m (→Visual concept) |
m (→Inside Unity) |
||
(16 intermediate revisions by the same user not shown) | |||
Line 7: | Line 7: | ||
== Collaborative Concept == | == Collaborative Concept == | ||
As a Virtual Reality experience in experimental sound practice, centring around poetry as spoken word performance, the VR poem is developed in collaboration with the poet, Malina Heinemann, who lives and works in Berlin. | As a Virtual Reality experience in experimental sound practice, centring around poetry as live spoken word performance, the VR poem is developed in collaboration with the poet, Malina Heinemann, who lives and works in Berlin. | ||
Our collaboration consist of a creative dialogue in which her work is the starting point for me to develop all visual aspects. Malina is in the process of slightly configuring the poem for its virtuelle rendition and recording the spoken word in studio quality. | Our collaboration consist of a creative dialogue in which her work is the starting point for me to develop all visual aspects. Malina is in the process of slightly configuring the poem for its virtuelle rendition and recording the spoken word in studio quality. | ||
Line 21: | Line 21: | ||
The project is based on the poem ‘twist’- as spoken/presented by the poet herself- the style of which will be analysed and relayed through a Data Patch in the visual programming software Pure Data. | The project is based on the poem ‘twist’- as spoken/presented by the poet herself- the style of which will be analysed and relayed through a Data Patch in the visual programming software Pure Data. | ||
The set-up of the patches follows the question: (how) Do small shifts in rhythm, tempo, pauses (and pitch) create a unique auditory experience? This generative sound experience is designed to be nuanced and slightly different every time it is played/performed by the algorithm. Just as it would be performed as spoken word by the human poet. | The set-up of the patches follows the question: (how) Do small shifts in rhythm, tempo, pauses (and pitch) create a unique auditory experience? This generative sound experience is designed to be nuanced and slightly different every time it is played/performed by the algorithm. Just as it would be performed as spoken word by the human poet. | ||
The data patch will identify pauses in the poem and randomise them. Hence, pulling words further apart, splitting them in two, or pushing them closer together, creating new sound-word-hybrids in the process, which will be based on the internal logic of the spoken poem as it was originally performed. In effect, this might leave a moment longer to understand, or make it a second harder to catch up with what is being said. Words that carry more weight because of their emphasised intonation might be pitched even lower, or inverse, their emphasis might be lifted completely. Sentences that are spoken quickly might be made even quicker or slowed. | The data patch will identify pauses in the poem and randomise them. Hence, pulling words further apart, splitting them in two, or pushing them closer together, creating new sound-word-hybrids in the process, which will be based on the internal logic of the spoken poem as it was originally performed. In effect, this might leave a moment longer to understand, or make it a second harder to catch up with what is being said. Words that carry more weight because of their emphasised intonation might be pitched even lower, or inverse, their emphasis might be lifted completely. Sentences that are spoken quickly might be made even quicker or slowed. | ||
By playing with these parameters of spoken word the poem is given a unique performative rhythm each time it is heard, which will (re-)shape the poem itself and influence the comprehensibility for the listener. In the process this will reveal to what degree these elements of spoken language play with language and cognition within the poetic genre. | By playing with these parameters of spoken word the poem is given a unique performative rhythm each time it is heard, which will (re-)shape the poem itself and influence the comprehensibility for the listener. In the process this will reveal to what degree these elements of spoken language play with language and cognition within the poetic genre. | ||
Line 27: | Line 28: | ||
== Visual concept == | == Visual concept == | ||
The VR experience will be a slow, fully guided, visual journey of the poem ‘twist’ and position the poem across and within the space, hereby creating a dynamic, shifting, spatial (sound) experience. | The VR experience will be a slow, fully guided, visual journey of the poem ‘twist’ and position the poem across and within the virtuelle space, hereby creating a dynamic, shifting, spatial (sound) experience. | ||
[[:File:twist_poem_pdf.pdf]] | [[:File:twist_poem_pdf.pdf]] | ||
The main visual components (bodies, street elements) are inspired directly by the poem | The main visual components (bodies, street elements) are inspired directly by the poem: | ||
''dry asphalt
| ''dry asphalt
| ||
Line 160: | Line 161: | ||
Possible approach 1: | Possible approach 1: | ||
Outline what should happen: | |||
(1) poem plays | (1) poem plays | ||
(2) rises and falls of the tone/inflection of individual words are detected by pitch | (2) rises and falls of the tone/inflection of individual words are detected by pitch | ||
Line 173: | Line 175: | ||
Possible approach 2: | Possible approach 2: | ||
Outline what should happen: | |||
(1) poem plays | (1) poem plays | ||
(2) rises and falls of the tone/inflection of individual words are detected | (2) rises and falls of the tone/inflection of individual words are detected | ||
Line 188: | Line 191: | ||
From these first patches it becomes clear that the single words that should be re-pitched need to be specified so that the effect is mimicking realistic speech. This needs to be linked to the changing breaks (and subsequent randomised position/timing of the words in the loaded sound file..) | From these first patches it becomes clear that the single words that should be re-pitched need to be specified so that the effect is mimicking realistic speech. This needs to be linked to the changing breaks (and subsequent randomised position/timing of the words in the loaded sound file..) | ||
Possibly continue working with > externals/ audiolab/ pp.sfplayer. pd | |||
Or Pitch Shifter: | Or Pitch Shifter: | ||
[[File:PD- Pitch Shifter.png|400px]] | [[File:PD- Pitch Shifter.png|400px]] | ||
Line 205: | Line 208: | ||
== Process == | == Process == | ||
This semester I mainly started work on creating my own visual elements | This semester I mainly started work on creating my own visual elements. The objective here is to create 'realistic' surfaces and objects that feel tangible. Because of this I enjoy using photogrammetry as technique to transport places, textures and people from the real into the rendered mesh. | ||
I used a range of different photogrammetry softwares: | I used a range of different photogrammetry softwares: | ||
Line 224: | Line 227: | ||
</gallery> | </gallery> | ||
During the shoot with (living, breathing, moving) people I used 3 different Softwares to compare and have a choice of results: | The first photogrammetry shoot was done outdoors with only one person. It became clear that more interesting poses could be captured with two people and that less post production would be needed to create a mesh of multiple bodies. The second shoot was styled with custom latex pieces by Florian Mathe and was captured indoors in an empty all white room. During the shoot with (living, breathing, moving) people I used 3 different Softwares to compare and have a choice of results: | ||
When using Trnio to scan people the results vary from recognisable to quite distorted. It is generally more unpredictable but the models are calculated on an external server, which gives very quick and ‘low effort’ results, as well as creating more comprehensive material maps. | |||
<gallery> | |||
File:Trnio1.PNG | |||
File:Trnio2.PNG | |||
</gallery> | |||
Metashape is a great photogrammetry software that creates very detailed point clouds and a realistic texture on the model. However, once creating a dense mesh the solid form (sans texture) has many irregularities and a very ‘bumpy’ surface, most probably resulting from the micro-movements of the models. This results in requiring a lot of cleaning up of the models afterwards. | |||
[[File:Metashape_pointcloud.png]] | |||
My personal preference is Agisofts Recap Photo (which sadly only runs on Windows). The software also sends the images to an external server were the models are created quickly. The way the software deals with ‘irregularities’ or missing information results in very organic colour transitions and fluid, soft shapes. Also the way the texture is displayed in the 3D viewport is ideal. | |||
<gallery> | <gallery> | ||
Line 251: | Line 268: | ||
[[:File:meshes_process.pdf]] | [[:File:meshes_process.pdf]] | ||
Line 282: | Line 292: | ||
The view of the camera is taken through the bodies, through their edges and foldings. | The view of the camera is taken through the bodies, through their edges and foldings. | ||
The camera/view transitions between the interior/exterior spaces that are twisted into each other, along their edges. The whole environment becomes a collage that isn't glued in the right way. | |||
Line 294: | Line 306: | ||
[[File:Bodyfoldings_unity_gameview04.mov]] | [[File:Bodyfoldings_unity_gameview04.mov]] | ||
This | This still needs to be done smoothly by animating the camera inside Unity. | ||
Line 308: | Line 320: | ||
'''Body-street-scapes''' | '''Body-street-scapes''' | ||
The street elements will function as facade-like foldings that collapse into themselves. | |||
The intensity of the external spaces is increased by working with simulating weather conditions that invoke sensory memories of a heatwave, the smell of hot pavement or cooling summer rain on the asphalt. A blurring, wavering heat distortion effect, a damp fog or wet puddles on the pavement. | The intensity of the external spaces is increased by working with simulating weather conditions that invoke sensory memories of a heatwave, the smell of hot pavement or cooling summer rain on the asphalt. A blurring, wavering heat distortion effect, a damp fog or wet puddles on the pavement. | ||
Line 327: | Line 341: | ||
Guided view/ journey with Cinemaschine camera | Guided view/ journey with Cinemaschine camera | ||
Animation of combined mesh-model | Animation of combined mesh-model i.e. collapsing and folding of street elements, distorting, twisting effects | ||
Effects on camera view | Effects on camera view i.e. blurring effect | ||
Sound | Sound implementation PD | ||
Spatial sound | Positioning the poem in the space i.e. Spatial sound |
edits