GMU:Bits, Beats & Pieces/Lalalolo/lalalolo/Projekt: Difference between revisions

Latest revision as of 15:26, 5 June 2019

<automata>

Context:

'Life is a simulation' or is it? The quest for finding a set of rules to simulate life in some shape or form has been around for ever. Whether it is in search for the free will or just to find out what makes things living. One mathematical approach got introduced to world in 1970 by the British mathematician John Horton Conway, the Game of Life. The so called Game of Life (or GoL in short) is a 'cellular automaton', a type of simulation that works on a regular grid of cells and in discrete time-steps. For every time-step a set of rules is enforced for every cell, determining it's state. In most cellular automaton there are only two states 'living' and 'dead'. Depending on the rules and other constrains of the simulation, complex patterns might occur and this project will try to take these patterns and represent them as notes in a 'cellular automaton synthesizer'.

cellular automaton in different dimensions

Concept:

For this project I will use a one-dimensional or so called elementary cellular automaton, represented as a line of 32 cells. In every time-step of the simulation the state of the cell (living or dead) is considered and than the state of the two neighboring cells, edge-cells will be wrapped around. Living cells will be represented as black and dead cells as white pixel. Rules are encoded by all of the possible combinations of the dead cell itself and its neighboring (2^3 = 8). To represent the rules, they are encoded in one-bit-array of length eight like the following graphic illustrates.

One rule-set and its representation

With theses 8-bits there are 256 possible combinations of rules that can be used for the simulation, these rules often form reoccurring patterns that will be important later.

Some elementary cellular automaton rules and their pattern

To interpret the simulation as music in every time-step the current line of the simulation is read and the different cells interpreted as notes in a musical scale. To minimize dissonance a anhemitonic pentatonic scale is used. Because there are only five notes per octave and no semitones in a pentatonic scale, they produce a simple harmonies that work well with random, pseudo-random and chaotic but structured input.

Different penatonic scales and the matching notes

Approach:

The synthesizer works like the following, the cellular automaton is simulated using the selected rule-set. The current state or line of the automaton is than read from left to right and the matching notes of the selected pentatonic scale played (low notes left, high notes right). To give the user more control over the sound, the notes are split up into eight channels of differing pitch.

One simulation step visualized

The synthesizer is web-based using HTML, CSS and JavaScript. To play notes the library Tone.js (js/tone.js)and their default synthesizer is used (js/synth.js). The simulation is visualized using the Canvas-Api and simulated using native JavaScript (js/automata.js). The User-Interface is written in HTML and CSS (index.html) using onclick-event handlers to interact with the simulation (js/gui.js).

The final synthesizer-ui

Downloads

File:Automata.zip Just open index.html with a modern browser

@@ Line 1: / Line 1: @@
-==Never ending match cuts==
+==<automata>==
 ===Context:===
-Match cuts, being the art of transitioning from one scene to another while continuing the context in some form, are a common trope in filmmaking.
+'Life is a simulation' or is it?
-Keeping the context using a match cut may be archived by keeping a similar frame-composition, motion-path or audio backdrop.
+The quest for finding a set of rules to simulate life in some shape or form has been around for ever.
-Match cuts may be spatial, temporal or metaphorical as long as continuity may be perceived.
+Whether it is in search for the free will or just to find out what makes things living.
+One mathematical approach got introduced to world in 1970 by the British mathematician John Horton Conway, the Game of Life.
+The so called Game of Life (or GoL in short) is a 'cellular automaton', a type of simulation that works on a regular grid of cells and in discrete time-steps. For every time-step a set of rules is enforced for every cell, determining it's state. In most cellular automaton there are only two states 'living' and 'dead'. Depending on the rules and other constrains of the simulation, complex patterns might occur and this project will try to take these patterns and represent them as notes in a 'cellular automaton synthesizer'.
-From cinematic classics like Stanley Kubrick's 2001 A Space Odyssey to  modern music videos like Radiohead’s Daydreaming match cuts create transitions without stopping the general flow of an audio-visual impression.
+[[File:dim.png|800px]]
+cellular automaton in different dimensions
 ===Concept:===
-Match cuts are usually crafted by hand with care and attention to detail. In contrast to that I will try to create match cuts programmatically and in the spirit of “Me and the Machine” use an algorithm to find fitting cuts in found footage videos to create an endless stream of matching videos.
+For this project I will use a one-dimensional or so called elementary cellular automaton, represented as a line of 32 cells. In every time-step of the simulation the state of the cell (living or dead) is considered and than the state of the two neighboring cells, edge-cells will be wrapped around. Living cells will be represented as black and dead cells as white pixel. Rules are encoded by all of the possible combinations of the dead cell itself and its neighboring (2^3 = 8). To represent the rules, they are encoded in one-bit-array of length eight like the following graphic illustrates.
-The idea is to use the endless stream of audio visual content of the modern day world to create something new and unique that looks and feels continuous and might resemble something handcrafted with context.
-Due to  technical constraints the techniques used might vary, but if possible, I would like to use all forms of match cuts from simple audio matches to complex contextual or even metaphorical matches.
+[[File:rules.png|800px]]
+One rule-set and its representation
-To further give the work meaning, I might limit the pool of videos to choose from or possibly even include interaction by the observer, while keeping the idea of an endless audio-visual stream.
+With theses 8-bits there are 256 possible combinations of rules that can be used for the simulation, these rules often form reoccurring patterns that will be important later.
+[[File:all_rules.png|800px]]
+Some elementary cellular automaton rules and their pattern
+To interpret the simulation as music in every time-step the current line of the simulation is read and the different cells interpreted as notes in a musical scale. To minimize dissonance a anhemitonic pentatonic scale is used.
+Because there are only five notes per octave and no semitones in a pentatonic scale, they produce a simple harmonies that work well with random, pseudo-random and chaotic but structured input.
+[[File:scales.png|800px]]
+Different penatonic scales and the matching notes
 ===Approach:===
-To achieve this endless stream continuous image and audio analysis is needed. Both of theses tasks are of high computational cost and no easily done in a live-context.
+The synthesizer works like the following, the cellular automaton is simulated using the selected rule-set. The current state or line of the automaton is than read from left to right and the matching notes of the selected pentatonic scale played (low notes left, high notes right). To give the user more control over the sound, the notes are split up into eight channels of differing pitch.
-Visual match cuts might be programmatically solved by comparing the last frame of a chosen sequence with frames of another video using different approaches.
+[[File:approach.png|800px]]
+One simulation step visualized
+The synthesizer is web-based using HTML, CSS and JavaScript. To play notes the library Tone.js (js/tone.js)and their default synthesizer is used (js/synth.js). The simulation is visualized using the Canvas-Api and simulated using native JavaScript (js/automata.js). The User-Interface is written in HTML and CSS (index.html) using onclick-event handlers to interact with the simulation (js/gui.js).
+[[File:finished.png|800px]]
+The final synthesizer-ui
+===Further Reading:===
+====Elementary Cellular Automaton====
+* http://mathworld.wolfram.com/ElementaryCellularAutomaton.html
+* http://atlas.wolfram.com/01/01/rulelist.html
+* https://plato.stanford.edu/entries/cellular-automata/supplement.html
-The first approach might be as easy as comparing histograms, which is fast, but is only really feasible for colored imagery and might fail to guarantee continuity.
+====Pentatonic Scale====
-Feature matching might be possible for finding similar image compositions and or things  of similar shape in two frames. This method might be able to recreate metaphorical and spatial match cuts.
+* [[wikipedia:Pentatonic scale]]
+* https://www.piano-keyboard-guide.com/pentatonic-scale.html
-Deep Ranking might give best results in finding similar images, creating the feeling of continuous imagery, but feasible solutions are scarce and training a neural network takes time and a lot of prepared data. This approach might still be possible using an existing framework or image-comparison API.
+====Web Stuff====
+* https://developer.mozilla.org/de/docs/Web/API/Web_Audio_API
+* https://developer.mozilla.org/de/docs/Web/HTML/Canvas
+* https://tonejs.github.io/
-Using audio feature extraction, classification and segmentation one might be able to find similar audio clips in videos thus creating audio match cut.
+===Downloads===
+[[File:automata.zip]]
+Just open index.html with a modern browser