Cartesian CV Sequencer with Intentional Interface: Difference between revisions

Latest revision as of 11:43, 2 September 2020

Acoustic Interfaces
Instructor: Clemens Wegener
Credits: 6 ECTS, 3 SWS

Project by Joel Schaefer

INTRODUCTION

In the course "Acoustic Interfaces" we wrote a C++ Machine-Learning library based on a K-Clustering Algorithm. It's purpose is to classify Microphone Inputs into different frequency based classes. After this cluster (training) process the algorithm is able to classify similiar microphone input in realtime. The algorithm is running on a teensy 4.0 with an audio shield.

I have to say, that it was quite hard for me to come up with an idea for the creative, musical use of this technique. As i dont wanted to implement this algorithm for the classic purpose of sound detection issues, i came up with a more abstract idea of use.

IDEA

My idea was a intention based gesture/timbre detection through the piezo microphone input. This input is expressed through Midi Notes and muscial scales. For example the aggresive touch on the interface surface should express in a, somehow, corresponding sequence of notes (or melody).

So the project i came up with is basically a sequencer.

INSPIRATION

In thought about what kind of sequencer i wanted to build, i was very inspired by the eurorack-sequencer by "Make Noise", called "Rene´ ". It's a sequencer based on motion inside a cartesian coordinate system. What i found interesting in this, is the fact that this way of sequencing doesn't work with one main tempo which is driving the sequences, but instead with more tempos (depending on the amount of dimensions). Each tempo or clock is driving the sequences only on one axis. That means when clocks (or gates) of different speeds (or ryhtms) are fed into the sequencer, the outcoming sequence can be very diverse. With an normal( 1-dimensional) sequencer, such or similiar results from only one sequence would need modulation of pitch values or radomnization in the rhythm patterns.

AUDIENCE

The audience this project is aiming at musicians that are using analog synthesizers. Especially at artists that use their modular/analog gear for live and improvisation performances, because the idea of the intentional interface is clearly standing for the spontaneous and imediate creation of melodies and patterns.

APPROACH

So i decided to design my own little grid-based sequencer. Im calling it grid-based because it only has 2 dimensions. My plan was to build it in a hardware format, with analog parts. That means i didn't wanted to build a Midi-Sequencer but instead a digital CV/Gate-sequencer for analog gear. For the UI Part i was inspired by the Teensy-based, opensource eurorack module "Ornament & Crime" by Patrick Dowling (aka pld), Max Stadler (aka mxmxmx) and Tim Churches (aka bennelong.bicyclist). This one is known for its many functions and flexibility. But the whole UI is based on an OLED Display, two rotary encoders and 2 buttons. I planned to do it similliar. That means also an OLED display and 2 rotary encoders but only one button. Concercing Analog In and Outputs i build two audio inputs for the 2 clocks or gates that drive the X-Axis and the Y-Axis. On the output side i integrated two outputs (0-4 Volts = Range of 4 Octaves) with external 12 Bit DAC Boards to get a proper resolution, voltage range and precision. One outputs the CV signal, the other one the Gate signals.

PROCESS OF WORK

To the process of building and coding this project i can honestly say i've learned a lot, but also that i heavily underestimated the work and knowledge that was needed. Unfortunately i got to face many problems in the coding of the sequencer library, some are minor bugs, some are stability issues, other are hardware debouncing issues. This and also the lack of knowledge of programming in C++ lead to the fact that the actual circuit and algorithm i build/wrote for the teensy can't do yet what my initial idea was. To be concrete - - the connection to our ML-Library isn't there yet and therefore the whole "intentional interface" is missing - the maximum size of the grid is 4, that means 16 steps (i wanted to make it flexible so that the grid size can be chosen) - i wasnt able to debounce the rotary encoders enough for acceptable results (so i replaced them by 3 usual potentiometers), that makes the menu diving rather impractical - the quantizer part is missing, so it outputs just all 12 semitones per octave - extra functions like playback direction, randomnization features, etc. are missing - overall running stability isn't good

In principle the algorithm is working as can be seen in the 2nd video at the top, but i would have needed much more time during the semester to get closer to what i wanted. So i decided to demonstrate my project kind of theoretically, in the sense that im using my PureData Patch to show a shortened version of my idea. ( Video 1 above is a musical demonstration) In that version most is working as imagined.

There is: - 6x6 Grid - Note Mapping from Teensy Serial Stream - Quantizer with a lot of musical Scales - ability to change playback direction of both axis separately

Whats missing is: - a good musical representation of sound timbres clustered by the teensy library - Mapping of clustered classes onto musical scales - flexible grid size - randomness fx

REFLECTION

I think at the end i have to say that its quite hard to define a fixed mapping for the representation of the still very abstract data the ml-algorithm is giving at this point. The direct frequency band mapping of the gestures captured by the piezo mic onto the whole sequence has also a kind of random character. As well as the direct mapping of certain frequency points onto certain steps gives a very randomly sounding result. The intention, intensity or timbre can not really be heard by the corresponding melody changes.

CONCLUSION

I consider my idea to map the raw fft-data from the teensy audio input onto note values as kind of a bad idea, but i think the mapping of the classification results onto musical scales has some potential. I guess not really in a fixed manner, but maybe if it would be able for the user to easily customize the connections between the different gestures and materials on the interface with its own musical interpretations. This would need a very sophisticated UI system and a very open algorithm structure for making it practical for users.

@@ Line 4: / Line 4: @@
 Project by Joel Schaefer
+ [[File:2020-08-30 19.27.46.jpg|PD-Proto|600px]]
 {{#ev:youtube|yQiy3417tNg|560|left}}
@@ Line 12: / Line 18: @@
 |-
+ [[File:pdseq.PNG|PD-Proto|600px]]
+|-
- [[File:freqbandmapping.PNG|PD-Proto|600px]]
- [[File:pdseq.PNG|PD-Proto|600px]]
- [[File:CVInterface1.jpeg|Prototype|600px]]
- [[File:CVInterface3.jpg |Prototype|600px]]
 |-
 |}
   <br/>
   <br/>
+INTRODUCTION
 In the course "Acoustic Interfaces" we wrote a C++ Machine-Learning library based on a K-Clustering Algorithm. It's purpose is to classify Microphone Inputs into different frequency based classes. After this cluster (training) process the algorithm is able to classify similiar microphone input in realtime. The algorithm is running on a teensy 4.0 with an audio shield.
 I have to say, that it was quite hard for me to come up with an idea for the creative, musical use of this technique. As i dont wanted to implement this algorithm for the classic purpose of sound detection issues, i came up with a more abstract idea of use.
+IDEA
 My idea was a intention based gesture/timbre detection through the piezo microphone input. This input is expressed through Midi Notes and muscial scales. For example the aggresive touch on the interface surface should express in a, somehow, corresponding sequence of notes (or melody).
@@ Line 31: / Line 42: @@
 So the project i came up with is basically a sequencer.
-In thinking about what kind of sequencer i wanted to build, i was very inspired by the sequencer eurorack-module by "Make Noise", which is called "Rene´ ". It's a sequencer based on motion inside a cartesian coordinate system. What i found interesting in this is the fact that this way of sequencing dont have one main tempo which is driving the sequences, but instead more tempos (depending on the amount of dimensions). Each tempo or clock is driving the sequences only on one axis. That means when clocks (or gates) of different speeds (or ryhtms) are fed into the sequencer, the outcoming sequence can be very diverse. With an normal( 1-dimensional) sequencer, such or similiar results from only one sequence would need modulation of pitch values or radomnization in the rhythm patterns.
+INSPIRATION
-So i decided to design my own little grid-based sequencer. Im calling it grid-based because it only has 2 dimensions. My plan was to build it in a hardware format, with analog parts. That means i didn't wanted to build a Midi-Sequencer but instead a digital CV/Gate-sequencer for analog gear. For the UI Part i was inspired by the Teensy-based, opensource eurorack module "Ornament & Crime" by Patrick Dowling (aka pld), Max Stadler (aka mxmxmx) and Tim Churches (aka bennelong.bicyclist). This one is known for its many functions and flexibility. But the whole UI is based on an OLED Display, two rotary encoders and 2 buttons. I planned to do it similliar. That means also an OLED display and 2 rotary encoders but only one button. Concercing Analog In and Outputs i build two audio inputs for the 2 clocks or gates that drive the X-Axis and the Y-Axis. On the output side i integrated two outputs (0-4 Volts = Range of 4 Octaves) with external 12 Bit DAC Boards to get a proper resolution, voltage range and precision. One outputs the CV signal, the other one the Gate signals.
+In thought about what kind of sequencer i wanted to build, i was very inspired by the eurorack-sequencer by "Make Noise", called "Rene´ ". It's a sequencer based on motion inside a cartesian coordinate system. What i found interesting in this, is the fact that this way of sequencing doesn't work with one main tempo which is driving the sequences, but instead with more tempos (depending on the amount of dimensions). Each tempo or clock is driving the sequences only on one axis. That means when clocks (or gates) of different speeds (or ryhtms) are fed into the sequencer, the outcoming sequence can be very diverse. With an normal( 1-dimensional) sequencer, such or similiar results from only one sequence would need modulation of pitch values or radomnization in the rhythm patterns.
-In the process of prototyping i faced heavy problems with the rotary encoders
+AUDIENCE
+The audience this project is aiming at musicians that are using analog synthesizers. Especially at artists that use their modular/analog gear for live and improvisation performances, because the idea of the intentional interface is clearly standing for the spontaneous and imediate creation of melodies and patterns.
+APPROACH
+So i decided to design my own little grid-based sequencer. Im calling it grid-based because it only has 2 dimensions. My plan was to build it in a hardware format, with analog parts. That means i didn't wanted to build a Midi-Sequencer but instead a digital CV/Gate-sequencer for analog gear. For the UI Part i was inspired by the Teensy-based, opensource eurorack module "Ornament & Crime" by Patrick Dowling (aka pld), Max Stadler (aka mxmxmx) and Tim Churches (aka bennelong.bicyclist). This one is known for its many functions and flexibility. But the whole UI is based on an OLED Display, two rotary encoders and 2 buttons. I planned to do it similliar. That means also an OLED display and 2 rotary encoders but only one button. Concercing Analog In and Outputs i build two audio inputs for the 2 clocks or gates that drive the X-Axis and the Y-Axis. On the output side i integrated two outputs (0-4 Volts = Range of 4 Octaves) with external 12 Bit DAC Boards to get a proper resolution, voltage range and precision. One outputs the CV signal, the other one the Gate signals.
+PROCESS OF WORK
+To the process of building and coding this project i can honestly say i've learned a lot, but also that i heavily underestimated the work and knowledge that was needed.
+Unfortunately i got to face many problems in the coding of the sequencer library, some are minor bugs, some are stability issues, other are hardware debouncing issues. This and also the lack of knowledge of programming in C++ lead to the fact that the actual circuit and algorithm i build/wrote for the teensy can't do yet what my initial idea was. To be concrete -
+- the connection to our ML-Library isn't there yet and therefore the whole "intentional interface" is missing
+- the maximum size of the grid is 4, that means 16 steps (i wanted to make it flexible so that the grid size can be chosen)
+- i wasnt able to debounce the rotary encoders enough for acceptable results (so i replaced them by 3 usual potentiometers), that makes the menu diving rather impractical
+- the quantizer part is missing, so it outputs just all 12 semitones per octave
+- extra functions like playback direction, randomnization features, etc. are missing
+- overall running stability isn't good
-So we we decided to use an Arduino Uno for our project. The starting point was this great, creative common released, open hardware device we found on github.
+In principle the algorithm is working as can be seen in the 2nd video at the top, but i would have needed much more time during the semester to get closer to what i wanted. So i decided to demonstrate my project kind of theoretically, in the sense that im using my PureData Patch to show a shortened version of my idea. ( Video 1 above is a musical demonstration)
+In that version most is working as imagined.
-https://github.com/eegsynth/eegsynth/tree/master/hardware/usb2cvgate_4channel
-Its designed by Robert Oostenveld for eegsynth.org.
-This is the original Schematic:
-{| border="1"
-|-
-|-
- [[File:Eegsynthschematics.png |600px]]
-|-
-|}
-It converts digital data from the arduino (connected through SDI) over an external 12Bit DAC to 0-4V and scales the range over an OpAmp to 0-10V. This device is able to do this on 4 channels, for us 2 was enough, so we cutted away the half of the original circuit. Also we skipped the 4 Gate Outputs. So what we were having then was basically one 12Bit DAC (MCP4822), one OpAmp (TL072) and one DC-DC Converter (SIM 0512D) and the needed resitors and capacitors which can be seen in the original schematic.
-The only thing that was missing for us in terms of Data Conversion was CV-Inputs. So an ADC was needed. As the resolution of the arduino 10Bit ADC's was to low for us, we buyed a 16Bit 4ch ADC. The ADS1115 Breakout Board by Adafruit. We connected this device also over SDI to the arduino. We installed 2 more Audio-Jacks for the input signals, set a little voltage divider with 100k resistors behind that Jack to scale the incoming Voltage to an acceptable Level for the ADC(max 5.5V). For sure this isn't the best way to do it, especially without an extra OpAmp as buffer and diode based protection circuit, but for this prototype it worked fine for us.
-So now, when we got the cv in- and outputs working it was time to get joystick and potentiometers. We just buyed a cheap Joystick and some 10k Potentiometers. In the first place we were disappointed by the Joystick. The accuracy was extremely low and the spring inside the joystick made it even worse for the usage of accurate voltage generation. So we decided to use 2 channels of the external 16 Bit ADC for the YX-Axis inputs for making the joystick more accurate. And we desoldered the Joystick Circuit, drilled a whole inside and took out the spring. Luckily we orded a 5-Pack of Joysticks instead of just one, because 2 Joysticks got broken in this procedure. The third one was working perfect.
- <br/>
- [[File:Joystickhack.jpeg|600px]] <br/>
- <br/>
-Then we decided to use the whole potential the arduino is giving us with its 6 internal ADC's and connected 4 Knob Potentiometers and 2 Slider Potentiometers to it. These should control different values of our functions on the arduino.
-Though we didn't had much functions written yet, we knew we would make different modes. So we got some LED's (1 blue, 3 red) for indicating different modes and also we took a switch button breakout board for changing modes. Another switch button we had available through the joystick itself, because this one had a push function.
-Hardware fully set up, now it's time for the digtal stuff.
-Our basic idea responds into the simple function to express the XY-Axis of the Joystick as 2 Values that are linearly mapped onto Out CV1 and Out CV2. So one axis gave us values between 0 and ~25000, which we simply mapped onto 0-10V. The following was to make a function which is summing the two axis into one value. We simply wrote a average function which, when activated, sums the the two Joystick values and maps them once again to 0-10V. We called this "Sum-Mode" and its accesible through pushing the joystick button one time. The State of these two Modes is indicated through the blue LED (on - SumMode [Out CV 1 = Out CV 2] , off - NormalMode [Out CV 1 = X-Axis ; Out CV2 = Y-Axis]). So the second CV Output becomes a copy of the first when the SumMode is active. So this Mode could also be used as a multiple.
-To not get users confused its important to say that these are global modi, which both work in the same way, no matter which of the following (local) modes is chosen.
-These are the Local Modes so far: <br/>
-. Sine LFO <br/>
-. Sample and Hold internal <br/>
-. Simplex Noise <br/>
-. Sample and Hold external <br/>
-<br/>
-<br/>
-<br/>
- [[File:Mode1.PNG|LFOSINE|600px]]
-<br/>
-<br/>
- [[File:Mode2.PNG|intSH|600px]]
-<br/>
-<br/>
- [[File:Mode3.PNG|Simplex|600px]]
-<br/>
-<br/>
- [[File:Mode4.png|extSH|600px]]
-<br/>
-The idea of these modes is to mix the voltage generated by the Joystick with different functions. Or, in other words, to make the joystick controlling an offset to different functions. Of course, if the Joystick is placed in the left-down corner the offset value is 0 on both axis'. That means if the amount value in any Mode is completely up, the output voltage is completely generated by the function.
-Conclusion <br/>
-This circuit is really a raw prototype, so there are many things to mention.
-On the hardware side we want to add an analog offset for the output Voltages. We think that would make sense because then we would be able to fade between the range between 0V to 10V and -5V to 5V, which definetely is useful for the use in modular systems. But as the interface got already pretty packed up with potentiometers and (digital) offset functionalities, we we will implement this feature with a simple switch to change the output range between bipolar and unipolar.
-Its extremely important to add a voltage limiting circuit with diodes to both CV/Clock Inputs because right now a voltage over 11V or below -1V could damage the ADC board. Though signals over 10V are pretty uncommon in modular synthesizers and eurorack, bipolar signals from -5V to 5V or even -10V to 10V are not.
-So for the harmless and free use of the CV input there need to be such circuit. As well as an additional OpAmp Buffer for those inputs.
-Then there need to be functionality added to the CV Input, because right now its only possible to use 1 Input as Gate Input for clocking the Sample Hold external Mode. But there are many uses for such Input. One could just be voltage conversion, so when the device is connected over usb, the, now digital data could be send via serial connection to other software as "Processing". Another way would be to implement OSC protocol to the code and make this incoming Voltage available as OSC data for the reuse by other software applications.
-Another way to use the CV input would be the modulation of internal parameters. For example the rate of the Sine LFO in that Mode(1) or the bitReduction Amount Parameter of the Simplex Noise Mode(3).
-Or, simply to add an offset by the joystick to an incoming signal.
-At this point it's right to mention that the parts we choose for our circuit are not really fitting each other. Because when we receive a incoming signal at 16Bits, process it in any way, and then send it out at 12Bits,  there is an enourmous amount of quality loss because of the resolution drop. So, perspectively it makes much more sense to use the same Resolution for the DAC's and ADC's. Or at least an higher Resolution on the DAC then on the ADC. So in the future we will exchange the current 12Bit DAC by an 16Bit DAC.
-Then, shortly before the final presentation we realized that two of our potentiometers weren't working. On the picture at the top of this page can only 1 still be seen, the free space in the right corner of the circuit was actually holded for the 4. Potentiometer. But as we figured out that the problem was laying inside two of the analog Inputs of the arduino itself, we decided to remove the 4. one for now. As we were having an indifference between the digital functionality and the possibilities our hardware setup gave us at this point, it wasn't a problem. The lack of more sophisticated functions was more connected to the fact that it took us too much time to write the functions. The lack of knowledge in writing code for such purposes was actually the biggest problem that occured in the realization of this project. Because our both amateurish coding knowledge doesn't include bitwise operators at all. And so doesn't our coding approach in this project. That means we used a ton of global variables to provide our functions working. This results in 80% cpu usage with really not that many or complex features yet. That is not practicible for the future at all. So the optimization of the code is really the most important thing for further developing this circuit.
+There is:
+- 6x6 Grid
+- Note Mapping from Teensy Serial Stream
+- Quantizer with a lot of musical Scales
+- ability to change playback direction of both axis separately
-After that said, we want to name and explain some functionalities we want to add in the future.
+Whats missing is:
-<br/>
+- a good musical representation of sound timbres clustered by the teensy library
-<br/>
+- Mapping of clustered classes onto musical scales
-Extend existing functions:<br/>
+- flexible grid size
-*General
+- randomness fx
--higher Range of available Speed for Audio Range Modulation Purposes <br/>
--Modulation of Parameters through CV in <br/>
--simultaneous Modes <br/>
-*Sine LFO
+REFLECTION
--more Shapes by using Wavetables
-*Sample & Hold
+I think at the end i have to say that its quite hard to define a fixed mapping for the representation of the still very abstract data the ml-algorithm is giving at this point. The direct frequency band mapping of the gestures captured by the piezo mic onto the whole sequence has also a kind of random character. As well as the direct mapping of certain frequency points onto certain steps gives a very randomly sounding result. The intention, intensity or timbre can not really be heard by the corresponding melody changes.
--Interpolation Amount for fading into Smoothed Random
-<br/>
-<br/>
-Add new functions:
-<br/>
-* "Quantizator" <br/>
-The, not yet existing "special knob" in the upper right corner of the circuit was intended for that. This function should not be another Mode which has to be selected like the other Modes. It should more be something like a filter at the end of the processing chain. When turned down, the output voltage isn't not affected. When turning up, the output voltage got slowy quantized into semitones. When it's halfway turned, the Output should spit out 12 Values per 1 Voltage, so 120 values or "notes" in the full range. That means it can play analog synthesizers musically over 10 octaves. When this "special knob" got turned above the half, it slowy fades the 12 semitones per 1 Volt into less and specific ones. By this, the device is able to turn any signal, no matter if generated, processed or just incoming into a musical scale. Which scale exactly shall be also choosable by the user over an extra button which switches between different scales.
-<br/>
+CONCLUSION
-*Envelopes <br/>
-Just a classic clockable AD-Envelope generator. Also loopable, when no Gate Signal is received. As expected, Decay time shall be modulatable by CV input.
-<br/>
-<br/>
-<br/>
-Further things:
+I consider my idea to map the raw fft-data from the teensy audio input onto note values as kind of a bad idea, but i think the mapping of the classification results onto musical scales has some potential. I guess not really in a fixed manner, but maybe if it would be able for the user to easily customize the connections between the different gestures and materials on the interface with its own musical interpretations. This would need a very sophisticated UI system and a very open algorithm structure for making it practical for users.
-<br/>
-*USB Audio CODEC<br/>
-To use this device directly without OSC or Serial data transfers with other Software like "CV Tools", like the Expert Sleeper Devices we mentioned in the beginning, it must be recognized as an audio interface.
-<br/>
-*Panel Design<br/>
-*Eurorack Format and Power Supply<br/>