12,297
edits
GMU:Autonomous-Collaborative/Hiyeon (view source)
Revision as of 16:31, 15 October 2019
, 15 October 2019no edit summary
No edit summary |
|||
| Line 16: | Line 16: | ||
Here are the steps how I thought it should work in the beginning. | Here are the steps how I thought it should work in the beginning. | ||
# The user taps on the drum | |||
# The piezo mic grabs the sound | |||
# The program detects the unique frequency of the sound | |||
# An equivalent drum sound is produced | |||
Following challenges immediately arose. | Following challenges immediately arose. | ||
* How to ensure the sounds being fed are unique enough to be classified into different patterns? | |||
* How to realize pedal movements? | |||
* How to allow multiple drums to be detected and played? | |||
* How to map the intensity and lasting of each output sound to the input sound? Would the texture of materials matter? | |||
===Implementation=== | ===Implementation=== | ||
I used the | I used the[https://github.com/wbrent/timbreID timbreID], a library for analyzing audio feature in Pure Data (Pd). In particular, its drum kit example already had the function to map the sample drum sounds to the real-time input sounds. | ||
===- Preparing the Instruments=== | ===- Preparing the Instruments=== | ||
| Line 48: | Line 42: | ||
[[File:2019-10-05 19.10.gif|400px]] | [[File:2019-10-05 19.10.gif|400px]] | ||
What happens here is that they are detected by ‘bark~’ in the ‘pd onsets’ sub-patch, which measures the amount of growth in bands of an input’s spectrum. Here I lowered the debounce setting to 50 ms for which it will be deaf. It was set as high as 200ms initially to prevent the sample playback to retrigger the playback, but it could be lowered to play in a bit quicker | What happens here is that they are detected by ‘bark~’ in the ‘pd onsets’ sub-patch, which measures the amount of growth in bands of an input’s spectrum. Here I lowered the debounce setting to 50 ms for which it will be deaf. It was set as high as 200ms initially to prevent the sample playback to retrigger the playback, but it could be lowered to play in a bit quicker rhythm by using the headphones. | ||
[[File:bark.png|400px]] | [[File:bark.png|400px]] | ||
| Line 66: | Line 60: | ||
Finally you can play the instruments by hitting one by one. This is because there is only one input channel at the moment. Notice how different instrument lead to playback of different sample. Here what happens is that the program tries to find the set of data previously trained that best matches the input sound. When the ‘id’ spigot is turned on, the cluster number will be printed out on the Pd’s window whenever the sample is played back. | Finally you can play the instruments by hitting one by one. This is because there is only one input channel at the moment. Notice how different instrument lead to playback of different sample. Here what happens is that the program tries to find the set of data previously trained that best matches the input sound. When the ‘id’ spigot is turned on, the cluster number will be printed out on the Pd’s window whenever the sample is played back. | ||
For the playback, I loaded my own drum samples to fit my mini drum set-up. | For the playback, I loaded my own drum samples to fit my mini drum set-up. | ||
I got each audio sample from the | I got each audio sample from the [https://sampleswap.org/filebrowser-new.php?d=DRUMS+%28FULL+KITS%29%2FDRUM+MACHINES%2F SampleSwap], an audio sample sharing site. Then I put them together into one audio file with even 1s interval, for it to be processed in the Pd. | ||
[[File:sampleswap.PNG|400px]] | [[File:sampleswap.PNG|400px]] | ||
| Line 77: | Line 71: | ||
===Limitations=== | ===Limitations=== | ||
==== Limited Timbral Differences==== | |||
Even with an effort to find materials with different enough textures to produce unique audio when tapped, some materials tend to fall into the same category of sounds. For example, the aluminium lid and the plastic gum container that I used for crash cymbal and open hi-hat respectively, sometimes were recognized to be in the same category. Obviously, classification gets more accurate as the number of audio items reduces. For this reason, I had to reduce the number of items, omitting the kick sound. | Even with an effort to find materials with different enough textures to produce unique audio when tapped, some materials tend to fall into the same category of sounds. For example, the aluminium lid and the plastic gum container that I used for crash cymbal and open hi-hat respectively, sometimes were recognized to be in the same category. Obviously, classification gets more accurate as the number of audio items reduces. For this reason, I had to reduce the number of items, omitting the kick sound. | ||
==== Lack of Pedal Movements==== | |||
This drum set, while being cute, cannot fully be used for practice because its missing pedals. I initially thought of introducing them by thumb tapping, but noticed that two thumb tap sounds would have exact same timbral sound and cannot be distinguished with the current single input setting. In the end, I realized it by attaching a wide aluminium juice lid onto the slipper and register the sound by tapping on the foot, but it cannot be used with the piezo mic setting which is a huge disadvantage. | This drum set, while being cute, cannot fully be used for practice because its missing pedals. I initially thought of introducing them by thumb tapping, but noticed that two thumb tap sounds would have exact same timbral sound and cannot be distinguished with the current single input setting. In the end, I realized it by attaching a wide aluminium juice lid onto the slipper and register the sound by tapping on the foot, but it cannot be used with the piezo mic setting which is a huge disadvantage. | ||
==== Singularity of an Input Sound==== | |||
Obviously, one of the biggest limitation when it comes to practicality is the fact that it can only process one sound at a time. Because you cannot play multiple drums at a time with some inherent latency in playback, it cannot quite produce useful beat in the end. This problem can be solved by introducing multiple piezo mics and feeding the inputs into different Pd programs. | Obviously, one of the biggest limitation when it comes to practicality is the fact that it can only process one sound at a time. Because you cannot play multiple drums at a time with some inherent latency in playback, it cannot quite produce useful beat in the end. This problem can be solved by introducing multiple piezo mics and feeding the inputs into different Pd programs. | ||
==== Lack of Ubiquity==== | |||
Due to the lack of time and technical issues, I had to process the patch on my laptop. It could have better fit the overall purpose of the course if I could utilize the Raspberry Pi, in the sense of ubiquity and collaboration of embedded systems. This can been done by installing the pd patch in a Raspberry Pi where the piezo mic is connected to its sound card, along with some sort of sound output device. | Due to the lack of time and technical issues, I had to process the patch on my laptop. It could have better fit the overall purpose of the course if I could utilize the Raspberry Pi, in the sense of ubiquity and collaboration of embedded systems. This can been done by installing the pd patch in a Raspberry Pi where the piezo mic is connected to its sound card, along with some sort of sound output device. | ||