Using our voice or even any other random sound as an instrument is not really a new thing. Music producers often make synths by sampling their own voice, which is a fun approach to sound design. Moreover, beatboxers are capable of creating quite realistic drum sounds with their voice. But, have you ever wondered if it is possible to turn any sound into an actual instrument- like a trumpet, violin or such? Well, now you can!
Two teams within Google Research- Magenta and AIUX collaborated on creating Tone Transfer, which took over a year to develop. This little software allows you to transform everyday sounds into instruments, like the trumpet, violin or flute. The way the algorithm works is by transferring the formants of the selected instrument onto the sound recording that is fed into the software. The results vary, but are sometimes very accurate. However, even if they give a weird result, most of the time the newly synthesised sounds are quite cool for integration into electronic music.
The project is based on Magenta’s Differentiable Digital Signal Processing (DDSP) technology. AIUX helped make it user-friendly, so that it can be used by a wider audience, not just experienced developers. Their current online web app has a limited array of 4 instruments, with the possibility to upload your own and train the algorithm yourself. The algorithm was trained with 10-minute long samples for each of the instruments that are currently available.
There is still a long way to go and the research teams state that this is just the beginning of their plans. Tone transfer is not perfect yet, as it struggles with volume/dynamics of samples it is fed with. It also cannot superimpose attenuations of each specific instrument on the input sound, so it may not always sound realistic. The trumpet sounds the best with vocals. I presume it is because pushing out air into trumpets and while signing produce similar dynamics and volume changes.
This project is a great example of how useful machine learning can be in a multitude of situations. Software like the Tone Transfer could open up possibilities of unique live performances with generated instrument sounds that cannot be achieved with traditional instruments. Below you can watch a demonstration of how this software looks and works:
Sources:
https://www.inputmag.com/culture/google-tone-transfer-turns-every-noise-into-an-instrument
