{"id":4242,"date":"2021-03-14T04:00:00","date_gmt":"2021-03-14T04:00:00","guid":{"rendered":"http:\/\/desres20.netornot.at\/?p=4242"},"modified":"2021-03-21T18:08:42","modified_gmt":"2021-03-21T18:08:42","slug":"processing-audio-data-for-use-in-machine-learning-with-python","status":"publish","type":"post","link":"http:\/\/desres20.netornot.at\/?p=4242","title":{"rendered":"Processing Audio Data for Use in Machine Learning with Python"},"content":{"rendered":"\n

I am currently working on a project where I am using machine learning to generate audio samples. One of the steps involved is pre-processing.<\/p>\n\n\n\n

What is pre-processing?<\/h3>\n\n\n\n

Pre-processing is a process where input data somehow gets modified to be more handleable. An easy everyday life example would be packing items in boxes to allow for easier storing. In my case, I use pre-processing to make sure all audio samples are equal before further working with them. By equal, in this case I mean same sample rate, same file type, same length and same time of peak. This is important because having a huge mess of samples makes it much harder for the algorithm to learn the dataset and not just return random noise but actually similar samples.<\/p>\n\n\n\n

The Code: Step by step<\/h3>\n\n\n\n

First, we need some samples to work with. Once downloaded and stored somewhere we need to specify a path. I import os<\/span> to store the path like so:<\/p>\n

 <\/span><\/p>\n

import<\/span> os<\/span><\/p>\n

 <\/span><\/p>\n

PATH<\/span> <\/span>=<\/span> <\/span>r<\/span><\/em>"C:\/Samples"<\/span><\/p>\n

DIR<\/span> <\/span>=<\/span> os<\/span>.<\/span>listdir<\/span>(<\/span> <\/span>PATH<\/span> <\/span>)<\/span><\/p>\n

 <\/span><\/p>\n


 Since we are already declaring constants, we can add the following:<\/p>\n

ATTACK_SEC<\/span> <\/span>=<\/span> <\/span>0.1<\/span><\/p>\n

LENGTH_SEC<\/span> <\/span>=<\/span> <\/span>0.4<\/span><\/p>\n

SAMPLERATE<\/span> <\/span>=<\/span> <\/span>44100<\/span><\/p>\n


 These are the “settings” for our pre-processing script. The values depend strongly on our data so when programming this on your own, try to figure out yourself what makes sense and what does not.<\/p>\n

Instead of ATTACK_SEC<\/span> we could use ATTACK_SAMPLES<\/span> as well, but I prefer to calculate the length in samples from the data above:<\/p>\n

import<\/span> numpy <\/span>as<\/span> np<\/span><\/p>\n

 <\/span><\/p>\n

attack_samples <\/span>=<\/span> <\/span>int<\/span>(<\/span>np<\/span>.<\/span>round<\/span>(<\/span>ATTACK_SEC<\/span> <\/span>*<\/span> <\/span>SAMPLERATE<\/span>,<\/span> <\/span>0<\/span>))<\/span><\/p>\n

length_samples <\/span>=<\/span> <\/span>int<\/span>(<\/span>np<\/span>.<\/span>round<\/span>(<\/span>LENGTH_SEC<\/span> <\/span>*<\/span> <\/span>SAMPLERATE<\/span>,<\/span> <\/span>0<\/span>))<\/span><\/p>\n


 One last thing: Since we usually do not want to do the pre-processing only once, form now on everything will run in a for<\/span>-loop:<\/p>\n

for<\/span> <\/span>file<\/span> <\/span>in<\/span> <\/span>DIR<\/span>:<\/span><\/p>\n


 Because we used the os<\/span> import to store the path, every file in the directory can now simply accessed by the file variable.<\/p>\n

Now the actual pre-processing begins. First, we make sure that we get a 2D array whether it is a stereo file or a mono file. Then we can resample the audio file with librosa<\/span>.<\/p>\n

import<\/span> librosa<\/span><\/p>\n

import<\/span> soundfile <\/span>as<\/span> sf<\/span><\/p>\n

 <\/span><\/p>\n

 <\/span><\/p>\n

try<\/span>:<\/span><\/p>\n

data<\/span>,<\/span> samplerate <\/span>=<\/span> sf<\/span>.<\/span>read<\/span>(<\/span>PATH<\/span> <\/span>+<\/span> <\/span>"\/"<\/span> <\/span>+<\/span> <\/span>file<\/span>,<\/span> <\/span>always_2d<\/span> <\/span>=<\/span> <\/span>True<\/span>)<\/span><\/p>\n

except<\/span>:<\/span><\/p>\n

continue<\/span><\/p>\n

 <\/span><\/p>\n

data <\/span>=<\/span> data<\/span>[:,<\/span> <\/span>0<\/span>]<\/span><\/p>\n

sample <\/span>=<\/span> librosa<\/span>.<\/span>resample<\/span>(<\/span>data<\/span>,<\/span> <\/span>orig_sr<\/span>=<\/span>samplerate<\/span>,<\/span> <\/span>target_sr<\/span>=<\/span>SAMPLERATE<\/span>)<\/span><\/p>\n


 The next step is to detect a peak and to align it to a fixed time. The time-to-peak is set by our constant ATTACK_SEC<\/span> and the actual peak time can be found with numpy<\/span>’s argmax. Now we only need to compare the two values and do different things depending on which is bigger:<\/p>\n

peak_timestamp <\/span>=<\/span> np<\/span>.<\/span>argmax<\/span>(<\/span>np<\/span>.<\/span>abs<\/span>(<\/span>sample<\/span>))<\/span><\/p>\n

 <\/span><\/p>\n

if<\/span> <\/span>(<\/span>peak_timestamp <\/span>><\/span> attack_samples<\/span>):<\/span><\/p>\n

new_start <\/span>=<\/span> peak_timestamp <\/span>–<\/span> attack_samples<\/span><\/p>\n

processed_sample <\/span>=<\/span> sample<\/span>[<\/span>new_start<\/span>:]<\/span><\/p>\n

 <\/span><\/p>\n

elif<\/span> <\/span>(<\/span>peak_timestamp <\/span><<\/span> attack_samples<\/span>):<\/span><\/p>\n

gap_to_start <\/span>=<\/span> attack_samples <\/span>–<\/span> peak_timestamp<\/span><\/p>\n

processed_sample <\/span>=<\/span> np<\/span>.<\/span>pad<\/span>(<\/span>sample<\/span>,<\/span> <\/span>pad_width<\/span>=[<\/span>gap_to_start<\/span>,<\/span> <\/span>0<\/span>])<\/span><\/p>\n

 <\/span><\/p>\n

else<\/span>:<\/span><\/p>\n

processed_sample <\/span>=<\/span> sample<\/span><\/p>\n


 And now we do something very similar but this time with the LENGTH_SEC<\/span> constant:<\/p>\n

if<\/span> <\/span>(<\/span>processed_sample<\/span>.<\/span>shape<\/span>[<\/span>0<\/span>]<\/span> <\/span>><\/span> length_samples<\/span>):<\/span><\/p>\n

processed_sample <\/span>=<\/span> processed_sample<\/span>[:<\/span>length_samples<\/span>]<\/span><\/p>\n

 <\/span><\/p>\n

elif<\/span> <\/span>(<\/span>processed_sample<\/span>.<\/span>shape<\/span>[<\/span>0<\/span>]<\/span> <\/span><<\/span> length_samples<\/span>):<\/span><\/p>\n

cut_length <\/span>=<\/span> length_samples <\/span>–<\/span> processed_sample<\/span>.<\/span>shape<\/span>[<\/span>0<\/span>]<\/span><\/p>\n

processed_sample <\/span>=<\/span> np<\/span>.<\/span>pad<\/span>(<\/span>processed_sample<\/span>,<\/span> <\/span>pad_width<\/span>=[<\/span>0<\/span>,<\/span> cut_length<\/span>])<\/span><\/p>\n

 <\/span><\/p>\n

else<\/span>:<\/span><\/p>\n

processed_sample <\/span>=<\/span> processed_sample<\/span><\/p>\n


 Note that we use the :<\/span> operator to cut away parts of the samples and np<\/span>.<\/span>pad<\/span>()<\/span> to add silence to either the beginning or the end (which is defined by the location of the 0<\/span> in pad_width<\/span>=[]<\/span>).<\/p>\n

With this the pre-processing is done. This script can be hooked into another program right away, which means you are done. But there is something more we can do. The following addition lets us preview the samples and the processed samples both via a plot and by just playing them:<\/p>\n

import<\/span> sounddevice <\/span>as<\/span> sd<\/span><\/p>\n

import<\/span> time<\/span><\/p>\n

import<\/span> matplotlib<\/span>.<\/span>pyplot <\/span>as<\/span> plt<\/span><\/p>\n

 <\/span><\/em><\/p>\n

#PLOT & PLAY<\/span><\/em><\/p>\n

 <\/span><\/p>\n

plt<\/span>.<\/span>plot<\/span>(<\/span>sample<\/span>)<\/span><\/p>\n

plt<\/span>.<\/span>show<\/span>()<\/span><\/p>\n

time<\/span>.<\/span>sleep<\/span>(<\/span>0.2<\/span>)<\/span><\/p>\n

sd<\/span>.<\/span>play<\/span>(<\/span>sample<\/span>)<\/span><\/p>\n

sd<\/span>.<\/span>wait<\/span>()<\/span><\/p>\n

 <\/span><\/p>\n

plt<\/span>.<\/span>plot<\/span>(<\/span>processed_sample<\/span>)<\/span><\/p>\n

plt<\/span>.<\/span>show<\/span>()<\/span><\/p>\n

time<\/span>.<\/span>sleep<\/span>(<\/span>0.2<\/span>)<\/span><\/p>\n

sd<\/span>.<\/span>play<\/span>(<\/span>processed_sample<\/span>)<\/span><\/p>\n

sd<\/span>.<\/span>wait<\/span>()<\/span><\/p>\n


 Alternatively, we can also just save the files somewhere using soundfile<\/span>:<\/p>\n

sf<\/span>.<\/span>write<\/span>(<\/span>os<\/span>.<\/span>path<\/span>.<\/span>join<\/span>(<\/span>os<\/span>.<\/span>path<\/span>.<\/span>join<\/span>(<\/span>PATH<\/span>,<\/span> <\/span>"preprocessed"<\/span>),<\/span> <\/span>file<\/span>),<\/span> processed_sample<\/span>,<\/span> <\/span>SAMPLERATE<\/span>,<\/span> <\/span>subtype<\/span>=<\/span>'FLOAT'<\/span>)<\/span><\/p>\n


 And now we are really done. If you have any comments or suggestions leave them down below!<\/p>\n","protected":false},"excerpt":{"rendered":"

I am currently working on a project where I am using machine learning to generate audio samples. One of the steps involved is pre-processing. What is pre-processing? Pre-processing is a process where input data somehow gets modified to be more handleable. An easy everyday life example would be packing items in boxes to allow for<\/p>\n