TorchAudio | PyTorch Developer Day 2020

Key Takeaways

[Music] hello everyone my name is vincent kenville bel-air and i'm the tech lead for tortugio which is what i'm going to talk about today the goal of torturio is to provide building blocks to other researchers and engineers that allows them to bring research to production this way tortuga can accelerate the development of other libraries in the open source ecosystem tortujo is built around the following core functionalities the first functionality is io to read and save tensors from various file formats like mp3 wav flac and sphere we can also download and use common audio data sets where samples are loaded in parallel using torch multi-processing workers the second functionality is transforms for audio and signal processing such as spectrogram and fcc and resampling the transforms are provided as neural network modules in torture.transforms since the transforms are written using pure pi torch operations the computations can be done on the gpu and it can be compiled using torch grip the third is socks and quality compatibility socks and caudi are audio processing library written in c plus plus for socks we provide an interface to use their transforms for coldly we provide reading and writing of quality binary files as well as equivalent features like spectrogram and ms nf bank the final functionality is the distribution of models along with canonical example pipelines for distributed training for major tasks as i said the first set of functionality revolves around i o here's a small snippet using tortuoload and transform the waveform variable is a tensor which is read from file and the corresponding sample rate of the file is read as a scalar the torture to transform spectrogram is given an input parameter to configure its behavior it is then passed the input tensor which computes the spectrogram tensors as output what's special here that i want to highlight is that not only are the transform standard torch n module and so can be compiled using jit but the load function uses torch bindings and so can also be compiled and ported wherever jit is supported the goal is thus to make it possible to get an entire pipeline to be around in production easily we support several data sets for different tasks for instance library speech for speech recognition libritts for text to speech the next set of functionalities i mentioned is transforms as i said before they're written in pure pi torch and as such support batching torch grip and gpu here's another example since each transform is a torch and in module they can be combined in a standard sequential wrapper for convenient data augmentation here we take a spectrogram apply a random time stretch compute the complex norm apply a random frequency masking and a random time masking and then convert the amplitude to decibel frequency masking and time masking are part of spec augment which is what i'm illustrating in the image a band of frequency and another in time are randomly masked the code is divided in functionals that perform the computation and a transform which is an nn module that wraps each functional and keeps their state here i'm listing a few new functionals that we added recently you can see for instance mask along axis that is used within the tor the frequency and time masking we also have several bi-quad filters that are used in signal processing or voice activity detection operation to detect voice the next functionality is the interface with socks and quality for socks we offer a way of using their efficiency plus plus operations directly within pi torch in a torscriptable manner for instance here i'm applying a sequence of gain speed rate change pad and trim using apply effects sensor directly on the pi torch tensor for caldi torture provides a wrapper for torch audio transforms that mimics the flags provided to quality binaries you can also read arc and scp files through tor trojo so that the processed output of quality can be used within your torture dual program kaldi is used quite a lot in the audio community so we want to make it easy to interface with it the final set of functionalities that i want to talk about is the addition of models within the library for space recognition we added a training example pipeline for speech recognition that uses libre speech data set and the wave to letter model for text to speech we added a vocoder based on the wave rnn model along with an example training pipeline in the example folder that uses libre's tts data set for source separation we added the kovtas net model and an example training pipeline with the wall street journal zero mix dataset before finishing i would like to highlight a few features that are on our roadmap first we would like to include the quality pitch feature extraction due to demand from the community second we are interested in including a beam surge decoder interface this is especially useful for speech recognition application and finally another loss that has been requested by users is the addition of the rnn transducer loss to use and learn about pytorch you can visit pytorch.org audio it contains documentation about the api installation instructions tutorials and links to the github page we also have a new tutorial for the recognition of speech command have fun playing with it torture is compatible with linux mac os windows and supports python 3.6 and up just like pytorch thank you for watching you