AI for Music Composition

Key Takeaways

hello world it's SJ and can a computer make better music than you maybe it's close to doing so in this video we're going to talk about AI to generate music and I've got a great demo for you it's using deep learning to generate music and what we can do is we can change the genre of music by tuning these sliders let's let's take a little demo sample [Music] [Laughter] [Music] all right anyway um sh okay there we go so that's this that's the demo for this video it's a simple web app it's using a deep learning model in the backend to to generate these sounds that was pre-trained on a data set of uh monophonic piano notes and we'll talk about all of that right so we we have to get to where we need to be that means we have to start off with um hidden marov models then move to recurrent networks and finally get to uh generative adversarial networks which is the demo that I'm going to talk about at the end of this video code included of course because we love code if you don't code you should code all right so anyway so let's start with the history just one little one little snippet of pre uh pre software so before there was any software uh we were music is older than language and before there was any softare Ware the first automatic music came from nature so Windchimes for example or the or ancient Greek wind powerered iolan harp soures Japanese water instruments etc etc etc but in the 1700s the first automatic music became algorithmic so there was a German game that generated short piano compositions from fragments with choices made by dice just want to give you a little fun little you know pre- software before we get into the good stuff just so you know that okay so that was then now we're getting into the 19 1900s and then onto onto now the marov chain very very cool model it's used in weather prediction it's used in it's been used in speech recognition uh it's been used in a lot of fields it's been it's been used in all of the Sciences for sure biology physics marov chains have been used extensively but anyway um what it's it's a very simple basic idea and I have this little slide right here to to show uh a little bit about Markov chains that I'm taking from Clemson basically the motivation is in this case it's it's always about predicting the next data in a sequence that data could be a word it could be a musical note it could be some speech but it's all about predicting the next data in a sequence and what it is is it's it's it's a basically a graph that shows probabilities of going from one point to another and by points those points could be data points like strings for example right so if you have the word what if I say the word what there's a certain probability that the next word will be is there's a certain probability the next word will be the' and based on those probabilities this the computer can Traverse that graph to maximize for those probabilities so that the most likely word is said or generated and that's the same thing goes for music and these have been used like I said in weather for example right so if we have three states Sunny rainy and cloudy and depending on the inputs it could be any of those things what a hidden Markov model does is it models all of those State transition probabilities from Sunny to to rainy to whatever it is right so let's say you know if it's rainy then there's a 60% chance it's going to be it's going to be cloudy right and it's going to be 60% less that it's going to be sunny right so there is there is some relationship between all these states that can be modeled as probability values and that's what that's what it's for right and so what it comes down to is just basic multiplication right because what a hidden Markov model does is it doesn't just it doesn't generate completely novel data it generates data that is a subsection of the existing data right so the the best way to explain what I mean by that is for us to look at a bit of a bit of code right so let's look at some code for this hidden markup model it's right here this 25 line snippet of python code is a hidden marov model okay so in this case we have two words okay these are strings um what we what we're going to do is we're going to create a table out of all these words and then what we're going to do is we're going to say let's generate some output by learning those those probability values between those words so it's a learning that the likelihood that one word will come after another and it's basically just picking a random word from the next word and we can print that out right so this is a very very simple model and it's it's it's just basic multiplication is what it boils down to multiplication from some existing set of data um and then learning the probabilities of these words just one by one by one right so U one example of an AR that used a marov chain to generate music was in 1958 his name was yanis zanakis and he used it in his 1958 composition and a logi which we can listen to thanks to the power of the [Music] internet yeah so that sounds terrible but anyway uh that was what he did so that's that's what it was it was it was it was modeling the probabilities of a of a note occurring after a certain sequence of notes that's what it did and so that basic idea of modeling the probability of a note occurring next is kind of like the constant of using AI to generate music right because when we have some piece of music it's a sequence of notes right all music is sequential and there is some likelihood that a certain note will come next after a certain set of notes have already occurred and we can model that with a hidden marov model but a hidden marov model can only produce sub sequences that also exist in the original data but what if we want to extrapolate Beyond those exact subsequences and create really novel um data that's generated and that's where recurrent networks come into play recurrent neural networks in 1989 the first attempt to generate music with the recurrent net was limited by their short-term coherence so someone tried to uh generate music uh in the style of Bach using recurrent networks in 1989 and uh it was limited because it could only grasp short-term sequences right so if if Bach had a a huge piece with you know four different uh sections it would stop it would it would forget about the beginning part and the beginning was very soft and the second part was very strong and the third part was like mellow and right what a recurrent network does normally without any kind of variation to it is it will so normally a neural network will learn U the hidden State based on every new data point that it's fed right every new data point that's it and it's learning it's it's it's like think of it like clay it's like molding its internal representation to become robust against all variations of that input data whether it's uh images of stop signs whether it's uh musical notes it's it's learning some abstraction of all of that so then using that abstraction we can generate new notes but what we current networks will do is in every time step it's not just learning based on the new data point it's learning based on the previously learned abstraction so at every time step that abstraction is fed not just a new data point but an an older version of itself so there's a recurrence there and what this does is it it let it lets it lets us learn from sequences the problem is that a recurrent Network can't learn long-term sequences because of what's called The Vanishing gradient problem I'm not going to talk about the vanishing gradient but just search Vanishing gradient SJ on on YouTube but basically we want a way to trap that memory that's being back propagated across time to get a little technical um and the solution to that was let's you know tldr the solution was to use what's called a long short-term memory Network an lstm Network and let's listen to a bit of this uh recurrent Network [Music] music okay yeah so that was better we have to admit that was better than before not necessarily Han Zimmer status but it was better um so so Doug E was one of the first in 2002 now we're like 16 years ago so we're getting closer to switch from using a standard recurrent Network to what's called a long shortterm memory Network and this improved the architecture it improved what the what the model could generate now Doug works for the magenta team at Google brain magenta's been developing a code for for generating music using machine learning and they they are really on the bleeding edge of this stuff magenta has made so many cool different little products that we can we can view in the in the browser um but the thing is even though recurrent networks have been around for a while it wasn't very much common knowledge until Andre karpathy posted the unreasonable effectiveness of recurrent networks in May 2015 which was more than a decade later which is incredible if you think about it the power of good documentation can move the masses to try some code what Andre used his recurrent Network on was generating Shakespearean text so it would train on Shakespeare text and then it would be able to generate text in the style of what it had just trained on Shakespeare uh so what I have here is a very simple lstm Network okay so for just for us to get an idea of what it looks like right so we are defining our nonlinearities which are our activation functions which makes sure our Network can learn both nonlinear and linear functions we then Define a training data set we have our hyperparameters listed up here we initialize the weights of our three- layer neural network and then inside of our training Loop we start training this thing we perform the simple addition problem we then update the Deltas to show that hey there's a certain change in the weight values and now that we've computed a certain change in the weight values we can we can update those weights by performing an optimization technique called back propagation also known as gradient descent see my video back propagation in 5 minutes what I need is basically just like a tree to just connect all of my videos across the because all of these Topics by the way if you don't understand just search SJ and then the name of that topic and I promise you I have a video on that so when we're updating those weights we're making sure that the gradient is not Vanishing that it's trapped properly inside of each of those synapses and it's being back propagated properly as well um but yeah so moving forward so September 2016 two years ago now we're getting really really close deep mine published their seminal paper on what's what they called wavenet okay so wavenet was a an an architecture that generated state-of-the-art human sounding speech okay so no model up to that point had made speech that sounded so humanlike and people were very surprised because no one had done that and and here's the real key to here's a real kicker to all of this wavenet was not a recurrent Network it was a convolutional network what right so convolutional networks are used for learning from images and recurrent networks are used for for learning from sequences but what Deep Mind did is they use a convolutional network to learn to process images by treating time like a spatial Dimension which is very interesting stuff anyway wavenet was was it required massive amounts of gpus like seriously massive and then later on someone made a fast version of wavenet which you can find on GitHub just search fast wavenet uh but then fast forward a year later magenta built on top of wavenets to create n synth which we can try from the browser it's basically a soundmaker that lets us try out different instruments and combine them together and make all new all new sounds out of it so up to this point there are some there are some there are certain important questions in this space that we have to answer right how do we decide on a proper representation of music what music data should we use whose music counts box or an EDM artist you know do we want to learn from the entire documented history of music with the vague goal of producing something similar or something novel or should we try to construct entire compositions or or to improvise with us do we want this to replace us do we want it to augment us right that's that's a major question for us to ask but when it comes to startups in this space and this is a part of the AI for business Series this is why I'm I'm I'm teaching you this stuff because we there's definitely a space to create businesses that that help artists improve their sounds using AI to help consumers be able to become artists in a way that they couldn't before by giving them the tools to make orchestral sounds for example that would that would require a hundred human people you know in the past and you could just do it with a web app now Juke deck is one example of a startup that lets anybody create music you can select the mood style Temple and in length and he uses AI to do this in the background and there's a there's a subscription model you can pay you can get your first five songs a month free and then you pay 7 USD a track for 150 bucks creators can even buy the exclusive copyright for their song and these are songs that are generated by AI which means their bottom line is zero which means that they are making some money this is a great business model genius and and there's there's a there's huge room in the space for for more of that iiva is another example right so what they do is they do music composition it was founded last year in London and it's it's taught to compose classical music which they already have clients film film directors advertising agencies game studios Amper is another example I talked about this with tan Southern in a you know a couple months now ago uh but that's what she used that was a tool that she used to to help generate music it was started by a film composer who wanted to make music that was more of a collaboration between humans and machines it does basically the same thing IBM's Watson so this is not a startup this is a big company but even there are going into the space and they're using their cognitive technology by the way IBM guys let's stop using the word cognitive let's start using the word you know deep learning or AI or machine learning there's nothing cognitive about what you're doing anyway don't hate me IBM I still love you okay so you know anyway just let's stop let's stop trying to make it seem like you know the human brain is like inside of some server that's not where we are right now anyway where were we so architecture okay so that's the history up until now a couple years ago Ian Goodfellow released a paper called generative adversarial networks that allows for a novel way to generate data and I think that and this is now now we're going into now we're going from objective to subjective right so I'm I'm telling you my opinion now my opinion is that the way forward is to use a generative adversarial Network to generate sounds that would give us better results than a recurrent Network an lstm Network a hidden Markov model a convolutional network so even better than wavenet because Gans are truly a a new technology that they have not properly been harnessed Nvidia harnessed them pretty well for this paper on generating faces like in real time was very cool but Gans are notoriously hard to train there's a lot that can be improved mov D the space is moving so fast um and what I have here by the way is this GitHub link you should definitely check it out it's called music generation with deep learning huge huge collection of resources for you to use if you want to learn more about the space papers from all across the board from all sorts of countries blog posts code uh conferences and workshops related to music generation with AI applications that you can play with in the browser super useful stuff definitely check it out but I want to talk about a gan right now okay so a Dam has two parts it's got a generator and it's got a discriminator and both of these both of these are neural networks okay so one's job is to generate data the other's job is to look at what the generator generated and discriminate it and say this is real or this is fake it's binary zero or one that's what it does and so with the power and ease of Kos we can create a generator in one line that's right one line of code we could create a neural network Define all those parameters and call it a generator okay so that's our generator what it will do so in this example it's generating an image in the style of this handwritten digit but we can generate music sounds right so what the generator will do is it will take in that input image okay it will learn some latent representation that's a collection of numbers in this in this image you can see that and then it will it will vary it a little bit so it generates something entirely new and this is a series of operations right and it's going to start off totally random right like how would it know how to vary this input data it doesn't know it's dumb it's just like oh let me just multiply by X and then divide okay the the real learning comes from this when it generates that new sample the discriminator will say that's real or it's fake at first the discriminator will be able to tell immediately that this is real and this is fake because the discriminator has been trained on the training data as well so it will know oh that's not a real sequence of notes that's fake that because I know that that sequence of notes is a part of the training data okay so talking about music when that discriminator makes that uh classification and says it's fake then the generator will need to update itself so using back propagation so both networks are back propagated so that the optimization technique will slowly shift those weight values and so the operations will slowly shift as well so the next time that some data is going through the generator it's going to be more likely to generate something that is harder to discern whether it's real or fake and so the whole point of this process I'm not like I'm not like I'm like not even reading the notes because I know Gan so well now but the whole point of this process is so that the generator becomes so good at generating something that the discriminator cannot tell if it's real or if it's fake and you could think of it as like a cat and mouse game you could think of it as like a police and a counterfeit eventually the counterfeit gets so good that the police cannot tell if the money is real or fake and that's what we're trying to do the point of generative adversarial network is not the discriminator the point is the generator that's what we want when the whole thing is trained end to end we remove the discriminator we take that generator and we just start generating in our case it would be pieces of music musical notes once we have those pieces of music that is our song that we can then play now it's easier to generate music that is monophonic right so the these are uh single notes piano notes you know whatever harp notes but most music comes in the form of chords right collections of notes polyphonic and that's a little bit harder to train but I think that Gans are the way forward to to to make that happen Okay so that's how that goes I have this sample here called Gan music and basically if you want an idea of what it looks like programmatically here's what it is right so these are our parameters for both D which is the discriminator and G which is the generator and these are just weight values and bias Valu input times weight add a bias activate that's how every neural network works just remember that so now we have our generator and our discriminator we Define them right here here's our generator here's a discriminator we we're using tensorflow to create both of them we can plot them to see the difference we can create a sample and say okay here's the real and here's the fake we have two loss functions one for the real one one for the fake and then we solve them both using gradient descent this is our optimization strategy and at the end we can print it out we can plot it it's very simple this was like 136 lines of code with tensorflow to create a gan and you just drag and drop your music data set and boom you're good um so yeah this um this demo it's on GitHub check it out as well check out my links I have some great links for you in the video description there's a lot of potential to use AI to create services for both artists and consumers regarding music and really if if you want to extrapolate to all sorts of entertainment everything I've talked about here can be applied to text to you know essays to songs to poetry to images to video even right all of this all of this is sequential data and because all of this is sequential data we can we can we can generate it we can learn from it and we can generate new sequences and this can help all sorts of people there's a huge business use case for this and uh now is a time because all these tools are just now starting to be democratized we now have access to gpus Cloud providers we now have access to algorithms we now have access to data sets we now have access to education right so all this is being democratized it's your responsibility to do something about it um that's the end of my Spiel and I hope you find this video useful Please Subscribe for more programming videos and for now I've got to listen to some music AI music so thanks for watching