Composing Bach-Like Music with Neural Networks in Python (Tensorflow)

Key Takeaways

Today we're going to train a neural network with convolutional as well as LSTM layers from scratch and TensorFlow to learn how to generate piano music similar to that of Johan Sebastian Bach. This is a pretty interesting and somewhat advanced machine learning project and I think you can learn a lot by implementing it. So let us get right into it. It's not a game. It's a red. [Music] >> Now, before we jump right into the coding, I would like to give you a motivational preview up front. This is the final result. This is what you're working towards in this video. By the way, the comments are my comments. These are not chat GBT comments. And the idea is that we're training a neural network on a data set that contains music from Johan Sebastian Bach. And the idea is we give it some seat chords. So this here is 5 seconds of music. That is actual uh music from Bach. So if I run this, there you go. I think this is a bit loud. So let me turn this down a bit. Um that is now the actual song. So this is the actual music from him. And that is what our AI generates based on the seat music. Now whether you liked it or not, it's definitely not random. It has some sort of intelligence in it because if you generate some random music as a comparison here, this is what it sounds like. Yeah, doesn't sound much like music. So, this is what we're going to build in this video today. All right. So, we're going to train a neural network in TensorFlow from scratch to learn how to compose music similar to that of Johan Sebastian Bach. Now I want to give credit here to this book called handson machine learning with scikitlearn kas and tensorflow. I think it's mirrored but yeah you can uh google the book and you can see what it is about. This is where I got this example from. Now the data set also exists independently of this book but this is where I got the example exercise from and also part of the solution. I do think I have my own solution, but the architecture of my neural network is very similar to that of the uh solution to the exercise. So, I don't want to claim that I came up with everything here on my own. I was heavily inspired to say the least by the exercise itself and also by the solution. But I'm going to guide you through it step by step. So, what we're going to do is we're going to first of all open our terminal and navigate to a directory where we want to be working. in my case is always the tutorial directory in here. Now I'm going to create my uh development environment. You can do that in a number of different ways. You can just install the packages to your basic Python installation on your system by saying pip or pip 3 install and then the package names that we're going to use today. I'm not going to do that. I'm going to use a virtual environment. In particular, I'm going to use uv as a package manager here. You don't have to do that. You can also say something like Python or Python 3-m ven to create a virtual environment if you're not interested in using uh a Rustbased package manager. In my case, since I'm using UV, I have to say UV init. And then instead of saying pip or pip 3 install, I say UV at. But the rest is exactly the same. You say pip or pip 3 install. And then the packages, I say UV at. And then the packages. We're going to need a bunch of the packages here today. We're going to need pandas. We're going to need uh TensorFlow obviously. We're going to need numpy. We're also going to need music 21 for showing the generated music and playing it. And I think that's actually it. So pandas for working with the data with a data frame, numpy for working with arrays, music 21 for displaying and playing the music. And TensorFlow for the whole machine learning stuff. So just say UV at in my case. Again, in your case, pip or pip 3 install if you don't using if you're not using UV. And in addition to that, I'm also going to install Jupyter Lab because this is my development environment of choice when it comes to machine learning. It's just interactive Python notebooks where you can run individual cells. I would highly recommend you also use the same development environment. So in my case, UV add Jupiter lab in addition to that. Same works with pip as well. And then I'm going to activate my virtual environment here and run Jupiter and LAP with a space in between. This is going to open up my development environment. And in addition to all of this, now what I need is also the data set. So this opens up now in my browser. I'm going to move my terminal to a different workspace. This is now how I can create a IPython notebook. I can click on Python 3 here. And now I can run individual cells. If you're seeing a Jupyter notebook or an IPython notebook for the first time, the idea is I don't have to run the whole code at once. I can run individual cells. I can say x= 10 in here. Then I can say print X in another cell. Then I can say X equals 20. Then I can go back and print X here. And now it changes to 20. So these work as individual cells which is very useful when working with data, looking at stuff, exploring stuff, pre-processing, training, evaluating and so on. So I would recommend you work with something like this. Let me store this as main.ipb. And the most important thing here of course now is the data. And for this we're going to use a Kaggle data set. You will find a link to it in the description down below. And all you have to do here is you have to download the data set. So you just click on download download data set as zip. And then you can unzip it. In my case here, I already have it in the directory called corals. I hope this is how it's pronounced. By the way, we're going to encounter a bit of musical theory here. So notes and uh chords and corral and all that. If I mispronounce something, if I say something that is not entirely true in the context of music theory, I know I have a keyboard uh next to me. I'm not good when it comes to music. So, I don't know how to play the keyboard, at least yet. I don't know how to play the guitar or anything. I have played around with the instruments, but I don't know a lot about music theory. So, if I get something wrong, uh yeah, please excuse that because this is mainly focusing on a machine learning part. So maybe I'm going to mispronounce something or explain something in a suboptimal way, let's say. So we basically have our data set here split into training, validation, and testing data. Each of these directories contains a bunch of CSV files. We can take a look at them by just saying import pandas as PD. Then we can say dataf frames equal to pd read csv and then corral train and corral 0000 for example. And when I look at that you will see that we have time steps here as the index. So time step 0 1 2 3 and so on. And then we have four notes here as columns. Now, I'm not sure if this is correct when I call this a chord, but basically four notes are always played together uh at any time step. So, we get these four notes played together. Then we get these four notes played together, which in this case is the same thing four times. Uh but then we get this uh set of four notes here played together. And that results in the music that you heard in the preview. That is how the data sets are structured. Now, what we need to do is we need to get this data. We need to pre-process it in a way that our neural network can train on it. So that as a result of that it can also generate data like this. So it can also generate music like this where we have four notes every second being played together and this results in music that sounds kind of good or similar to that of uh Bach. So I'm going to start here by loading the data. I'm going to import OS and I'm going to say that the train files are equal to sorted. And now we're going to do a list comprehension in here. OS path join and we're going to go of course to corals. Then we're going to go to train and then we're going to go to a file foss list directory and here again ospath join corals train but only if the file ends with csv. So only if f.ends with csv. So these are now the train files. I can copy that. I can paste that. I can change that to be actually let me use substitution here with vimb binding. So I'm going to say train becomes test. So we basically just change this to test files. Corals test corals test. And here we can do the same thing with the validation. So train slashv validg and now we renamed the occurrences here. So these are now just the file names. I can take a look at that and see these are just the paths to the individual CSV files. And now we need to actually load them with another list comprehension. So I'm going to say train data is equal to list comprehension of PD read CSV f values to list for f in train files. Now we can do the same substitution here as before train test and valid. And now we have our data loaded in list format. Now one thing we need to understand here is how the data is structured. So I'm going to use a markdown field to explain that. Basically our notes are numbers. So train data or as you can see here we have 74 70 65. These are notes and the lowest note we have is actually 36. So 36 is equal to a C1 note. And then we also have the highest note which is 81. And that is an A5 note. Everything in between is C D E F G A B C D E F G and so on increasing the number. And we also have a special symbol or character or number you could say for silence. This is zero. So zero is silence. Everything else is a note from C1 to A5. And we can also listen to that. We can listen to what it sounds like by using the music 21 package. So I can say from music 21 import stream and court and then I can say here corral is equal let's say to train data let's go with 20 for example just one of the pieces that we have here in the training data and now we can say s is equal to stream and now for row in corral we're going to say sappend and here now we're going to append four notes as a chord so I'm going to Say here court do. Court and in here a list comprehension n for n in row ifn. So if n is not zero and then quarter length is equal to 1. And finally we're going to say s show medi. And that is going to allow us to listen to this music in our notebook. So now we're listening to an actual piece from Bach. If I click on it, that's the idea. So, what you hear every second is four notes being played at the same time. And our goal now is to train a neural network to generate music that sounds similar to the music that we find here in the training data. Now, for this, we first need to do some pre-processing. So I'm going to use this as a markdown cell here to say we're now starting with a pre-processing. And there's a couple of things that we want to do. First of all, we need to prepare our data for training. So we need to have some input and some output. And also we want to rescale the range of our features. So now we have values from 36 to 81 and also a zero for silence. I would like to have this from 1 to 46 which is the same range. So the same distance but we start at one. So this is the rescaling that we're going to do. Also, we want to have the training data where I'm going to look at a certain number of notes and then predict how the sequence continues. So I'm going to basically get an input sequence and produce an output sequence that is shifted by one in our case. And we're going to do that with a bunch of parameters. So I want to start here by saying import numpy as np. Then I'm going to say here min node max node is equal to 3681. This is important for rescaling. And now we need to decide on the window size, on the window offset, and on the batch size. So the window size basically means how many notes am I going to look at to make the decision to predict the next note. The window offset is how much am I going to jump in order to look at the next sequence. Um, so I'm going to say here window size is or let's say window size, window offset, and batch size in one. We're going to say 32 for the window size. Look at 32 notes which divided by four is eight chords and the window offset is 16 in our case. So we're going to jump 16 notes for the next window. This is to minimize overlap. And I also want to use 32 as the batch size here. So we're using eight chords for context, you could say for window size and four chords for offset. So we're going to define now a function make X Y. So make the training data basically given our corals as input and we're going to start by creating the windows. So the windows are going to be taking our corals and just looking at 32 or 33 because we need to also have the prediction in there. Uh looking at the 32 initial nodes and the one after that and doing that with a specific offset so with a specific step size. So what we're going to do here is we're going to have a list comprehension for an individual corral. We're going to start at I and go to I plus window size + one because we have our prediction as well. So our next element we're going to do that for corral in corals for i in range starting at zero going up until the length of corals minus the window size so that we don't go out of bounds and the step size is the window offset and of course it's not length corral but length sleep. So the length of the individual corral not the number of corrals in the data set. Then we take all of this and we turn it into a numpy array. So, np array of windows d type will be equal to integer because we only have integer values. And now we're going to do the rescaling. So, we want to take every value uh that is not zero and we want to scale it down by subtracting the minimum note from it. So, 36 is going to become or actually subtracting the minimum note and adding one. So, 36 is going to become 1, 37 is going to become two and so on. So we're going to do that by saying data is equal to np where data is equal to zero. There we're going to keep it at zero. And otherwise we're going to say data minus minimum note + one. And to make sure that this actually stays in this range, we're going to say clip clip the data with zero as the lower boundary and the upper boundary is going to be max node minus min node + one. Now we're going to flatten our data by saying data reshape data shape zero and then negative -1. And what we're going to return now is for the x data, we're going to return the sequence of all the notes that we look at. And then we're going to return uh the same number of notes shifted by one. So I'm going to say here flat everything and then everything up until the last element. So except for the last element and then everything starting from the second element. So one until the end. This basically means if I have something like 1 2 3 4 5 6 then I would get 1 2 3 4 5 as uh training data and as a or as X data and the Y data would be 2 3 4 5 6. That is the structure of our data set here. So then we can do this quite easily. We can say x train y train is equal to make xy or actually since we have a sequence is y as well we're going to do y uh uppercase as well. So make xy is going to be train data here and then we do the same thing here for for testing and also for validation. Now we can take a look at the shape of our data. So if I look at the training data what do I have here? I have 3,111 windows with 131 nodes each. And if I do the same thing for the testing data, you will see or not for testing data, sorry, if I do the same thing for the Y data, uh you're going to see that the shape is the same. It's just shifted by one. Now we get to the interesting part which is training the model. So let me add another markdown cell here where I'm going to say training the model. And here we're going to start with some TensorFlow imports. We're going to say from TensorFlow.caras domodels import sequential from TensorFlow.caras.layers. We're going to import the convolutional layered one-dimensional. So one-dimensional convolutional layer, the dense layer, the embedding layer. I'm going to explain what each of them are for when we get to the architecture. The LSTM layer and the dropout layer. And finally the batch normalization. And then we say from tensorflow.caras.optimizers we're going to go with anatom optimizer. So let us now define the architecture. We're going to say the model is a simple sequential model. And now we're going to add layers to it. So the first layer is going to be an embedding layer which means we're going to allow the model to learn vector representation of the notes. So the notes are just integers, but some notes may be closer together conceptually even though they're further apart numerically. So we could have uh the same tone. I think this is the correct term. I'm not sure. Again, this is musical theory. But you have notes that if they are a certain distance apart, I think eight if I'm not mistaken, an octave or something like that. uh but basically you have nodes that are a certain distance apart that belong together that are close together and it makes sense to allow the model to learn the representation in vector space instead of just using the raw numbers. So what we're going to do is we're going to say model at and then I'm going to add an embedding layer. So embedding we're going to say that the input dimensions of this embedding layer is going to be 47 because we have 47 different notes 46 and the silence the output dimension so the complexity of the vector space the dimensionality of the vector space is going to be five. The reasoning behind this is that we don't really need to have a highdimensional vector space because we only have 47 nodes. That's not too complicated. And for the input shape we're going to just say none. So this layer is for learning representations of the input of the notes. Then we're going to add a convolutional layer with one dimension. So one-dimensional convolutional layer and this basically slides a one-dimensional window across the data which is used for extracting temporal patterns. It can also be done in parallel which is something that you cannot do with RNN layers or LSTM layers. So convolutional layers can be uh used in parallel. And what we're going to do here for the first layer is we're going to use 32 kernels or filters. So windows that we slide across the data to extract temporal patterns. We're going to use a kernel size of two. So we're going to have just um a window of size two that we slide over the data to extract patterns. We do that 32 times. So we're able to train 32 filters to find patterns. Uh we're going to say padding is equal to causal. So we cannot look into the future. So we can only focus on past data. So we are not biased by future data. And then we're also going to say the activation this is classic is a rectified linear unit. So just a simple value where everything you get in is what you everything you get out is what you get in unless the number is negative then you're going to get zero all the time just to break the linearity in a simple way. So that is our convolutional layer one. Now we're also going to add a batch normalization layer. This just keeps the scale and everything consistent and counteracts exploding and vanishing gradients. So batch normalization layer is going to be put in between the next convolutional layer and the first one. So after this we're going to have another convolutional layer. This time we're going to increase the number of uh kernels and filters. The reason we do that is because we're going to get more and more long-term patterns that we extract. we're going to see more and more types of patterns because we get a longer um context you could say or a longer time span that we can examine and because of that we don't want to have a bottleneck in terms of the number of filters. We're going to increase the number of filters all the time. So 48 for this one and we're also going to add something called the dilation rate. The dilation rate basically means oh sorry this is the wrong the wrong place. It's in here. dilation rate is going to be two. And basically, this means that we're going to uh look back not just to the previous note, we're going to look back two notes. So, we're going to have a longer um a longer time span between the notes that we look at. And we're going to do that, we're going to also increase that with the next layer to four and then to eight. And in a combined way or combining all this, we're going to have a window or a context of 32 nodes even though we don't have to explode in the number of parameters. So this is a nice trick that we can use to look further back and extract more long-term patterns or mid-term patterns you could say. So this is what we do here. And now we can basically copy this structure. We're going to use another batch normalization layer. We're going to increase the number here again 64. Kernel size remains the same. and everything else too. The only thing that changes is dilation rate and number of filters. So here we have four. Then we can copy that. We can paste here. We're going to go with 96. And then the dilation rate here is going to be 8. And finally we're going to add another one batch normalization. And then 128. And the dilation rate is going to be 16. Combining all these now together we have we we're able to connect nodes that are 32 nodes apart or 32 positions apart and we don't have to explode the number of parameters and all of this is parallelizable. So this is a very nice architecture to extract short to mid-range patterns. Now with these features extracted with all these patterns extracted what we're going to do is we're going to add some regularization so we cannot overly depend on individual patterns or features. So, we're going to say model at dropout, just a very tiny dropout of 0.05. And then we're going to combine all of these short to mid-range patterns, and we're going to extract long-term patterns by using an LSTM layer. So, model at LSTM, and in here now, we're going to say 256 and return return sequences is equal to true. So this is our recurrent layer that is going to extract long-term patterns or long range patterns. And finally, we need to have a layer that projects all this into our solution space or into uh the final result that we're interested in, which is a soft max activation for each individual node. So we want to have a probability distribution knowing okay, how likely is it that this is the correct next note for every single note. So we're going to say here model.add add dense layer with the number of nodes. So in our case that is 47 same as the input dimensions and then we're going to say activation this time is softmax. So softmax is just used for taking the logits the individual raw values and uh yeah basically turning them into values so that all of them scale up uh add up to one. So that we have 100% total and every single value in the output layer is the percentage the probability how likely is it that this particular node is the next node to be played. So we can then say model summary to take a look at that. So I can run this and this is our uh neural network. We have almost half a million parameters here. And all we need to do now to train this is we need to say optimizer is equal to the natam optimizer. The learning rate is going to be equal to 1e minus 3. And then we're going to say model compile loss is equal to sparse categorical cross entropy. Optimizer is equal to optimizer and then metrics is equal to accuracy like this. And then finally I need to close the bracket here. Finally, model fit and we're going to fit on X-rain. Y train, we're going to train for 20 epochs. You can also increase this if you see that the loss is going down all the time. Then maybe you have more capacity here for training. So you can increase this to 30, 40, 50 epochs. And we also want to look at validation data. The idea of validation data is we don't want to use it for testing because we're using it to already tweak some stuff maybe hyperparameters and see what the best configuration is but you want to have something to evaluate on which is not the training data because you don't want to overfitit the training training data you want to actually generalize to unseen data. So X valid Y valid uh uppercase Y of course batch size is going to be equal to batch size. And now I can run this and this is going to train our model if of course I provide the correct name. And if I don't misspell categorical. There you go. Now it's training. I'm going to skip that part and I'm going to come back to you once the training is done. So as you can see now our model is done training. It has a validation loss of 0.637. What's interesting here is that you can see the validation loss is increasing since epoch 12. So here it was at the lowest point. Then it went up and up and up again. Whereas the training loss decreases. This is a sign of overfitting. We're getting too focused on the training data. We're not generalizing well anymore. So that would probably be the optimal place to stop here. But it's still okay. So you can see this doesn't require us to train for more epochs. If you see the validation loss going down and down and down, maybe you want to keep training. In this case, it's okay to stop here. And now the question is how can we generate music from this model? The model is now trained. It can accurately predict or somewhat accurately here 80% around 80% accurately predict the next note. How do we actually get music that we can listen to out of this model? For this, I'm going to define two functions. One is going to be a helper function that is going to sample from a probability because we don't just always want to go with the quote unquote correct or most likely prediction because then we can get into repetitive patterns that can sound very boring or even horrible after a time if we get stuck in a loop. So what we're going to do is we're going to do sampling based on the probabilities that we get from the model. And for this I'm going to define a function sample next note based on probabilities that we get from the model. So first of all we're going to say probabilities is equal to np as array of props d type is going to be equal to float and then we're going to calculate the probability sum because for our sampling with numpy we need to have probabilities that add up to one. This is not always the case. So we're going to say probability sum is equal to probabilities dosum. And if there is a problem with the probability sum if it is negative, if it's not finite, if it's zero, we're going to just return the arcmax. So the largest uh the the note with the largest probability otherwise we're going to do random sampling. So I'm going to say if the probability sum is less than or equal to zero or if it is not finite. So, np is finite, not np is finite probability sum. Then we're going to just return int np arc max uh for the probabilities. It's as simple as that. And otherwise, we're going to do random sampling. We're going to say first of all, normalize the probabilities to make sure they add up to one. So, divide it equal to so just divide all the probabilities by the probability sum. This is going to normalize them. Not props but prop sum. And then we just return np random choice length probabilities. So we want to have the position. We don't want to have the number the probability but we want to have the index because that is going to also allow us to pick the note. And the probability for this sampling here is going to be probabilities. And this is why we have to add this uh have this add up to one because if you pass something here a list of probabilities that don't add up to one you're going to get a problem. You're going to get an error from numpy. So this is why we need that. That is our first method or function. And now we're going to also get the second function which is the generate corral function. So this function here is going to take in the model some seed chords. So some starting music and the length of the piece that we want to generate. So how long should the song be? So what we're going to do is we're going to start with a token sequence which is going to be the seat chords. This is going to be what we start with. It's going to be np array seat chords. Data type is still int because we're just dealing with notes. And what we're going to do now is we're going to step by step extend this token sequence to include new nodes generated by the model. However, we need to do some pre-processing here because our model is trained on a different range. Remember, we rescaled the individual note value. So, we need to do that here as well. We need to say token sequence is equal to np where and then the token sequence where is it zero there? put it to zero and in all other places just rescale it. So do token sequence minus min not node that we still have from before + one and then we're going to say token sequence is equal to token sequence reshape 1 -1 and now we're going to generate the chords note by note. So the length specifies the number of chords that we want to generate. So we have to take it time four to generate as many notes. So we're going to say here four underscore since we don't need the control variable here in range length time 4. This many nodes have to be generated. We're going to do it by saying next token probabilities. This is what we get now from the model. This is where the model does a prediction. So we say model.predict and we feed it the last tokens that were already generated. So we're going to say here token sequence and then I'm going to say here 0 -1. Based on these next token probabilities that we get, we're going to do the sampling. So we're going to say here the next token that was chosen by randomness is going to be sample next note next token probabilities. And then we're going to just concatenate the current sequence with this new token. So we're going to say here np concatenate token sequence um for that in a list. So token sequence in a list and then also next token in a list here and then access equals 1. So this basically adds the new token to the token sequence and we can continue generating with that next token being already part of uh of the ground truth or of the initial stuff that we feed into the model and then we're going to say here finally token sequence is equal to npware we need to basically scale it back so that we can listen to the music we need to map it back to MIDI here we uh subtract minote and add one now what we need to do is we need to go back to the range of 63 to 81. So we're going to say where token sequence is equal to zero keep it at zero and uh in all other cases add the minote and subtract one basically reversing the operation from above and then we just return the token sequence reshaped so that we have quartz. So -14 that is the generation function that is going to use the sampling function and with this now we can generate some music. So all we need for that is we need some seed chords. I'm going to say seed chords is equal to test data. So this is unseen data for our model. Let's go with instance two. The first eight notes are going to be our seat chords. We can listen to them. We can say coral as before is equal to seed chords. Then s is equal to stream.stream and then for row in corell s.append cord.cord and here n for n in row if n is not zero basically and a quarter length is equal to one. And finally just s.ow show medi and of course it makes sense to close all brackets. So this is the seed chord that we listen to. So that is definitely going to be part of our generated music. This is the beginning and everything after that is going to be what our model generates. We can also listen to the ground truth. So, we can also uh remove the eight here and just listen to the whole thing [Music] and so on. And now all we need to do to generate new music is we need to take the seat chords again. So, colon 8 here, the first eight notes, the first two chords. And now we just say new corral is equal to generate corral based on the model with the seat chords and the length is 56. Let's say by the way we can also remove all of this by setting the generation or the prediction here to verbose equals zero. Let's actually do that. So verbose equals zero. Make sure we don't get this printed all the time. Let's let me rerun this. Now you can see there's no output until we're done. There you go. And now if I print new corral, you can see this is what it looks like. This is our generated music here. And now we can listen to it. So I can basically copy this here. The only thing I have to change is what we use as the corral. So the corral here is going to be new corral.2 list. And the rest stays the same. Now we can listen to what our model generated. [Music] [Applause] This actually doesn't sound too bad as I already showed you in the preview. I'm going to copy paste here now again the random generation code. So here you have generate random corral and you have also basically the same code for playing the output. If I generate some random music, you're going to see that this really sounds differently than what we have here. This sounds like someone is just randomly smashing a keyboard. And this sounds like music. There you go. So this is how you train a neural network from scratch in TensorFlow using convolutional layers using dropout layer. I mean that's just for regularization using an LSDM layer using also an embedding layer to learn the representation. This is how you train a neural network to generate music like Johan Sebastian Bach. So that's it for today's video. I hope you enjoyed it and hope you learned something. Let me know in the comment section down below if you like these machine learning projects that are a little bit more comprehensive going into a little bit more detail. Let me know if you also have other ideas for projects, other things that you would like me to implement in a video. And besides that, of course, don't forget to subscribe and hit the notification bell to not miss a single future video for free. Other than that, thank you much for watching. See you in the next video and bye.