Store Sales Prediction in Python - Time Series Machine Learning Project

Key Takeaways

Today we're going to go through the full machine learning process. To be precise, we're going to do time series prediction using one of the classic Kaggle data sets. Now, when you do the first Kaggle challenge for classification, usually you go with a Titanic data set. When you do the first regression, you usually go with something like house prediction, house price prediction, and when you do time series, you go with the store sales data set. So, you want to predict the store sales in different stores using different categories of items. And that is what we're going to do in this video today. We're going to do that with PyTorch to be precise with a temporal convolutional neural network. So we're going to use one-dimensional convolutional layers to do this time series prediction and we're going to have this encoder decoder architecture. So we're going to take a look at the last 120 days to predict the next 16 days which is what we're going to apply to the test data. Then we're going to make predictions and that is what we're going to upload to Kaggle to get a score. This is a competition. So I think you can learn a lot today. If you like this video let me know by hitting a like button and subscribing. But now let us get right into it. [music] >> All right. So I already mentioned it in the introduction. We're going to use the store sales data set from Kaggle, the one for time series forecasting here. This is also a competition. This is a permanent competition. So it doesn't have an expiration date. You can submit your predictions. How this basically works is you can go to the data tab and there you can see what the data set looks like. a bunch of CSV files. You have a train CSV which contains the training information obviously. Then you have a test CSV which has the same structure but it doesn't have the sales data because that's your job to predict the sales data. And then you can see what the format of the submission should look like. You have the IDs which are just the numbers that we have here for the individual rows. And we just have to submit our predicted prices. And then once we do that, we're going to get a score in the leaderboard. So, if you scroll down here, I should be somewhere around 104 with my prepared code. We're not going to overengineer this. We're not going to include all the information that we could because I want to keep this video at a level where most people watching can follow it, where we can get results quickly without overengineering, without including all the possible information and doing feature engineering for an hour. I just want to focus on the time series data itself. So, what we're going to do precisely is we're going to take the train CSV data. We're going to focus on the different stores, the different categories, and we're just going to do time series prediction based on the sales data. So, we're going to take in the last 120 days for example, and we're going to produce the next 16 days. Why 16 days? Because the days that we have to predict here in test CSV are 16 days for all these different stores and categories. That is the basic idea. So, what you need to do to get started is you need to download the data set. You can click on download all here using Kaggle. This is going to open up this dialogue here for saving the file. I'm going to save it in my tutorial directory. And there we're going to unpack it. So I can also open this up here. I can click on uh extract here in my case. Then I have this data here which is a bunch of CSV files. I'm going to rename this directory to be called data. And now we can start setting up the environment. We're going to work with a Jupyter notebook. To be precise, I'm going to work with Jupyter Lab, which is just a different, more advanced, more sophisticated, I would say, environment for working with Jupyter notebooks than just the ordinary notebooks. And we're going to use the command line to create our project. So, I'm going to zoom in here. I'm going to go to my tutorial directory. And I'm personally going to use UV. So, you can go and say pip or pip 3 install, whatever you want. You can use your VN virtual environment, whatever. You can use your base installation of Python. There's no need to copy my setup. If you want to use my setup, you just have to install UV. So pip or pip 3 install UV and then instead of doing pip install, you can just do UV at. But for this, you need to create first a project. So UV innit is the first command. This creates our tutorial project here. And then I'm going to say UV at which is exactly the same as saying pip or pip 3 install. For example, pandas, numpy, and so on. All the packages we're going to use today, we can install them using uv. So just UV add and we're going to need the following packages. Pandas to work with the data itself. Numpy for a quick helper function that we're going to write. We're going to also use torch. So PyTorch for the whole deep learning stuff. We're going to build a temporal convolutional neural network. So we're going to use 1D convolutional layers with uh dilation. So we're going to look at quite a bit of data as context. Then we're also going to use scikitlearn for the pre-processing for the scaling and I think that should be it. What we of course also need is jupit lab and also I forgot mattplot lip for some visualizations to explore the data. So these are the packages that we need uv them pip install them whatever and then we can just say if you're using uv uv run jupiter lab if you're not using uv just jupitter lab. This is going to open up the Jupyter Lab environment in the browser automatically. In my case here on the second screen and now I can close this and I can move this to a different workspace. I can go full screen. And here now I have my data and I have yeah [snorts] some files that I don't need. I don't need the main py file. So let me get rid of that. And we can open up this Jupyter notebook here. Let me save it as main.ipb. So I'm going to try to keep this quite casual today. I don't want to just go and give you the perfect code without explaining anything. I want to go through this as if we're looking at this for the first time. So I'm not going to already say, hey, I know what the data looks like. Let's just go and do it. I'm going to explore this with you to see how we could approach this from a machine learning perspective. So the first thing that we want to do is we want to take a look at the training data. How do we do that? I'm going to just say import pandas as pd and I'm going to say df is equal to pd maybe let me zoom in a little bit more here like this read csv and I'm going to read data slash train csv and then I can just take a look at the data frame here. So what do we have? We have some ID. Okay, that's just going to be uh zero up until the maximum just an incremental auto increment ID. Then we have the date which occurs multiple times. Why? Because we have different stores in different families. So I can have the same store, the same date, but a different category of items. Then for the next store, I'm going to also have all the dates and all the categories of items. So I need to find a way to structure this data in a reasonable way. For this, I need to think about my architecture. What do I actually want to do? I want to feed in information about the time series data into my network. And on the other side, I want to get the prediction, the forecasting for the same sort of data. So for the same sort of time series data. Now what we have here is not just sales. Now of course we could go and aggregate everything and say we just have the total number of sales everywhere across all categories. This is possible. But that is not what we're trying to predict. This is also not what we're going to submit to Kaggle. What we need is for every single store, for every single uh family or category and for every single date, I need to have 16 days of predictions. So I need to make a prediction for every single time series, which means I basically have number of stores times number of family time series, different time series that I have to predict individually. So how do I do that is I'm going to have X, let's call this X input channels. Maybe let's use a comment for this. I'm going to have X input channels and they're going to produce also X output channels. For every series I take in 120, so X input channels with 120 data points each are going to be taken into our model and our model is going to produce X output channels with 16 data points each. That is the structure of our architecture. This is what we're trying to do here. With that in mind, we need to think about this. How are we going to do this? So, I think it makes a lot of sense to have every column represent one of these channels. So, what I actually want to have in my final data frame that I'm going to be working with is I want to have the date here as a unique index. So, I want to have just one row per date. And then the column should be the combination of family and store. So, I don't want to have automotive and one, I want to have automotive and one as one value. So you could say 1 automotive 1 babycare and these would be the columns of my data frame and I would get rid of on promotion. I would get rid of ID and the actual cell values would be the sales. So for example when I go to this final data frame and I look up the row 2017 15th of August and then I go to the column 9 school and office supplies I want to see the value 121 in this cell. That's the basic idea. Now before we do that let us briefly see how many different values we have. So I want to go and say df family and then I should say unique and actually the length of that that is the number of unique family values 33 and I want to do the same thing also for the store numbers. So unique like this 54. So basically we're going to have that many channels, 1782. These are going to be this is the number of time series that we have that we need to make a prediction for. So we actually do have quite a lot of data and we're going to do all this in a single model. So how can we get our data into the desired format? We can do that by using a method from pandas called pivot. With pivot we can provide three things. The index, the columns and the values. So index in our case would be date. Columns would be the combination of store NBR and family. So this needs to be engineered first. we need to create this feature first and then the values are going to be the sales values. So what I'm going to do here is I'm going to go down and I'm going to say df and let's call this now stored family is going to be equal to df apply. We're applying a function to the entire data frame rowby row. This is going to be a lambda expression taking in the row x and producing as output x store nbrr and I'm going to make this an fstring. So that is going to be the first part and then underscore x family and that is going to be our string that is the return value. We then also need to say access is equal to one to make this on a column basis. And now if I take a look, this will take some time, but if we then take a look at the data frame, we're going to see a new column that combines the information of family and store number. So we have one feature that includes both as you can see here. So these are the three columns that we're going to need. Date, sales, and store family. So I'm going to say now df.pivot index is going to be date. Then columns is going to be equal to store family and then the values are going to be equal to sales. So that results in a data frame like this. And if I want to store this, I'm now going to say df_pivoted. That is our resulting pivot table. So let me print this again so that you can see it. Now every date only occurs a single time and all the different values are here now as columns. And of course we can also see here that we have 1,782 columns which is what we calculated above here. Now one more thing before we get into the actual pre-processing um of the data. Although this was already the first step of the pre-processing I want to briefly visualize just a bunch of these time series so we can see what we're dealing with. Do we have crazy data? Do we have data with regular patterns? at least some of them let us visualize uh a portion of the data for this we're going to say import mattplot lip piplot as plt and here now I'm going to say figure and axes are going to be equal to plt subplots let's just go with five and five the idea is I want to have a grid with five rows five columns where I'm going to say the rows are the families and the columns are the store numbers for example and then I just want to see the first 25 pieces or maybe let's go eight and uh three which is going to be 24 but I want to have like eight different categories and three different stores and I want to see the sales just so I get an idea of the entire structure for this I'm going to say figure size is 1812 maybe and then we're going to keep it very simple for i in range 8 and then 4 j in range three I'm just going to say axis is equal to axis i and J. So we're just going to iterate over them and then we're going to say the I is going to represent the um family and the J is going to represent the store number. So I'm going to say axis plot and what I want to plot is the data. So the data frame the sales essentially but which sales. So I'm interested in DF sales but in between I need to specify a condition and the condition is on the one hand that DF family has to be equal to DF family unique position I and also DF store NBR has to be equal to DF store NBR unique and then J. So you want to make sure that both of these are met and from that resulting data frame I want to plot the sales in this particular axis. And then I also want to say axis title to keep it simple. Fring will be the store number. Actually we can use no we cannot use that. So let's just craft it here manually. Let's go with DF store NBR unique J as before underscore and then DF family don't mistype all the time unique and then that I I think that should work and finally to have a good layout let's go plt tight layout out and that should probably No, I did a uh I did something wrong here which is what exactly? Yeah, of course I need to use set title not title. There you go. So now we have some data that we visualize. You can see that books for example didn't sell for most of the time in store 1. Then in store 10 we got no sales. In store 11 likewise beverages have pretty similar patterns across the stores. then probably some of the stuff is going to be seasonal. Some of it is not going to be seasonal. So you can see what the data looks like in general. Some of them are not sold a lot. Some of them are not sold in specific stores. Some of them have very chaotic patterns. Some of them have more structure, more um yeah, seasonality you could say to them. This is just so we get an idea of what kind of data we're working with. Now let us move on to the pre-processing. And the first step of the pre-processing is to scale the data. Since we're going to use neural networks, we're not working with something like decision trees. We need to scale the data because the scale matters. Otherwise, features that have a large scale or are on a large scale will be more important when it comes to calculating losses and then back propagating the losses. So, you want to make sure that the data is scaled. How do we do that? We're going to use scikitlearn for that. From sklearn.processing preprocessing import standard scaler. Now you can also go with a minmax scaler if you want to. And then I'm going to say scalar is equal to standard scaler. And now we need to think about what we're actually trying to do here. We need to remember that the testing data that we're actually doing the predictions on is unlabeled. So we cannot use it for evaluation. Which means that for our purposes, if we just want to play around with the data, it makes sense to train on a portion of the training data to test and evaluate on a portion of the training data as well on the remaining portion, but to then in the end go and train on the entire training data to do the actual predictions on the test data. So you could say that we use part of the training data as validation data and the test data is to actually do the submission. So we're going to define a train size of 80%. So the train size is going to be the integer. So the type casting integer 0.8 times so 80% of the length of the pivot data frame that is going to be the number of instances we use for training. And then we're going to say that the train data is going to be our pivoted data frame. So up until position train size and the test data is going to be everything from that position to the end. Now why do we not shuffle here? It doesn't make a lot of sense to shuffle when we're working with time series data because we actually want to use the past data as the training and then for the future data we do the predictions. This makes a lot of sense. If we were doing classification or regression it makes sense to shuffle. In this case, since we're depending um on the data or the data is dependent on a temporal level, it doesn't make sense to shuffle it around. It makes sense to use the first 80% for training and the last 20% for evaluation. So now what we do is we actually take this data and scale it. So train data scale is going to be equal to scalar.fit transform. This is going to learn the scale and also transform using the training data. And to do the same thing on the test data, we only do transform because we're not going to learn again on the test data. We're going to use the learned scaling if you want to call it that um from the train data. We're going to use that and apply it just to the test data because in reality we cannot do anything else. We cannot just scale data that we don't have if we work with unseen data. So that is how we do this. Then we're going to define a helper function that takes this and turns it into the X and Y arrays that we need for training. So I'm going to say import numpy snp and I'm going to define a function. Let's call it create x and y given the data. Then we also want to have an input length. So how much data do we want to take a look at? Then we also want to have an output length which is how much data do we want to predict. Data in this case meaning how many days do we want to look at and how many days do we want to predict. I'm going to say here x and y start as empty lists. And then I'm just going to fill these lists up with um yeah by by iterating over the time frame. So I'm going to say 4 I in range. And we want to go now up until length of data makes sense. But we also want to make sure that we have enough input data to still make predictions. So we want to have uh minus input length minus output length because I need to be able in our case to look at 120 instances and I need to be able to predict 16 instances but I still also want to have a ground truth. So I need to have enough data in our case now 136 instances that I can evaluate on otherwise there's no reason to do it. There's no um yeah it doesn't make a lot of sense to do it. So I'm going to say here minus input length minus output length + one. This just because we want to go up until the last element. That is going to be my iteration. And the data I actually want to append now to the X list is going to be the data from I. So from the very beginning because we start at zero up until I plus input length obviously. So we're looking at 120 days of data. And what is now the correct output? So what is the value that should be predicted? That is whatever comes after that. The 16 days that come after that. So we go to the beginning of the end. So we go to I + input length. So where exactly did the last sequence end, which is this here. So I plus input length. And we go up until the same point. So I plus input length. But we also want to take a look at the next 16 days. So plus output length and I'm just saying 120 and 16 because these are the values that we're going to pass here but I want to keep this variable. So we can also use 60 days we can use uh more than that less than that whatever but that is going to be how we craft our data and the only thing that we now need to do is we need to return them as numpy arrays. So np array x np array y that is the basic idea. Now we can actually apply this to get the X train and Y train data. So I can say create XY. I can pass train data scaled here 120 60. I can copy that. I can change this to be X test Y test. And of course also applied onto the test data. Same parameters. And then we can print here just so we can take a look at the shape. We can print X train shape. Um then Y train or actually let's go with yeah train shape copy and then we're going to use a fancy substitution because we're Vim users. There you go. And now you can see what the data looks like. We have 1,168 instances where we take a look at 120 days of data where each uh day has 1,782 channels, time series channels that we take a look at. Okay, I was just wondering why my uh shape is not the same as in my prepared code. Obviously, because I shouldn't go with 60, but with 16. So, this needs to be changed. Now, we can run this again. And now we should probably see There you go. We have actually 1,212 packs you could say or yeah data points where we have 120 uh days we look at and we have per day this number of different time series that we're looking at and then for this input we produce 16 days worth of output for all these ch uh channels. That's the basic idea and then we have less data points for testing for evaluation. So that is our data. Now what do we need to do? We need to turn all this into a torch compatible format. We need to turn this into pietorrch tensors and then we're going to turn them into tensor data sets and data loaders. So what I'm going to say now here is import torch and we just need to say xrain tensor is equal to torch float tensor. I think it was called float tensor but I don't get autocomp completion. That's fine. Uh the flow tens tensor is going to be based on x-ra and we're going to send it to CUDA to our GPU. If you don't have a CUDA GPU, just use the CPU. It will be quite a bit slower, but I'm sure you can handle that. So X test tensor for X test. And then the same thing needs to be done with the Y tensors too. So Y train tensor, Y train, and then Y test tensor and then Y test. I hope I didn't mess anything up, but this should be Yeah, I get CUDA out of memory. I'm not sure if this is because probably it is because a kernel is running on my prepared notebook. So, let me fix that. Okay, so let's see if this works now. There you go. I do think, however, it will most likely crash uh while I'm recording just because OBS takes like 1 GB of data. I only have 8 GB of VRAM. So, maybe I have to turn off the recording for the training, but I'm going to provide the code uh on screen here. So now we're almost there. We have the tensors. Now we need to turn this into tensor data sets and data loaders. For this we're going to say from torch.youutils.data import the tensor data set and also the data loader. And we're going to say that the train data set is equal to a tensor data set which is created by using the xrain tensor and the yra tensor. And then we just need to say train loader is going to be a data loader based on the train data set. And we're going to have a batch size of 32. And we're going to shuffle the data. Shuffling is useful for training to have some variety. It doesn't make a lot of sense for testing, which means we're going to copy this. We're going to paste this. We're going to replace train with test. But we're also not going to shuffle because it doesn't make sense to shuffle if you're just doing inference. That is the pre-processing part. So let us create a markdown field here. Let's say training uh and maybe built building and training model something like this. And here now we get to the interesting part which is defining the architecture and then doing the actual training. So that is now the part where you need to turn on your brain because what we're going to do is we're going to create a temporal convolutional neural network model. I don't want to go extremely deep into the theory of convolution and everything. I will explain the basic idea, but I'm not going to go super deep into it. The focus here today should be on coding. I plan to do some theoretical neural network stuff in the future, but now we're going to focus on the application. So we're going to import torch.n nn uh as nn and then we're also going to say import torchn functional I think sf and the class that we use is tcn model we're going to call it tcn model and it's going to inherit from nnn module we're going to have a basic constructor now um with this constructor here I'm just going to pass self you can also make this more flexible by passing the horizon, so the 16 days. You can also pass the input size or the input window of 120 if you want to. I'm just going to fix these values. I'm just going to say these are the values. Also, the dimensions here, which are 33 * 54. I think they're also going to be hardcoded into my model just to keep it simple. You can of course do stuff like number of inputs and then a default and then number of uh outputs or channel size or whatever. I'm just going to hardcode it here in my code. First thing of course, we call the super constructor like this. And now I'm just going to define the architecture. So we're going to start with a simple convolutional layer which is going to take in our data and then it's going to produce 64 kernels. So very very basic explanation of convolution um especially of 1D convolution. You basically have something called a kernel or a filter which is like a window that you slide over data. I actually have a link here. Maybe I can show that as um as a visualization. This is a GitHub link that shows you how convolution basically works. This is 2D convolution. This is not a 1D convolution. And you can see we're getting some input data and we're sliding a window which is a kernel. This would be a 3x3 kernel. We're sliding this 3x3 kernel or filter over the data. And we usually do something like uh mean or just summing up everything or averaging or what yeah which is mean actually uh and then we get the value. So for these first nine blocks here we get this output whatever we do and then you slide and you get for the next nine you get this value and on a 1D level you can do the same thing just that you don't do this with an image you do this with time series data. So you slide your one-dimensional window, your one-dimensional kernel over the time series and then you make predictions and uh you get yeah information that you can use and then there's also other stuff which I'm going to talk about here in a second. But that's the basic idea of convolution and we can do that in parallel. It's better than recurrent neural networks as um or at least from the perspective of parallelization because with convolution we don't have to do it one after the other. If you use something like an LSTM, a long short short-term memory network, or if you use something like a gated recurrent unit GRU, you have to do it one after the other because you have this temporal dependence. Whereas with convolution, you can do it in parallel. That's the main benefit of using convolution. So we're going to say here self.con 1 is going to be equal to nn con 1d. And the first parameter is the input size. In our case, now this is as we saw going to be 33 * 54. So we basically have that many input channels. 1,700 something. This here is the number of channels that we feed into this convolutional uh layer. Then how many channels do we want to put out? I'm just going to go with 64 here. You can play around with this. You can try 32, 121, something else. I'm just going to go with 64. It works in a decent way. Then we're going to decide on the kernel size. So the kernel size if we go back here is the size of this thing again in one dimension not in two. But here it would be three as well because we have three by three. If we had one dimension it would just be the first row. So just three cells that we look at that is going to be the uh the kernel size here um which is going to be three in our case. Now we get to padding and dilation. So I think the visual explanation for dilation and padding makes more sense. Padding basically means we have some nonvalue padding. This this is yeah it's just padding. So you have no values here but you can still apply the filter in the kernel to this range. So in this case if you go 3x3 with this specific um area here you would only consider this one square but it results in you having the same dimensions here. So you can still go and keep the number of uh data points if you want to call it that. And dilation basically is like spreading out your focus. So dilation is you're not just using the kernel as one thing that sticks together. You're splitting it up like this essentially. And what you do by doing this is you increase the size. You increase the window of data that you look at to then make a prediction. So without going deeper and deeper, without increasing the depth, complexity or kernel size of your convolutional neural network, you can take into consideration pretty long time frames to make a prediction. That's the in a nutshell explanation. I'm not going to go deeper into that. And last thing maybe in order to do that effectively, we're going to incrementally or we're going to always double the dilation and the padding. So I'm going to say here padding is going to start as two. dilation is going to be one. And then I'm going to say for the second layer, now we need to take in 64 because the previous layer produced 64. So now we have 64 channels. We take in 64 channels. We produce another 64 channels. And now the padding is going to be four. Dilation is going to be two. Copy. And the third layer is going to be 64 again. Um padding is going to be eight. Dilation is going to be four. And finally, we have a linear layer, which you could say is sort of the final final reasoning if you want. And what we're going to do with this one is fully connected layer. So FC is going to be equal to linear. It takes in 64 uh channels. And what do we need to produce in the end? We need to again produce 16 days of um of these this number of channels. So we have to produce 1,782 channels of data for 16 days. So that is going to be 16 * 33 * 45. That is what our layer does. Now that is just the architecture. We now need to take this and we need to actually feed the data through this process. So for that we define a forward method which takes in x. So first of all, I'm going to swap the dimensions. As you can see here, we have this order. I'm going to swap this and this. So I'm going to swap the second and the third dimension, which in programming terms is 1 and two because we start counting at zero. So I'm going to say here x is equal to um x.transpose 1 2. And then we're going to feed the data through our layers. But we also need to break linearity with an activation function. If you don't know why, I have a video on this channel where I explain explain why this is necessary. And I'm going to say now freu rectified linear unit self.convolution one applied onto x. And important, we want to get rid of the padding. So we want to crop out the padding again to get the same temporal length as in the beginning. So I'm going to say colon colon and then colon -2. I'm going to copy this. I'm going to just increase the number of the convolution layer and I'm going to say here four and eight always the same as the padding. That is the idea. And then finally once we have that we're going to return the fully connected layers. So self.fc applied on to x and then here um -1. So the last um the last value here. And then we're going to apply the view function. And this function here basically means okay now we're going going to get this into the format that we want again because we don't just want to have this this one number. We don't just want to have batch size and and then just this huge number that is unstructured. I want to have the following shape. The first dimension. So the batch size should be inferred automatically. So negative -1. Then I want to have 16 days. And then for each day I want to have 33 by 45 channels. That is our model. Now uh and of course I have some problem here which is that I'm not closing the brackets here. That should be it. This is now our temporal convolutional neural network model. So now we can just initialize it. I can say model is equal to TCN model. I'm going to send this to the GPU immediately. Then the optimizer is going to be torch.optimatom applied onto the model parameters. Why do I not have autocomp completion model parameters learning rate we're going to go with 00001 and then we're going to define a loss function criterion is going to be equal to nn.mse mean squared error loss. Actually, I'm going to use the root mean square error, but we don't have this as a function in PyTorch. So, I have to apply torch.square root manually, but that's fine. So, we create all of this. And now we get to the training logic. How do we train? We say for epoch in range number of epochs that we want to train. Let's go with 30 for example to not overfit the data because if you go too far, you can maybe perfectly predict your training data, but then you fail on the testing data. This is also not very useful. So we don't want to do that. And then you want to say model.train. So put the model into training mode. And now we just need to go batch by batch. So I'm going to keep track of the epoch loss. Let's start with zero here. And then I'm going to say for xbatch ybatch in train loader. So I'm going to load the data batch by batch. We had a batch size of 32 remember. And I'm going to do the prediction. I'm going to calculate the loss. I'm going to back propagate the loss. I'm going to take a step with the optimizer. Basic training stuff. So first of all, I'm going to zero out the gradients of the optimizer to reset it basically. Then I'm going to make the predictions. The Y predictions are going to be the model applied onto the X batch. Then we need to compare our predictions with the actual ground truth. So the loss is going to be torch. square root since I want to use the RSME RMSE not the MSE. Um so square root applied onto our criterion which is taking in the prediction and the actual batch data and then we need to back propagate the loss. So lossbackward and then I need to take a step with the optimizer. That is essentially it. Then of course also epoch loss needs to be added here. So that is going to be plus equal to loss do item. And I think the only thing left to do here is to print occasionally. So if the epoch plus one if that modulo 5 is equal to zero we're going to just print here an f string which says epoch epoch + one loss is equal to epoch loss divided by length of train loader think that makes sense and that is going to now train our model if I don't have a mistake again So, what did I do wrong this time? Okay, I think I see the problem. Not 45, but 54. Need to get the number straight. So, let's run this again. And now we get the loss decreasing. As you can see, you could also keep going, but again, we don't want to overfit. I think 30 epochs is enough. And now we can go and evaluate at least on our training data. So, of course, we cannot evaluate on the test data of Kaggle unless we submit. But I can go and evaluate on the remaining portion of data that we didn't use for training. So I can say here model in eval mode with torch no gradients. Since we're just doing inference, the test prediction is going to be our model applied onto X test tensor directly. We actually don't did I actually create a data loader for testing? That's kind of useless. Not going to use this, I think. Yeah. So actually I'm just going to go and feed the tensor here directly into um into the model. This is going to give me the predictions. Then the test loss is going to be equal to torch. square root dot criterion uh test prediction and y test tensor and then we can print that test loss. Actually this number is quite high compared to the loss that we got here. But this is as far as I understood it because I looked at the data. I'm not going to do this now. Uh because we have a pretty large outlier. So if you take a look at the maximum data the maximum number in Y test it's significantly larger like way larger than what we see in Y train but the median and the mean I mean the mean is a little bit affected by this but the median is basically quite similar. So that's actually something we shouldn't pay too much attention to. I think we still get decent results then on Kaggle. So let's just skip this part. But now of course we saw that the process um worked. So we got a model. But of course want to do the exact same process now on the full data set. And then we want to do the predictions on test CSV. So on this data that doesn't have any sales data associated with it. So I'm going to create now here another cell markdown. let's say uh full data set training where we're going to essentially do the same thing as before. So I'm going to go up here and um I'm going to actually say copy this here is actually I could have copied it. I could have also just wrote it but doesn't really matter. I'm going to say full data scaled is going to be equal to fit transforming the entire pivoted data frame. And now I don't need to do a split anymore. What I need to do is I just need to create the XY data. So I'm going to say XF Y full is going to be equal to create XY based on the full data scaled 120 and 16 are still the proper parameters here. Then I still need to turn all this into a PyTorch tensor. So x full tensor is going to be equal to torch dotflat tensor of x full send it to cuda. Same thing should be done with y here. Same as before. Then the full data set is going to be equal to tensor data set of x full tensor and yf full tensor. So we basically just repeat what we had before. Then full loader is going to be equal to tensor not tensor data set actually data loader based on the full data set with a batch size of 32 shuffle equal to true because we're training again and essentially we can then just take this part here copy it it's going to stay the same I'm just going to call this now final model and the optimizer will tweak the final final model parameters run this. And now essentially also we can just copy this here the training loop. That's going to stay the same for the most part. Just replace final model here. Then also not train loader but full loader. Then also final model here. Then I think that is it here. We need to say full loader. But besides that I think everything is the same. So let's go and run this. This is going to now train on the entire data set. And now the last thing that we need to do is we need to apply this onto our test CSV file. So onto this file. We need to load it. We need to get it into a similar format. And then you just you know predict and store the information in this format here. ID sales. So for this we say final model in eval mode. And we need to get the last sequence the last data that we have the last 120 days that we have uh in terms of data and we need to do a prediction on that because that is going to produce the next 16 days. So with torch no gradient we're going to say the last sequence is going to be equal to torch float tensor applied onto the full data scaled that we have the last 120. So minus 120 until the end. That is that we're going to unsqueeze. So unsqueeze uh unsqueeze to add this additional dimension unsqueeze zero and then we're going to send this to CUDA to the GPU. So again, why do we do this? Because in the training data, we have the input for the testing data. So we need to get the last 120 days of training data as input to produce the first 16 days of testing data. So we're going to do it like this. predictions are going to then be full model or sorry final model applied onto the last sequence that is going to give us the prediction. So we can take a look at that. This is the information. This is the predicted data the 16 days that we're predicting here all the channels. And now we need to take this we need to get it back to the CPU back to numpy. We need to squeeze it again. So we need to remove this additional dimension. Then we're going to inverse scale it. This is also important because now we have these values that are not actually sales. We need to inverse scale them. We need to get rid of all the negative values because they're going to be wrong for certain. We're going to cap this. The lower cap is going to be zero. And then we're going to take this final data and write it into CSV. So we're going to say now uh predictions is going to be equal to predictions. Take it back to the CPU turn it into numpy uh squeeze it. So remove this additional dimension. Then predictions is going to be equal to scalar inverse transform whatever we have. And then we're going to cap we're going to say is equal to np maximum whatever the prediction is or zero. So we're not going to have any negative numbers. If we have negative numbers they become zero. So now the data looks like this. doesn't really give us anything that we can really analyze or look at. But that is now uh the shape we have 16 days of all the channels. That is what we need. Now we need to take this and put it into a data frame for the test data. So we're going to say here test df is going to be equal to read this test cvv file using pandas. So test CSV is going to be read and we can actually just uh not test CSV sorry data test CSV. We can take a look at that. We already saw what this looks like. Basically like this. And now we need to first craft again the column that we had before. Store number and family combined. So here I'm going to say that I want to have this what did I call it? Store family. I I think I called it store family is going to be equal to test data frame apply again lambda expression x and the formatted string which was I forgot already it was x uh actually I used double quotations here x store number and then this whole thing underscore and then x family I think that's it of course access one because we're doing this on a column basis here. Now we have this crafted feature again and now we just need to go and sort the dates. So dates is going to be equal to sorted test data frame dates but only the unique date. So we have all the dates that we want to do prediction on not dates but date of course that gives us the list of dates that we do predictions for the 16 days of predictions. And now long story short we're going to say prediction data frame is a pandas dataf frame crafted from scratch here which is going to take the predictions as data. The index is going to be the dates and the columns is going to be equal to df pivoted dot columns. That is now our prediction data frame. This is what it looks like. For every day we have the sale prediction. But of course that is not the format that we have in the submission file. So we need to also map this to an ID. How do we do that? This is done by using a fancy function called stack. So stack is essentially the reverse operation of pivot. So if we take a look at that, this is the pivot table. And if I now go and say prediction dataf frame.stack, you will see that basically it does the complete opposite. So it turns it back into this group format. And if I now say reset index, I'm going to basically get the structure that we had in the very beginning where the dates are duplicated, where I have the store family like this, and I have the sales price. Now of course we should also say that that here so let's call this prediction long prediction long dotc columns should be equal to and now we're going to say date store family and sales and we can take a look at this. This is now already in a pretty nice format. And as you can see now, we just need to add the ID, the proper ID. So we do have some ID, but that is not uh mapped to what we're interested in. But we don't actually need to craft this ID. We can just uh map on the respective data. So since I already have in my test data frame also the store family, I can just map on store family and date uh or I can just uh join on that. I can merge on that. So I can say here that the submission data frame is going to be equal to taking the test data frame and merging it with the prediction long data frame on the following columns that is going to be a list date and store family. That is going to be the columns that I merge on. How is going to be left because I want to keep all the data that is part of the test data frame. That is the most important part. And then I'm going to say the only thing I actually need from all of this, we can actually do this. Store family is missing. Why is that? Because I called it a different way. Store fem. Yeah, I have a typo in here. I need to remove that. Then run this again. Now we can take a look at submission df. It's already in the perfect format. I have ID. I have sales. The only thing I need to do is I need to just select that. So that thing is going to be ID sales. And there you go. I have all the information that I need. The only thing I need to do now is I need to say to CSV that thing I want to export it to a CSV file. I'm going to call this one submission.csv. I'm going to say index equals false because I don't want to export this index column here. I just want to do it actually. Okay, I messed this up. I just want to do it like this. And as a result of that, I should now have created a submission CSV file with sales data. And this should be exactly as long as the sample submission. If I go to the bottom, we can see they have the same number of rows. And that's essentially it. We created our submission. So I can now go back to the Kaggle challenge. So I can go to the store sales challenge. I can go submit prediction. I can go and browse files. Then it opens up this file dialogue here. And here I have in my tutorial directory the submission.csv file. So I can just select it. And if I want to I can provide a description, submit it and see what my score is. This time I even beat my old score. This is probably just random. There's no really uh not really a reason for why we beat the score. But now I have 0.48. And if we go to leaderboard, that means that I get place 94. Cool. So challenge to you guys. We did not really overengineer this. We did not really include all the information. Our architecture was quite simple. We can do a lot more. Maybe you can combine this with some recurrent layers. Maybe you can try an entirely different architecture. Maybe you can engineer more features, use more features. I'm sure you can beat this score without investing too much effort. So that's what I would recommend you guys do now as a practice uh or for practice. But yeah, this is how you can process time series data. This is how you can do predictions on time series data using convolutional neural networks in Python. So that's it for this video today. I hope you enjoyed it and I hope you learned something. If so, let me know by hitting a like button and leaving a comment in the comment section down below. Also, if you are interested in any personal one-on-one tutoring or if you're interested in any services, you can go to my website. I have a tutoring tab and a services tab. There you can see what I have to offer. At the very bottom, you can contact me via email or LinkedIn just if you're interested in that. Besides that, don't forget to subscribe to this channel and hit the notification bell to not miss a single future video for free. Other than that, thank you much for watching. See you in the next video and bye.