🔴 Live Competitive Data Science

Key Takeaways

hello everyone it is Thursday February 9 2023 and we're here to stream and do some coding I hope you're doing well tonight um and it's been since Sunday since I streamed last so we're gonna pick up where we left off I believe as long as nothing major has changed I haven't really worked on uh what we were working on before uh odinson what's up welcome to the chat yeah I haven't worked on this Chicago competition since we streamed last so what you know is what I know and we are going to uh kind of see how we are and if uh where we sit with this so let's go ahead and switch over to this View and I need to make sure I get my chat out of there for a second and get this out of here just let me know if the audio hey Flex what's up welcome to the chat let me know if the audio is sounds okay and if there's anything else messed up good to know that as soon as possible so we've been working on this kaggle competition for um the last few streams on and off but we're gonna pick this up and uh continue working on it it looks like uh there has been there there are a lot of people with 1.018 look at this all the way up to fifth place all have zero point or 1.018 on the leaderboard and I also have that score because that is the score of the public notebook that was released to us let's see here best score 1.018 which uses graph net Baseline and that's what we're going to go ahead and dive back into again I want neural network tutorials I need to start making some of those Sprite lyrics bright lyrics what would you um What specifically would you like it on Pi torch or just in general hey treb welcome to the chat on what days you stream usually Tuesdays Thursdays or Sundays are the days that I would stream if I'm streaming so we didn't look at anyone else's submission but it does look like um like data source here who's using this dine Edge model that was very close in its score to the graph net um well also not I guess not that close but better than anything else that was publicly released data source is now in first place only three submissions to the leaderboard that's very impressive we I also see some uh familiar faces here and other competitions these are very s uh strong competitors on kaggle so we're gonna try to catch up as much as we can and uh just to remind you we did set up our environment to run this graph net model last time and we were able to run inference on some of the data but it was taking a while to actually create all of the um create all of the SQL Lite database features or I guess data and I created this script this simple script after the stream last time and man I need to get this set up so this simple script allowed me to Loop over all the training data and just start converting it into this database format but the problem was I left for a few days to go to a work trip and when I came back my hard drive was all full so let me get over to where that sits I think it's here no it's not let's look in the script where did I save it on my disk oh my my disc in this directory and if I do d u all right d f Dash H you'll see here that the my disk hard drive that I have on here which is almost one terabyte is completely full um zero available megabytes and if I go into the Ice Cube DB folder which I was running this on it's 546 gigabytes of data there that's a lot uh tensorflow Keras would be great bright like lyric says okay maybe at some point I need a most of the stuff I've been doing recently is pi torch so uh Oaky your vids are great thanks I will try to join your stream but it's 8 A.M here and I gotta go to school okay have a good time at school yeah put me up on the screen at school show all your friends I'm cool with that um so I think this is where I was running the script yeah this is where I was running the script no this is where I was tailing the log the logs of the script so let me exit out of this what is this this was I think that's just garbage okay so this is my webcam and that's running um now I'm all situated I know what's going on I think what I would like to do is see if I can explore this database I'm not that familiar with mySQL uh uh MySQL query how do we query MySQL just from from the command line that I don't really know how to do okay so MySQL MySQL is not found do I need to install it all right let's try this actually conda activate Ice Cube conda let's see what the conda environment was conda EMV the list and then we'll grep on Ice Cube MySQL U admin Dash p I need a MySQL quiet yeah I I do need to install it I think it would be conda list conda EnV list okay so what did I call this oh graphnet conda activate graph net I didn't call it Ice Cube and then MySQL is it installed in here no sudo apt install MySQL client core Dash 8.0 what's Maria DB client use MySQL from notebook yeah all right so MySQL Dash U admin Dash p do I need to give it this file name Maria DB is the mysql's open source things MMD you know a lot about this I'm used to working with like uh Oracle or postgres databases that are actually running so do I do this train date but MySQL oh it this is SQL light isn't it let's go over here and double check graphnet helpers create train database so convert to SQL Lite yeah SQL light yeah it's definitely a SQL like uh should I be updating everything on stream right now then we could see okay it's it's running or we have this installed now sqlite3 um how do I exit out this Dash backslash Q all right control C got me out um so then I do SQL Lite 3 of this train database.db select star from how do I see what tables there are equal light list tables tables Johnny drop tables do a help oh wait everything in SQL do I need to do semicolon to end it the dot tables okay so there's a DOT table there's a pulse table and a metal table select so let's do some SQL here select star from meta table do it can I do like limit 10 I can so this is all the metadata and then select star from pulse table limit 10. this looks like it has X Y and Z coordinates I'm not quite sure what this is so why does this pulse table start at 24 so now we see these are the tables that this code is writing to when it creates this database so add to table this is what it's running um create table and then it's using Panda's 2 SQL to write this I don't think this is the fastest way to do it also um so it creates this and then let's see what is the data frame that it's writing that's this add to table okay so here's here's in the code where it actually writes this so we have this convert to sqlite function which finds the database file it reads in the metadata then it does this iterating over the metadata parquet file and then it adds to The Meta table this metadata batch so let's let's go here and do below this I'm going to look at the actual parquet files to make sure I have this right Ice Cube neutrinos train all right so conda activate graph net IPython import pandas pandas as PD always and then let's uh read parquet which is PD read parquet why isn't it Auto Complete that's kind of annoying it shows me the suggestion but tab completion's not working all right so if I go to batch one and I do a head on this command what do we have we have this event ID which is 24 so that matches up with this the event ID is the index which is also the index here then the sensor ID then the time then the charge then auxiliary which is true or false which are ones or zeros and then I believe this is really inefficient but I believe it also saves because it's merged is it merged yeah I thought this geometry table is also merged on it but it doesn't look like it is it is when it makes the pulse table so it creates this metadata batch it creates the database path sorry it creates this uh basically appends this table with first the metadata and then the metadata loaded with the geometry table input data folder let's see what this why would it save it twice here why would it save The Meta table are these the same okay so this one saves the pulses but that also does this load input let's see what the load input's all about load input will load the corresponding detector readings associated with the metadata batch okay so up until this point it's only loaded the metadata that makes sense so the metadata table is only going to have one row per yeah these are the angles that we're trying to predict this is like our Target table so this is 24 so if I do select star from pulse table where event ID equals 41 which is the next event yeah we got all these in here so there is an event ID equals 41. and that's the next event in our metadata table so these are all of our targets so basically it's saving it in a why save it in a database can't we save them just as as numpy files that's one thing I want to maybe modify about this code because it seems like let's do some let's do some testing for seeing how how fast things are to read time test dot ipy notebook no let's call this data IO test data i o test uh speed test data IO speed test test speed and size I don't know if we can do size of SQL I want to I want to rip out this sequel light stuff from this whole thing hey we got a subscriber sorry clipped clipped 10 months clipped thank you so much for being there for me and now I'm gonna be here for you as I go ahead and go here and spin this wheel let's get the Picker wheel up let's spin it for clip thank you let's see what people are saying people are saying your PC died hola Argentina nice big fan Pablo awesome little Davis is here tell a dad joke let me tell you all a dad joke here because clipped did subscribe on Twitch which all of you could do for free if you have Amazon Prime by using your Prime what did one plate say to another plate tonight dinner's on me that's a dad joke you can write that one down so we're saving it to a SQL Lite database we could do parquet or probably the best would just be like numpy gzipped because before we start going and training these models we have to ask ourselves the question of is it worth uh can we speed up this process at all and is the data I O gonna be the the limiting factor because we don't want it to be the limiting factor we want the limiting factor to be the GPU now when we were testing it during inference it seemed like it seemed like it was pretty fast so maybe and I'm sure these people know what they're doing and that's why they've done it this way let me load up the graphnet API so let me see how they're loading the okay so this is loading in the input let's see how the inference works we looked at the model this is our graphnet prep work convert to sqlite but now we want to look at when we're training this model how do we do that and we I think that one thing we were looking at before was uh graph net the GitHub of grat graph net which helped us to see how they train so train they have an example here of how they train the model so it does use pie Church torch lightning data set config path model config path now is this SQL stuff built into built into graphnet that's the question and I'm looking here and I don't see it it's saving the model it's writing the results all right this is a pretty um straightforward training Loop there's this training config SQL has a extra L oops gotta fix that so what are they doing here they make this data loader batch size config okay OS path join example output directory with train model maybe we should look at that they didn't really provide inference or training code we can go back and see how the inference works so it says prepare for data frame load pre-trained model with this config and then inference this inference function it makes a data loader using their where does this where is that I think we put in this helpers make data loaders that runs from make data loader which is pulled from graphnet training utils so let's see if the data loader part maybe we need to start making our own branch of graph net gravnet training utils make data loader pulse Maps features and Truth this is a list of strings pulse map is a string and a list of strings and then it does take in the database which is the main thing okay so SQL Lite data set this is the this is the thing that constructs our data loader so let's load this bad boy up in another tab let's go let light data set oh come on graphnet data sqlite so they do have like a parquet interface it looks like they've already built a parquet data set version of it so it could be in parquet why do they convert it to SQL Lite data set it's like an intermediary step that doesn't seem necessary why not just use this parquet data set it must be faster right I'm not quite sure yet so let's look at SQL Lite what's your goal for today's stream I'm just trying to get further down the line of understanding this um graph net which is the top scoring submission for this ice cube competition so it looks like graph net a lot of the stuff we need to focus on is optimizing and understanding how the data is loaded into these models for training and for inference it looks like this Library saves stuff off as SQL Lite and we're looking at that here let's let's select the distinct event ID from metadata table meta table I don't know why you call the table a table so let's do count of distinct event ID hello this is my first time catching your live stream and loving your vids though I'm retired and I felt like learning stuff so I'm trying to learn d a type of thing that's awesome supper slash that's great I'm glad to hear it you know what I'm realizing is I needed to change this environment to graphnet and now I should be able to explore through the graphnet code base that was that was a silly thing I was doing before I have an idea I have an idea let's look at the code section here best score so all graph net is doing is training a dine Edge model for us let's see how datasaurus did this trans Ice Cube transparency what is that all about torch cluster so this is where he installs all the packages oh this is what I've done before too this is really smart so he's identifying if this code is run locally or if it's been run in The Notebook and if it's being run in the notebook then he installs all these packages otherwise he assumes it's installed on his local machine now he is using graphnet also um and he made this dine Edge graph net model and a models.pi I kind of like his setup better than the graph net one even though it does the other one does score better class dine Edge I bet he just didn't train it as long as the other one dine Edge model where's his data loaders add weight decay angular distance score this is his loss function Ice Cube submission data set okay so the data set that he's using actually does this is the actual data set it doesn't use any SQL MySQL it just straight up pulls from the parquet file y subtract this this I'm not quite sure why he's doing that sure there's a reason is this like the average making it he's also doing a log 10 formation of the charge he's just subtracting the average value of this so it's centered around zero so when it gets it it merges the event with the sensor data frame sensor data frame I'm guessing is yeah where all the the location of each um data yeah each uh sensor on the Ice net yeah so he uses this sensor geometry he's dividing everything what's this GitHub graph net assume that rde relative Dom efficiency is equivalent to QE pre-processing oh so he's so he's dividing by 500 TTA this is test time augmentation wrapper okay so for test time augmentation he's taking both angles he's flipping everything 180 degrees and then evaluating it that way as well this is good stuff make predictions let's see where the training is he using pytorch lightning also yes pytorch lightning module let's see what the disc so his stuff is really good and it's a and it doesn't use any sequel light let's read what he wrote up about it my solution is now public for code competitions I actually use dot Pi files rather than notebooks as I just copy and paste relevant parts of my code base into a single submission script this means that it is very light on comments so I'll summarize here I use the dine Edge baseline from graphnet as already shared by Rasmus here so this is the one that scored better on public leaderboard I added some batch Norm later layers and switched out the activation function wow I trained on five percent of the data roughly 33 batches validation score was 1.048 and 1.046 so it looks like the validation is super stable here isn't he converted to degrees your continent is helping me not much Ilya thank you so much um so he trained on 33 batches he trained on 33 batches and got this score the top scoring one just trained on more batches it trained on 50 batches and scored much better zero one point zero one eight I also use the following loss function Von misses feaster for asmuth L1 for Zenith and cosine similarity this is some really good stuff uh 180 degree TTA about the z-axis also brings some game so that's what we're looking at there TTA stands for test time augmentation augmentation means is when you take your data set and you apply some transformation that you don't believe changes the fundamental label or you you rotate it in a way that you think the label also would rotate like in this he's doing 180 degree rotation why not do 90s also but anyways he's doing that to um gain some um some boost in his test predictions so the it doesn't look like the augmentations are being applied when training just because there's so much training data that there's not necessarily A Reason To Train um I'm guessing that there isn't reason to train with augmentations yet uh very nice we'll be interested to see how Gat for others and other flavors of gnns perform what's g-a-t this person's an ace eighth place g-a-t Ice Cube mod no that I don't know what that is thanks for this you share your model training code I use Pi torch licensing so you just call PL trainer with the model maybe it's easier than I realized but I was learning precisely how to create the data set from roughly 33 batches you simply pass a single python list containing 30 Ice Cube submission data set objects or something terrible let's see what these oh my gosh there's so many replies just to this one and no other comments actually plus one this if you give some light have you also a parquet data set for training do you recall the ram requirements I've tried to modify the parquet submission data set to a suitable train data set I'm able to concat multiple batches but facing out of memory errors greater than five batches in kaggle kernels so this person's working on kaggle kernel yes I forgot about this I'm training on my local machine and created a PT files for each data object using torch.save before using the output of the prepare samples function everything else is the same other than the Y values which are also saved in the data object hmm I limit the train batches to 0.05 in trainer it was quite tricky doing this since even on a large Drive I hit the I node limit for my partition so I had to split files over two drives I suspect that doing it in this way hosts have done it and the Baseline code might be better though I remember seeing some data set related to code specific on this competition recently added to the graphnet repo so he's saying he pre-processed kind of like we've converted everything to SQL light or not we but the graphnet codes converts everything to secret light he's just using torch.save to create all these PT files for each data object and let's go back to his code for the data yeah they're saying that um so everyone's running into the problem where the parquet files loading in is too slow so you do have to do it intermediate intermediate step to save this data files for each data set using out of the prepare samples so I guess we speak for greater than one terabyte could you also give us a rough estimate on training Epoch time sorry I meant to say prepare centers after joining with the event data and then saving the PT file the total total size is just under one terabyte so they're running in the same issue we are but the large number of small files you may hit the I node limit before your storage limit if you were using the Linux e XT partition does anyone know about I the people are asking what's graph net graph net is all right just to back up here a second anyone new here in the Stream right now we are looking at this kaggle competition um Ice Cube neutrinos deep ice I'm gonna put this in the chat so you guys can see it it's a huge data set huge-ish data set with um all this data gathered from this device or this site that's located on the South Pole that tries to detect neutrinos it like actually has sensors that go into the ground like this it's amazing don't ask me why they're trying to use this to kind of get a sense of what's going out on in the universe so they have these big sensors that go into the ground and we're trying to predict what direction the neutrinos are going using two different angles that's what we're trying to predict given all this sensor data values so people are discussing this one public solution that was released and pretty cool thing about this competition is they had uh submission an early sharing prize for whoever shared their best results early on in the competition so we're kind of learning from that I don't know what the ice transparency data is measurement of South Pole ice transparent with Ice Cube LED calibration system the Ice Cube neutrino Observatory is now complete with 86 strings of diploid and Antarctic Ice Ice Cube detects the chernek cough radiation emitted by charged particles passing through or created in the ice to realize the full potential of the detectors the properties of light propagation in the ice and around the detector must be well understood this report represents a new method of fitting the model of light propagation in the ice to a data set of in situ light resource events collected with Ice Cube the resulting set of derived parameters namely measured values of shattering scattering and absorption coefficients versus depth is presented and a comparison of Ice Cube data with simulations based on pneumo so it looks like these people have analyzed the transparency of the ice to the point where you can use those parameters so there's this transparency txt file that he's using which has the shattering scattering and absorbs absorption of the ice so that can even be modeled let's see how he adds it and I'm really interested to see that so the scattering comes in here this is in the data set submission data set and the absorption how does he know to do this these are both roughly equivalent after scaling ice transparency data from page 31 of this the datum is footnote 8 of page 25 what great comments here robust scalar fit transform so he's not just reading this in he's reading this in subtracting 1950 from it dividing by 500 then applying the scalar to it geez this stuff is intense Jared worth mentioning that there's a dust layer halfway down oh so maybe that's what this is picking up for someone who is learning python to become a data analyst who already mastered SQL is true that polars is easier than pandas to master I don't think I've I think polars is a little bit more constrained in what it can do but I don't think that either of them are necessarily easier or harder but I'm definitely more proficient in pandas I've only used polars a few times and I've been surprised by how great fast it works but without needing to for any reason like I unless I really needed stuff to be faster I still am sticking with pandas hope that answered your question foreign this code is really good so let's try to understand how he's doing his data saving prepare sensors he's saving before prepares sensors repair sensors is here sensors takes assume that RDA relative Dom frequency this is an intense competition guys there's a lot going on here this is not this is not like Titanic data set um so this is Reading in the data if train batch IDs is here load from checkpoint this is for ensembling wow I hope this is anyone's getting it's an easy tabular data set they said it'll be fun they said yeah that's right um it's really fascinating but it's really overwhelming too like this is a type of competition where I would need to be completely engrossed in it and just only thinking about it for a long time clearly data source has done a great job with this and he has um read a lot of papers in order to understand what's going on and I actually think his submission is is really um clever and or well not clever but really neat and cleanly rent written and clear to read but it's overwhelming it's overwhelming so much um let me try to think of what I'm gonna try to do here so he said the problem with hat saving each one as a file is you actually run out of you have to start to partition your thing to run he doesn't even Epoch times was 20 minutes on my machine using five percent of the data how many ebooks did he train for all right so grab the graph example notebook trains on a single batch which ends up as 2.5 gigabytes with which is a real restriction regarding memory do you know if we can train on a single batch save model weights then low save model weights train on another batch repeat yeah you can do that I think he exhausted the I node yes Beyond here lies nothing you're right what's that limitation you may hit the I node limit approximately 4.3 billion inotes there are many inodes in every system there are a couple of numbers to be aware of first up in less important ethereum theoretical maximum number of inodes is equal to 2 to the 32. I noticed like a little database built into the Linux file system so it's just the maximum size the file can be or the maximum number of files that you can have so he's reaching he's getting more than four billion files I didn't think there were that many there's not that much data in the metadata so uh is he creating each I'm just lost in why that would why that would be the case let's look at the data set again Ice Cube submission data set pulse limit of 300. I want to see how this works Ice Cube submission data set that's the key right now is just figuring out how to work with all of this data it's crazy we almost need like so we're doing uh uh I made this for doing a comparison of IO data i o speed test so from torch import actually let's just do where's the data set data set data loader here's all the stuff he's importing so let's load that then let's load this data loader and let's see how it works submission data set so we're going to make a train data set which is this ice cube submission data set which the with the event ID equals what's this first one 24 and then sensor data frame comes from sensors which comes from prepare sensors which loads the geometry and then applies all this stuff jeez so we're slowly getting there all right so we need a low prepare sensors all right input path is going to be input here all right so this is a path now we have sensors look like this this is just all those 5000 sensors in the data set so prepare sensors gives us the sensor data then when he creates then he runs it pulls in the data set the metadata metadata like this and this is going to be train mode is equal to train and we also need this underscore d-type so we saved this globally that's smart to do it that way underscore D type is not defined now it is so let's look at this metadata yep meta he's only pulling in these the batch ID and the event ID from this and then he's getting the unique batch IDs like this which is here building the nearest neighbor stuff has his infer function which takes in the data set all right almost reminds me of uh how I skimmed the paper the other night and in bed and it was kind of helpful from what I could understand yeah so maybe that's where I need to go back to is read the paper almost reminds me of how people were trying to combine the ocean depth the water refraction for the starfish starfish counting competition I did not know they did that that's pretty awesome yeah I would think that they would do like some sort of water augmentation maybe but that's intense so this pre-transformed is uh the k n graph filter which comes from k n graph filter whereas I thought this was imported K and N graph oh that's all down here graph net pre transform so this is going to transform our data into a graph event ID is equal match okay so this is just a unique list of of event IDs for the batch so we're gonna do batch IDs which are this we're going to just do the first batch so b equals zero right batch IDs starts at one now so this should be one uh so we're gonna we're just testing on the first batch he's looping through all of them uh batch preds that okay so we don't care about that we just want to create the first batch and see what this data set looks like object init takes exactly one argument mode equals mode pre-transform pre-filter why isn't it like this we're doing good we're doing so good come on we have sensors the metadata mode so it's looking in I think it can't find this file no that's down here super init transform pre-transform pre-filter we built this pre-transform here what's this super init my mom's intent why'd you say that Camus guys I don't know if this is good stream material this is just too intense and I know what you said but it is it is way too intense so patch two models here P equals empath oh that's just to load the model data source why is your code breaking here ice so it's creating the ice submission data set it's creating super init transform pre-filter super init transform pre-filter pre-transform oh I need to these are none so data set should be fine with that good stuff Carlos you like seeing me struggle keep going if you struggle it makes us feel better okay okay okay I guess ice transparency so in order for this I know in order for that function to work I'm going to need to get this ice transparency working might be over complicating this think what I need to do is I need to go into here and actually copy his code over as if it was its own module am I doing the data s-a-u-r-u-s [Music] making a directory called datasaurus and then let's go through this and see what python files he has uh let's make one called okay so let's go into this in here so make a file called run.pi and it looks like his run script is basically all this I think maybe this like utils or something or a knit I need to install Transformers everyone loves using this get cosine schedule with warm up from Transformers Super popular okay so everything else is here then he has a models that pie models that Pi has the actual Ice Cube model from pytorch lightning all right so clearly models up high actually needs a lot of this stuff to run unless he does like from run import Star but that's kind of not cool where does Dine Edge come that's lower down here so yeah there's definitely a lot of imports that are aren't being shown here because you put it all in one script all right models there's the utils is this all models I just didn't copy at all modules yeah we're gonna have to do a lot of imports modules the pi it was this and then utils losses utils add weight decay um Carlos says uh it shows real life learning is so good move from YouTube Feels less laggy and stream oh so twitch is better hey why is this camera look like I'm I always start leaning down as the stream goes on and then I'm just like at the bottom of the screen all right so this is our utils it's just add weight decay function losses to find angular distance score right data sets that pi that's all of this pre-processing not pi tta.pi as inference NTTA I don't know why all this would be in TTA dot Pi but maybe it is and then he has an if main is name if name is Main in here if colonel so this stuff looks like it goes into the Run nut pie it looks like it should go in here then we need to start asking ourselves all right if I run get pass get user what is it going to do it's going to be my name so I need to go in here and modify this to be my name if not kernel all right so this is where he puts his input path if it's local so my local path is is here so I'm going to do this directory what's the direct location of it Mount Robin my disk this kaggle data and then the competition name so he also has a mounted storage drive that then he puts us in kaggle data and I'll do output I guess model cache I have no more space on this disk so I can't have the output B here because there's no more space at least give myself a little bit of room here um so let's do output here is going to be in a directory called out here in my repo I don't know if that's right if that's smart to do it but transparency path is an input directory and you also put ice transparency that txt in there so I'm going to have to go into this data set download this save it to repos no actually save it to let's just save it to my desktop and then move this into this folder or else it's not going to work unless it's there so now it's there move that over we have this output directory which is not going to work it's not going to be enough room for it we don't have a a storage drive for our model cache I don't know all right so prepare sensors where is that bad boy prepare sensors is in pre-processing from pre-processing repair sensors from TTA import this from from Models import Ice Cube model Ice Ice Baby from data sets support and none of this is gonna work because all these other things don't have their required Imports like this loss needs to be imported here we need to import pytorch lightning basically we should just do this import everything into models.py and just remove all the stuff we don't need because it's grayed out oops where's the Von mishes that's down here that's even lower okay everything in here should be working okay and then I can delete stuff later let's try to get this modules this doesn't have tensor did this not import all the torch stuff or maybe I deleted it global pollings that's the stuff he said he added right global poolings we should put this here remove that from here oh geez Louise this utils is fine wait models is not fine it needs dine Edge and that's in modules angular distance score this is our loss function in our loss yeah so import import and then in losses import torch this one's pretty simple why is it yellow add weight decay where's add weight to gate in here from utils import add weight to K now that one's good to go these are data sets all right so this needs a lot of imports robust scaler all of this stuff too transparency path and input why is this global why make that a global variable that doesn't make sense to me after all this making this just a global variable I guess I can do input is input path transparency path is here boom this is not going to work there's no way this is going to work if I try running it is it how would I recommend to transcoin from data engineer to data science maybe try starting out on kaggle uh Camas is just calling out my mom dude not cool those comments are not appreciated what are the odds this will work python run let's just see TTA rapper is in our tta.pi oh we never went to this far all right so important numpy as NP import torch from torch import and end is that how it works uh no it's import torchnn as nnn and then this should be good to go because it that's all that needs and then pre-processing need to input path why is this not here for pandas as PD import numpy SNP and then prepare sensors where's prepare sensor is called run and prepare sensors this is going to take our input path that's kind of nasty but input path is determined here when it checks to see if it's my local user which it is which should be here all right let's give it a try let's give it a try hi Rob hey Skid Row what's up hey this looks quite Advanced topic yeah this is not this is not for the faint of heart AKA not for me from Models import Ice Cube model cannot import okay so I have a circular import accidentally imported yeah basically we can go in here everything that's grayed out actually isn't required so let's do that let's clean this out clean this up this is not needed so now our models is nice and clean we need to change this to also run from a config file the data loader is used here but not the data set doesn't look like this is used or numpy or any of these let's go ahead and save this run is done models looks like the Imports are clean modules don't need pandas don't need get path or this stuff that looks good utils doesn't need anything you're nice and lightweight this only needs torch this doesn't need pie torch lightning or any of this stuff this graph net stuff can't even tell what it needs it does need numpy it does need pandas it needs this data set this is our data sets that pi accidentally pasted models up high there at the top alrighty alrighty alrighty let's try this index out of range no it didn't work because I don't have any model paths that's because I need to download his models his model weights it's currently running in evaluation mode so we need to go to here find its data this is his model I believe fold zero it's 100 wait what 117 gigs this is model download to kaggle clean file oh cool he has a script that automatically will automatically downloads his stuff from to man that's intense all right so download this and let's now let's go back into data source and make a directory called Model saves and let's move this from downloads here results that zip to here and unzip results now we have dine Edge let's remove this result zip now we have the dine Edge directory there early sharing prize dine Edge and this input here is from for the model cache input path where's the model cache hey alexio thank you so much for subscribing perfect time to get my brain off of this stuff is this enjoying enjoy him full so I'm spinning the wheel for you type Pizza 10 times perfect perfect let's go here and type Pizza 10 times pizza pizza pizza pizza pizza pizza pizza pizza pizza pizza pizza pizza pizza pizza pizza ah I can't even type Pizza there we go I'm rounding It Off I typed that for you thank you so much for for the sub Bing with prime for free I love it really appreciate it is anyone enjoying this which keyboard do I use uh this is a Das Keyboard you're loving it super slash okay okay okay I'm surprised by that honestly so what are we trying to do we're trying to see where the model cache is model cash I think he's just saving model cash doesn't get used so where is he loading the model he's loading it from a checkpoint P which is empaths empath equals data set path our glob suffix our glob suffix what's the suffix is metric what what where's the data set path when you make predictions data set paths input f for f in model folders model folders is here okay okay Beyond here lies nothing thank you for subscribing subscribing with Prime let's spin that wheel I think you have to print out items from empaths list just watching me figure out other people's code oh this is an easy one sigh into the mic Serenity Now it says good other person's code though it's not it's not like messy code it's really it's really smart well laid out code I'm just I'm just um there's always like stuff you need to just work out in your own head though if kernel data set paths equals this otherwise oh let's just make it easy on ourselves here and in this run dot pi by the way we didn't clean this stuff up so in this run.pi he has his model folders let's say data set paths is going to be equal to this which we downloaded from this folder let's just keep everything the way that he's doing it let's make a directory called this and move this dine Edge into ice cube and then we'll just do a PWD on this which is this long path we're just gonna hard code that in here and actually this could we could just make this a little simpler like this all right let's give it a try if this works we're done I'm just done figure out the training later and like undo everything that I messed up python run string object has no attribute or glob of course it doesn't because he makes this into a path because he's smart and I'm not and I just I'm using strings Serenity Now for sure why am I getting the same issue takes exactly one argument it's the same thing in the data set is it because I'm using a different version of Pi torch or something that could be it that easily could be it so we noticed that in the um in The graphnet Notebook they install a different version of Pi torch here and if we do the kaggle docker image so kaggle maintains I believe yeah look they just had a patch 11 hours ago so kaggle maintains their Docker image that we're running in the kaggle notebook in this repo this is how they get pie torch to work so we need to see the docker file this is old um so this does installs torch Vision Plus CPU this first CPU version ARG torch version that's an argument let's try running this Nvidia um this Docker file here locally runtime Nvidia hmm it's been a while since I ran Docker locally let's just do this let's just do this copy and edit I really could have opened any oh this is script too so this is going to give me a heartache so let's just do create new notebook let's go import torch print towards that version this is so we could figure out what version they're running in the notebook I'm guessing it's going to be different than the version that I'm using for the graph Network um all right 1.11 point zero plus CPU oh that's CPU version and if I activate graph net and I print the torch version 1.11 is it different in Transformers let's see your import Transformers maybe that's it like the older version 4.20.1 so I'm now downgrading Transformers to a different version I doubt that's it but it could be it's got to be something with pi torch data set foreign because it's inheriting this from PI torch's data set let's see if there's this pre-transform let's search this exact phrase any other ideas people what is going on what is going on that would make this not work this is where you guys help me out get cosigned schedule with warm up this is not needed here let me make sure I'm saving things it's in data set it inherits from the transform pre-transform pre-filter these are none none none and it and it airs here and oh torch geometric dot data that's where it loads in data set so it could be due to the torch geometric versions let's see oh geez Okay so let's see in the run.pi he's installing torch geometrics 2.2.0 now import torch geometric and then we'll print this version 2.0 .4 as chap gbt is chat gbt up right now like I saw people complaining about it being down um let's see what it says if I could ask a specific question hey man let's see hey man been struggling with this kaggle competition so I'm using data source 's code and I and there is something with wrong with the data loader so I think it has to do with the torch geometric version because he uses version 2.2 which version is he using 2.2.0 I'm trying to confuse chat GPT right now he's using version 2.2.0 but locally I have installed torch geometric version 2.0 point four the error I'm getting is when I init the data set and it gives me this error what should I do should I just give up this stream is wants to know seems like you're encountering a compatibility issue between Co and the version of The George geometric I told you that you're just repeating back to me what I said I'm not chat to VT but I'm definitely confused chat TP only traded data up to 2001. not 2001. later than 2001 right 2021 if you're unable to upgrade the library I would suggest checking if the code is compatible with version or reaching out to the author of the code oh thanks however you're not able to operate I would recommend unsolving 2.20 testing code again and seeing if it resolves error if you're still unable don't give up wow it's telling me to keep going there are many resources available including forums GitHub issues and online communities where you can seek help from others thanks so much Chachi PT you're always there for me thanks so much so encouraging let's temporarily let's hope that torch geometric is not have any breaking changes and let's try to install this version that they have that he installed so 2.2.0 and see if this resolves the issue we've upgraded we're upgraded no no we have 2.2.0 now and it's still not working maybe I need to make sure I install all of these exactly the same versions as what he has like this torch cluster they're probably all inheriting from each other so it basically reply to whatever you said and said in the end don't stop yep this is the last thing I'm gonna try and then I'm giving up for now why does it need to build Wheels first torch scatter so I have torch cluster 1.6.0 towards scatter maybe I didn't have yet and now it's having to build it building Wheels is annoying though yeah so I have 2.0.9 and that's 2.1.0 and the build failed but it's still trying to install it using the setup.pi installer normally this is where I would go for a walk listen let's be frank here my name is Rob not Frank but I hope this has been fun for you guys I need to take a break from this foreign if anyone knows if anyone's watching this later on they know what this error is specifically and they could tell me I would love I would love your help what did we achieve today I've learned honestly more about how data source tracks his experiments and code it that makes me inspired it definitely inspires me to do stuff or to be as organized as that and look at this this it had a failure oh nope nope this out of failure um but I I need a I need a break in order to think about this and try to get this uh to run Kebab I think I just realized maybe what's going on I think it's importing the wrong data set it's importing data set like this but in the actual code where does he run import data set so here it Imports towards geometric data set is it always from torch geometric yes it is so never mind that wasn't it that was not it what's the exact air again it's in this one line where it's doing this super in net for the data set class it's saying it takes exactly one argument but we're passing it three and we're importing from torch Geo geometric data set so they'd say in addition each data set can be passed to transform a pre-transform a pre-filter function which by default are none I could force it through by doing so the only thing we actually need is this pre-transform will it work if I do this actually let's just comment out this line instead of screwing it all up I don't think this will work it has no length because it didn't uh super I'm overriding it with torch utils data set I am why am I why why am I all right now hey hey now we're on to something right when you're about to I should have listened to chat GPT don't give up there's a plethora of people like Guillermo who helped me out here Tuple Tuple John Henderson do the Arts do the three Arts need to be listed oh args or a triple one model found batch finished oh so it ran yes we're successful where was I making the out directory oh so here's our submission.csv so it created this it created the CSV file yes let's remove that submission.csv yeah that was it so I'm I'm not sure how I imported data set from a different location from um just torch directly but I must have typed that on my own go back and re-watch like someone can point out when I did it but I should have been loading torch geometric from a data set from torch geometric man I don't know who's watching this because this is not fun all right next time we might take a break from this and stream something different um but I hope you guys learned something here tonight I hope you had fun and I think we might just end the stream right now I'm gonna give you a couple of things a couple of homework assignments youtube.com that's my name you can check it out you can subscribe if you feel so interested to do um you can join our Discord you can also find me on Twitter for Rob underscore Mulla I think that one's me you can find me on Twitch if you're watching on whatever here um and that's about it so go do those those are your homework assignments be kind to each other love you guys I'll see you next time and uh next stream will probably be Sunday I'm guessing or maybe next Tuesday so be on the lookout and thanks for watching