EXTREME PYTHON & DATA SCIENCE LIVE STREAM

over foreign all right all right coming to you live October 16th it's Sunday October 16th 2022. hope everyone's doing great out there hope you had a great weekend and we're going to jump into some coding here if you want to stick around with us that'd be really fun hope you're doing well let me know in chat if you're around I'm going to test it out test one two three good evening etsw good to see you in chat right over here over here there you are hey just to show you uh extended out the follower goal on Twitch to five thousand it's now bigger I.T speaker and Raju welcome you are on our chat in uh in YouTube so you can jump over to Twitch if you want or you can hang out there um let me get my my link here we go hello it speaker welcome or it's peaker are you its peaker or I.T speaker welcome and hanging out tonight with us I appreciate it all right let's get into the real stuff how extreme is this going to get so extreme Emmanuel you have no idea you have no idea um let's just get on to it to show you how extreme it's gonna be I'm gonna load up the data which I had pulled earlier today and we're going to just jump right into it you said my name I feel so special hey there's like 10 people watching right now so you and me both feel special by that I feel more special than you do trust me all right let's open up our Jupiter notebook like always that's the good stuff where the good stuff happens Heidi Ho another reasonable guy so we've been doing a lot of stuff on this stream that I feel like is related to chess by not by Design as much as just by like I'm working on things that I think are interesting so we might jump around to other things later but I figure we just continue go to with this data Basics welcome to the chat how's it going um my data friend another twitch streamer he is so I was here's the thought last stream we looked a little bit into the chess.com um playing history of none other than Hans Neiman but we didn't really have anything to compare it against so I think what I've done here is I've ran some code which pulls before I did it manually went to chess.com and downloaded all the games of his since then I found someone else's code uh let me load up Studio code to show you what it looks like so this code no this is my code this parsing code uh allowed me to automate this using the chess.com API so I realized there was a chess.com API for the developers and instead of writing it from scratch I saw that someone else had had um already used this code or already written a python package which could pull down everything from a given username so then what I did is I went on chess.com and I went to the top leaderboard for all of Blitz and I went and I got a list of the top 50 players and I pulled all of their games mostly Grand Masters I guess some non-titled players are in the top 50 which is surprising um hey nice to see you little Davis welcome to the street hey Vish vishy Vish welcome to the stream how you doing so anyway so I've downloaded all this data so I ran the script which pulls it all in and I haven't looked at it at all and I thought maybe we could look at it tonight what do you guys think maybe we could make a kaggle data set so we can make a new data set based on this or maybe we should explore it first um chess.com top 50 Eda Eda of the playing history for the top 50 players let's call this top 50 chess.com Eda exploration into the games of players in the chess.com top 50. all right so let's check out what this data looks like uh before we do that let's uh of course do our Imports import matplotlib import Seaborn and let's also import the chess API shall we let's go we're not we're not joking around tonight I'm going straight into it everyone no chance to even catch your breath um so this is the directory with all the data that I downloaded what I did was I have all the PGN files downloaded it looks like they're saved by month so it has the year and month of each playing history players playing history it does download the version with the clock time and it has some interesting metadata that I'm not sure if we had before so it has like the um the the opening URL it also has a link to the actual game and it has the start time and end time and end date we didn't have all this information before so using the API gives us a little bit more data than we download this um by scratch or by hand hey Mr Gabriel Blends welcome to the family how are you doing today um also thanks for the follow vishy Vish I've turned off the sound for when I get a new follower so I thought it was getting a little bit distracting maybe I'll turn it back on but uh if I miss you let me know in the chat so I can uh so I can say something what are we going to do today we're going to look at this data that I downloaded we can make a data set from it it's of the top 50 players on the blitz Leaderboard on chess.com all of their playing history so um what does the data look like what it looked like all right so we have a folder with parquet files summary data of each game name of player per file and then we have folder with the per player with all the PGN game files with with what else with the Oh by month all right so let's look at an example player um let's pull in no read parquet hey N3 n34r welcome to the family thanks for following let me know in chat if you end up following me and you're enjoying what you're seeing let me know why or or let me know in chat chess.com uh who should we look at let's start at the top nahil Larson is currently at the top of the leaderboard on chess.com so let's start with nahil so we can look at this parquet file and what does this have oh we should do the PD set option Max uh display.max columns is 500 something big then let's look at what we have here it's all the metadata that we would find in those PGN files so this is the result oh yeah we need to do that code which which determines if they won or not oh looks looks like some of the earliest games played by this person was against Magnus Carlson is that right just happens to say Magnus Carlson one by resignation huh and then we even have all the mainline moves here include Blitz game mode in the description in What description what twizzers why I like numbers Kobe super helpful interesting and you have a nice soft voice thanks thanks little Davis um okay so tournament no vet let's see if the tournament stuff okay so maybe the first thing we look at is only 746 games let's see if that matches up with what we see here on the actual website if we go to completed games uh um that's not good so why doesn't it have all the games maybe I was getting blocked out 41 000 games on here maybe I didn't download everyone's games let's look at read parquet I was under the impression that I was getting everyone's games but this could be bogus what's the shape of this 746 also oh did they cap me out 746 per player why 746 746 such a weird number anyone I'm looking at the wrong person oh I'm looking at nausea all right let's look here thank you for you're right I need the hills here hey hey mgx just subscribe thanks so much all right spin the wheel for hey mgx how's it going hey mgx why are you subscribing gotta spin the wheel oh wait let's I want to restart this I want to add some more fun stuff to this I need to scream something all right let's make some other options for when I get a subscriber scream Kevin as loud as I can PSI into mic let's make this a 10. any other ideas for what to add here can I quickly show how the data was pulled yeah let's go into that next let's go into that next scream lululu okay let's make this Loop I like saying something short come in like that but I'd do it louder hey mgx I'm back in Action I haven't been watching twitch as much the last couple of weeks because of work so I haven't caught you in a hot minute well welcome back mgx floss dance do the floss all right make that a 10. let's make this a 10 second thing is I need to make these a little bit faster drink water I gotta make that a low one okay I want to make it exciting enough I do have dance on here all right now now we have some more stuff on the Wheel it's a little more interesting to work I don't know how to twerk that I get technically that's dancing I guess flossing and dancing oh boring 10 10 second hamstring sets hey mgx this is for you thanks so much for subscribing if you subscribe on Twitch I'll spin the wheel fewer push-ups I kind of like the push-ups though I like I like having a little bit one that's got to be hard okay thank you for the hamstring stretch that's uh my stream doesn't show that oh it doesn't show the YouTube comments am I aware um yeah let me switch that over I'm just realizing that thanks for the heads up gotta grab this alert bring it over here and do that okay now you should be able to see them see everything now it's kind of hard to read now it's a lot smaller but it is what it is okay back to the data shall we YouTube guys wait uh shoot this is all getting messed up I joined twitch to view your content but why twitch and not YouTube YouTube just feels much more familiar to me that's why I've started streaming on both but I guess it's against the rules I don't know I'm trying to figure this all out as we go okay so I'm realizing I think that no matter what all these data shapes are seven 46. so it's a good thing when we figure this figured this out early um let's look at the dates I think I have a clean version of the dates yeah let's sort the index to so this does go all the way back to 2014 but it's not every game so let me try to figure out why that is um let's look at the code that I used to pull this down so this was uh publicly available someone had put up there which was supposed to pull down all the PGN files this function access to the chess.com public API and downloads all the pgns to a folder so let's kind of redo what this function does in our own code example of pulling down PGN from chess.com API all right so what do we have here imported we have OS import Json and import requests pick a random look at how many were played um I can also just look at their most recent games so it looks like there was a game played today right their bunch of games played today a bunch of games played yesterday so we should see that here but we don't we don't is it because I've only downloaded certain types of games just delete this one it's definitely not all the games that were downloaded they're of their live chess let's see if the events all the same uh there are some 960 games in here but if we go in his page like going back how many of these are on one page like 10 of them what is that more like 50 and if I go to live I should be able to go way back and see each day's worth okay so there are some gaps I guess in data but something weird is going on for sure maybe there's API limitations that's probably it that's probably it and I was relying on this too much to retrieve so let's see if we can run this API this request this git PGN archive links so the username here is username here should be this and I was getting this okay so verify is false so we get this 200 response so as a as a appropriate response and if we look at the content this has all the archives of data okay so the archives look like this huh this looks like pretty easy to pull right the archives are in the format of year and month maybe they don't archive everything could that be hello tickets Please welcome to the family um we are trying to use the chesacom API to pull the data down and trying to data for different players and something funky is going on because we're Limited in the amount that we can get okay so this is the URL of all the PGN names let's look at their API documentation archive monthly archives I don't know why this page looks weird yeah this page looks a little weird but um complete monthly archives array of Live And daily chess games that a player has finished so it just says array so this should get all of the games I think of the you of that player for this date let's go so let's go for an example we know that this player played today and yesterday so if we go to if we just went to the URL of them in this month that's what this file is right um so now it's just only pulling from The Archives which basically is only the response that we get so these are all the archives and then what is it doing after that for URL in Json loads this then it's splitting it and getting the PGN file for that URL seems like we could have just done that with getting we could have gotten this indirectly a different way so if we go into this we actually see that this API we see that there is huh why is he doing this why is he doing this user archives dot PGN oh then he's just that's just to write out to make sure that the file name that's written to go to October 22nd and try to find the latest games okay so here we go uh we will go for URL in here what it's doing is URL lib recruit retrieve this does have every month going back to then you all retrieve this will be my URL slash P PGN file path will be temp PGN hey welcome to the family pleakly import your lib all right so it downloaded that locally to this temp PGN all right so that worked fine now we have the PGN version of this but how many games are here is a question and are the games going all the way up until today well they're going up until yesterday that's the 15th 15th or the late are the lower down ones that more recent no this is going back so at least goes up until yesterday it should have all the games from up until yesterday hey pleakly welcome to chat hey Rob can you show a video later on how to parse PGN files yeah so the parsing part I just use the chess API so let's actually do that let's like is it possible to use any databases for this data set what do you advise um well once we have it in a tabular form format I guess it could be put in any sort of database but I'm probably just going to put flat files of it into upload it onto kaggle because it kind of is an archive that I'm not sure people would would pull up otherwise why the question is what format is what is causing the shape of this data frame to only be 746. so how many games are in this temp file that I just pulled in maybe one thing I should check first is just the Roth the files that I downloaded so we've gotten that far and now if I go into my data directory chess.com and I go into one of these folders and I do a word count let's see 75 files 75 files so this is going back until yeah then and if I go into a different folder there's only okay so this player I only got one two three four five six files but maybe that's because they only played for these months probably they're probably only played in big events yeah they have 570 completed games so if I pull in this player's parquet file 746 again okay oh I think I know what's wrong and it might have not been the API this is the other file doesn't have the games yesterday which other file I think I know what I did wrong it could be something just in my download code yeah this was it this is it this is good this is a good problem that I figured out so what I'm realizing I did was I downloaded for each player I downloaded their p GN files then I ran this combined pgns to combine all their pgns together and I did that using this function now what I didn't do is update every time I was looping through to the new list of pgns oh wait I did here let's see if the file sizes are the same for the parquet files foreign parquet yeah look at this all of these files are identical in size so I think what I've done is I've actually accidentally um combined all the results in the Arc in the parquet format incorrectly I combined them all for the same player that's why when I was looking at this to start with I was confused because we saw Magnus Carlson was at the in the first game okay so let's delete this API stuff and go back to reading in this parquet file and see yeah look at this these are all Magnus Carlson games so this parquet file is not correct it does not have the archive data that we had hoped it would so I'll just redo that part from glob we'll import glob this will let us list all of the files in the directory that we want to copy over and why is this acting weird okay yeah so this is like a this is the best problem that we could have had because now I don't have to read if everything's correct I don't have to re-download data so let's let's do star PGN for this seven ant now let's let's stick with the same first player nehal all right this will be pgns and then um I'm gonna also import sis sis path append this will let me pull in the script that I wrote I think that's how I did it and then from I can pull in import the code that I've written that's parses this pgns uh to data frame come on Scripts PGN to DF I need a c in there there we go yeah thanks thanks for catching that guys that keeping me honest all right so now we have those here pgns and I should be able to like take this first one and do PGN to data frame and it will split out everything into a data frame format all right this is good this is what we're hoping to see and if we look at just the white and black moves value counts well value counts wasn't a good idea um all right so this is good this is good I think that we've pulled it in correctly so now what I have to do is I need to take this list which I had before and basically I need to go in here and I need to remove startup parquet now all those files those parquet files are gone and I'm going to recreate them from the beginning I'm awesome for creating these tutorials thank you for watching Emmanuel you're awesome for watching so what I'm gonna try to do here is Loop through ola nihals games and then zip them together so I was doing this with lip list comprehension before read PGN to data frame 4p in pgns and then I can concatenate these together and this would be the this should be all of the files for this one player put together hello angel welcome to the Stream currently at work but always enjoy the streams oh you're watching while you're at work that's great double work that's considered double work you're working double time he works hard for his money do you all know that song he works hard for his money he works hard for his money it's kind of old song when I was in high school late high school early college I did valet parking at some big venues here near where I live um it was a fun job it was a really fun job but there was one time where I was working like at a nightclub and I had to run like people would come out of the nightclub they'd give me their ticket I'd have to run to the garage find their car and drive it up and I would always just like Sprint to the garage as fast as I could she works hard for the money yeah yeah that's a I was saying it wrong so I I did that maybe like 40 times throughout the whole night and there was a homeless guy like right by the the um garage that I would go into and every time he would be singing that song to me he works hard for his money I thought that was fun when it happened um so this is actually taking a lot longer than I thought it would let's see what's going on here let's see what's going on with my memory use another thing that added spin wheel what to Sprint and drive and find someone's car sing a song you don't want to hear it you've already heard enough for tonight all right so this is still going all right how many games did oh yeah we saw nihil games has 41 000 that's a lot of games to pull in in archive here together so this is gonna be one of the largest ones that we hey large score welcome to the family thanks for following so maybe doing this all in the background would be a better way for me to do this or maybe I could do this asynchronously or I could do this for each file and save it as its own parquet file but then what do I do with that did you take the IQ test what's your result I'm sure my IQ test would be fairly low I don't know IQ tests are kind of to don't they seem like an oversimplification of a lot of factors um all right so this data frame thing is still still chugging along should I stop this yes I will all right let's pick a player pick a piper with a lot less games Magnus Carlson by the way there two m a uppercase and then let's open up the lowercase one they look identical it's just one is uppercase and one is lower it looks like I might have created this one manually like one of the earlier ones yeah so let's remove this lowercase one and let's try to pull these in so the length of the pgns that we had before and I guess it's not about just the length is just the number of months but at least we know Magnus has a lot less and let's at least see if we can get this to run through and parse all of these four months or whatever it is yeah there we go that happened real quick show so maybe I should do and that's the 746 that's the one with 746 games so maybe we could pick a medium-sized player and pull in theirs or maybe we can only go back to a certain date um so let's go to the top leaderboard again ah hikaru's up there how many games has he played probably a lot right 106 months of games explore games four thousand forty five thousand I've now joined your stream can you explain to me what you are trying to do yeah we're just pulling down all so I guess I've already done this before but abder abder him we have pulled down all the past games of the top 50 players on chess.com all of their archived history we pulled it down we have it here and I'm trying to do some analysis to compare players against each other but in order to do that I need to um I need to combine all their files up and so I can have some metadata to do analysis of um so like for instance we look at this player that we pulled down this is magnus's if we look at their ELO so okay so what we're going to have to do here is create a function to make metadata related to specific player um so let's say this player name is Magnus now we need to find out what is magnus's Elo at the time of the game so we need to see if which player that Magnus is so if this so what this will do is if Magnus is playing as black we will locate where that occurs and we will make uh their ELO equals to black yellow and then other times it will take white yellows and pull that in so now what we have is the Players ELO over time and let's set this index to be the date and plot this not numeric okay okay so the ELO is not numeric as type int this should work there we go now we should be able to plot it all right this style interesting so his ELO started around 20 800 28.50 back in 2015 it was 28.50 and then kind of is converged up here there's a bit a lot of big gaps here where I guess he didn't play and I also am curious about all these games where the ratings stayed the same uh let's see what Chad is saying you're looking at the top 50 today right or you can considering the top 50 per year slash month oh this is just the top 50 of today so I took whoever is rated in top 50 today and Polo there BTS World welcome to the stream new subscriber never program in my life but I'm getting into it Vegas you're in the right spot then welcome uh what's the end result I'm aiming for just to have fun and to learn about this data and to maybe gain some insights we started looking at this back when all the Fiasco controversy around the cheating scandal in chess occurred and then uh chess.com released this report so we were trying to think through like as a data scientist what are some of the factors that we would use to explore the data and try to determine if someone might have been cheating or not now I don't think we could ever definitively say that someone was cheating or not but that's kind of what spurred on this interest right and um after we started doing that then chess.com released this report where they said they had strong evidence to suggest that they believed Hans Neiman was cheating in these online games and they list all of them out here so we've done a few things on stream before we pulled in all the games of his and we actually ran the chess engine against all of the moves that he chose to play and did some analysis of that so that's like here might be in one of these where we're running the analysis against what he did and coming up with plots like this where you kind of see like when did each player either black or white make the wrong move or where when were they way off the top engine move because that might be one way that you determined someone was cheat cheating or not um but then we're just kind of interested in from a meta analysis sort of idea is there a way to see if a player is cheating based on just the fact that they just win a bunch of times in a row so Hans Neiman admitted to cheating online before and he said he did so in order just to increase his rating to the point where he's playing people at his level um so on one of our streams we were going through and we're looking at plots like this where we could see his rating over time so we're right now we're looking at magnuses here we had just pulled up but this is Hans and we noticed a few interesting things and this may be that we were just looking for uh proof when it wasn't there but this one really stood out to us this plot so what we have here is a series of three minute games being played over five hours or so in which he just was consistently winning and this might have been because he started a new account after he was kicked off and or was given a new account and then was way below his rating at that point and this was just him catching up or it could have been kind of something fishy and what I was interested in seeing is can we compare this against he can lose on purpose yeah how deep are we getting in this analysis can we should we get my b uh I Davis you owe you if I'm wrong there's a period where Magnus was sponsored by Lee chess and didn't play at chess.com yeah I don't think he plays on it much because he oh he has a partnership with chess 24 also so chess 24 is like magnus's owned thing and it's being acquired or the whole Magnus operation is being acquired by chess.com or is in the process of being acquired so that would explain the big gaps in him not playing this is this is expected uh chess.com is kind of like the Walmart some people will call it the Walmart of Chess like big Corporation there they are making money they sell subscriptions and they're not like lead chests where it's kind of like more open source everything's free so some players just don't choose not to play on chess.com a lot how would you find out if he's losing a game on purpose or doing a wrong move on purpose to between perfect moves by the engine it's really hard Rama and that's kind of why we're kind of looking into it to try to understand if there was any way we could figure it out so the way that chess.com explains they looked into it in their report is they they looked at the playing history of the player and then they also looked at um like moves that were made in key times of the game uh they ignore forced moves what's the other and then the other big thing that we don't have access to is they actually because they're running like cookies or whatever on the browser they can tell when a player is toggling between Windows so you can imagine if you have a suspect move that a player made and they also happen to be toggling between Windows and that's what they're claiming with Hans the big one I would say that McDonald's have tests some people said uh yeah so if you're toggling between Windows you're already suspected those Moon and then the big thing is that he also admitted to it so their way of dealing with it hey Jesus welcome yeah so that's kind of what's going on here he has to be lazy to run the engine on his phone do I speak another language no I just speak English as my main language off topic can you explain conflict drift and can I do adaptive learning or is it mainly with deep learning neural networks yeah I haven't seen any good good transfer learning is what I'd call it maybe with anything other than deep learning models but that's a good question I don't know anything about conflict drift I know about the general idea of drift like if your model is trained on historic data and then it's like there's a systematic change in the way the data is either sourced or the um the thing that you're predicting then you'll have a drift where your model won't be able to predict that well in the future what major Gene learning models we use to find patterns and games I to identify cheating that's I mean that's part of the question we're just looking into this for and this is all for fun too how far off he was on most of them but cookies is tough that one is also almost foolproof isn't it yeah so the interesting thing here is his chest.com strength score which they gave him wasn't necessarily like perfect C-section below entitled Jessica's best in chess cheat detection and selective cheating cheat detection it's part of our cheat detection process we attuned to recognizing different kinds of cheating some often newer players use an engine like stockfish decide every move they make this four machine is obviously easy to detect other players especially those that play at Hans level are more sophisticated and engage in selective cheesy cheating using a chess engine to give advice only in key moments and often intentionally making subpar moves to mask their engine move top players only need to cheat three times a game oh these are quotes uh uh to effectively identify the vast majority of Gene chess.com compute an aggregate strength score strength score is a measurement of the similarity between moves made by the player in the movie suggested by the strongest move the engine so we've done this on stream we've taken all the moves within a game ran the engine against it and um and computed the difference like so you take the sent upon difference between the top engine move and the move played um in a sense it's better the accuracy of the play the longer chess game control the higher the string score will be expected to be since players are more often timed to okay it ranges from zero to 150 where 150 is the closest to perfect chess with the chess engine at maximum depth and performance a score of 100 is properly approximately the highest we have measured for human chess players that have cheese over several game span and 90 is the highest score we have seen the top player sustain over time in classical chess controls pure engine usage alone would typically show scores between 125 and 150 Depending on time device so when he admitted cheating on Jessica in 2020 hunt at IM Tittle with at least one GM norm and the performance was ultimately led to the action of having his account closed having a strength score of 85 point five in three zero games this performance which as as mentioned Hans confirmed was attained via cheating was within the range of strength scores obtained by Tesla comes reports of various DMS in their games at the time of closure so this so what I don't understand is if this okay so this is his strength score when he admitted to cheating and these are all other strength scores of of admitted cheaters how come in the the tournaments that had cash money in them I guess his strength score on this one was pretty high but in the 70s here I guess what they're saying is 70s is still pretty high high 70s is like is Grand Master territory hey man how's it going what does that mean he would make okayish move 98 and then two percent stock fish depth 20 moves I think that's what they're saying do you have a data set with all his games and moves all of his online ones and uh all of his over-the-board games is it only online cheating so this report fell short of saying that there was any proof of over the board cheating so the sample size also with over the board cheating is so much lower that it's obviously harder to to say if anything was luck or if that was not legitimate but yeah I think the accusation is that there was also cheating over the board all right let's take this let's take this uh Magnus data frame and see the day he played the most so the day he played the most was actually very recently 10 12. um was this day now now since we have this times I think we have the times the start and end time this is the cool thing and I've always wanted to look at this more we can actually see the UTC date and UTC time so we can create like a date time column and look at throughout the day how his games were going so let's add this to UTC time and then convert this to a date time we'll call this UTC date time we'll set the index to be the UTC date time and then we'll do player ELO and plot this okay so this is almost the opposite of of what we saw going on with Hans going on with uh with Magnus on this day like a downward trend does the music bother you during programming um it when I'm streaming it definitely helps when I'm not streaming I I uh tend to uh I can go either way is it music bothering you out because I can turn it down I have it pretty low though all right so this is the trend throughout the day downward downward downward trend um one other thing is we could see oh we did this before no that's not it is we figured out if the player was winning yeah get player result so I wrote this little function where I can apply get player results I think oh and then axis equals one how do I oh I then I can do args player equals hmm this should be a tuple me oh maybe the arguments just need to be passed like this foreign arguments were given is that because it's taken the length of this args are let me just change it in here all right this is the player win loss let's Group by this player win loss and I got a group after setting this index there we go okay so now we add in this legend legend the blue dots are draws the greens are winds okay so there's a big uptick in winds here and the oranges are losses the player WL variable is not a rate that's just the um the result of the game because if you think about it this data frame we have um a data frame that is specific to one player right now we're looking at magnus's games so we're like specific to magnus's games what are this data frame tell us and Magnus can be either playing the black pieces or the white pieces for each row we could have another player that we care about in another data frame which has the exact same game and we only care about aggregating up their results but for this data frame we care about magnus's results so we're looking to see what what color was he playing and then who won so this would be a draw this would mean that the white piece is one and if they were playing as the white pieces this would be a win if he was playing as the black pieces and it was zero zero one then it would be a win otherwise it's a loss so it's like finding we could find the rate in this for the player but it would take a little while so um uh so Magnus on chess.com has one 498 of all of his games uh so we could take out some of this with his 746 we could do this divided by so 66 of the games wins 17 losses the weird thing is just how much back and forth this is it's kind of like if you he's almost 50 50 winning and losing at here in the left part um but he's going down in rating because everyone else he's playing is probably a lot a lot worse than him uh what are people saying in chat your current rating David J more difficult to maintain 3200 ELO than 27 I think you should plot mid-range Grandmasters like keycore oh that's a good idea yeah so let's look up uh let's look up uh I mean I've only pulled the data for the top 50. but you're saying to pull like I guess these are based on their their um fee day ratings but let's uh let's pull another player let's see if I can pull another player's data So eventually offline I should probably go through and have this parse through all the stuff like hikaru's stuff but yeah let's oh what I can do is I could pull in hikaru's pull in hikaru's 2020 one plus games right so maybe it was too much going all the way back into history for certain players but what I can do is pull in pgns when the file name starts with 2022. uh 2020 I guess anything that starts with 202 would be 2021 2020 on did he play much before that see what the length of this 106 versus 34 yes so definitely a lot less pgns to parse but probably a lot more games within those pgns someone asked about my parsing code so I basically pull in this PGN I use the chess API to read it with all the metadata and then I convert that into a data frame and aggregate each of them on um just so we don't sit here like looking at the screen not showing how long it'll take let's import tqdm and not do this in lip list comprehi yeah list comprehension so 4p and pgns and we'll wrap this in tqdm uh uh we'll make this will append this reading and then our data frame will be concatenation of these data frames with the reset index and copying them got it oh no did I double click it did I double tap this all right so it's going through 34 files and it's actually taken a wow so this will take four minutes just to run for and it all depends on how many games he played per month so if you played a lot in a certain month it's going to take a lot longer to process just to show you how this PGN to data frame code works and I probably could make this run faster I'm guessing the time that it it's slow is when it uses the chess API to read the PGN but basically we're taking in all the Header information and in a a PGN file that header looks like this again let's open maybe this recent one this is all the Header information so it reads that in it reads all these Mainline moves which is everything here these are all the moves and we actually have the clock time too and then it Loops through each of them in the PGN file so the file will have everything for the month just back to back in here and it appends these lists which I then concatenate the headers into a data frame and I add in these made line moves I also try to coerce the date column so we're using this code when I was parsing some some games that were over the board from the uh this week in chess archives and this seemed to work okay because the date was not always a consistent format we had to coerce the errors so it's a pretty simple script this script that runs input is a tensor that encodes the board State and output is a tensor encoding oh you should make a Neiman I missed the beginning of that joke BB astronaut you should make a Neiman neural net input is a tensor that encodes the board State and output is a tensor encoding all the possible moves and the probability of each being made the games where he cheats the neural net would perform poorly because he makes uncharacteristic moves it's hard to train models like that because I don't know if you're just making a joke or but if you it's hard to train your mouse like that because we don't know the ground truth by the way we have some new followers um optron Trini Lil Anna Yusuf zippy kig Kali welcome to the family you're all part of the family now ever thought of getting this data and joining the data visualize into a flask website maybe eventually I'm thinking first just aggregate that at all and put it on kaggle that's my MO so on on a previous I mean on a lot of streams I've been making data sets on kaggle by the way you can go to exclamate shin kaggle to get the link to my kygo webs uh profile but I've been making data sets with a bunch of different stuff and like this weekend chess archive I put on chess and then they're on kaggle and then we can do analytics and stuff on here that we can share with the world and they can copy it because our notebooks will all be public hey Yousef what's up welcome to the chat your favorite topic is mushroom slash pepperoni perfect um other things you can check out exclamation point Discord will show the Discord um YouTube if you're watching on Twitch that'll take you to my YouTube page which I would love it if you would like to subscribe on there and um yeah I think that's about it we also had a competition just so you guys know uh we may probably run another one of these soon but we had a it's corn big lumps and knobs seed image classification competition this was an open competition for anyone who wanted to join it a lot of people who watched the stream joined it we had over 90 participants and we gave away the prize was this RTX 3080 TI like beautiful GPU that I mailed by the way I need to check with Andrew to make sure that it got to him I haven't checked the um tracking recently but I hope it got to him all right with all that talking done now we should have a data frame which is all of hikaru's games taking a while to concatenate maybe that's what took so long and last time when it's concatenating so what happens when these data frames get really big and you try to concatenate them you can have like memory issues I know I have like personally I have memory issues but I'm talking with the computer it can have issues with running out of memory or really exploding the memory up it doesn't look like that's a problem here but you know Yusuf says I'm a junior data science really nice to see someone doing data science stuff on Twitch awesome yeah there are a few of us there's there's Nick when there's um data Basics who's on here uh David J probably a few others that I'm missing but yeah the it's a small tight-knit community of a data scientists here can you give me fbid what does that mean Pro 24. what does fbid mean FDIC do you want me to insure you I'm very confused about why this has taken so long oh shoot I'm an idiot I'm an idiot I just ran the same old version here at the same time so I should have deleted that I did the concatenation here already now if we look at the shape of this Hikaru has played 17 000 games this might be a better way to do it we actually will we could run for each player and make a parquet file for the player year that way it's a little bit easier to to pull it in uh Graham says I think it's nice to see someone doing data science on YouTube thank you I'm glad that you're hanging out and watching oh Facebook uh I don't really do much on Facebook oh I also have Twitter I haven't put my Twitter handle here if you want to reach out to me that's probably a better way to do it yeah not a big facebooker over here one more question can I do adaptive learning with svm or is it always deep learning there are umer there are some papers out there that you could read about it but it's not something commonly done to do any sort of transfer learning other than deep learning from what I've seen it might be out there I think I I saw some papers come out where they claimed it worked well but I haven't seen it usually for me my my judge of if it's legit or if it's just a paper someone put out there um that mayor may not be reproducible my when I start buying into things is when I see it working on a kaggle competition I might be biased in that way but I know then that it's probably a legit approach okay so here we have this now we need to we should make a a function called yeah we should make a function called format for player maybe where we give it the data frame and then it does a few of these things like it finds out the win losses so like this all together oh we can also have it do the UTC daytime part so we also need the player name it'll do this it should do this date time part and then I need to figure out how to do this mapping with a variable in my apply statement because I want to also do this basically but apply this with the player name so pandas apply with arguments so they have it like this foreign apply get player result axis equals one and args will be equal to Hikaru do I need a comma after this yes that was it I needed a comma after that all right so now we should be able to do this where the args has the player player name in it and then this and then return the data frame look good format for player and then we could do player name equals Auto this would be kind of fun so if White so what this will do is take the the top player in that data set and assume that's the player name if player name is Auto then it will make the player name equal to this which is the top existing player in that data frame then we don't even need to feed that in look at that there we go uh if you had to learn statistics and Maths for data science where would you start from uh I I learned well I think having a good foundation in math just in general is good and then and then uh taking a look at some of the online courses because I learned through grad school it was really helpful to take that intro the statistics course it was really helpful for me I forget what book we used um but I'm sure there are a lot of good resources out there I'm not sure of a specific course that I would recommend but I'm sure you can find something good out there sorry my answer isn't very good is this NBA 2K Jimmy snap says no it's not transfer learning for tabulator doesn't make too much sense tabular data isn't reproducible and as images are images yeah a lot of good points uh Jeremy Howard talks about it in fast AI course that's cool it all starts with stats yep that's true uh what are your grad courses yeah a lot of good chat a lot of good chat going on player win loss value counts okay so he's played a lot more games still a lot of wins ah I wonder if because a lot of them are not rated that's my that's my guess with him so like okay what's this over so remember how it looked like like Magnus was like 66 win rate Hikaru is 76 win rate but also this is for like a lot more games than what we had for Magnus uh let's see if this shows rated or not as one of the there is no rated it might be variant no variant's just going to be like these crazy um chest 960 and then these other things like the duck chest that everyone's playing these days you guys heard of the duck chest does everyone know what duck chess is um maybe it's the event name maybe it wouldn't be considered live chess but I think no maybe these are rated games you've never heard of duck chess oh okay so duck chess is this variant that they have on chess.com now where you make your move uh uh like Eric rosen's big on it where what they do is they make you make a move and then you have to move the duck somewhere and when you move the duck somewhere that completes your turn you must move it too you can't leave it where it is and uh it blocks your opponent from making any moves on that square so like when he moved it here to uh when he's moving it here to right in front of the king the queen is stuck in and this bishop is stuck from moving onto that square and I guess the king too um but those are like the variants that I think would show up in this data set as variants what I want to see is if they're rated or non-rated games I guess I could see if the following game if the rating had changed but let's just assume these are all rated because they I know they play a lot of like against their followers against their their chat and then they uh yeah oh this is interesting is it really that the first game is versus Hans Neiman black White date yeah just so happens that one of the first games in 2020 played against Hans random all right so let's look at his uh let's look at time controls uh what type of time controls does he play Hikaru free uh most popular most played time controls there we go we'll keep the top at the top and let's do uh let's just do like 10. because some of these timer controls he barely plays it looks like there so mostly 180 which is two minutes wait three minutes so this is like the most popular uh Blitz games to play when it was three minutes then these are one minute games and this is three minute with a one second increment I'm so I'm surprised that there's not a lot of um 180 plus 2 because it seems like that's that's a pretty common type of game so if we did DF uh Group by no let's query where time control equals 180. set in uh group I result set index to be the UTC date time uh player ELO and then plot this style is this there we go still need oh so we need to figure out what these really low ratings are about why would there be jumps down there but you could see that his play is much more consistent much more filled in than what we saw with what we saw with uh Hikaru but obviously because Hikaru doesn't play uh sorry that's what we saw the Magnus but Magnus doesn't play on chest.com a lot Hikaru plays on it a ton so it looks like he was really killing it oh this is right when covid hit look can you tell when covid hit right here in March he just started playing non-stop online and here I thought it was impossible to over complicate chess that's funny what else are you guys saying maybe pandas how to open files simpler eda's a little math and stats oh you're saying you're suggesting what to do what kind of example projects are you guys recently recommending to start for doing for data science losing all the knowledge I have check out my uh YouTube exclamation at exclamation YouTube I have some example like sentiment analysis project I have one where I'm doing economic data analysis okay now Mr Gabriel is speaking in a language I do not recommend or I do not recognize so I will let you guys sort that stuff out we are speaking in Portuguese okay now I get it all right so what else is going on here we could look at we could look at so we can look at hikaru's date uh when the date if we Group by date and then we look at player ELO or we can just do it this AG player ELO min max oh I'm just realizing someone asked me what IDE am I using I'm using Jupiter lab if you check out my YouTube channel and you will see oh we're getting spammed if you check out my YouTube channel I have report Ed and removed but uh yeah I have one with nltk and with using Transformers um let me switch back and forth and see if that makes it go away no so yeah I'm using this IDE I sorry that just threw me off there for a second but if you I also am using vs code to write some of this code here in a different window so usually what I try to do is like develop my code in Jupiter lab and then I'll move it out into its own script or if I'm running a script I'll run it here in vs code and then I um yeah but I'm doing the exploratory stuff here so this is the for each date the Min and the max ELO so if we plot this yeah so what this is is for each date how high does the ELO go and how low did it go oh let's first let's look at these examples where the ELO is very low so let's query these time controls of this I better some weird variant maybe that's why the ELO is so low where the time control equals 180 and player ELO is less than three thousand let's do less than 2500 so we can see some of these weird ones that are really low so here yeah this is a three check event so it's a special variant of Chess where your goal is just to check your opponent three times so you can see he won by three check variants so maybe we should add to this query where this is 180 and variant is n a now we still have values that are in there that are low but much less let's see what these are about let's see what these are about okay so these are all under 2900 delete this one this is odds chess shouldn't that show up as a variant so I guess we could filter these out based on the based on the event name and sheds not in I don't know if this is going to work an event that kind of did not work uh Kyoto hi I'm new to programming and I'm learning C I don't understand what you're doing here but it's fun to watch anyways yeah thanks for hanging out and watching anyone know wrda I don't know that try to plot a can of plot for stocks opening close oh that's a good idea that's a good idea um let's see what the best candlestick yeah okay I knew plotly had it so maybe we could just do this um the thing is that I guess yeah I guess what um stocks usually have for the day is they Group by the date and then they have like the start so that would be like the first I think and then last would be the last value and then they have the Min and the max value so the Min and max value could happen in the middle of the day which people care about for stocks so we're basically like aggregating this as if it was stocks we're going to reset our index Maybe uh things I want to flatten these column names so this is our DF aggregation we're going to call these column names open High uh no this low high close so we're we're pretending like Hikaru is a stock here and then we do have the date column we're going to set the um yeah we should probably want to take this DF AG date I haven't figured out a good way to do this inline make this to a date time all right now our data frame aggregation shows on that given date the open the low the high and the close we'll take out this whole Apple stock ticker thing which is open oh I don't oh shoot this was opening up a new data frame which we definitely don't want to do oh shoot I just overwrote our whole Hikaru data frame I should have been smarter about that well this takes like three minutes to load so let's go ahead and do that while that's happening we can talk a little bit about this you will subscribe to you thank you so much slow welcome D Leo is that D Leo is live is that your name they got min max and close I want to buy a car stock price how much does it cost it depends Buy Low problems with naming everything data frame yeah true true and that eventually what I'm going to be doing is after the stream is I'll let this sort of thing run for each player and we won't have to wait around for it that's what I honestly thought I was doing before but I had messed up my code so I thought all this was going to be aggregated for us and we would have had a data frame but I probably right here or maybe down here after I run this processing and I'll run this I'll write all this in a script where I'll Loop through each one and run the script somewhere else um but we should probably save this as a parquet file and this should be in our data folder it should be in chess dot com and then it should be the player name dot parquet and we talked about doing this by year so that's another thing it could be like player name year like this don't uh forget to change data frame to data frame AG here yes you're right this is going to be data frame aggregation also I should delete this line of code right thought I already did that but yeah so this will save out do we actually put the player name anywhere um we did not because we have it doing the Auto player name but let's just write that here and then the year is going to be 2020s it's just an example we'll actually do it by year next time so what I like about this channel real world stuff happens yeah you live and you learn so I'm a little disappointed in myself that I didn't have all these data frames loaded before um but I think we learned a lot and I kind of walked through the process of how we can do that now uh we looked a little bit at Hikaru we did see that there are certain days where the high and the low can be drastic there we go here's our candlestick good idea with the candlestick this seems kind of crazy oh I know what's going on here I know what's going on here huh we need to query all this stuff out out or else it's not going to work we have all the variants in there that's that's why it's looking so weird um so we want all the situations where the event name does not contain odds chess the way I was doing it in this query was not working and then we also want time control is 180 and the variant to be n a this should give us a good data set to aggregate and why don't we just load usually I let this do all the formatting for me and now if we do the high lows now it looks a little bit more reasonable so if we're looking at this correct the close close of one day 31.90 should be the open of the next day but that's not the case here could that be because we need to sort values by UTC date time and then reset the index and copy that yes now things look like they line up a little bit more so this closes at 32.82 and then it opens the next day at 32.85 which is the ELO so this is the ELO right before the last game so there's a little bit of an offset but though only difference between days should be that one last game score what do you think you're going to end up using to detect cheating I I don't know BB astronaut we may not actually get to it ever I have some cool stuff I want to show you guys do you want to sneak pre peek at the stuff I was I've been doing with the chess Vision stuff the computer vision stuff okay you guys want to switch gears here for the last like 15 minutes all right glad you're all here you're going to see this so we've been doing stuff with trying to do computer vision on a chessboard right if you look at my channel on YouTube nope that's not me um if you do exclamation point YouTube Hey Arda and I don't know we do this one you could see back when we were trying to do computer vision on the chess board and have it detect the positions so taking in the positions try to try to guess the position based on computer vision now one of the problems is we don't have a big data set to do that but I've been working on this I hope this doesn't Crash My Stream using blender to actually create these data sets so there's this chess COG algorithm out there that supposedly that someone released as their Master's thesis where they had all these generated boards and then they tried to detect the position based on these images that they generated however it didn't really translate well to the type of pieces that I was using and even when I try to use a a wooden chessboard it doesn't really adapt well to that I also wanted to see if I could create this for um the standard chess set and I want to also wanted to create an algorithm that would try to detect this end to end as a neural network instead of trying to detect the board and then detect each square and then detect the piece on each Square might not end up working but the main thing I wanted to do was actually use blender to do this for me and the cool thing is I actually got this person's code running and I was able to modify some of the watch this not work I was able to modify how the images looked in the materials within the image why is it freezing up on me well I can at least show you here yeah this is when we're doing it so we wanted to work on a chessboard type like this like the standard tournament boards because that's the most common use just like on the street that you would see um so they these are what the generated chess boards look like that I was creating there we go it's now it's showing in blender too so now we have this 3D chessboard right and we can have it set up it takes a little while to render that's why it's it's a little bit odd looking here with the light refractoring but um this code can pull in F any Fen position and then set up the board in that position and what I haven't done what I haven't done is how to actually change the camera location and change the lighting but with with doing that then we would get a really good synthetic data set to train a neural network on so that's the idea with this maybe we could talk about this next stream uh but definitely need to figure out that lighting issue I had to change some of the code to count for the sizes of the pieces being different and yeah this in the end would hopefully be pretty cool to to get running and at least to validate models on now one thing I noticed too though is uh like the pieces don't necessarily exactly match like these these type of pieces especially the queen I don't know if there's a queen here yeah this queen looks really different than the queen in this 3D board which I think is Frozen is it Frozen just lagging so like look at that Queen in this so I paid for this model yeah I I try and I have found other models that work now the other thing that's kind of cool that I started looking into was closed blender because it's freaking out right now the other thing I started looking into was okay what if can I actually get my exact same pieces almost identical and there's this program called mesh room and I don't know why it's not working now okay oh I just opened up like 10 versions of mushroom let's quit this quit this and what this is supposed to do so mushroom you you can give it let's Show an example so you're supposed to be able to take a bunch of different pictures of an object just from from different anger angles and then have it display in 3D now I wasn't getting it to work very well when I tried this on my own but in theory if I was able to do this well then I could take those pieces and replace the pieces that I'm using in the 3D rendering right now Overkill Maybe um I just heard a weird sound but I think it's our dishwasher I thought you said your live sessions are Tuesdays and Thursdays Sundays Tuesdays and Thursdays Thursdays prom sorry and not every time every week um okay so what's the other thing with that that just the last thing to touch on with that was I did train a YOLO model on it um so if I go to YOLO V5 I should probably use YOLO V7 but YOLO V5 I don't know I'm kind of partial too because it seemed to stream well when I run from a camera um so if I open this up this is what the training batches look like so I was having it it was having issues detecting the colors so I thought maybe just to have it try to detect piece type like this and then the results and I think it's definitely overfitting uh but the results of this if I open this this is our validation loss was just like getting really low the longer I traded it it looked like it was probably overfitting to the fact that I didn't have the camera angle changing and all the pieces were identical um isn't yellow better than mobilenet SSD I don't know I haven't used mobile net much in theory you should have iPhone you should be able to do mesh really good with its laser camera thingy yeah I need to figure that out I need to figure that out um so yeah so I did that and then I could show you example of me trying to run the detection see if this is a good one so you can kind of see what's going on here I mean it's not it's not horrible it's definitely picking up on pieces but there's a lot of overlap and I think I think part of this is the fact that when it's using these training samples there can be overlap between pieces like uh if a really tall here this queen is in the way of The Rook so then the Rooks bounding box actually has a queen in it in theory that with the more training data that shouldn't matter because the model should learn which part is the rook and which part isn't The Rook but it's definitely having trouble with the amount that I've trained it here uh let me show you another example let me show you an example maybe where I had the different piece colors oh yeah if I didn't set the threshold really high too it was just picking up pieces everywhere yeah so this is what I was trying to predict the actual color of the piece also so each class was like a different piece and color and it was having a hard time with this and I figured if you can predict the piece then doing the coloring would be like a you take each piece you identify it and then you take like the average color within it and you could cluster it pretty easy gotta train it on the tops of every piece yeah but what what about when the top of the piece is covered by another I guess there needs to be like a minimum angle that will that this is viable on if it's if you're too far horizontal like I think some of these game data which I was hoping that the like this I don't know if it would be able to pick this up the thing is I'm also convinced that anything that a human can identify on the board should with enough data the model should be able to pick up like a human could easily well with time see that the these pieces are here like it's a little hard to see this black Pond back here um but otherwise you should be able to identify where the pieces are and then place them on a board in theory I don't know I guess that's why it's an interesting problem because we don't know how how to solve it plant vases being classified as a pond yeah so I don't know why all the analysis I'm doing on stream lately has been chess related I don't mean it to be that way maybe we'll snap out of it here soon but um so far it's been fun I I think these projects are kind of fun if you enjoyed watching if you could do a few things for me that'd be awesome um there's a model know where the board is currently no so currently the model just is trying to identify pieces but I don't think the final model would even need YOLO it shouldn't need to go to YOLO it should just I do love chess I guess I'm so bad at it is a thing so if you guys enjoyed this do a few things if you haven't already subscribed to me on YouTube please do that give me a follow on Twitch so I'm putting these links here give me a follow on Twitch if you don't mind link is here I'm currently working on any competition I don't have the time I just wish I had the time to work on kaggle competitions but I haven't I need to do that on one of the streams some sometime here soon is just dig into one of the new competitions usually this time of year really interesting ones come out uh Michael says you enjoyed the stream first one you watch you need ideas future box a temporal Fusion Transformer seems fascinated yeah let's look at that um thank you all who followed on Twitch too I think we have like a bunch recently is 2K1 uh Universe Ry call Gonza beat De Beer we're gonna get we're gonna get to 5 000 followers like in five years never used twitch stream but might set up to follow you yeah if you put like the Bell on that you'll get alerted every time that I stream uh we got steel lever so everyone who joins twitch family who follows me on on YouTube and all that stuff um yeah so if you do all that you're considered part of my family you're part of the family and I appreciate you having spending time with me and hanging out um so what are we gonna do here do you have some side uh pipeline to use in competitions or do I do it all from scratch it depends on the competition and also I have it depends on who you team up with um but I did kind of have a way that I would do it at least for offline competitions where I'd kind of track all my out of folds and I track all of my predictions and I went through that in one of my streams but also someone's adapted that code into their own kind of code base I'll try to find and uh put out there but I think they they made it public I think your stream randomly showed up on YouTube homepage tonight because you were doing this chest stuff oh Graham nice so I guess I'm reaching people that are um that have similar interests as me very cool all right so the goal is to get um to get interesting enough that we get Hikaru to mention one of these streams so I need to look into hikaru's data and find something interesting that we can share um hey Mitchell thank you for subscribing I gotta spin the wheel for you look at that we can end tonight on a spin of the wheel and then I will also oh I'm not showing the wheel um scream Kevin as loud as I can here we go Kevin I'm gonna wake up my wife and kids upstairs they're gonna come down screaming look into his speed runs oh yeah I need to look into his but he creates other accounts for their speedruns thanks for subscribing Michael I appreciate that I yelled Kevin um we got another subscriber on Twitch AK rwp thank you you guys really don't need to but hey if you want to subscribe and especially if you subscribe using prime uh it's essentially free if you have Amazon Prime by the way what's the score of that football game I was gonna another spin we got play a one minute chess game okay that's a low odds one I guess I'm streaming a little bit longer no one wants to see me play a horrible game Jake programming thank you for subscribing four months let's spin the wheel for you so I can see what that is before I play this chess game I really don't want to play this chess game it's gonna be horrible Kevin all right I yelled Kevin again I like that it landed on that one twice oh man we got spammed again someone blocked the spam people okay let's go here we go one minute chess game uh I usually play one minute on my um on my iPad so I'm not not super solid when it comes to playing quickly with a mouse oh no here we go here we go pin this try to get there and then get this here we go here take this no where are you running off to I want to trade Queens let's go if I can get this here that night there then I get yeah tack this oh no that Pawn's hanging that pond was hanging oh that's this is bad yeah I know I'm about to get checked here check made it here I'm losing this Pawn let's go try to flag them you didn't need to do that dude oh yeah he would have been mated if he got it with the other Rook yes get him on time let's go there come on come on no that was a bad move by me uh I let him off the hook two seconds uh I won on time I went on time who is Kevin by the way I just think it's a funny name to yell all right so that's the end of the Stream let's find someone on Twitch that we're gonna raid thanks everyone who subscribed everyone who hung out tonight it was a lot of fun let's see who else is coding coding with strangers is on coding with strangers I like he's a cool guy we've raided him before he looks like he's coding python perfect coding with strangers man you're getting a raid uh please stick around keep the Positive Vibes let's sent him a lot of energy if you're in twitch if not um thanks for hanging out and I really appreciate you guys have you ever played chess throughout my 30 years I have never played chess on my throughout my 30 years of existence promising I don't understand it you'll get it you should you should learn it it's beautiful game all right have a good one bye everyone all right bye YouTube bye YouTube have a good night