Hugging Face Datasets #2 | Dataset Builder Scripts (for Beginners)

Key Takeaways

today we're going to continue with the face datasets series and we're going to have a look at how to use the Builder scripts so with the build scripts we can either we can do a few things so we can include data pre-processing within the data loading pipeline we can stream from a another sort of remote data source which is pretty useful if you are using a data set where the owners of that data set want the data to be streamed from their server which happens quite a lot or if you know maybe you have your data set split into multiple files or you have images in your data set or something along those lines and in those cases you always need to use one of these data set building scripts so what I'm first going to do very quickly is show you how I created a compressed file for this demo so we're going to create Shades so we're gonna go over here so into this James Callum hey check data sets repo on GitHub all right zero one build script we're going to here and you you will see this file here so this data set tar gz file so this is a zit or compressed file and we're actually going to stream our data from this exact location so if you we go on here uh you see we have this download link download button we're just going to copy that link address and we're going to use that to team our data into into the data data set building script so very quickly um how do I build that you can actually have a look at this file here so all I'm doing is I'm taking this the Reddit topics data set that I've built already very similar to the dates that we used in the last video um it's just a little bit bigger so it's not massive it's 3.7 3.8 000 rows um OLED converts pandas convert to a dictionary or using the the records orientation there and then save that as a Json l or Json lines file then we compressed it using this so you'll probably if you have your own data set and you want to compress it and and kind of follow the same steps to doing here this is what you will need um so you need to add your data set file to the zipped compressed file here and you just use top file that this I I believe is is actually installed by default with python so you won't have to Pivot install that so with all of that we can go ahead and actually have a look at how we build our data set building script so we start with a template first so we come over to hugging face here and go data sets and let's go over Squad Okay so go here this one's just a very popular data set and I think I'm on the tutorials I use it as a as like a template for building your own scripts um and that's probably where I got this from but I I just by default I go to this data set and use this as my template if I'm building a new data set loading script so come over here so within the build script I have a few things here um Json Lines Just you can see what what is in there I can actually delete that I don't need that anymore so let's remove that um what I want to do is create a python file and I'm going to name it the same as my data set so I'm I'm going to call this um the Reddit topics tar gz um because that's what I'm going to call this this data set okay this oh okay we're gonna modify a lot of this but for now I'm not going to touch too much we just want to um what's focused on the essential things that we need here so first thing we don't need this it's like added complexity that isn't necessary plus um we'll call it sort of reddits charge easy I suppose it's fine already topics Builder configs uh this doesn't matter this does matter but we'll we will mess around with that later not now so what does let's focus on what actually matters right now so the here we have this download manager and we're gonna we're gonna look at download manager a bit more in the next video but um for now download manager is essentially a hugging face data sets utility that allows us to given a particular file either local or on the Internet we can download it and extract the contents of it so this is why I formatted the data sets file as a charge easy file because I want to use this download and extract function or method so what we need to do is in Europe so I'm going to change this to URL I'm going to come up here where we Define URL um actually remove that and here I'm going to replace that with the location that I copied earlier so see if that actually that won't work so I need to copy it again so if I go to here um the repair again go to the wrong Builder script the location of the compress file go there and then where's the download I'm going to copy that link and I'm going to put in here okay so with that um description we can you know call it demo I don't know um we'll change the other things later with that we or this here will kind of almost work so there's just one thing so we're downloading this URL but um with Squad there were two Euros okay so if I actually go back a little bit you see that there was these two Euros one for the train set one for the development sir um we only have one so we actually need to modify this a little bit to deal with just one data set not two so here we need to return split generator we actually just remove this one the validation display because we just have a training split and the download files is actually not going to be this it's going to be um so this is basically going to show us a path to a particular location let me show you exactly what it's doing okay so we're going to do from uh transfer data set sorry import from data sets utils import Download Manager it might not be there maybe it's here see okay it was there so DL manager and I'm just going to initialize it this is kind of happening in the background of our of our Builder script so we don't actually do this in the build script it just kind of happens so we do that and then let's just copy what we have elsewhere so we have the URL um and it is this okay that is the Euro and let's just see what this output so download manager and download and extract URL okay let's see what we get um we'll call this okay so we see we get a final path from that now okay interesting so what let's have a look at what is in that file path so OS Lister out so now we can see we actually have that Json lines file that we that we put inside our compressed tar file so okay what what does that mean for us it means we can actually just load that from here based on what the download manager is giving us okay so this is like a cached location for our particular data set so return to the Builder script uh we have download files I I don't really like the name so I'm just going to call it path um I'm gonna say path here as well remove train and if we just have a look at the path um it is it is just the directory that contains our dataset.json okay or Json lines file so actually what we need to do is we need to do like out plus dataset dot Json l okay this this will give us a full path to our file so that is what we're going to do here so where are we um so path let's I mean it's a bit easy to read okay come here zoom out a little bit okay so it will be PA and then here we have data set dot Json out okay and yeah that's how that's a split generators function here and what that will do is you see that we have this file path here that is going to get past along to this generate examples um function or method and is this method that is going to kind of output the rows of the data set to us so what we need to do is actually um just use this to kind of read our data set now it you know just kind of doing that from from scratch without seeing what is happening is kind of hard so let's let's return to the The Notebook file and see how we can do that so we're here we have our let's call this file now because this is what we've created in the other file it's a file path and what we're going to do file path is well first we need to import Json because it's a Json Lions file so we're going to have to um read that and exactly we actually want to do this so as FP and I don't think we're using these but encoding there but we'll put it to be safe um and what we're going to do is we're going to go through that so for line in FP it's a Json lines file so there's just lines of data condos lines represents adjacent object so we are going to well we can we can just print it for now in in here so let's put a count on this so we'll print out a few items but not too many so if count is five break okay let's see what we get Okay cool so we can see that we we get a few items here right so we're just kind of going through there's a red file and we're just looping through and and printing them so we can do the same over and out of there file in the Builder script so so let's come to here copy this in for now um all of this we can see here okay some of this we will need not all of it so let's go ahead and just remove what we don't need okay this is all we need um this yield so because this uh generator exam was creating a generator function right so let's come here remove um remove parts of this so the line or the should call it a record is equal to json.loads line maybe we'll call it object um okay and within that object we have a few few different key value pairs right so what are those we can have a look at the the make tar file them file and we have we have all these here so we have subtitles subtypes up vote ratio ID and created UTC now we can actually just pass all of these directly onto the next so we can yield all of these so let me show you what I mean by that so we come here and you see that we're just yielding and then we're yielding this dictionary type structure Squad all right for us we already have that dictionary type structure because we use the Json lines file this is one of the reasons I like using them so we can actually just do yield key object like that now okay what is um what is key key is actually the index value or ID value if you if you want her but it's an index value so I'm going to rename it index because that makes more sense the meeting key and yeah yeah here we go so we have set everything up here we're going to open the file located oh let's let's do that and read the lines um load file object or Json object your demo so [Music] when we are loading the function or when we are loading the data set over in hooking face data sets this is going to be the thing that generates all of those all those items so what we should do now is maybe we can maybe we can test it and see what happens and it won't work straight away we'll see but let's try so what I'm going to do is I'm just going to copy all of this then I'm going to come over to hooking face I'm going to click on my or icon right over here click new data set I'm going to call it Reddit topics tar GC create that and I'm going to come to files I'm going to add a file create new file and this is just going to be Reddit topics Todd GC so the exact same file we created before and I'm just going to paste all that code in there okay so you can see we have all this code let's let's remove this I think it's that important uh it's not Squad anymore so let's just call it Reddit topics rgz demo data set one thing we do need is we need to import Json so it's good that it's already there um we don't need this anymore but let's keep it in there for now before we start removing everything crank more errors so let's commit that and then let's just try and see what happens okay so I'm going to create a new file to test it so right so test test data set and what we're going to do is from data sets import load data set and the data set we'll just call it data load data set and we can find the dates and names over here so I'm just going to copy click here copy that and there is just one split in this data set so split equals train okay let's see what happens okay the downloads build a script so far so good download the data and then we get this okay what is this OS Arrow come down here okay so if we had a look at this so without this dot here uh we can see that the data file is there so we have our first error um which was not on purpose but that's fine so the reason we have that is because here I we've got a DOT I'm not sure why I did that so let's save that and actually let's just edit it in the and web editor here as well so let's remove that commit changes and then let's try again okay let's come up here let's clear everything restart and let's go again okay so now we get this key error so what does that mean Kiara contacts okay I don't remember putting context anyway so let's have a look at the Builder script and if we okay let's have a look okay here we have this so we haven't modified this yet now what is this telling us um it's basically telling the data set builder which features to expect in the data set so basically uh down here we're kind of feeding in these these different features we're feeding these these records each record is a key value pair so the keys are the feature names and the values are obviously values which have a particular data type now here we have the the feature names so the keys but they are not aligned to our actual data these are using the squad data set key value pairs so need to come over to this file and we can or get those features specific to our data set from there so let's take these I'm going to copy them across here and all I'm going to do is actually just those here okay so we have subtitles self-taste update ratio uh we have or we have ID and we also have another one so let's create another well actually let's make this one more normal first ID is this and then we have one more which is created UTC okay now we can try this it's not going to work again but let's try okay let's re-run this and see what happens okay so actually it does work but it's not working in the way that we might expect so if we have a look at data and zero okay we have sub title self text and then there's a lot in this self text um but so just look at this so the upvert ratio which is a floating Point number is now a string the IDE does fine we should expect that and the credit UTC which is also floating Point number is now a string as well so there's a bit of an issue here basically if we if we go back to our script when we are feeding the features through this sort of feature specification um it's seeing that we're saying everything should be a string and it's converting everything into a string we don't actually want everything to be a string so what we need to do here is use a specific Apache Arrow data type identifiers for different things so for example flow that we have here okay so let's go ahead and have a look at how or what that might be so to find that I'm just going to type like Apache Arrow data types are here so Apache our data types and Shimmers schemas Maybe uh we come here and we can see we can see a lot of these so we have we have integer values unsigned integers and then we have floats so I'm going to say okay single Precision floating Point type is perfect okay so I'm just going to copy that float 32 and I'm going to put that for created UTC and also the operator ratio okay I'm going to save that I'm going to change a few things that we don't we don't actually need so I'm going to remove this task template because we can't do question answering with this data set um no we can't it's at least not extractive question answering or can't train with that uh for the home page let's let's put this I suppose okay supervised Keys is none and what else do we have here so description uh it's a it's a demo we know that okay okay let's save this and and try again okay so I'm going to copy this over into looking face from here uh not here here edit and come here select all paste and I am going to commit those changes now let's have a look at what happens if we load the data set so back over here to data set uh let's run this let's see what happens okay it loaded well it loaded correctly that's a good sign come down here and now we can see that these are no longer strings but they're actually floating Point numbers Okay so that is that's everything there are maybe a few aesthetic things to change here so the like the citation will change that up here I can change this as well but we're not going to go through that in this uh in this video I don't think you want to watch me change citations so yeah that's everything for this video in the next video what we're going to do is take a look at taking this a a little bit further and adding more advanced data types like images into our data sets so until then I hope this has been useful thank you very much for watching and I will see you again in the next one bye [Music]