GPT-4 Code Interpreter for Data Science - DEMO

Key Takeaways

hello Community today we're gonna play a little bit with our gpt4 code interpreter if you want you can look over my shoulders when I explore the possibilities here so how do I start I have here a zip file of a CSV of multiple CSV files that describe European r d projects and I say hey do you find a project description in those multiple unzipped files and tbd4 comes let's start by unzipping a file and upload it and then we can explore the content to look for the biologic descriptions so you see here zip file date unzipping file results usually so it seems there's a directory named CSV in the unzipped files but further explore this so you see what it does okay this is the directory and it gets the result back those are our multiple CSV file and as you can see we have here organizations and where is it projects those are the two most important CSV files so gives me the answer yes we have it here based on the file name says GPT for the project CSV seems to be a good starting point to look for the project description let's load this file and inspect its structure to confirm okay now we get an error thus the error indicates that there's an issue with the data structure it is likely that some of the rows have more columns than others this is a common issue when reading CSV files as commas are used to delimit the fields but they can also pee within the field and substance especially within the text so we use a different thing and we use a CSV parser and this is exactly what happens here because as you will see I used a semicolon instead of a colon as here stated the data has been successfully loaded now we find that this is it beautiful and now we have here if you want a very simple CSV file here with the acronym the status the program name the topic the framework program the title of the project the start date the end date a specific URL the objective the cost the contribution the funding contribution when the call was the issue the funding scheme the coordinator the coordinate the country to different participants in the eu27 countries and different subjects so the project description can mostly be found in the objective column let's verify this by having a look at this with the ah not a problem that is there's no objective column which contradicts my early absorption but let's check the actual columns present in the data set I have done nothing it's just GPT doing all of this so it looks at the columns gets back the columns because it seems like the data was not properly separated into a different column instead of treating this semicolon as a delimiter the entire row was treated as one column we need to specify the correct delimit of a lawyer when loading the data let's try again and this time specifying a semicolon as a delimiter finally as you can see here always the delimiter here is a semicolon here in the European case so here we go and we read now the CSV and we have here now the correct delimiter this is done by gpd4 I've done nothing at all so here you get back the result here it's the First Column second column third column then I have here the title column the start date the end date the contribution the costs and then finally finally here I have now the objective here we have now the text of the documents tbr4 says now that the data is properly loaded we can see that the objective column indeed contains description of the project here the objective for the first projects and it just shows me the first project descriptions great let's say great I say now can you take all project prescriptions by the way how many are they and find thematic clusters in their content can you analyze the top 10 thematic clusters in these project descriptions so can you produce a list also of the topics and as a short we can use text clustering techniques for this task one approach is TF IDF the term frequency inverse document frequency to convert the text Data into vectors and then we use a clustering algorithm and the simplest is of course K means to group the vector into clusters this could be a potentially a memory intensive operation yes yes yes so that's okay let's start but by finding how many projects we have so very simple as here looks here just at the project description unexpected reset of the code execution environment interesting then again again okay name PDF is not defined yes this happens to me my pandas sometimes got lost so Tubidy comes back apology is very confusing seems that the state of the was reset and we need to import the necessary libraries and load the data again let's do that now so we have to start all Omega import pandas great then we have the zip path then we have the unzipped and we read the series wheel we have here the correct delimiter now and now we have a look how many come on the OS is not defined I forgot to import the OS module let's correct that and proceed I mean he knows exactly from the error name what's happened and he takes an action to correct it so he Imports now OS congratulations we run through everything again and we have a result we have 5765 project description in the data set great as now let's proceed with the text pre-processing this is usually steps like lowercasing removing punctuation stop words stamming lamatization after pre-processing the text we will convert the text into a matrix for TF IDF features and then we apply the clustering to find a cluster in the data perfect due to the size of the data and the computational resources required this is interesting I have no idea what is here running in the background I super yeah I know it's pipe now I've seen it's a python code but I don't know about what we have so tbd4 says I'll use mini batch k-means variation that you reduce the computation time okay and we use here the elbow mechanism to find here the perfect number of different clusters so let's start so tbd4 writes you the code from sklearn we have it here PF IDF vectorizer. mini batch the import is upwards oh no and we have I didn't show you but currently in gpt4 we do not have the Bing functionality so we have no connectivity to the internet Bing has been deactivated by gpd4 by openai so we will get a problem here so we pre-process the text lower translate remove the numbers wow cool remove the stop words it's the split the text here PF idea for Optimizer fit transform yes absolutely the optimal cluster of numbers from 2 to 15 yes minibatch K means yes and plot it yeah that's exactly what I thought yeah nltk data are not available it seems that an old stopwatch Corpus is not available in this environment is a list of common words yes I can manually Define a basic list of English stop word to use this task it would cover as many words as the nltk Corpus but it should still help us with the text pre-processing so now I mean this is done by gbd4 without me telling it anything sorry I'm excited I think it's gorgeous you find a basic list of English top part chibd4 defines top words then we have the again the pre-processing again after saying what PLT no apologies for the oversight it seems I forgot I mean gpd4 forgot to import the matplot library for plotting will we import here matplotlib okay yes okay and then we have here the album we say yeah 10 is perfect the elbow is around 10 this is 10 is a good choice for the number of clusters beautiful so let's use k-means to Cluster the project descriptions into 10 specific clusters after the clustering we'll analyze the top terms in each cluster to get an idea of the themes or topics of the projects in that cluster and look 24 writes the code of cleaning distances yes we need the metric the number of clusters we have here are mini patch yes we have here the prediction yes beautiful yes cluster terms and yeah it's it's start from the very beginning so cluster 1 has the words learning data machine machine learning a social algorithm AI privacy ah cool research European Innovation social science okay cluster three new energy data Quantum materials technology system research for climate change water biodiversity policy application yeah you get it so we have to ah here we are 10 terms the top 10 terms for each cluster okay so he says okay this is my cluster one and those are my main terms here learning data machine machine learning social algorithms great great I mean I have done nothing I've encoded a single line of python code from this we can roughly infer the following topics for each cluster and now we have the headlines for the cluster machine learning and AI European research and Innovation policy energy and Quantum materials wow climate change and biodiversity heart disease and bioengineers brain and cognitive neuroscience building retrofitting okay Alzheimer's disease therapy cancer and immunotherapy prostate cancer and diagnostic procedures so those five thousand projects are clustered in those 10 clusters but you know that we have no idea about the size of the cluster about the distribution whatever so I say great can you plot as 3D visualization with each project is a little little circle that is color coded by YouTube before so that all 10 clusters are visualized and print their names within the 3D plot 254 comes back I have 3D plot would indeed be nice to visualize the cluster however since our OTF IDF Matrix has many dimensions we have high dimensional Vector space we need to reduce the dimensionality of our data to 3 before we can plot it because we have a 3D plot so either we go with PCA or we go with here our t-distributed stochastic neighbor embedding tisne or of course the best would be Yuma app DB scan and umap and then plot the Clusters please note that this nice computational intensive processing may take some time as a yes Banner so girls SK learn manifold is fit transform great and if a panda data frame now with x y z coordinates create a 3D plot beautiful we have here now a specific color distribution the spectral colors and we plot our things great okay and we get an error back we get a memory Arrow back unable to allocate 2.4 gigabytes for an array with the shape 5K to 55k and data type float64. so gbd4 comes back I apologize appears that there's not enough memory available to convert the anti-tf IDF Matic scenario which is required by tisney so lgbt4 comes up with an alternative great we can use truncated SVD variant of the principle component kind of works with sports Matrix to reduce the dimensionality of our data to down to three could be more memory efficient so you see we import from ASCII learn how a truncated SVD we do the same and here we have our 3D distribution and you see you see nothing because this color coding is well not really optimized and we cannot move in our 3D space so but at least for doing no coding I think the data scientists really have a powerful tool at their hands is the 3D blood of the projects color coded for each cluster each point represents a project the colors represent to the 10 clusters identified this is a simplified representation since we do not add the compute time I know and here it gives us again our 10 thematic clusters machine learning and research energy and Quantum materials climate change heart disease brain Alzheimer and I say the blood is hardly visible please change the color encoding to Stronger colors and find the perfect viewing angle to separate as much clusters as possible so heat changes now here to a chat color map okay and he changes the viewing angle here to a specific viewing angle but honestly you can see the blue cluster the light blue cluster is just dominant so cluster 3 here is is Extreme color-coded yes yes yes cluster again yes so I say okay give me a chart give me a 2d chart with the number of projects in each cluster and it says yes so here we go print and now you see our problem exactly a cluster number three look at the number of projects in each cluster so either there was a specific call here for topic number three or topic number three has a very interesting European Public Funding here for universities or but you can see topic number three great so we have an absolutely yeah this is not even a distribution in any real form so it says we can see that a number of project varies across the Clusters yes some clusters contain more projects than other this is a common outcome of unsupervised clustering yes I know we do not have enough memory uh so I say the cluster segmentation is not helpful we have to recalculate almost assessing equally distributed cluster density in size can you apply umap for this Activity 4 comes back and says U map this uniform manifold approximation and projection is a dimension reduction technique that can be used for visualization similar to Disney but Advanced but it can also be used for General non-linear Dimension reduction yes this is what we're looking for it has some advantage of it isn't yes I know to use umap to create an equal density clustering you use here the hdb scan the hierarchical density based spatial clustering of application with noise this is a beautiful algorithm if you don't know it use it which works well with umap httpscan unlike k-means does not require us to specify the number of clusters ahead of the time and it can produce a more evenly distributed class designs so let's apply umap for dimensionality reduction HTTP scan for clustering these are computational intense processes may take some time might not be perfectly evenly sized never mind so any goes to work look import your map import HTTP scan python library is great use umap to reduce the dimensionality 2D okay 3D yeah 2D beautiful umap reduce the fit transform and then we use the HTTP scan to perform the clustering great great Perfect Party clusters this is great and we get not found error the no module named umap it seems that you might have an HTTP skin lovers are not available in this coding environment and those libraries are necessary for the proposed approach as an alternative you could try the simple K means algorithm again no but with a different strategy to get more evenly sized clusters one such strategy could be to run K means multiple times with different numbers of clusters and then choose the one that results in the most evenly distributed cluster size so you see he says okay currently I do not have the professional python libraries not available but he comes up with an alternative approach so yes okay here's the alternative approach comes up with 30 clusters which is more or less nonsense but so we have here situation where we have simply not enough compute power but for the execution of the Python code but the gpt4 system analyzing coming back telling us what's happening interpreting here error messages coming up with Alternatives coming up with Alternatives that are really focused on the problem like not enough memory where was it this is great and coming up with cluster labels is amazing and this shows us here really the power of a large language model like gpt4 it would be great to have here an interactive 3D representation I can zoom in and zoom out or maybe the end use here augmented reality to move into this High dimensional cluster here but okay but I think for the very first start here I have with the code interpreter with here a more complex file for here a typical job of a data scientist I missing gpt4 code interpreter is an extremely powerful tool it shows you exactly what code it is doing you could say hey copy code and you go here and I don't know in your call up notebook or in your Jupiter notebook that you run locally and you could do it perfectly you could actually run it without any limitations if you have a powerful notebook of course so I think for a first look at gpd4 code interpreter I'm absolutely impressed and it is a powerful tool for any data scientist for any data analyst coming up with new ways how to make this analysis really fast okay it has only the basic tools like principle component analysis like tisne it is missing a more advanced topologically mathematic mathematically challenging and computational more intense python libraries but I think we are just at the beginning and if we would be able to to upload for example here the new python library and tell the model the compute model here the python environment here to to Simply import the new models and run with it I think the code interpreter has a magnificent future out of this thing so I'm really impressed try it out you looked over my shoulders you see what I have seen and I have to tell you wow I hope this video was informative I hope you got an idea about code interpreter and it would be great to see you in my next video