Code Interpreter GPT-4: multiple files upload Adventure

Key Takeaways

hello Community gpd4 code interpreter I upload two CSV files and I ask hey can you analyze now both file and gpd4 comes back and says yes of course so let's display the content begin with the first CSV file and you see there is a problem parsing here the CSV file now ask me hey another topic another way to open this is yes source and I pretend to be a no coder so I do not look here at the python information that is provided here you know you see here in this white and green boxes when you click show work you would see the python code but I mean no code I'm just using this to get the information that normally I would use a database for so let's have a look and I say hey can you find a cross relation between both files and GPS work I'm certainly to perform a cross-relation we need to identify it shared keys or attribute so we need columns that have a unique identifier normally it's a numerical identifier so they look here at project ID no there was an error I apologize for the oversight let's correct this and gbt4 comes back and tries another way another problem okay and we can see here cross relation I need to split these columns at the semicolons to extract the relevant data points let's start by extracting the project ID or ID from both data frames and tells me hey it appears there are no shared entries between the two data sets I can provide a few sample entries for both data sets so let's inspect it manually and say okay let's go for it so look here it shows us identical numbers absolutely identical numbers but it is not able to understand that those numbers are identical and are the unique identifier that's okay I'll I clean the project ID columns by stripping any unwanted characters and then I recheck the overlap before comes back it's surprising that even after cleaning there still appears no overlap my goodness comes back and says hey should I try something else yes so gp4 says okay I'll start by tracking if this seemingly matching project IDs values from our sample truly exist in both data sets and the result is does not contain any of the samples and the second file also contains all three of the sample ID values okay okay I'll activities first as I'll re-examine here the project so so another time okay yes yes yes I checked now if there's an overlap and it is working tbd4 is syncing with all of its parts like unexpectedly even after corrected extraction of the ID values there still appears to be no other overlap between the two data set given this surprising result I'll display a sample of newly extracted project ID values the complexity of the data and the project CSV file is causing extraction challenges GPD 4S challenges given the complexities I'll take a different approach I'll display the first lines as raw text and this will allow us to manually identify the correct position and a pattern facilitating a more precise extraction process so gpd4 does not give up it's fighting it's fighting to find here away goes on 234 says I'm genuinely surprised even after the robust extraction process there still appears to be no overlap between the two data set gbd4 is surprised can you say that now I think that minimum 10 different approaches for me as a non-coder the gbd4 failed so I tell now gpd4 hey both files have a very special delimiter and it is not a column like in America but we here in Europe we use a semicolon delimiter and now this is the information that gpd4 needs I'll Repros pre-process both files using a semicolon as the delimiter this was it the difference between a colon and a semicolon made all the problems to D4 could not resolve imagine yes we successfully structured the data frames yes finally yes I have it now now that we have structured the data correctly I'll attempt to merge the data sets again when it comes back and says great I use this semicolon delimiter and we identified 5765 projects now that we have merged the data set what would you like how would you like to proceed and I'm now really shy and say hey can you perform a simple ex exploratory data analysis and have here some topic clusters that are in the title and in the project descriptions and 234 comes to access a certainly topic clustering or topic modeling can be performed by an LDA so we do the pre-processing we do the LDA training of the model and then we assign the most probable topic to each project there's how many projects how many clusters do you want 10 for example and I say well I'm generous start with 20 different topic clusters and here we go so we do the pre-process we do the cleaning here of our data set we do the tokenization we do the stop word removing a bag of words wow a bag of words is really something ancient and the Stone Age we did a bag of words but okay okay it's it's free I'm a non-coder I don't know anything of this so I have here an LDA model 20 topics display the top words for each topic let's begin with the pre-processing 24 comes back and says hey there's an issue with an ltk stop words because it cannot load them from the internet it does not have access to the internet so I create my own basic set of English stop words for the pre-processing step so let's continue with the tokenization and the vectorization process of all those title and all those project description yes the text has been successfully tokenized vectorized we have no representation of 20 000 unique terms across all the 5700 projects now let's proceed to train the LDA model with 20 topics and then display the top words for each topic and after 30 seconds tp4 comes back and says yes I have identified 20 topics look well each topic here you see the main the most important words would you like to take any other options or any other exploration from the data and I say yes create a vertical bar chart with labels and the number of project as the labels in this chart so gpd4 is working again and as a non-coder I have no idea what's going to happen I don't know nothing about matte plot lip or any plotly algorithm I have no idea and and yep uh here we have I mean it's it's a vertical I mean the the the the the bar is vertical but not the chart okay yeah never mind so here we have 20 clusters and on the x-axis you have to three dominant word and you have here the bars according to the number of projects in our 5765 projects not bad for a non-professional and non-coder is armed today I have here a very good overview of over 5000 projects not knowing anything about SQL not knowing anything about databases or python coding anything at all and I say hey I want the bar chart a vertical visualization and says Ah you want the horizontal bar chart so okay okay I'm not gonna fight with gbd4 about this I just want my bar charts yeah the the bars are horizontally Yuppie yeah we have it have a look at this this is exactly what I wanted the top three words for each of the 20 clusters I have now an idea how those 5700 projects are clustered and I say hey can you dive into one and only one of the cluster Quantum new system and divide it into 20 subclusters same method same visualization gb4 says of course here I go and yippee so it's filtering out the project only for this specific clusters thus again an LDA model and here for Quantum new system with 459 project we get now here all relevant sub clusters let's have a look if tpt4 is able to do this you know and non-coder today I have no idea I'm not looking at a python code what goes right what goes wrong I'm just using here the complete and overwhelming intelligence and here we are here are for the subcluster quantum new systems all detailed categories learning new systems control topological astrophysics data Learning Theory new flat gravity astrophysics again lightning project synthesis are k so 24 gives me okay and I say hey can you give me instead of three words give me the six top words per subcluster comes back and says sure top close to one stop cluster 2 Sub close to three okay and so on it just gives me five sub clusters here and I say hey find a perfect title for each subcluster and print a complete list of all 20 subcluster titles so it says okay so from those three words can he create a sentence what was I not precise enough to say Hey you have to go and look at the project description and then do here the analysis yeah and as you can see here now this perfect title has only three words so I was not precise enough instructing gpd4 what to do this is not a good idea so I have to reformulate my command for gpd4 okay let's do this so I say redo the complete project title analysis of all 459 projects and design valid and informative sentence for each of the 20 subclusters 24 says okay certainly so let's do the interpretation let's do the sentence grafting with write a coherent in descriptive sentence for each subcluster and here we are subcluster one investigate active processes and her mission system and explore the nuances of matter research the interaction between dust and DNA formation process in Learning System understanding new methodologies Dynamics and delving into Project Specific challenges it's not really informative and here we see that an LDA is not the right methodology what I would need a minimum I would need a bird system or a sentence birth system to do exactly this analysis but I can also search here in the whole 5000 projects for a specific topic like artificial intelligence and I say hey find old projects that are concerning artificial intelligence and cluster them in 10 topics and show me their distribution and of course here in the very first stages of code interpreter we only have an LDA model we do not have a sentence Bird model but we could ask for the python code but we are non-coder so we let the machine do all the work including here the python execution so it comes back I have identified 278 projects related to artificial intelligence great and now I proceed to Cluster these AI related projects into 10 cluster unfortunately only using the LDA model and then I visualize the distribution of projects among these clusters let's continue with the clustering process and tells me a there was an issue I'll investigate the issue and correct here so let's do it again I'm not gonna look what a python code says no I'm an encoder I yes okay I have to look at area just okay okay no we're not gonna look yeah this is it so AI projects per cluster cancer AI intelligence artificial intelligence welfare term polymerization cancer European technology research AI intelligence interesting research and let's see if we can dive into one subcluster here at AI artificial intelligence that's why I say hey dive into the topic AI artificial intelligence and show me relevant projects tbd4 comes back certainly I'll filter out a project that specifically match artificial intelligence in their titles or project description and after filtering I provide a list of the relevant projects for you I made another arrow gpd4 oh my goodness but here we come back and we have now oh it is not the title but it is the project objective so more or less the description okay now I just wanted a title so maybe I have not been precise enough to say objective is nice but I want the project title so I say hey give me the titles of these projects only not the project objectives yes okay here the title acronyms of the top 10 projects no that okay I get now the acronym no I don't want the acronym I said no not the acronyms I want their titles 254 comes back and says I apologize for the oversight so let's extract the actual project titles finished work yippee now we oh no I apologize for the confusion I mistakenly try to access a column name project title which does not exist to correct this I'll review the available columns again and retrieve the titles thank you for your patience now let's extract the titles of the AI related projects and here are now the titles of the top 10 ear related projects okay brain surgery medical imaging AI for medical imaging neurons yes data-driven integrated approaches chemical safety drug development interesting interesting what about this eight monitoring of earthquake nuclearization for false new urban areas so I tell Jimmy before hey how do you use here with earthquake nucleation for false new urban areas AI says okay gbd Force comes back and says I'll extract the details of the project real-time monitoring of earthquake nucleation for faults in urban areas I was like I apologize there's an error let me correct her array again it's working it's searching I apologize for the confusing I encountered an issue retrieving the project details why it's a CSV file it says CSV file and comes back okay I can come up with something no no this is not okay gpd4 that you come up with something that is related I want the official documentation so I say hey I want your official project description the official one not what you came up with and clunk we reached here the current usage cap for gpd4 Saved by the Bell gpd4 Saved by the Bell but I think this gives you a good idea how gpd4 analyzes with code interpreter two different files and what it is able but it is not able to do where I make a mistake not being precise enough I think those are fascinating topics to learn for your next encounter with gpd4 code interpreter