Visualize - Bringing Structure to Unstructured Data // Markus Stoll // MLOps Podcast #258

Key Takeaways

I'm Marcus I'm from romic I'm founder co-founder and City off and romic and I like my coffee as a l maiato sometimes with Van little flavor welcome back to another mlops Community podcast I'm your host demitrios today talking with Marcus has been enlightening on the car simulation front well just simulations in general and how his been doing some really cool stuff with embeddings we got into it at the end on how messy this data can be and what's been taken up a ton of his time and why he feels there aren't the right tools for the job when it comes to some of these data pieces that he has and it's not only for simulations but just for the immense amount of data that he's been gathering with these use cases that he's working on in partnership with car companies so we got to geek out a little bit on car companies and Roads and you're you're Italian I mean no harm when I say that you have shitty roads it's just a fact of life also Greek I threw you under the bus let's be honest there's some places that have really nice roads and me living in Germany I'm going to look down on all of you that do not have the nice roads cuz Germany's got some nice roads I'm going to give it up to Holland Holland they might have some of the best roads in all of Europe they spend a lot of money and those roads are super smooth all right let's get into this conversation already enough about them roads if you like it share it with one friend [Music] please all right real quick I want to tell you about our virtual conference that's coming up on September 12th this time we are going against the grain and we are doing it all about data engineer ing for ML and AI you're not going to hear rag talks but you are going to hear very valuable talks we've got some incredible guests and speakers lined up you know how we do it for these virtual conferences it's going to be a blast check it out right now you can go to home. ml ops. community and register let's get back into the show and let's just start from Square One what are you up to these days currently uh I'm working on getting uh our company uh comply so I'm CTO so my role actually is to take care of scalability at the current correct time andr correct level in the project and currently we are working on getting um some certificate uh for for projects and that's what I am working at the moment uh at the most uh but uh aside I'm also working at a lot of projects um for example uh I worked very hard and had a lot of fun working on it uh for realization for R so I applied um and other visualization techniques for R to to give the possibility uh to take a people look uh with visualization into uh Rec data on the um document side but also on the query side and yeah I publ that visualizing the embeddings yeah I visualized the embeddings so I reduced embeddings using a um map to um two dimensional map and um this helps a lot um when you try to get a good overview over your data so you can see forming uh clusters of different topics for example or you can find um a lot of time you can find some anomalies where you see that um you have included some text uh that you actually didn't want to be in your database for example in the um in this demo I wrote I uh s that um Wikipedia footer and header text us where also included in the database and I popped up like uh little clusters with a lot of points and uh I solved that only because I was using this uh a visualization technique and and you're exploring the data sets through this visualization the basically the embeddings visualization on a I think it's a 3D graph right and so you can see a lot a lot of people are using 3D graph so they redu the high dimensionality from of like think almost thousand Dimensions to three dimensions uh but I prefer to use a 2d presentation because that is a little bit easier to to overview it's so it's reduced more so maybe you can see a little bit less but it's um much easier to present so if you are creating slid through it or show someone specific clusters it's much easier from my perspective if you have reduce it to to uh 2D because you can use a plane image and it's very hard to um represent the 3D Graphics uh in a presentation or in a team and work together on it on the team that's uh little bit hard but yeah I just so I use this 2D two dimensional map a lot of points on it and uh check out the different clusters look for um clusters that are somewhere outside the other clusters and check if is this expected what's going on there is this data that's actually useful or is this some anomaly and it gets even more interesting if you include them uh the question with it so you can for example project the question the same space the question all your questions reverence questions or or question from from your users to find out which part of the of your document space is uh for example important for your users or is important when you evaluate your model so you can find for example also where is um where clusters in your document documents that are not covered by your reference questions for your information and then you can think about it and maybe add more questions or maybe this cluster isn't not that relevant and you can drop it from the your document store and if I'm understanding this correctly you're also creating embeddings for the questions or you're just looking at which embeddings the questions are pointing to yeah you can use both you you um use uh do you embed the question to find the relevant documents for this question so you often bring have a embedding for for question but yes of course it's also um very important to find that actually the nearest neighbor of this question uh in the embedding space which other documents that are relevant for this question maybe the five or 20 next neighbors are usually take into account when answering the question and then you're getting a better idea of what documents are getting high usage and what documents are lower usage exactly here you can maybe use that to color color your um similarity map so you can see how the how often the documents are used or referenced by question and color code that to find some kind of a heat map and see what area in your embedding space is important or is reference a lot and which is not so how have you been using this like what are you able to do once you have this heat map once you know that the documents are being used more frequently what do you do with that information I think the most interesting part for me at the moment was to to see how my how the project evaluation questions are represented there I've seen a lot of cases where the um reference questions only uh span a very a very limited area in the embedding space and uh there are other document in documents in the space and it's very important to talk to the customer or whoever is interested in the data uh is here something missing do we don't need the data or should we try to add new questions so I think the the coverage idea for the reference question is very very import is a very efficient um step in a project um to to do to get it better understanding for for the expectation of the customer maybe yet then I haven't used it yet for uh in production to see uh how it works out um with the question of the end users to see what documents are important for them but yes that's also a um very interesting point you could use that for to to optimize your model or to optimize the whole approach or to reduce the data for um for the use case based on this information how it's actually used and maybe also compare the reference questions that you have um worked out with the the customer and compare it against the questions that they really uh asked by the users in the end system yeah you're able to figure out what are those 20% of the documents that are getting 80% of the queries yeah and really make sure those are pristine I also like this idea of as you're building a product you gain insights into what the expectations of your users are yeah so this visualization is going to help you see in a different way this data that is coming through as people are interacting with your Rag and I think also um what important to also collect uh information from the um maybe the the customer or the in the end the user how good the results uh the rec system where for specific F to get this feedback and also use this this feedback in the visualization again to maybe find then um areas or clusters in the document space where uh the system doesn't work as uh as expected not only to find which is covered but also uh find out which cluster in the document space has which uh quality level based on the feedback so so you're taking it a step further it's not just what questions are coming through what documents are relevant what are the best ways to answer these questions but also so what is the actual output and then using some kind of evaluation criteria on top of that and being able to see what the output was and visualize it differently than just a bunch of question answer pairs yeah exactly and I think the important step is not to use um some automatic measurements or criterias but real user feedback so we always try to include very early in our projects some feedback mechanism for the users is this either good or bad why should we discuss maybe this answer again together with you I think that's that's much more important than uh using some rag metrics uh to to find uh to find to decide if the quality is okay yeah so this just so everybody knows this is an open source project right we can all go and download this andig and start playing around with it today um we have uh different projects so the uh we have Spotlight Spotlight is our um tool that we use to look into uh different kind of data so it has support for 3D data like uh three measures that's where we actually come from we did a lot of uh development for um simulation engineering so that's supported there and video and audio and images but also text and it has support for visualization of embeddings uh as a umap and you can use different embeddings uh and visualize for example also rack data and visualize uh the questions in the embedding space and the document documents in the embedding space and that's what I have um written about uh at too data science a few months ago and you can check it out it's all open source and um there's also a rack project from us uh which includes some limited um very simple uh rag techniques it's a vanilla Rag and it doesn't include the uh feedback me mechanisms actually but you can use it for uh BAS B projects and um visualize uh embeddings of documents and questions with it this article but it's not only for rag right you can also do classifications or is this spe is Spotlight specifically for rag so Spotlight is specifically for I would say machine learning data so it has support for text has support for images videos a lot of audio because uh we have uh customers that use it for for audio and yeah also for 3D uh animations actually we use it used it for um exploration and classification of simulation data of car crashs oh wow so where you can um compare the end result of a car crash based on similarity of the uh deformation for example but that that's something you have to prepare in the project so the the idea of Spotlight is only to visualize three different kinds of data like 3D data or audio or images and a lot of different stuff uh but together with embeddings and uh you can use this embeddings to create some kind of a similarity map and then you can use the similarity map to browse or go through through your data can zoom in select a specific point which can be for example the result of a car crash and then you can compare two points in the similarity map that are next to each other uh by U visualizing the actual simulation so like the 3D uh animation of the car crash so the the idea is to have uh have a tool that uh uh combines the machine learning view of the data like umap or statistics of the individual of of each points in the data set but also have the possibility to very deep zoom into each point to find out what the actual difference is so it's the idea to bring the machine learning engineer on the same page together the domain expert for example a simulation engineer for forour cases okay and let's let's talk a little bit more about the simulation engine because this is fascinating to me the idea of creating a 3D space and then simulating car crashes and creating embeddings out of those simulations so you can know which car crashes failed in which ways in similar ways and if you wanted to then search for failures or car crashes I guess you could search through the embedding space y that's the idea so if it works well um you have to of course you have to do some work to get this embeddings it's not the best idea to just take the whole animation and the whole uh part or the whole car into the embedding space but um often very easy to just Define this is our region of Interest defined by um maybe this thousand points and use this thousand points to uh create embedding for the this area of of the mesh and then you can uh cluster uh the result of the of the simulations uh using a p map for example based on the embeddings and find um clusters of similar uh defects where the the results of the cage were very similar and maybe you can find some some outliers then you have uh found a very very good configuration for some material so that's that's the idea this is working with car companies on testing new materials or testing new designs yeah that's a use case for for that for example something we we don't have this uh specific case I'm now talking about in a uh customer use case but uh that's a a Showcase from us and there people that uh work work like that not with Spotlight every all of them but but some of them but we there are of course also different areas we have a lot of projects in this um the the enging space of car companies this starts with maybe this um simulation postprocessing where you can organize the results uh using embeddings but uh also in the for example pre-processing of the simulation it's there's a lot of manual work to do for example you have to um Define how if you combine hundreds or thousands of Parts together in a simulation how these are are connected and you you do not only have always have the the information how this parts are con parts are connected you have have to look into and compare it okay this uh this is part A this is part B and this is usually with uh connected with this kind of connection and we have uh in a very early or very old project now uh we had uh build a classifier that automatically detects the different parts and uh gets the idea uh What uh the connection of this part uh should be and then the engine engineer can use this tool to create hundreds of connections automatically and reviewed words and then uh use this in a simulation oh nice and and so this is it again going to this specific use case it is basically so that when someone is creating a new style of connections or new parts in a car they're able to to keep that going so that the simulation can understand how the Dynamics would be and how physics would work with that or is it so that the next person who comes Andes to do stuff with it knows the way that those parts were put together in this project it was about um creating the the correct uh representation for the simulation for example um if you have maybe a crew in in in reality uh you would in the simulation just tie the points where this crew actually would be together using a um constrain so we wouldn't model the the connection itself as how it is really is but with very simple model and you can uh have a cuton look of this this um simple models and train a classifier to find in which situations this uh connectors uh Court wants to take and warm that up I've got to come clean with you I absolutely am addicted to these simulation videos on Tik Tok when you watch but only when you get somebody good that's giving like a really nice voice over and they're making it really funny because you have different cars getting smashed by semi trucks and it's like would you survive and then you have people giving voiceovers that are hilarious but for the most part you're not surviving when you're getting hit by a semi truck that's the key takeaway there as if people didn't know that one already but so I let's talk more about what some of the other pieces that you have been working on are what some like cool use cases that you've been seeing out there are a very cool use case for us at the moment is the evaluation of test data um so if you build for example a car then you test it on test tracks you with test trials and you get a lot of results and it's very hard to get all the results into a representation that that really allows you to find anomalies very easily or find uh good setups or find errors it's actually I think at the moment a lot of manual work you have an expert that has to Pro do a lot of data and because there so much data that you're getting yeah so much data so it's a lot of work to do and I think the tools are not um not that good how they could be actually from a ma sheine learning engineer perspective so yeah we're currently working on projects where we use um anomal detection or classification to um classify bad behaviors uh so or errors during the test drive of of cars um where you where we can use um automatic uh approaches and reduce the time that is necessary to uh review the data mhm and so this is basically like well you and I are both in Germany so let's take some German cars right we'll go with Porsche just cuz so this is Porsche saying we've got a real car with a real driver we're going to go take it onto the track that has a lot of different styles of roads it's got the Italian roads with all the bumps in it it's got the German roads with the smooth sailing where you can go on the autoban uh 200 kmers an hour it's got many different scenarios maybe dirt roads whatever you have that test drive in real life it's no longer we're no longer talking about simulations right so it's in real life it comes back after and it probably does various of these I imagine it's going through and doing many of them and you're trying to just take it to the Limit sometimes or you're trying to just see how it drives normally and change gears make sure it does everything that are table Stakes well and then when it gets to those edge cases see how it performs there I can see how if you have the minimal amount of sensors on a car that would just be so much data that you have to sift through and see if there's any kind of anomalies but now I can only imagine that Porsche has more than the minimal amount of sensors right they probably there's probably sensors in everything that you now are looking at and you're trying to see is this working properly or is this working less than properly because I imagine if you're a human you don't necessarily feel if something's off but you want to make sure that all right it works for 10,000 km or 10,000 Mi but we want it to work for 100,000 mil so we want to be able to catch things early and fast and now you're trying to sift through the data and see if there's going to be something that breaks how do you even go about that it's it very depends on uh the data that you have already so of course you have very low records you have a lot of chanels you have different modalities you have sensor data for acceleration for example or or sound maybe also a camera so a lot of different data also and the first thing to do if you set up the product like that is um to see is there historical data do that is already La do we have some data uh that we can use then we can uh for example uh go for a classifier for for specific um arrows that we have already seen in the past to find them again in the new data no actually um automatically or do we have to if we don't have the data then we could go uh approaches for anomality detection and find where some very very rare cases and look deeper deep into the uh point in time in all channels and all videos and find out what's going on here and of course you have to just the idea that um that I often prefer to do some um visualization of it so you use embeddings for again for for the different signals and maybe for the images and um for the audio data and project that into different similarity maps and go for your clusters and find out what's going on in a different cluster then you can filter out one cluster if that's not interesting and uh find out what you'll end up with um being interesting for you and maybe you can use Simple concatenation of the different embeddings from the different modalities into one visualization or you can use multiple visualizations that are linked together so can select on one side the uh the data points based on the audio signal and on the other side you see okay the if the cluster of the audio signals uh also form the subcluster in the um the different sensor um space and use that together to find some specific um clusters again so you can actually use use the embeddings for data exploration but I think in the real projects at the moment we are doing we are focusing on very specific cases to find where we can use classification or uh maybe also an Mone detection for example to find out that the simplest thing is that something is wrong with your setup so that's a very uh yeah common case just to to find out before you start the actual measurement to some animal detection if uh maybe you have uh connected your sensor in the wrong way you have switched to cables oop some up yeah yeah I can imagine that happens more than we would like to admit and it's good to find that early so that you don't waste your time on bad dat data and you're trying to figure out what's going on here why is this data not giving me these answers and you recognize okay well there were some cable switched or there was some type of sensor that was broken or whatever and one thing that we are currently start look looking into is the combination of of the direct world and this um time serious data world so that you can use questions not all for your do but also you can include the data by the simple generation of sqa statements maybe or by code generation to give also the engineer or the test driver the possibility to explore the the data very easily wait so explain this more because this seems awesome so basically you're combining these two worlds yeah but I didn't quite understand what that entails for example you could ask them the system uh which was my best round on the test drive and the system gets H creates an SQL statement to query this data from from the store and um find the fastest round or you can try to find uh or try to ask more difficult questions maybe for for to find what could be the reason for something I heard something was different than usual there was a anous sound in round five what could this be and then the system uh can try to uh bring the relevant data to you so it's interacting with the tabular data in natural language y That's yeah that's the idea maybe by simp SQL statements Maybe by code generation but the idea is to make the tempate data accessible for for test drivers or test Engineers not not only for ML or data Engineers yeah so it's it's that test driver who says heard when I was going around that fourth lap there was something that was making a weird sound or the brakes weren't working as well as I would have hoped can you pull up why that could have been and so then you you have that type of um yeah that's that's very cool yeah and the the the big point I think is what is then the result of the system at the moment maybe you get an answer in text but I think it would be much better than to to get some visualization of the different channels that were relevant for this event and maybe a highlight of of the specific EV event in this channels and yeah maybe also the visualization of the whole data point in in your similarity map with the U map representation yeah so not not I think the the just answering the question with text is okay but very good user experience would be that you get then uh the automatic Zoom it into the data that interesting for you yeah more than you could ever want and then you can go and you can use what's relevant to you so then now talk to me about how you're seeing multimodal being used because you mentioned there's audio data there's visual data there's also this the sensor data and it feels like all of that is playing a part to give you a much better and robust picture and so what do these things entail that that's almost like what you were just saying thing where at the moment when you're asking this text to SQL type of question you're just getting text back as an answer but in reality what you really want is all the relevant data and so talk to me about the multimodality I think at the moment um for us the the common approach to have different models so also different embeddings for the different modalities maybe it would be in the future also possible to have all the different modalities combined into one model so trained on the um the behavior of the car uh simultaneously in the signal space in the audio space and in the corresponding image space but at the moment maybe that's a little bit too too tough for us so usually try to keep the things simple so uh I prefer to use one model that is specifically trained for audio and use this uh maybe for embedding uh the results or for classification uh add a second model trained on video or images for the second part second group of embeddings or class classifications again of course for classifications it would make sense to to maybe uh trade a classifier on the combined output of both models the embeddings actually that could be a good idea but yeah I'm cently not working on on one model for all modalities together but on different models each um pre-trained on uh data from uh from the internet uh and maybe find C on data from the different projects but usually specific for one modality or for audio or other signals and a different one for images yeah it feels like the ROI of trying to figure out if if one model can rule them all might not be there especially if you're finding success with each model being very specific to the specific modality I could see that and I think the the big problem in the this domain we are currently working in an industrial AI domain is that you can't easily use a pre-trained model that is trained on data from the internet because the sensor data or audio data that is uh relevant in the project is very very different from uh from the data you can find in in Internet and the pre-ra pre trade models on that data is something I think is very very very clear if you look into for example image processing so you get a classifier pre-trained on internet data it's for cats and dogs and persons or all the large data sets industry food and the industry you would then have very very uh maybe boring images from production machine where you want to compare uh products uh if they classify products if the result looks okay or does have a defect so it's a very very different uh uh use case than uh the models are trained on you usually get you creating all your own custom embedding models and or you fine-tuning different models that already are have a like you're just grabbing base models from hugging face and then fine-tuning them uh both so usually we we try to find you well models again under very different data but I'm not sure if there's one strategy for us that always works we only we always have to figure out what's best for the specific project it's always different sometimes we can have a very good results based on pre-trade models but sometimes it's clear that we have to do a lot of Uh custom fine tuning or even trade from from scratch you also also Train sometimes Ms from scratch for specific use case for example this um detection of part connections for the simulation was uh um completely trade by us without the initial weights yeah it sounds like it can get messy especially because you're not getting that much data out there on the free internet yeah exact L it's also always tough to get enough data in the in the projects because there there's often a lot of data available but quality is always challenging and of course uh you often want to have enabled data and that's it's rare yeah that's expensive rare yeah so usually we have to set up a a label process in the in the projects at the moment not for the rec uh projects of course it's a little bit easier but in the uh engineering domain with test data or 3D mesh data there's always lot of to-do on on labeling set well yeah it's it's a little bit counterintuitive because as we were just talking about you get so much data when you go for a test drive in one of these cars but then you need to label that dat you need to clean that data and just one test drive I imagine will take you so long to get actual gold standard data from yeah exactly and also the data you have is often um hacked into different systems based on the the measurement for example for your sensor maybe you have the data in a different subsystem uh then for for your audio data because it's uh the device from a different V so you have to combine into signals again uh and you have maybe to synchronize to resynchronize them because you don't have you don't know how to align them no wait so what do you do there because I just think about when I make music and I have two microphones I clap so that I can easily synchronize them after the fact in post- production but I don't see you going before a test drive and clapping so that you can easily synchronize them yeah you have to to do something like that have to to get them synchronized or maybe you can try to have some really good uh clocks so for uh where you can use the time stamps for the synchronization or you can if you don't have that then you can go for specific events where you can use uh uh to to rearrange um the the signals because often you also not only have the the mismatch between them you also have a drift so because the the frequencies is not of the measurements are not perfect in re World sometimes and then you have to do this uh but only the beginning but I can and they can yeah and so that feels like just a whole lot of time that you're spending trying to get that data so that it's synchronized and then you label it and then maybe you can do something with it and a different um problem usually that is occurring is also that you have uh not always a good standardization of the test drive so you have to this different test drives with signals and maybe some different captions for the signals and you have to uh combine this all into if you want to train from it into uh some data set that is standardized and this can also be very challenging if you have recorded that for a long time we that's actually something we also want to try using uh L&M so to find out which is uh uh the current name of the channel and what could be the last name in know maybe the test drive from 5 years ago to do this matching uh how which column has what uh uh what meaning is something we try to figure out with nms at the moment yeah it's like hey given these attributes given this type of data given this last data set give us your best guess yeah exactly and if that doesn't work out maybe also by comparing the data looking uh what ends up for similar impeding results for example you have so based on similarity you can try to do this match just to get a good uh representation in the beginning of the data set yeah so now I'm understanding more clearly and vividly why you were mentioning at the beginning of this conversation how you felt like there weren't proper tools in place for this type of thing it feels like you're prob doing a lot of stuff manually it's taking you a ton of time there's a bit of pain whether it's the types of data being unsynchronized or coming from different vendors so the sensors are going to be completely different you have that data drift and being able to get that data in order is the first step before you can even think about any of this other cool stuff that you're doing with it and the embeddings and all that fun stuff you got to just get that data in order yeah usually we try to focus on specific questions in the project so we and and we only have to clean up the data for the specific use case so I don't think it's a good idea to start with cleaning up the data until it's perfect but try to clean clean up the data and get all the data you need for a specific use case like a classification for for a very specific thing in the data because if you try to get the perfect data in the beginning beginning maybe you uh take a lot of time uh to uh create a very good data set and I think often you don't know enough uh how the perfect data set would look like until you start trying to use it for example by trading uh by trading for classifier and using this classifier all also so I think it's very important to uh go through the whole steps of the project through the whole pipeline very fast and then iterate at the beginning again so don't create a perfect data set but just give us it a start and use it and find what is the real problem in the data set and fix that because you can't fix all the program sta so that's fun to think about it's taking that same mentality of having an MVP yeah and I think I've heard somewhere someone saying if you're not embarrassed by your first version of your product you waited too long to ship and so you're saying hey just get something for one use case so you can start to understand what the data should look like and then slowly expand your use cases and by the time time that you have more and more use cases then you'll understand what the larger Corpus of data should look like because if you aren't doing it use case driven you are at risk of spending a lot of time to curate that data in a way that isn't going to be useful for the use case yeah and maybe also that I think very important point is that maybe in the beginning you don't even understand the use case I think it's very important to close the loop to get an understanding what the the customer or the users actually need in the end that's also true for for our rack product at the moment we we usually start with vanilla very simple rack system just to that can be implemented in a few days or a few weeks and then give it to the customer and look how they use it what questions are are being asked and then we can try to optimize and do some multi-step rag or all the fancy stuff you can use to to increase the quality but we really start with a vanilla rack yeah keep it simple and fast yeah something out there fast see what is hitting and then learn and iterate from there it's a great strategy for doing that I also like that I think you can do that with just about anything and it if you are coming into it with that mindset of let's just test and iterate and test and iterate and eventually we're going to start to hit the direction that we want to be going in or as I've heard other people put it it's it's almost like in the beginning you don't have to know exactly where you want to go you just want to be directionally correct yeah so as long as you're moving in the right direction you're good right real quick question for you are you a Microsoft fabric user well if you are you are in luck because we are introducing SAS decision Builder it's a decision intelligence solution that is so good it makes your models useful because let's face it your data means nothing unless you use it to drive business outcomes it's something we say time and time again on this very show but wait what do you mean by nothing Well SAS decision Builder integrates seamlessly with Microsoft fabric to create effortless business intelligent flows it's like having a team of geniuses you manage in your pocket without all that awkward small talk with decision Builder you'll turn data into insights faster than brewing a double espresso and you know how much we like coffee on this show visually construct decision strategies align your business and call external language models leverage this decision Builder to intuitively flex your data models and other capabilities at break neck speeds there's use cases for every industry including Finance retail education and customer service so stop making decisions in the dark turn the lights on with SAS decision Builder yes I did just make that joke watch your business shine SAS decision Builder because your business deserves better than guesswork want to be one of the first to experience the future of decisions well sign up now for our exclusive preview visit sas.com Fabric or click the link below well this has been awesome man I appreciate you coming on here and taking the time to have a second conversation with me because our first one didn't get recorded but I'm just lucky because I get to chat with you two times now instead of warm well thanks for the invitation me [Music]