Synthetic Data Generation using Generative AI

Key Takeaways

hello everyone welcome to AI anytime channel in today's video we are going to explore grater AI so greater AI is a platform that helps us generate synthetic data so it's a synthetic data generation platform so we are going to also see that what are the other features that Gretel currently provides so they have multiple features like generating synthetic data and also you can use the some of the models that they have to train on that generated data also on your custom data that you have right so it provide multiple features so that we can leverage and nowadays synthetic data generation has been entrained due to generative AI because one of the most prominent use cases of generative AI is to generate synthetic data now that can be you know generated design for example now you have multiple designs like cloth materials your in automotive you have car designs Etc how you can use this fascinating technology to generate designs which don't exist right maybe you can use large language models to generate CSV data and tabular data for example you can ask jigpt that hey here is my data model this is how it looks like these are the columns and I need 1000 record on on to generate a CSV file you get a python code and you just execute that python code and based on your data model these generative AI models will kind of generate the data for you it's extremely you know useful when we talk about you know generating uh synthetic data it has huge cases in Industries like bfsi in general banking Finance insurance and you know in these industries because for example I'll just take an example of fraud detection where you want to train a machine learning model you would like to have a data from the banking or financial institutions they would not like to share their data with you because the confidential data right those are their customers data right so in that cases it's better to have some kind of mechanism that will help you generate synthetic data you can train a machine learning model to you know detect fraud similarly you can also see some of the use cases in empty money laundering you know tools that's called AML system or tool because my learning is kind of combination of different sub components like because they are for re-routing there are financial documents there are you know money has been divided into smaller chunks and have been you know have been laundered through different banks and in use example that's that's for example right so how you can you know use this data to generate okay some synthetic data on top of this and then you use that to train a machine learning model for your task right so that's that's how generative AI is currently being used in industry and synthetic data generation has been the use cases that the clients are looking at apart from the tasks that we do like chatbots conversational AI you know knowledge GPT of whatever GPT that we call it right it's domain specific GPT that we call so that's why I'm going to explore in this video and will also programmatically see that how we can use grater AI API you know to set up a workflow take a sample data like a bike bio data that we have okay and I'll show you on the GitHub repository it's completely interest is very interesting as well like they have a GitHub repository there on their platform they have good documentation maybe you can follow that to you know generate generate some synthetic data so in this we have identified a huge case guys okay uh like we'll see that how we can you know take a sample data for example and on that we can uh we can have a look at the personal identifiable information that's called pii right because if you talk about countries like you know gdpr uh countries like you know European countries and also in the United States there are regulations and if you talk about EU for example they have uh they have gdpr for example which is global data protection regulation which falls under EU so just to protect the Privacy data privacy rights for the EU citizens right and if you talk about uh mainly the CCPA which is California consumer Privacy Act which is mainly for if it is an U.S state law to protect the data and privacy rights for the California citizens right so there are multiple regulations and that's why you will see also see the trend of uh privacy preserving AI That's for ppai nowadays a lot of algorithms are being you know trained like that that how we can use uh these models mainly to have a to protect the data protection and privacy right because clients are not looking at this thing because these are all being audited and there are a lot of compliance related to it it's kind of regulatory Norms are there when you when the clients are operating in Europe and also in the US that's where this kind of techniques helps from the data standpoint and also when you go to modeling when we use ppai for example therefore we're also going to you know uh basically see in this you can currently see I am on this griddle website it says the synthetic data platform for Developers and it says generate artificial data sets with same characteristics as real data so you can develop and test AI models without compromising privacy privacy is extremely important nowadays guys because the the you can see the speed right of this technology being coupled you see not not weeks or not months it's every day we see some kind of models are being released you know in this ecosystem of genitive AI that we're talking about so it's very it's very important now to that we look at the data privacy of our consumers or the client that we are working with so you can see a lot of companies are already working with you know greater AI okay HSBC and you know sap for example and a lot of other sales let me just give you a walkthrough guys okay what I'm talking about it so you know if you see that it says synthetic data is the future of AI and I do believe that okay you want to generate x-ray data from NIH data set go on category take an Nia sample data said then use some kind of Gan models you know to generate on top of that you can also do that you know generate synthetic data on top of data that we already have so how does this work guys let's based on algorithms and statistical models right that we have it looks at the patterns the characteristics and the relationship between your data that you have so there are different use cases and the use cases that we are going to look at is the second one which is protecting private and confidential data right so basically on the pii right so that's what we're going to look at in in this video guys right so for gdpr data privacy training AIML models mitigating bias as well so it's a it's a two-way sword guys right when I talk about buyers so when you generate synthetic data okay it can also generate bias data okay if you haven't Define your data model in a way that it should you know it would handle that bias but it can also help you reduce the bias you know the nature of bias in your data that you have because that's completely depend on the data model that you define and then you generate the data so these are the tools and stack okay uh this is tools and stack that you can see here it says we can also see gridel for example we see beta data we see why data a lot of other companies who are working with the you know a structured privacy preserving synthetic data and trust me ppai is the go forward right it's privacy preserving AI lot of big companies are working and if you are watching this video if you are just you know if you are if you're a data scientist or a machine learning engineer you know I will recommend you to have a look at ppai if you haven't you know explored yet and when you talk about structured synthetic data you can you know you can see a lot of organizations are doing like for example gym rocket or Informatica which also helps you generate synthetic data and on unstructured which is very important so we have synthesis AI which have just you know uh got up the hype recently with genitive AI you know we have UI AI we have Zumo Labs a lot of other companies who are doing it neurolab for example okay so our focus is on that how we can you know Leverage uh this grater Ai and to build something this is what we are going to do so you can see this is a website it will come down you'll see 150b records into site no SDK downloads blah blah blah print validate generate all this documentation and you know everything the information you can find it over here on the website now here's the GitHub manipulatory and as we are focusing more on the gdpr part in this video you know in the next video I'll also use something to you know compare or the machine Any model performance on the synthetic data and also the the real the actual data that we have that will come in the upcoming video guys but in this video our focus is to set up a greater workflow you know and generate some kind of data you know we use the classified transform and synthetics I'll explain that three different steps when I'm when I start writing the code so this is the purpose of this video guys that how we can leverage greater gdpr helpers to set up a workflow and generate some kind of data on top of it this is the whole purpose and if you come over here you can see genetic models to automatically analyze data to meet gdpr and ccpi standards documentation are given how to install you know how to you know add your API Keys blah blah blah you can have a look at that as well okay so I'll give the link in the description of this GitHub repository on gdpr helpers and let's jump in so here you can see I don't have any projects here now it says new project create your first project uh it's from greater dashboard you just have to do a Google login to get the access to the dashboard okay it's completely gives you in free you can see you need an API key that I already have here in the notepad so that's what you need and you can see they have you know you can also use this Cloud interface because again running on some kind of cloud here so generate synthetic data create highly accurate synthetic training data so you can also select here and join it just directly on this on this interface itself you don't have to do it programmatically if you don't want to do that then you can also connect to any normalized SQL database now you have some SQL database where you already have some kind of data format or something right you can just connect it to there and then you can also generate custom images it's in Early Access but it's really fascinating right to now generate custom images now you have some sample image that can be any domain specific now you want to generate some custom images on top of it you can also leverage this generate custom images or features from a grater now create probably private versions of sensitive data like tabular DP and you graph based genetic models which is ideally looking forward to it you know maybe in some upcoming videos when I explore it fully I'll I'll create one video on this guys on graph based genetic model which is we also I'm doing some research on how to combine graph neural networks with large language model to do something in these areas and it's really interesting and then we have generate natural language text using GPT genetic pre-trained Transformer which has been in Hive due to chat GPT nowadays right so generate natural language text also using grater and redact pii personal identifiable information now what does pii means like you know your your name your maybe for example credentials you know something like that it all falls under pii right you can also redact that right if you want to sanitize and do that as well or replace with something else you can also do that so it is your dashboard and on the project I don't have any project I will do it programmatically maybe for example so let's do that so you can see I am on this uh my Google collab notebook and here I'll just start writing it so let me just first do one thing let me just uh I think I close that gdpr helpers I have to go inside this and I'll just I'm just gonna clone this guys I'm just gonna click on this I'm gonna come here and I'm just going to clone this repository inside this runtime environment of Google collab which is GitHub HTTP I have to write git clone by the way sorry so git clone git clone and https github.com greater AI gdpr helpers dot git okay so let's run this now once I run this you will see that it says cloning and it has cloned it now here comes on the left hand side just do a refresh you will see the folder here called gdpr helpers now if you you know maximize or extend the gdpr helpers you know you'll see something there you know you have data so if you click on data let me just show you there okay so you have data here so this is the data that we are looking at currently right Adventure Works bike buying this is the data that we will take and say hey look now you generate the synthetic data okay use some kind of Gan models there and follow the gdpr compliant okay takes care of it right so now we'll get a data there uh synthesized or generated data that whatever you call it so they have other data as well Google meet my company and my Trace in the electronics something but I'm going to use this adventure Works bike buying data set now if you come on examples you'll have some files you know of course you can have a look at that files and then I use files py and something like you know anonymized file school app okay so we are basically going to load these modules okay so now let's do that so what I'm going to do here now is I'm gonna just do CD and let's go inside it so CD gdpr okay excuse me helpers this is what I'm gonna do and in this only let me just do a pip install as well so pip install so whatever requirement is required you know all these required libraries and modules you know that's what we have to do in this video guys right so let's do that as well let me just turn off the volume of my monitor so click install and then we're gonna do so basically upgrade and also just to you know uh the verbose right so CD GDP are helper so we are inside it and now we are also installing all the required libraries and modules okay with less verbose okay and you can see the updated one okay or the upgraded one so this will install it you know installing build dependencies and something something that that you see here all right currently you see on the projects we don't see anything at this moment because create your first project now that you are here it's time to create your first Gradle project you don't see anything over here right now what I'm gonna do you can see it's have been done now let's import couple of things so import OS and if not in OS dot gate CWD if not in OS gets current walking directory dot ends with I hope it ends with gdpr helper we are just doing a validation so gdpr helpers and inside this if not I think it's not I think I'm writing something wrong here guys if not OS yes now excuse me if not OS get current working right ends with gdpr helpers this looks good with DOT chain directory so chtir gen directory gdpr helpers excuse me gdpr helpers this makes sense you can just run this now you can see now the search pattern right whatever data data that we are going to explore so let me just do one thing so let me just going to do here search so let me just write something like search patterns so if search pattern and then we have to go inside data because we're already inside GDP as helper so you can see in this data folder we have this file so I'm going to just do data which is slash Adventure so it is Adventure works by buying dot CSV let me just do a copy path guys I can't see that so I'm just going to do here data Adventure we don't need this thing because we already have defined so that's it so buying csb so this looks okay now let me just run this so this is the data that we are looking at guys currently search pattern Okay so now let's have a variable called anonymizer which will anonymize the data right so you're going to anonymize this data so I'm gonna use this class so let me just do I think we have to also first import this we haven't imported yet so what I'm going to do here from or let's do one thing uh let's define it over here so import hello and then from gdpr underscore helpers import anonymizer I hope I'm spelling it right anonymizer yeah this looks okay so we have input globe and now here we'll write our function so that I'm going to use this anonymizer class now so am a variable now let's use anal this makes in and inside this I'm going to write let's create a project name for our project on this you know project name so let's keep it gdpr workflow so DDP a workflow which is my project name on this currently see we have no project here so let's keep gdpr workflow project name and then run model on cloud so I'm just going to use this parameter called run model and it should be Cloud we're going to use their crowd and then we have we need couple of yaml files so transform yes transform config which should be in this Source you can see if you come if you come down in the source here guys if you have config file and in this config we have two files that we need which is synthetics dot config and transforms.config now it's very important for you to understand right so so we have there are three different steps let me just write it over here how the workflow basically let me just do one thing let me remove this click on text how this basically works the workflow setup on greater and this basically works like this the first is classify so in classify what we do is it set up our setup of policy to find and label sensitive data find and label that which is for fine and label which one to you know uh basically redact or sanitize right or synthesize right sensitive data sensitive excuse me sensitive data that's the first one centigrator you know and this can be your pii credentials blah blah blah right something is credentials Etc the second is transform now in the Second Step what it does it kind of transform it so how it's basically learns how to define that policy that you have set up in the above State how to define the policy how to define the policy to label and then transform so that can be that has to transform through different you know techniques your algorithms for example so transform our data for example with you know regular expression so with your re or data shifting it's a shifting Etc etcetera and the last step is synthetics so synthetics is the last step where we know generate the synthetic data guys so this is the step that on the Gradle that we follow so if you want to replace greater gdpr helper this is how you set up the workflow so generate synthetic data so this is the workflow setup on grid and now we have couple of files here that we need so let me just do one thing let me just come over here you know copy the work path so I'm just going to do you know uh copy that transform config copy path transform config copy path let me just come over here I just I just don't need the complete thing I just need from SRC because we're already in gdpi helpers this makes sense and now the next one is synthetics synthetics config and this is nothing but again come up but can come back here and copy this and when I copy this I think we don't need this by the way again and this looks nice so we have yamul now the end point which is that's how you connect with that cloud right uh The Griddle Cloud that we have so https and it should it is API dot grater and Dot Cloud this is what we have to do here in this thing guys so now we are what we are doing we are using this anonymizer class in this anonymizer class we are passing the project name that we want to create on The Grater dashboard the dashboard that we have given and the Run model is in Cloud we want to leverage that and then we have to config files one is the transforms config and the other is synthetics config and then we have the end point which is https API dot griddle.cloud now when you do this it will ask you to put your API key so let me first run this now when I run this it says anonymizer got an unexpected keyword run model it's not run model my bad it's run mode it's not a model so we are going on cloud mode so now let me run again and when I run it again it says during handling the above exception greater API Key password input may be e-code cannot control Eco on the terminal so I think there were some issues with my API Keys guys so I have re-logged Into The Grater dashboard okay and I have created a new API keys and we will try with that API keys right so now what I'm gonna do here I'm going to run this and it asks for the grital API key so in this place you have to basically enter your API key and the API key that you will get it from The Grater dashboard so now let me just do that and once I log into into it it says caching grater config to disk using endpoint API griddle Cloud logged in and follow along with model training at this right so if you currently see I have no project over here and this is what I'm gonna do in this case so let me just minimize this and now I'm gonna just gonna do for data set path so for data set underscore path in now let's look at clocked at clock blog and we're just going to look at the search pattern so in that search pattern basically we store our data set that bike data set that you see from gdpr helpers inside this data and inside this adventure Works bike buying right this is the data that we have search pattern and inside this I'm just gonna use am.anomize so this is the file that I'm going to use if you see it over there so there's an example they have SRC and srct we already have used the config file right and what I'm gonna do here is anonymize so anonymize and here my data set path is nothing but the data set underscore path so this remains data set underscore path so this is what we are doing in this guys right so for data set underscore path in Global Group search pattern this is our source pattern if you see data Adventure Works bike buying csb and if I'm just running it over here so let's do that so once I started doing it you can see it says anonymizing data Adventure Works bike buying CSV now what it will do it will anonymize according to gdpr like the uh the protection regulation that we have So currently it's in the transform States as we have discussed the three steps you know the uh the transform and then we have synthetics right so we have classified transform and synthetics in this case you can see the job is currently active now let me show you what I'm talking about this here it will automatically show on the dashboard in the projects now if you come over here let me just do a refresh it is still so just go back to here in this you can see it's transform job pending it's doing it it will take little time okay so let me just come over here on this uh project there you can see now gdpr workflow let's go inside it so now in this gdpr workflow you'll be able to track everything right you can see it has been created 40 seconds ago this is the transform ID this is the workflow gdpr transforms within this workflow gdpr workflow this is a transform step so it's completed it says now if you come back it's the starting transform it transform job activity still happening job completed you can see now it uses name entity recognition in the hood guys okay so it had to look at your whatever column that you have wherever you can it can you know apply named entity recognition any r that we have seen in Spacey and online earlier you can see it's currently doing that sample 100 records 24 columns processing time it's also printing it out everything transform had been finished and the column has been transformed you can see the transformed count transform type has been passed through fake data shift as we have discussed right so we have discussed about regular expression and data shifting we have discussed about hashing Etc you know to basically help this data to become more privacy preserving that's what we are doing in this case with grater now it will take little time so let me just you can see now the ACT again your object you know the modeling part is coming into this the synthetics step the step that we have generating generating the synthetic data that's the step where you can see it over here you know if I come back you will see on the gdpr workflow we have two things our gdpr transforms has been successfully completed now you can see the tabular ACT Gan okay some kind of Gan that model has been used under hood if you go inside it you'll be able to see you know training has been happening you know it started one minute ago processing this data set these are the data set that it has processed you know some 21st records you can see it over here looks fascinating isn't you can see the creating synthetic model you can find I find out all the logs you can see the schema version the model name the models their privacy filters like outliers which is automatic the similarity which is auto so it has to look at the columns and then it has to look at that what's similar it can generate right similar uh rules that it can generate the values that inside that rule then we have data sources this is data source that we are using Adventure Works bike buying transform data CSV and you can have a look at that parameter the embedding Dimension the generated diamonds and the discriminated Dimensions right the generator part and the discriminator part the evaluation metric Etc everything has been identified and the training Epoch has been completed so it will take little time guys you can you can completely track it over on this dashboard you can also see similar on this collab notebook as well you can see the entire step so what I will do I'll pause the video it will take around five to seven minutes depending on the length of your data once it gets completed I will resume the video from there guys so now you can see guys our anonymization has been completed it says data has been anomized and synthetic data is stored to artifacts Adventure work works by buying synthetic data CSV anonymization report is stored to you know artifacts and for example right so let me just click on this artifacts and you can see it your things here right so let me just download these things so when I download it I'll keep it on desktop guys so this is a report because this is an HTML document and if you download on this one sorry excuse me I downloaded from the wrong data so I'll just cancel it this one it will take little time to download that it is you can see the synthetic data Adventure works by buying synthetic data that's the data we wanted to synthesize and generate right and let me also download this so they will have three files you know so it will basically keep three files in the artifacts the transform data as a synthesized data and also the report so let me just save it over here now you can see we have saved all these files and you can see the verbose the logs everything over in this collab notebook now if you come over here you will see the tabular ACT again has been completed and you can see it basically says you know two minutes ago completed gdpr transforms completed if you click on this you can also find everything here you can track it out you can see the generate was there you can also click generate and click it from there you can see the quality score the quality score the data that has been generated the Privacy protection level is good in this case and makes your you know to also explore the paid version if you want to utilize the paid version of it okay they have you know premium as well you can basically leverage more features there I guess now you have additional downloads We have this gif file I guess if you click on this it will download as well you can also download it from here so you can see I believe it's a GI it's a zip file so let me just keep it in the download file I'm not so we've already have downloaded so here you can see that right everything has been tracked the model type act can Cloud completed training time it took around nine minutes but it makes sense right because it has to do a lot of things over there and then we have additional download so let me just go back to models and this you can see there's two works so in this gdpr workflow we have two models tabular at Gan and gdpr transforms now if I come back on the desktop you can see over here adventure bike synthetic data let me open that for you guys so this is the data that we have you know generated okay so miss Stephanie you know first name and the middle name and the last name if you come back over here in the bike buyer you can see the total records I think it's around more than 16 000 okay more than sixteen thousand of records has been synthesized anonymized according to gdpr regulations guys this is what we need you know to work with privacy preserving AI algorithms that's how you look at you know up to if it comes to gdpr and CC right so this this makes sense now let me just do one thing as well for you let me come back here in this data this is the data that we have taken let me just to download this data so I click on this electricity time let me just save this on desktop I just need to save this on desktop I'm going to open this now you can see let me just come back over here you can see the similarities I just want to show you the first name middle everything remains same it has been transformed it has been anonymized wherever required so these are the algorithm that works you know under the hood guys name entity recognition Okay you can see everything over here okay in this data now you can leverage this data okay and you can train some kind of NLP engine recommendation machine learning model whatever required whatever you want to do with the data whatever task that has been assigned to you you can do that now this one so in the next video what we can also do guys we can compare the machine learning model performance on the synthetic data and this real data we would like to compare with the evaluation metric right what kind of you know performance changes that we see we will do that in the next uh video guys so if you see this is what I wanted to do in this video programmatically that how we can you know anonymize data generate synthetic data with with grater Ai and you can see it over here The Grater workflow so I hope you like the video all the resources will be given in the YouTube video description like the collab notebook the GitHub repositories the Gretel console or the dashboard a link and also the gdpr helpers GitHub repository and all the data also as everything will be given you know let me know if you are extending this further or if you have any other use cases that you know you're exploring uh a synthetic data generation using you know generative AI for example this is what I wanted to do in this video guys I hope you like the video if you if you like the video if you like the content please hit the thumbs up icon there if you haven't subscribed the channel yet guys please do subscribe the channel and you know also share that uh share the channel and the video to your friends and to peer so more contents will be you know published uh ingenitive AI large language models and also the classical uh all things AI guys in general so there's a lot of contents coming up okay and mainly end-to-end projects will I'm also planning end-to-end projects how I can create videos from data pre-processing to model deployments end-to-end videos for that I also have some suggestions from you guys the subscribers and I am currently working on those videos once those videos are done I'll tag you of course in those videos that have been requested that I'm currently working on so I hope you like the content you know please subscribe the channel guys and thank you so much for watching see you in the next one