DataOps for Computer Vision | Machine Learning Applications | Community Webinar

Key Takeaways

yeah um as um Nathan introduce me um my name is James Le and it's my pleasure today to um you know present the audience at datao about um data Ops for the modern computer vision stack um so just a little bit about myself um I um my my expertise realize in this whole new field of U machine learning infrastructure also known as mops um tooling um currently I'm a data Advocate at Super PI um I did a lot of writing and and um speaking at at events and um with the goal of black driving product awareness for for the company also um you know r a lot have personal blog on on medium that basically cover different industry application um research areas in machine learning and advice from practitioner in the field uh and finally also have a podcast actually when I interview people like practitioner researchers in ml and data to unpack the career lesson along the way um before all that I um complete my uh graduate degree in computer science from RIT where I did some research work on uh deep learning and recommend system and U before that I was in data science doing stuff for fintech and e-commerce startup um and some of my other interests besides just uh ml infrastructure uh um you know just just fundraising um venture capital and Community lro I think to to these areas that are relevant to to the war you know startups so this is like a brief agenda of uh what I want to talk about today uh you know first I will Define you know what is dat Ops second I will uh give a reason couple reasons why we need to bring that Ops to the field of computer vision third I will uh unpack some the key principles to dat Ops thought I will propose an ideal dat spotline for the computer vision St fif I will talk about the several challenges related to data that any competivision teams have to deal with and finally I will talk about the future of the mod computer vision stack uh so yeah let's get started uh by seeing what is data Ops so essentially um data Ops is a term that being Borrowed by the few devops devops um is the transformation in the delivery capability of software teams and then data Ops focused on the transformation in the delivery capabilities of analytical teams and as you see here in in this image devop is sort of the Synergy between engineering QA and uh operations right and the goal is to can collab together to release the software in a faster base and then data Ops is like adding an additional lay of data on top of all that they it it it it bring about um you know different data teams to work at various level to you know check data quality engineering data um ensuring data security with the eventual go to obtain you know um actionable insights from data they have and I believe that's similar to how devops has transformed the way in which the software development softare works I think de Ops is also changing changing the um best practices of um you know uh handling data and actually another representative of the operations family is called mops which basically imer operation with machine learning so mops um apply de BS F to automate the life cycle of machine leing models and uh as you see here in the slide um both dataops and mops can be seen as a extension of the deop methodology dat Ops cover the data Journey like data inje transformation to to data preparation then mops cover the second half of the you know Journey which include things like mod training M validation motor serving Etc and then um these two loops work uh together to complete the whole uh ml life cycle development so you might be wondering what has led to the rise of data Ops so um there are three main reasons uh first one is the massive volume of comp complex data so um especially in the 2010s you know there's so new new Revolution about Big Data with tooling like um you know Hadoop or um you know RP or high my SQ hdfs those kind Technologies they um they being Bor out to enable businesses to deal with um large volume of data coming from various sources right the second reason is what I call technology overload like in know to answer a business question the data needs to be in a format that you can understand and use for analysis and you basically have to a lot of things ranging from um you know data transformation to data cleaning to uh data security data Integrity Etc and for each of those tasks you need specific tools to handle you know different step and um there's so many tools that that you have to think about and you might be overwhelmed with what choice to to make right and then final reason I could say is the divers roles and mandates so basically um the people using the tools in technology to work on your data are very diverse right we have data Engineers who focus on um data preparation and data transission we have data scientists who need to worry about getting the right data with the algorithms there's data analysts who care about building daily reports and data visualization that's um business manager who only think about um the the business impact using the data and then you know bringing together these different Technologies processes and people with different mandates will creates collaboration overload and friction between teams and that's why we need a DAT framework in practice to make sure that these different person work together productively now this fig and slide provides a partial list of companies that are providing solution in different areas in that form the ecosystem of data Ops so we have the um infrastructure Ops layer which um are consisting of tooling that uh mon monitors the data plan on top of it and initiating a respond to the evl we have the data plan here which includes tools from Opera system that's ETL tools data warehous tools as well as you know visualization and um you know uh Native as system to um there's also a metadata uh stack that creates a map of organization data flow tracks the for the data and enforc access and data quality uh there a development layer which included a lot of the um development tools that you can use to uh manage complex processes that may includeed things like experiment and iteration um and then collaboration across teams and unit and sitting all the way on the top uh data products and services which included uh tools that you can use to track business kpi and maching products and services so hopefully this SL um allows you to understand like you know this a you know emerging um slack and there's a lot of different company that building tooling to support the data Ops landscape so next I want to argue for the case of why do we need to bring data Ops as a discipline into the domain of computer vision the first reason is that in most real world machine learning including computer vision the data is more important than models and this contrast with academic machine learning which emphasize the modeling component rather than rethinking the data pram there a lot of like low hanging footage use um to you know improve your data all right um instead of like fixing only on the model you can you know do small things like data exploration data curation or data collection to get better data samples um and that eventually can can improve your more performance instead of like only focusing on you know um investigating your more architecture or tuning hyers for instance um so I think that as a framework data Ops can help you you know visualize sample and label all the important data parts that are what being labeled and you most value respect to a given task so that's the first reason the second reason is that um you know unstructured data preparation is very challenging um you know as you can see here in the slide um these are some examples of some data sets red planes you know that in incorporates a lot of real and synthetic gener and synthetically generate cell satellite imagery and you can see that this you have to be very you know meticulous about how to label this data Bo and overall like when you're working with image and visual data you know the the sample is high but you also need to be very careful about like you know draw drawing the right Bing boxes for instance right um and a lot of these things are it requires domain expertise to to understanding how to like you know what made up like an actual uh labels and because of that um preparing you know um preparing data for like computer vision use cases and generally like not for the phom part and the third reason I would say is that buing computer vision application is alterative so um this diagram that I put here in the slide represent the two Loops two separate Loops of you know building computer vision application so you got the algorithm development Lo where you first um view your algorithms then you measure the algorithms again in the data set and then then you um you know measure that um algorithms and learning from fillow cases right so that's the first Lo and separate from that you have what called the product development Lo where you first view a product using some sort of microservice uh infrastructure inhouse and then you measure your product performance in production with some sort of monitoring or loging tools then you learn from some failure cases uh with some uh error analysis using um you know maybe tooling to um to analyze your product performance in live performance data so these two Loops must work well together in order to you know complete the whole computer vision application Live cycle things right and data Ops as a practice can have speed up iteration of these two Loops you know for example after building the algorithms right here you can provide those algorithms into the engineers so that they can incorporate it into the computer vision products right and uh another way is that um after you learning from fow G in production you can tell the um you know the the data labelers to sample and annotate mod data set only the one that um your product M filler cases on right and then incorporate that into the algorithm development Loop to BU a new set and then bu a new algorithms and then with best practice data Ops you can basically yeah like I said speed up these two uh different ways that these two Loop can work together um so now I made the argument for the need of bringing dat Ops into computer vision I want to kind of went over some of the key principles of dat Ops and I think it's pretty crucial to understand these principles and then journalize that into the domain of computer vision which I go in deeper details so the first principle is to implement best practices for development so basically um you know if work data right if you're a data engineer or data scientist it's important to follow this life cycle guidelines as I outline here in this line because you need you need to write a lot of code right so it's it's good to borrow best practices sub engineering to make sure that um you know your your code is good enough to be put into production uh so in practice there are things that included vision control code reviews unit testing artifacts management release Automation and infrastructure scode um and these some the these are some of the open source tools that can use to uh you know um follow this Skyline right and um applying the software development best practices for competivision specifically is quite straightforward uh to get started I recommend this document by Google called R of machine learning as well as this article called engineering best practices for machine learning and these uh two resources they gather a set of engineering best practices for developing Software System with a machine learning component the second principle is to automate and orchestrate all data flows from the initial data source to final deployment so in practice you want to automate your department with a cicd pipeline and then discourage any C of manual data rankl and the best the best way to go about this is to run your data flows using an orchestrator for tasks such as back filling scheding and um Gathering pipeline metrics there are a couple well-known open source tools that can help you with um the or orchestration task that I put here in the slide including aach a flow lxer and prefect and applying CD for compe vision is also quite straightforward a great resource I recommend getting started is this article by the team of talkworks um that talk about bringing continuous delivery principles and practices to Machining application the third principle is to test dat quality in all stage of the data life cycle in practice you basically want to test your data at The Source by writing either data in test or you know depending on the data sources that chopar is connected to you might want to write SK test SQL test or streaming test and you then want to validate your data at different stage in your data flow and then finally you want to capture and publish some of the metrix across some of the stage that I just mentioned about and and the best way to go about this is you want to you know um reuse your testing tooling by building a common testing framework that can be standardized across your data team and great expectation is a wellknown open source framework that helps you with you know better quality testing um for competive vision I think that continuous testing best practices is quite tricky because um you you only you not only have to deal with the quality of the data source but you have also think about like the quality of your label because when you working in mission video you need to label the data part and making sure that the label is a high qu right so for things like this you need you need to answer questions like does the work of all your labelers look the same is labeling consistently accurate across your data set and given that human in the loop component when you need the actual labelers to collaborate with building a testing framework for label quality is definitely um you know I think not an easy thing to do the part principle is to monitor the quality and Performing Matrix across the data flows then in practice the first step is to Define data quality Matrix which can be broken down into technical Matrix functional Matrix and performance Matrix the second step is to visualize your metrix to to appropriate data stakeholders so that they can take meaningful actions using meaningful ads um and in comp Vision I could say that um you know this concept of monitoring quality and Performing metrics is also called observability and there's a couple of like you know good vendors in space that have published a have published a lot of thought leadership content about you know bringing observability practices to um machine learning in general um the team at monteo you know defines the five pillars of data offs availability to be included freshness distribution volume schema and lineage and in the context of computer visions that means like you know um how up to date the labels are updated how the label can be distributed within an image how big the number of images is how the labels are visualized and formatted and then finally how the image are currently being versioned uh the teamate arise AI uh come up with a concept they call model observability which is the process of collecting model evaluation in training validation and production environments and then tying them with different analytics workflow to allow a practitioner to connect Otis SP to solve the M Engineering problems together then finally the team gu apps proposed anatomy of an Enterprise AI off stability platform which include key capabilities like T collection a Time series database a Deb backing engine an anomal detection engine and a visualiz and a visualization layer as you can see like there's a lot of companies in this space that are doing nov work with the go like you know how can we monitor these different metrics that allows you to be more robust and reliable um Machining system in the real world the P principles is to be a common data and metadata model um you know most organization in the real world they face a significant challenge because the data comes from different sources and then everybody looks at the system from different angle and practice you want to uh you know create a common data model so that um different people can identify similar terminology different uh ac across different areas and across different teams right and then a more Enterprise focused approach to do that to do that is to invest in the data catalog to share that knowledge more broadly across the DAT organization and I put here in in the slide you know couple of the open source tools can help you with you know this specific principle so there are tools like DBT very well known uh for allows me to combine different data sources and put them into a common data warehouse and there are tools like Amazon data Hub and Mar case or open source tooling that um do the task of data catalog right in the context of Compu Vision I would say that U data catalogs mean creating a repository of O your image and video data from different sources and that include also the details of you know the data structure the data qu data quality the definition of data resarch the data right and ideally you also can allow the users to access the metadata alongside the data itself and what when we're talking about image metadata really means like Json object that describe specific objects right and one thing that I think is still very hard is Version Control for unstructured data right version controls want to keep track of how draw data evolve um across um across you know um time you know maybe previous week like present and in the future how does it evolve and when working with image and video it can be hard to tell exactly how does it evolve um and you know solution like PM for instance um is a promising option that is currently on my Horizon ideally though I could say that a sophisticated solution for vision control can um know key track of you know the the data lineage over time and then that can also part with future data governance and compliance concern and this is the Sixth and the final principle of data Ops which is Empower collaboration among data stakeholders basically there are different ways to structure your data teams but the best approach I think is to embed data thinking into every functional team and this is quite similar to the concept Azure principle in uh sub engineering where you want to build a cross functional team with no division between key function and this team can Define important kpi and matri together and then crop a share objective with the business goal and furthermore to remove potential B for data usage you want to use like you also want to set up self- service data monitor data monitoring and then democr access to data so that's got the high level principle of how you can build cross functional team when working with data so why building computer vision application I believe that an ideal that op team should be composed of the core function all here in the slide and I I go over in detail what these different teams and their responsibility in a second you have the um data labeling manager and these people work in-house or offshore labeling teams to help scale the throughput of data labeling they Define labeling instruction inspect the work of the labeling team and decide how to handle comp Lex or ambiguous scenarios you have the data Engineers who are responsible for Designing building and maintaining data sets that can be leveraged across your machine learing projects and then these data Engineers work very closely which are machine Engineers you have the data curator who are the expert in their respective data domains they are capable of visualizing and manipulating your data set and um their knowledge of the best well the knowledge of the business goal as well as the machine capabilities can inform you know the organization to prioritize the data cation efforts to improve your machine system and finally uh I could say you know basically the head of data Ops the the leader who overse you know this strategic data practices um the goal their goal is to create an environment that allows different parties to access the data they need painlessly build the skill of the business to draw meaningful insights from data and ensure data governance and they also act as a bridge between the dats team and other functional units serving as a um a Leone basically between this team and then other teams right um as as my this data teams like work with different ml application team right across the organization maybe if you build you know like a um different sort of computation application then this that is responsible for coling data for these different use cases um overall I think collaboration process between um you know the team and these different teams are still very ad hoc at most organization currently there's no standard or best practices on how to fac facilitate this collaboration and I think there's definitely room for um tooling that can make this process more efficient so um in comp division as I alluded through our conversation so far like that Ops mean building a high quality train de set um so the next line is my attempt to kind of like draw a diagram picture of how this dat Ops prob look like for modern computer vision stack so here here's like kind of like my my high level thinking of how it looks like and this results from a lot of uh different conversation with practitioners in the field as so uh you know uh tooling Builder who are building tools to solve this problem so uh let's let's go over this slide in more detail and kind of like zooming in in each of the building block right um the first way is what I call acquisition so that basically means you want to collect the data to use for your um Machining efforts when you talk about thata collection that can range from like you know um you have to answer a question like you know where to collect the data and how much to collect initially right besides data collection you can also do things like synthetic data generation and especially in computer vision you can use technology to like generate Sy data um and and you know this can be useful for some of the more first for some use case that doesn't have a lot of data at the first place and finally uh what I call data Scavenging is the idea like you can script internet or use open source data set you know online to get data for your um development efforts the second building block is data labeling and in the context of data labeling you have to answer a question like who should label the data um how should the data be labeled and what data should be labeled right and then getting this data labeling step right is extremely complicated because it is very slow expensive eror prone and often impr practical right and I think that uh like my company we start out with the dat dat labeling solution and so we think that efficient labeling operation could requires a like a very rigorous process with like high performance tooling its own life cycle um and as well as a validation process to make sure that you know you can actually get good labels with your data the third phase is data debuging which entails providing uh expectation test to address the data pre-processing and data star system and essentially these test are what I call unit test fure data they are designed to catch data quality issues before this issue make their way into the datop spline the fourth phase is data augmentation which is a scientific process where you can manipulate the data by flipping it rotating it translating it changing the color a bit right um and this is really novel in general like you want to augment data in different ways I think that um there's still some fundamental issue with data augmentation though like you have to think about how to scale it to B data sets how to um handling some of the bies or Corner cases right things of that nature the fif phase is data transformation which include three SE separate step uh data formatting which is basically you want to uh format the data in a correct um format right you have to inter interface with your data PES to you know make sure that you get the right format out of your data storage you have to do things like FAL engineering right you want to engineer relevant features and um recently there's been a whole new category Co feature store that just came out to you know address this problem and then you have do things like data Fusion which basically means fusing your data from different modalities different sensors and different time together um the six phase is data curation and I think this is the most like underrated step in this whole life cycle because it a very critical part that uh you know Bri briding the gap between the DAT off SPL and M SPL data coration is the idea that um you know you only want to create the the most relevant data boice to use for your development efforts and in practice you can do things like um you know cut data C data cataloging basically you want to um you know um sort the data by use cases right or uh make it searchable by tax or you know cluster data by a specific metadata and then after you do the data catalog you can do data structuring data selection you will select the one that most relevant for use cases I will talk about the data cion in a more few slide but I think this is like something that is emerging that we need to pay more attention to after data you go into your M SP you do everything that you want to do with your model you want to view algorithms you your model in production you monage your model then finally after you get a model in production you you want to close the loop right by do by doing a final step what I call fillers and HK detection and the idea here is one to um you know measure your model in production identify the the cases where your model underperform and then um after that collect more data on those fow cases to get back um collect more data and then go back to the data acquisition loop again to like you know um iterate your mod development efforts basically the second like V2 right of your of your life cycle um and yeah got of kind of complete the life cycle and and is important because machine learning development is always going to be alterated so you want to constantly got like a new version of model all the time and this is how you want to U accelerate that feedback look so yeah I hope I hope this you know this diagram make sense to you and happy to share more more details um up after the the webinar on how I conceptualize this effort um for the next three slide I want to talk about the three specific data Centric challenges that any competition teams have to deal with the first one is data curation which is the part that I really just made in the previous slide right so to to double click on this why um data cation is the process of discovering examining and sampling the data for specific U analytics or prediction task and then in the context of computer vision data curation is like massively underrated because currently there's no streamline method to understand what kind of data has been collected and curated into a well balanced hardare data set it is very time consuming you know because um it often requires lot of domain knowledge about like you know what what does it mean what what what this data means in this context and then existing solution cannot keep up with the 4V of big data which are volume velocity variety and veracity in in a very you know M highly evolving Ever Changing data equ system and finally the solution are quite narrow because they primarily learn from the correlation Cur currently being present in the training data I think that the function of datos platform for division should have data cation capabilities that can enable data data and ml Engineers um to um you know visualize massive data set discover and reach data with ease Cate divers scenarios and integrate seamlessly with existing workflows and tooling the Second Challenge is data labeling which I also talked about in my previous slide uh three three slide ago yeah so um the idea here is like training a computer vision model right requires like like a constant fit of large and accurately labeled data set however this process requires a considerable time and capital commitment especially since most of the labeling and QA is done manually by humans so my team at superbi we work on this problem for like several years now and uh two major pain boys as I like here in the slide like manual labeling nqa is just painfully slow like the like human annotators go into each image and annotate them Hand by hand right it's just not really scalable and and also when you're dealing with very domain specific data set the the the QI is going to be you know very low you think about like later pause like image in video that You' never seen it before right very uh rare cases then you cannot like label it correctly and and so uh basically the label Co is going to be very bad so I think that an effective dat platform for Compu Vision must address these two very expensive step of labeling data and auditing the label and in in more detail like the um this platform should be able to automatically label the data identify and audit hard labels and then use no techniques such as active learning to verify you know the the human labels um and you know my company do a lot of this work on on you know using automation as practices to to address this issue and then the the third and the final challenge is what I call accounting for data drip so um the idea of data drip is that um basically um the the training data set that you use to train your model um in in your in your local environment it's going to be different from the production data that you use when you put your model into production right uh because you know the data change all the time it drips because because it's come from a dynamic and time evolving distribution and that means like when the drift your Motor Performance going to go down and it suffer right and it going to decrease over time leading to like business uh losing money for instance there are different causes for data DP like uh Upstream process range data quality issues natural drip in the data coar stri for instance and um I think that a robust datos platform for competition should be able to detect their drips and raise alert accordingly analyze where and why the drip happens and then adapt the drip and improve modor performance and yeah like and this this SP Bo it kind of went back into um what I said earlier about observability platform right these companies building tooling to us to address this their drip um issues and I think there's a lot of interesting you know work and and technologies that currently being spun out to to address this issue what the final three slide I will talk about the future the mod competition stack and basically like what does it look like you know in the future I I say that the modern competivision stack going to follow the footstep of the modern data stack so if you like if work in data science I could assume like most people in data science don't show do you might have heard of this term called the M Stack you know being um everywhere in the past two years or so so the idea of mod data stack is that is a collection of cloud native tooling center around cloud data warehouse um the benefits of using mod data stack are are like four four number one is e abuse so these different Technologies allow your teams to not worry about installing and maintaining technology everything is BU around a data warehouse and this minimize integration pain points and Sal data platforms that requires lots of effort spend shipping data around second benefits isqu adoption so the modern data St is constructed with the intention of upscaling data workers and removing the barriers between workflows third benefit is automation so you know the this St allows you to automate a lot of different workflows um you know when when when uh integrated alongside different tooling and the final benefit is um cost efficiency uh in the cloud if you use like a cloud services for your technology you only pay for for your usage right you pay for what you use and then um that has to benefit you know allowing to spend money on other more business critical activity and then as the modern data St continue to grow and evolve there are different Technologies and vendors that that entering the conversation and I put here in my slide a diagram created by the TM Contin which is a Tooling in in this St um that encapsulate the some of the main functional areas of the data St with have the cloud ATA Warehouse data integration even tracking data transformation uh bi analytics data catalog data off ability uh and data orchestration and this a very robust ecosystem of tooling that made up this St now I will say that um machine learning and computer vision is sub machine learning is also evolving and and converging in the same par what I what what I call the Canal St for machine learning um so my company super we we are part of this new newly form organization um lasto AI infrastructure Alliance and this Alliance are dedicated to bring together the essential building blocks for AI application and this kind of cost for machine learning brings together like efforts of different people projects and organization and it it acts as a focal point uh that you know um allow these people to talk together right and um the idea is to establish clean API integration Pino and open standard for how this different you know Machining component of a complete Enterprise Machining sty can interoperate together and the diagram that I put here in the slide is a an early draft of some the emersion patterns that seeing across like 100 members of The L um you know you can see here like you know there's there's tools on data engineering to um you know orchestration pipeline to feature store foundation on Prem infrastructure to like motor security so there's a lot of different like component that made up this this modern ml stack and then I guess my my my hope is to you know in the next few years we can come up in with a clear Paradigm of how this two can work together just like how the mod data set has been um in the last two years as I show as I show in the previous slide and and with that like I also want to kind of like put this final slide here um which showcase the different startup opportunities in the machine infrastructure ecosystem like um this diagram broadcast some of the times of ml infrastructure companies divided into five major categories so we have tooling that can make call workflow better faster and cheaper web tools to that optimize Communication web tools that improve production model reliability we also have tools to augment data programmatically as well as tools to remove the need for special Engineers so these categories are meant to deal with some of the existing mainstream ways of doing machine learning right right and then um My Hope Is that as more noval techniques um you know take off um this infrastructure will need to app as well like recently we have we have seen a lot of Novel Technologies in like lar language models or um you know open releas stable diffusion for instance like this technology um they have changing the way we're doing ML and the infrastructure to adap with those need need to adapt to to um satisfy that and so if you like working data and you think about the pain power you currently have in your workflow and you want to think about way to build new tools and even do a startup AR this whole ecosystem now is a great time to do that and you'll be in in the same hand with like a lot of different other vendors that are solving the same problem which is to improve the the productivity of um ml engineers and computer vision Engineers Etc to to uh get more of those application uh into the real world um yeah and so with that uh I want to conclude my talk thanks a lot for um you know attending the webinar um reach out to me via email if you want to get a copy of slide I'm also available Twitter if want to chat about you know high level ideas um about data Ops about conversation in general and I also have a website when I publish a lot of um rock B you know about this topic so you want to check it out feel free to follow me on there as well um yeah I I pass it back to Nathan right thank you James um so I'm not seeing any questions yet so what I'm going to do is I'm G to steal the screen from you um I'm gonna do my little bit on the next uh uh webinar we have coming up next week and then uh we'll see if any questions surface at that point so um so Sam is asking about the session recording so the recording well there is a recording um and we'll be posting it we'll do a couple edits of it uh uh take out any awkward pauses that sort of thing um and then we'll uh we'll make the recording available on our YouTube channel as well as uh data science Do's website so um FMA if you could post the link to past events again because Sam that is where where um uh you can find all of our events very easily so I'll have FMA post that or actually yeah is that it yep so FMA just posted the link so you can you can find the recording there um probably in a day or two and then uh but let's get my screen shared because we have a crash course on transfer learning coming up next week um that is the right screen perfect so next week October 4th at 12 Pacific time so the same time as today uh we will be having a crash course on transfer learning where we're going to be understanding the idea of transfer learning we're going to learn how deep learning models communicate with each other explore World applications and then uh compare transfer learning with uh the human uh continuous growth model so um if transfer learning is something that you're interested in uh feel free to join our crash of course we'll have wasif Nadim he's one of our data science doas analytics content Engineers um he's a real math nerd so um uh really fun guy to talk to about transfer learning so October 4th uh 12 Pacific Time crash course on transfer learning I hope to see some of you there and uh James I am seeing a question coming so I'm gonna stop sharing if you want to feel free to pull back up your final slide again if if you still have it open um just know how to get touch with you y uh Adama is asking is it possible to get the the presentation slides and I think you mentioned for people to email you if they wanted a copy of the slides is that right um yeah um I mean I I made the slide available so yeah just just email me in the slide yeah so all righty so thank you James uh really appreciate you taking you know time out of your data be here with us and thank you for everyone else who joined appreciate you being here as well