MLconfEU 2020: Machine Learning Conference for Software Engineers

Key Takeaways

thank you [Music] [Music] [Music] thank you [Music] he is suffocating I'm waiting and always hesitating Kryptonite desires set my heart of Fire [Music] heart on fire [Music] thank you [Music] with Crimson Secrets can't stay still don't stop that Thrills my bones [Music] foreign [Music] [Music] [Music] [Music] thank you [Music] [Music] foreign [Music] laughs [Music] [Music] thank you [Music] thank you [Music] foreign [Music] [Music] [Music] thank you foreign [Music] [Music] laughs [Music] [Applause] [Music] [Applause] [Music] all right [Music] [Music] [Music] [Music] thank you [Music] [Music] foreign [Music] foreign [Music] foreign [Music] [Music] [Music] welcome back [Music] [Music] foreign [Music] thank you foreign [Music] thank you [Music] [Music] [Music] thank you [Music] [Music] thank you [Music] [Music] [Music] thank you [Music] I don't know [Music] foreign [Music] [Music] thank you [Music] [Music] thank you [Music] thank you [Music] thank you foreign [Music] [Music] [Music] foreign [Music] [Music] [Music] [Music] thank you hold tight dripping lights paint the skies All Because Of You dripping lights Skies only [Music] can make me lose [Music] Make It Last Forever [Music] hold tight [Music] [Music] [Music] thank you [Music] [Music] foreign [Music] [Music] [Music] [Music] [Music] foreign [Music] stranded in the open eyes [Music] [Music] [Music] [Music] foreign [Music] [Music] thank you Heroes tonight we will fly above the sky we are heroes tonight [Applause] [Music] [Music] [Applause] [Music] all right [Music] ready to explore right up in the sky [Music] I've been flying from town to town [Music] [Music] We Will Fly Above the sky we are heroes [Music] [Music] they were doing [Music] tonight [Applause] [Music] thank you [Music] heat is suffocating I'm waiting and always hesitating Kryptonite desires set my heart of Fire [Music] set my heart [Music] thank you [Music] my bones [Music] [Music] thank you [Music] [Music] foreign [Music] [Music] [Music] thank you [Music] thank you foreign [Music] [Music] [Music] foreign [Music] [Music] thank you hi [Music] [Applause] [Music] [Applause] [Music] foreign [Music] [Music] [Music] thank you [Music] foreign [Music] thank you [Music] foreign [Music] [Music] foreign [Music] [Music] foreign [Music] [Music] thank you foreign [Music] [Music] [Music] thank you [Music] thank you [Music] [Music] foreign [Music] [Music] [Music] thank you foreign [Music] foreign [Music] [Music] [Music] [Music] foreign [Music] foreign [Music] [Music] thank you [Music] foreign [Music] [Music] [Music] laughs [Music] [Music] [Music] foreign [Music] [Music] thank you [Music] Make It Last Forever [Music] [Music] when I went down [Music] [Music] [Music] foreign [Music] [Music] [Music] [Music] [Music] stranded in the open eyes thank you [Music] [Music] [Music] foreign [Music] [Music] Heroes tonight we will fly above the sky we are heroes tonight [Applause] [Music] [Music] [Applause] [Music] [Applause] [Music] ready to explore right up in the sky [Music] I've been flying from town to town [Music] [Music] thank you [Music] thank you [Music] hey good morning evening Day to everyone how you doing good morning are we excited today guys sure am we are super excited and the day is still out so looking forward to the next hours no it's almost dinner time Sergey it's almost any time so what I'm looking forward to today is a conference about machine learning and as I'll tell you later on I'm not into machine learning yet but why is it not called machine teaching we're teaching a machine stuff we're not learning anything we're teaching the machine you can somebody find the machine something that is so true but the machine itself is learning right so it's from the perspective of machine you know machines have feelings too or do they should I call you Wally then please do I love that nickname oh what do you think about this circle yeah you can only teach machine if you have something to cheat there's nothing but who are you like who are you to teach this machine well I'll explain Let's uh get to it so let's get the show on the road um I am Matthew prasinski I am a front-end developer at passion people here in the Netherlands in the lovely city of bermarent I um I've been working with machine learning since this Monday since I started at a company called Aiden's agency is a med tech startup that analyzes CT scans to detect early stage lung cancer so I'm really happy to be using my forces for good and yeah now this this whole event became super relevant for me to join and I would talk Corona thing going on but I think it's really cool that I just learned this Monday that aidens is also trying to learn to have these Technologies to look at long scans from City and now they're also trying to detect covet so that's super nice and I'm really looking forward to working there and doing some machine learning so Sergey who are you yeah it's funny that you mentioned like patient people actually I work with fashion people that's kind of close by but for different direction right so during the day I build products with help of machine learning and data and during the evening times and machine learning Google developer expert engaging with communities and do some open source stuff so definitely looking forward to see what's going to be in the next days what about you AJ like how are you connected to machine learning me I also have like a Jekyll and Hyde relationship with machine learning too so my name is a Jay Hawthorne I am an associate data scientist at the goat Group which is an online sneaker marketplace where you can buy and sell shoes so on the day I do that day job and in the night I manage an educational YouTube channel where I teach artificial intelligence machine learning data science deep learning all that good stuff and I'm on a mission to make this world a more intellectual one so are you talking about this world I guess that's what you're talking about right but as you can see here there's this world map with all sorts of nice dots and this represents the people that registered to watch here today and as you can see we have basically every time zone represented so if you're having breakfast enjoy if you're having lunch enjoy everyone brunch enjoy maybe you're having dinner or a midnight snack well enjoy today and happy that you set your alarm to join us or you're staying awake so if you're like in Australia and it's in the middle of the night for you just five more hours before you can go to bed happy to have you and if you think those are just blue dots on the map random photos you have runs those are actually real people all around the globe trying to learn something about machine learning so they can teach their machines as many mentioned before so I'm really excited to have like more than 5 000 people in the same conference right because it's actually quite um democratizing to have so many people at one event if you would remember pre-color on that time for fitting 5 000 people we would need quite a big room and with today's regulations we would need like have even more space so no safety regulations to break today and hopefully a lot of content to enjoy that's awesome guys and those are actually some really amazing stats and we're able to have so much of you so many of you here today because of our diversity sponsor Firebase they allow people of different social and economic backgrounds to have more easy access to the paid tier and maintaining this diversity is extremely important to us so thanks a lot Firebase yeah so we discussed already about a dentist right but what about speakers how did they select them so I would tell you that it's not an easy task right I was trying to help the team to do that as well so how could we solve it we collected a bunch of people that we call program committee and those bunch of people were meeting up almost like on a video base and trying to select best of the best speakers so I just want to say thank you again to uh Gan sayak Dmitry if and Yuri for doing a great work I'm looking forward to see how you're going to like our speakers and uh please give us your feedback afterwards so besides our diversity sponsor we have normal sponsors also and once again we would not be able to have this kind of events without them so big thanks to Microsoft MathWorks infinite red Academy Focus reactive and once again Firebase thanks a bunch yeah and there's no conferences it's not only about watching people on the stage over your screen is to do right now it's also about talking to the people in the hall and for normal conferences we actually have rooms and Halls right for online conference we have the internet so if you would like to engage in the conversation and you use Twitter please use hashtag ml.com view this would make your conversation more visible to other people and hopefully gonna get you some free tweets as well wow that's some great connectivity via social media guys and guys and girls and for more information about the conference and agenda with timings visit mlconf.eu now here's a screenshot of like what the navigation on your website is going to look like so the main screen here is the video player that you'll see the live feed towards the bottom we've got a timeline of events that happen in the conference on the top we have you can access like the Remo which is the social media conferencing tool you can access the Discord and also even the conference schedule so basic question how does this conference work great question audience well we'll first start each session of the conference by first introducing the speaker then the speaker will give their talk then join a q a session with dmcs now if you're a part of the full access tier you will also have access to speaker rooms and discussion rooms which we're going to explain shortly now this is probably a lot of information but if you are lost at any point feel free to click on the little question mark on the top and will definitely help you out because we're very nice people speaking of nice people there's more than us there's all sorts of nice people so we have virtual watch parties prepared for you for this we have a tool called Remo you can find the link up top above the player on the website so if you join Remo you'll see basically this View and it's like a big conference hall right so you can this is the top view you can sit on a table with other people and enjoy the conference together so when you join you'll be seated at a random table with other people if you want to switch your table you can just double click on another table and join that group of people you can join with a without video and with your microphone so that you can interact with the other people on your table while you're watching the video stream together so you can have a discussion about the content and the learnings you're doing so once again this is the top view and you can click on the video to expand that video of the talk that's currently happily happening so this is a lean forward type of situation even though you're watching at home maybe you're not at home maybe you're on a couch somewhere in the middle of the park I don't know but however you're watching you can still socialize just like in real life yeah if the table sounds too intimidating for you or you don't know like what table to join because like too many of them we have you covered as well so we have Discord that allowing you to send some textual messages and talk to all people at the same time Additionally the same tool is going to be used for um asking questions to the speakers there is a special Channel called ml q a and during the talk please write the questions there at the end of the talk we're gonna ask your questions to real speakers and your question is going to get answered so um yeah it's the tool that we have fun fact this card originally was used by Gamers and those days AI is actually using pretty actively games this environment for intelligent agents so Navi part of the game is artificial intelligence so let's win wow we really came full circle on that and speaking of Discord and any kind of social media interaction we want to assure you that everyone should feel safe and have a good time during our event so first we encourage you to read our code of conduct at mlconf.eu COC and if at any time that you see somebody be harassed or you are harass yourself either on Discord the discussion rooms or anywhere in the conference please reach out to us at COC at get nation.org and we'll help you right away we want you to have a fun educational and carefree experience yeah like I mentioned we have the remote to be lean forward but there's also discussion rooms which are also super nice so today there's two discussion rooms and tomorrow another two today we have one about Caress and tensorflow and another one about computer vision Tomorrow there's one about reinforcement learning and another one about fairness and interpretability so if you're into any of these topics be sure to check out the timeline when these discussion rooms are taking place and join them if you want to join our experts with our moderating these discussion rooms for a nice discussion about these topics this this discussion rooms will be taking place on spatial.chat if you're not familiar with spatial.jet I'll explain how it works so when you join you can once again turn on your camera and or microphone and you'll see a top-down view of a space errors will be floating avatars and you can move your avatar around and here's the cool part just like in real life if you get your avatar closer to someone I can realize it would be your person you can hear their audio more clearly so just like in real life if I get close to you you'll be able to hear me better if you move further away vice versa so Jeff this is really cool and a really unique experience and I invite everyone to join there on spatial.chat for these discussion rooms and there's more stuff happening on spatial.jat which we'll get back to later yeah floating over to our sun school but what can be even cooler is like floating and Blasting avatars for that we have quick tournament so later after the talk once you have like a good you know good understanding of new topics meeting new people please join our great tournament just in your browser right so you can make even closer connections with your people if you don't like gaming or you're not aware how to play Quake please fix it tonight you can still join special chat rooms this is also amazing opportunity to just like hang around with the people maybe meet the new ones and remember the conference is not only about watching videos or people on the stage it's actually about making new connections and enjoying new people so maybe one or two or five hundred of you are secretly looking for a new position good thing we get you covered so some of our sponsors have amazing positions available so if you're looking for something new be sure to check out moconf.eu jobs and check out the amazing positions we have available at our sponsors and once again our sponsors are the ones that made this event possible we have a lot of people also watching for free and as you can imagine it's not free to host this kind of events so if you want to work somewhere that cares about these kind of events in the community check out the jobs speaking of our sponsors again these same sponsors also have offered us some few goodies and some perks that you could claim at mlconf.eu perks now if you are interested in attending a lot of conferences like ml coffee you like but you're interested in like graphql or like node.js or any of them or like anything else right consider getting a conference multi-pass so you can save a ton of money and participate in our amazing conferences to come so visit getnation.org conference multipass for more information on prices and events it's a really slick deal and will be completely worth your while and we also have a way to get these conference multi-passes for free at this moment so you can play along for a competition that will start now just get your phone just get your little phone take a little selfie of you and your entire setup and just show us how you're watching the conference and whether you're watching on like a TV a computer a laptop whatever it is just tweet this selfie at us under the hashtag mlconf EU at the end of this conference we are going to pick two winners and these two winners well well first we're actually going to shortlist a set of 10 winners where we think okay that was a pretty good selfie and from this we're going to spin a wheel from which we select two winners for a final raffle ticket and the winners will get access to a Free Conference multipass it's fun there's nothing to lose there's only to gain so let's just have fun with this and about having more fun um it's also fun to receive feedback right from this fund machine learning can actually learn or you can teach them depending like who you're gonna ask um so please send your feedback also to us in order to do that you need to go to ml.com feedback please put what do you like what did you maybe you want to improve for the next time we're gonna listen to those things and makes even better than next year yeah and again without those sponsors that you can see on the site it would be really hard to pull off this amazing event so thank you again for joining us and helping us to make this amazing conference cool I mean we had quite a bit of like meta discussions and mentioning all things about organization but all of you are here not because of that you are here because to would like to listen to amazing speakers and I'm really happy to introduce the first speakers this is Thomas Wolfe and he is going to be talking about NLP and why NLP right because on one hand if you're getting burned as a kids right first we start seeing some things right but once we start seeing we also start talking and this is about language right so language is like really fundamental part of us being humans and Thomas together with his company hugging face did quite a bit of uh good things to this community and natural language processing so please welcome Thomas this is amazing talk about how to use NLP for transfer learning and making it easy for other people thank you [Music] foreign [Music] welcome to my talk and today we're going to talk about transfer learning in NLP I'll start to talk a little bit about the concept the history then talk to then present you the tools about the tools that we are developing at hugging face and um and then I hope you have a lot of question for me so q a session okay let's start by Concepts what is transfer learning that's a very good question so here is the traditional way we do transfer learning yeah sorry is it is the traditional way we do machine learning usually when we face with the first task in machine learning We Gather a set of data we randomly initialize our model and we train it on our data sets to to get the the machine Learning System that will use to predict and to predict you know to for instance to work in production now when we faced with a second test usually we start from scratch again we'll gather another set of data another data sets we'll randomly initialized from our model and we'll train it from scratch again to get a second Learning System and the same way we Face refreshold task okay we'll have a user data set we have a third machine Learning System again initialized from scratch and that we'll use in production so this is not the way we humans do learning you see when we face with a new task we reuse all the knowledge we've learned in the past task or the things we've learned in life all the things we've learned in University classes and we use that to bootstrap our learning okay so you can see that as having a lot of data a lot of data set that we've already used to generate a knowledge base okay and now this gives us two main advantages the first one is that we can learn with just a few data just a few data points because we can interpolate between today's data approach and this can Um this can help us do so for that augmentation if you want naturally and the second Advantage is is just is that we can also leverage all this knowledge to reach better performances so humans are typically monetary efficient and have better performances than machine Learning Systems so transfer learning is one way to try to do the same for statistical learning for machine learning so we've done last summer very long tutorial there was a three hours tutorial so you can check out these links there are 300 slides a lot of Hands-On exercise and open source code base so if you want more information we should go there so there are a lot of way you can do transfer learning but today I'm going to talk about sequential transfer learning which is the currently uh most used flavor if you want of course for learning so sequential transfer learning like the name says it's a sequence of steps so at least two steps the first step is called pre-training and doing these steps you'll try to gather as much data as you can okay we'll try to build basically some kind of knowledge base like the knowledge base we human builds and the ID is that we can add gen end up with a general purpose model so there's a lot of different general purpose model you've probably heard about many of them where to work on Globe were the first kind of model leverage leveraging transfer learning they were word embeddings but two days we we use models which have a lot more parameters which are fully pre-trained okay like the GPT or distilled birds and these models they are pre-trained as general purpose model they are not focused on one specific task they can be used on a lot of different tasks so how we do that we we do a second step of adaptation or fine-tuning usually on which we will select the task we want to use our model for and we'll find tune it on this test okay so here you have a few examples text satisfication we're labeling question answering but let's start by the first step free training so the way we train our models today is called language modeling so language modeling is a is a free training objective which has many advantages the main one is that it is self-supervised which means that we use the text as its own label we can decompose the text here the probability of the text as a product of the probability of the words for instance and we try to maximize that so you can see that as given some context you will try to maximize the probability of the next word or the probability of a master okay the nine things that we don't have to annotate the data so in many languages just be just by leveraging the internet we can have enough text to train really high capacity model okay so this is great for many things any particular low resource languages it's also very versatile as I told you you can decompose this probability as the product of probability of a various view of your text and this is very interesting from a research point of view now how are the models looking so there are two main flavors of model yeah they are both Transformers because Transformers are kind of interesting from a scalability point of view the first one is called bird so to train a bird model you will do uh what we call mask language model which much log wedge modeling which is a denoising objective so we take a sentence here we mask one token and we'll try to predict this token back so we project all these all these words in vectors then we will actually use what is the attention layer in Transformer which will do a weighted average of these vectors to end up with also vectors but these vectors that now depend on the context which mean that the vector Associated to Mass have enough information from the context to be able to predict back or at least to to try to predict back the missing the mass 12 okay so this is how birds is trained you train this model to predict back the mass token now there is another flavor which is called Auto autoregressive or causal model and these models they look pretty much the same but you can see that with to one token here is associated the next token so my dog the token that will be at the end of the hyper power either of the hypo column Associated to dog will be is the next token okay so we need to tweak the attention here we have to mask the right context otherwise the model can just see the label here on the right so these models they are inherently um less powerful because I cannot use the right context to do their prediction but they have a lot more strange signal here so they usually train faster so these are the two main flavors of model these are the two main famous of general purpose model now how do you do yeah the patient steps it's pretty easy and yeah these are the two advantages I show you the first step is that we will remove the pre-training head so we'll take these pink boxes that I showed you on the on the previous slides and we remove them and we will replace them with a task specific head so if you do text classification it can just be a linear projection to the number of classes that you have if you're doing a more complex task you can add a full neural network on top of it if your task is very complex like circularly different for instance you do matching translation then you have to do more complex things we'll talk about that a little bit in the next session section so let me show you two main examples of Downstream tasks that we can tackle text classification and generation so here is an example for text specification to kind of show you how it works let's say we have a sentence here Jim Henson was a puppeteer and the task for our machine learning model is to predict if the sentence is true or false so this is just binary classification if you want okay so the first step would be to convert this input sentence in something that our model can digest so now models they are they can tackle uh integrals floats they need numbers so the first step will be to convert these strings into numbers and there is one thing interesting here is that our models are made to be trained and through process basically open domain vocabulary open domain copies they are made to be trained on the internet so we have to handle some uncommon words here is Preparatory is a word that is definitely not very common in English so the way we handle that is that we will catch this word in prefix and suffix until we know every sub part of the word okay so here for instance per pattern error how to kind of common prefix and suffix so they are in our vocabulary and we can then just convert all this in our vocabulary indices this is now the inputs of the model we saw two slides ago the birth of GPT and as you remember the output was a vector here the output before the pink boxes was a vector and we then just have to protect that back this Vector to two classes so for instance we pull them put them into a linear classifier and then we get the classes so this linear classifier will be trained from scratch on our Downstream task okay so it has to be quite a small in terms of parameters if you want to be that type issue but this one is pre-trained on the huge Data Corpus that we've gathered so this one can be really use okay so here are two um I mean here is a practical example on a classification test called track six so you can see more details in the slides of our tutorials last year but basically you will see that just after one Epoch we already have an accuracy over 90 percent which means that this is really that efficient this data set this tracks is is really a small data sets 2500 example so this is really that efficient and when you train for three bucks you'll get down to an error rate of 3.6 which is the state of the art well it was the state of the art last year so you get the two things that we're talking about data efficiency and high performances okay now here is a totally different example so you see the variety the diversity of tasks that this model can face um this is a chatbot setting where we have kind of a knowledge base the chatbot our chatbot is pretending to be an artist before children who get a cat and like to watch Game of Thrones and now he's discussing with a user in an open domain setting so the user said hey Chad button sir hello how are you then do users say I'm good thank you how are you and our dialogue agents our machine learning model is supposed to generate a reply which makes sense given the first night okay so you see this is quite a different task because we have a lot of different type of inputs we have kind of a knowledge base here we have a dialogue history as well and we have these two channels and even the beginning of the reply because we're generating the reply word by word so we have the beginning of the reply that we should also take into account or generate the next word so how do we handle that well actually we had a paper last year at ACL that you can check out but basically you can do two main ways to handle that you can concatenate all this in a single input this is one simple way to tackle this and it actually works really well or you can duplicate your model and have several parts that process each type of input and then you need to connect them together with cross attention or a way to to connect them okay this is very different but the idea is that overall as you can see here this is a competition we participated in a talking phase two years ago now and you see we were really leading on the automatic matrix by a strong merging using this type of transfer learning approach so if you were done Transit limit I go fast but yeah there is a trend to bigger model this is amazing this is quite a big problem because uh it's narrowing the competition there is some way we can reduce the size of the smaller distillation printing quantization won't spend too much time on this but this is something we are like a lot these models have generation the generalization problem even though they are big they have really difficulties to tackle out of domain generalization and there is basically a limit to text as a medium which is that a lot of common sense is just not written down in text and so this is a limit to just what this model can learn and the main way we can overcome this is to use some database or image or even to use human in the loop okay and now there is the last main problem which is that these models they are changes uh one time and they can it's very hard to make them learn new knowledge later on because they have this problem of catastrophic forgetting and so for instance gpg3 which was very expensive to Trend Trend costs like 10 million gpt3 has no idea what covet is well Kobe is such a strong element of our daily life today okay so this is too bad so there's a lot of shortcomings a networking phase we try to tackle some of these shortcuts so where do we do a twigging phase well we try to democratize an LP we start with as a chatbot company General building a game and we we open source a lot of tools while we were doing that and actually this tool catch so much interest in the community that we are now fully focused on catalyzing and democratizing all this research level work so we do that in two main way the first way is to knowledge sharing that's why I'm talking today and the second way is to open source a lot of code and to open source some Library which lets people leverage all these developments and develop better models on top of them okay so let's see a few of our open source libraries and this will be the open door for the question in the second part okay so the first main library that you probably know is called the Transformers Library which is a way to access all the state of these uh of this set of the art model and they can be used both for nlu and LLG in many languages we have today read a lot of models in the library there is all the model you know probably bird GPT and also more interesting model and more recent DPR Pegasus XM Roberta is a great multilingual model by Alex ikono on team and there are actually uh G5 is a really huge model so there are a lot of them very simple to use we can talk about that in the question and we even have a model Hub when people can share their model we have more than two three thousand models and in many many languages so if you want to use a model for finey short German they have like 132 different models you can expect and you can play with them even in the live the in the web interface you can try them all and see how they behave this is at huggingface.com now we open source more recently a second Library called tokenizer which do this first part that I was talking about like the string splitting the string in integers so why did we did that this was really a bottleneck in time of speed and so we decided to to use some very low level risk code super fast to do that to solve this problem so this this is now really a very fast step is available in Python node rests rest is a great language and more recently a few months ago we opened sources their Library which is called data sets so data set is our library and this is a to tackle um one last problem we discover which is that data set themselves they're kind of hard to access for people they have to share and we spend a lot of time rewriting the same processing code that people have actually already written a lot in there for the on that set so we decided to to make your library to to solve this problem make it easier to share data set and also to share metrics some metrics right now some novel Matrix are actually very complex to use or like to at least to install and to to set up Matrix for instance to evaluate natural language generation can be really complex and so we decided it could be also good to make something very easy to share them to upload them to share them so this is the latter set library with that set on metrics and it has a few really cool stuff for instance there is building interoperability with numpy panda python tensorflow too so you can use basically any type of framework if you want well any type of modern framework it's also made specifically to tackle really large data sets so if you have gigabytes of data if you want to for instance train your model on the Wikipedia and some data that are your data set maybe for instance bigger than your RAM memory you can use this library because it has a lot of smart way to do um on disk zero sization so you just take nine megabytes from for instance to train Wikipedia there's a lot of smart caching if you process your data set once it's all right it's already good and you you won't spend time reprocessing it again so you can check this one here is an example how you prepare this is the full preparation of training set to train a model and glue with the tokenization with the padding and you see this is just like 20 lines of code and here you're ready to train this model so it's made to be very efficient and everything be very visible at the same time so same as the model we also have a hub where you can access and you can explore the data sets so there is like a model 160 data set now there are also multi-modal data sets with images and so you can go to hackingface.com explore all the data sets you can read see the station everything and you can even like see what's inside the data sets in the web interface okay so same here you can go and check it out at hackingface.com okay so these are the three main libraries and I'm really happy to talk about any question you may have on them and also any question you may have on the first part the concept and the history [Music] foreign [Music] yeah that was amazing talk thank you Thomas and for our audience please give him a round of clubs right but since he cannot hear your clubs right maybe please do virtual clubs in community on Discord so if you actually share a bit of a love and some feedback as well um yeah so with the talk we really got like a good insights but we also have like one question to you to ask before we're gonna go to a q a so for that we're going to use mentee.com so please open your browser on your laptop or mobile phone and enter mentee.com there you're gonna get asked for the code your code can be also read on the screen that you see right now but I will also read it for you this 7967042 one more time seven nine six seven zero Fourier two and the question that we asked today in order to get your opinion is about leaky relu if you're playing a leaky relu to Adam what you expecting or what are you doing basically please watch it here and after sessions we're going to get results for this one and now um let's welcome to this visual stage Thomas for your questions Hey Thomas how's it going for you hi good good how's it going for you yeah it's also pretty nice I mean sun is already a bit like out in Munich but uh it's still quite a bunch of exciting talks and you did kick off this conference pretty nicely because I do believe that natural language processing or language in general is like one of those indicators how good we are at the you know understanding this machine learning how do you feel yeah the Fiji I think what we've seen in NLP right now I guess over the last two years probably that it has become really what we would have expected to become from the start which means uh really the way to process knowledge and the way to kind of do reasoning or what we hope would be like resulting and so when we talk about AGI I think right now a lot of people think about gpt3 which is a full text model so I think this is really an impressive thing about how NLP is now the most exciting field to be in AI yeah and it's kind of funny right because you're almost predicting the first question right and the first question is actually about gpt3 and people asking like hey since you're an expert in NLP right and your company driving such a good efforts in this regard what what do you think is like uh advance of gpt3 comparison to egbt2 yeah it's it's a good it's a good question and I think well one of the problem of gpt3 is that it's it's quite difficult to access it so I think we have not really evaluated the capability of gpt3 like we've done it from for other products like birds or GP Distributors because like a lot of academics did not really have full access to be able to investigate what's happening what can you do with that to be able to test it fully on a lot of tasks so it's kind of hard to give you really an answer right um what I think is is what we can do is that gpts really behave in in some way like an interesting thing which is a retrieval like a smooth retrieval engine over a really large data set so you can do like it's like having a huge Google search where you can search every page is on the internet and be being able to smoothly interpolate between all these pages so I think this is very interesting and what we see that you can do some pretty cool application with that right you can smoothly interpolate to generate like also code and to generate like a realistic looking blog posts now when you talk about like really reasoning meaning or something like that um I don't think there is really any deep breaks going GPS but that's my personal opinion and yeah no it's really good that at least for me it does resonate that you separate reasoning right from having a big database right because sometimes you have feelings that our community or parts of our community going like hey if database is bigger right you can just solve all of the problems right and sometimes it's not really you know having a bigger model right doesn't really mean that you're suddenly like having like a AGI right and it's good to remember basically yeah yeah so another question would be um just like from the audiences also there are like so many different sides of the model right even Transformer ways right this gpd3 erasers like Transformers there is like Berta Roberta all kind of things right there is also more distilled versions of them right and uh when one machine learning engineer kind of like starts to work on the task right um what is a good like rule of thumb how to make this decision process right and obviously there is no like one clear answer right but do you have like any I don't know mental model or like a framework especially for beginners who may be working in the company that doesn't have like you know a big machine learning group right but they're like one of a person to actually make the calls right how can we help and support such a person yeah that's a good question I think that's that's definitely something a lot of people's face and I mean the very practical thing about me about our team at making face I think is that we should we should help people do that because I I understand we have like we're providing a lot of models we're providing a lot of checkpoints but it's really hard to actually see the one you should select the one you should use so so the first thing is that we we will try to build some better tools on this and we'll try to help you but now for the quick answer I think it's good to to keep your good reflex like your good um uh routine is that you should start with something simple like you should start with a smaller model like starting with distilled bird for instance instead of bird like starting with kind of something like small and see how far you can go with that with this compute efficient model like a distributed distort beta you test a little bit with them and if it's not enough then you scale up then you then you start to use bigger models and you try to see if you need like a G5 or something like that yeah but that's yeah that's the typical thing yeah it's the same you should always start with like logistic projection because before you you move to like something very fancy under here yeah definitely I can only agree with you right because once you start with gpt3 right you can only make it like worse from there right so okay no way to explain it right but maybe just like a random idea because you're also building quite a bunch of tools right um so right now many companies building this like a model so right maybe you can also have something as like a use case right that's kind of like showing like hey what is the space of problems right and how does they map like a space of models right and uh um yeah solution but again just like a random idea well that's a very good idea I think you should come build that with us yeah yeah I will try Okay so next question would be um Transformers right we have seen the Transformers did reinvent or revolutionize the NLP industry at all right but how do you feel um like about future of Transformers um there was like some papers also about using uh Transformers for like Vision right do you see that uh soon Transformer is going to be like uh all around the machine learning or deep learning in particular or is that kind of financial Niche right I'm gonna plot off from there basically yeah yeah that's a good question that's a that's a timely question I think this Vision Transformer was was a very uh impressive paper because it actually showed that what we've seen for for NLP was also true for for or Vision which is that Transformers are quite efficient and they are they are really a nice model to scale to large data sets so as as soon as you start talking about transfer learning from a large data set then you you really want the most efficient application that you can still scale and yeah yeah I mean I'm not a computer vision guy so yeah I don't really know how why we only sing that today for instance so I don't know if there was something unlocked but yeah I think they're nice I'm pretty model agnostic I would say I think the the underlying like strategy like transfer learning has something very deep and important regarding the precise model that we select to use I don't I don't care that much if somebody finds something better than Transformer I would be happy to switch to these new models but I think the idea of reusing pre-trained model is a lot deeper deeper with deep yeah with deep consequences yeah I I guess also something happening in Industry right now it's not even using Transformers or architecture right but more like use like weekly or unsupervised like learning right because the big part correct program of a transformer or Bird right or this direction was like hey we don't need to have like a big label data set right we can just reuse what we have right and by using the same ideas of like hey can we learn representation from the image itself right and all of ideas of contrast it was right might be something that will Sparkle like another different directions right um yeah cool one more question from the audience is basically around I will rephrase it but I hope I'm not going to lose as a sensor so what you suggested also like in your talk right if you can use transfer learning right you can try to start there right and try to you know unfreeze some layers right and train it basically like um gradually basically took as a task right and the person is asking like um essentially what is How likely rated by you doing transfer learning you would end up in just like a local Optima right of convergence right so you don't get like really something good right in comparison if you start with a model from stretch right and you start basically training it all together right because I mean usually people hope that the more you like on freezing right the better you're kind of like getting and it's almost like a smooth transition from you know like really uh out of the box transfer learning right into this model that has uh has been trained from stretch do you agree with this one right or is it like anything that people should be aware of and um yeah like any tips and tricks basically of advanced transfer learning and NLP yeah yeah that's there's a lot of things here um so the best the best resource I mean it's probably uh so I I had a link in my in my first slide on a long tutorial where we talked about that and I also made a video a video I could share uh earlier this year on this question but there is a lot of risk to fall in local local Optima so we see that when you fine-tune bird for instance you you can get like none of behavior for some kind of so for some random scene the the model just datum doesn't converge it's the states that can be Minima and for some other Rhythm seed it really works well which is a which was a surprising in the beginning and a lot of people have been investigating what was happening and you have a lot of tools now to try to avoid that you have tools like um something related to what you're saying which is called um mix out which is uh the idea that a bit like Drop Out you probably want to like you only you probably want to regularize your model with dropouts you you regularized right towards zero like you you cancel some weights and with with this these other things you can regularize to my to what the pre-train models so you can have a smooth operation from the pre-train model to the model it is fine tune but that but there's a lot of other other techniques that try to make some Spruce incorporation to not this deck in local Minima that's a very active area of research and the other option is to try to do some assembles which work very well as well but yeah that's still quite open research question I think and we definitely have not really understand everything what was happening here and while it's failing sometime while it's working very well in other cases that looks really close yeah so maybe for whoever did ask this question it's like a good topic also for research right so once you find some answers right please write an article or paper and we're all gonna read it later yeah cool uh maybe like the last question because it was like that this conversation can be going on like for a very long time but we still have a bit of like schedule so how do you feel about NLP and smartphones like is it something that already kind of like sold out and we already have like pretty good distilling or are they like still uh great breakthroughs like ahead of us yeah I think it's a cool cool area we have a few open source Library actually on switch switch core ml with birds that you can run on your iPhone and you have a lot of very recent paper uh regarding the talk uh for next week NLP emnlp activity on that you can quantize model very efficiently so yeah I can see a lot of breaks who come in there with very efficient model keeping a lot of the performances that we have on PC on the computer but on smartphone yeah foreign thank you again like for uh your amazing answers right and also your talk but also I would like to mention again like the work that face does in open source it is uh really helpful and it does make uh you know entry barrier like lower for many people in this industry so thank you again and uh hope to see you somewhere around the internet thanks yeah yeah and if you still have some questions right and you would like to address them to Thomas directly remember that we have the speakers room um for that you just go to website as AJ will explain to you early and after you will find the link to the conversation with Thomas that's going to be starting more or less right now so please go ahead if you wanna continue your conversation on LP yes yeah cool now we are back to other questions if you remember we use mentee and the question was if you're apply leaky relu to Adam what are you uh trying to do right or what are you doing basically um and it looks like that uh yeah this is actually funny but uh it feels like it's the reality of life right 68 of you saying that you're trying to figure out ml Concepts right yeah there's like too many um nice words and in machine learning and sometimes people not really aware hey how liquor will do like working with its comparison with relu right and what's the implication to Adam so it's definitely a good thing to discover cool um the next one is uh AJ all right thank you so much Sarah again thank you Thomas for that amazing talk on NLP Transformers and hugging phase now before moving on to our next speaker I want to also just give a shout out to our gold sponsor Microsoft now ml confu would not be possible without our sponsors like Microsoft and they provide an end-to-end service for machine learning from from development to deployment and it's a service called Azure machine learning and I highly recommend you check it out if you want to stay informed about Dev news tutorials and other events then subscribe to Microsoft dot source which is their monthly newsletter subscription alrighty so without further Ado let us now introduce our next speaker Beryl servichek so there are two social causes that she cares about Healthcare and Earth care or environmental care she's finding ways to leverage artificial intelligence and computer vision to address both of these concerns and her passion um her her actual passion to save the environment is seconded by the fact that her favorite movie is a movie called vegan which was a 2019 production she is self-taught in augmented reality and uses this with AI to you know for apps that help farmers in um basically a vegetation and soil help and this is a part of all her initi this is a part of her initiative in her company called create 4D fun fact when she was young she wanted to be a computer scientist and here we are full circle coming here to listen to a talk from her on Cutting Edge computer vision technology and well it's amazing to see that we've come full circle and with that I'll pass the mic over to you girl and while we're in while we're inviting her over here once you all go to Discord give her in the in the community Channel give her a little Applause make her feel welcome make her feel special and hope to talk to you guys soon just after a talk so with that Brill I'll hand the mic over to you foreign [Music] [Music] thank you very much for registering to this machine learning conference whether you are located in Europe or anywhere else on the earth thanks a lot for being here I'm very and today we are going to see how we can make software machine learning software see things like human does we will try to introduce some image processing and computer vision topics in this very limited short time because of the time limitation I cannot show you so much programming examples but within my slides if you notice I put some algorithms within boxes so I hope you can turn back and look around later that could be useful and we can discuss more examples maybe after the presentation at the Q a session so this is my agenda for today after a small introduction I will first introduce you what is image processing some image pros and algorithms and then we will look at computer vision and we will see some classical computer vision algorithms finally I want to address how we can combine computer vision and machine learning and how it is different than regular AI so we know Problems by itself help so how is it different is computer vision irrelevant now because AI is doing everything by itself so we will discuss these topics and lastly I will give you a piece of code again you can Implement yourself to jumpstart into doing some computer vision algorithms hopefully a little bit introduction about myself I'm currently an assistant professor at the Swedish university called young shopping University it's an AI lab just focusing on developing Ai and explainable AI algorithms besides I've got my own company which has done a lot of computer vision applications in the past in the Netherlands I'm located in the Netherlands also and I've got experience over 15 years in the area of computer vision machine learning and also artificial intelligence is the area where I'm putting my a lot of efforts these days if I want to introduce myself I always need to add that I'm an environmentalist so what is an image let's start with what is an image before processing these images image if it's a digital image we are talking about digital images not alt analog style images if it's a digital image we are talking about a metrics The Source can be anything The Source can be your smartphone a regular camera it can be a satellite sensor it can be a heat sensor it can be dermatologic or microscopic image telescope image whatever if we are talking about digital image we are talking about the metrics all right if you've got a matrix you know how to do metrics of operations and welcome to image processing you know how to do image processing now if we see a picture with a t within it let's say and we can represent this t as an image by putting one when it's bright and put in zero when it's dark now we've got T represented as a digital image when the numbers are only 0 and 1 of course this Matrix is binary matrix it's a binary image it's called but normally when we've got our smartphones make a picture we've got RGB red green blue color images so that means that we don't have one image as a one Matrix for each image we've got three matrices but in order to make things operation simple now we will try to look at one Matrix at once assume that we talk about brightness or grayscale image which is combination of this red green blue bands all together let's assume that a weighted sum is obtained then you've got the grayscale image so before going further I said I will talk about image person I will talk about computer vision are they the same thing or not I say that's the same is it true or false give an answer from your head now it's false well even though people use these terms interchangeably some people say image processing for computer vision some people can say computer vision for an image person application but in fact there are different things when we say image processing we assume that we've got a software which we say image processing software where the input is an image and output is a processed image then we've got an image process software however we are talking about comp when we are talking about computer vision then our input again is an image but the output is a value it can be a number of people in the scene or a vector a position a GPS location that we found in a satellite image a boundary a shape it can be any class for instance so if we have a value at the end then we we said we've done computer vision and most of the time these are done all together because most of the time the image for computer vision is not suitable to process and to be processed immediately so it's a good idea to do first image processing to process the image maybe removing some noise and make it more suitable to be processed by computer vision and afterwards computer vision extracts some values shapes vectors that's later on you can pick them up and put them into a machine learning algorithm learn something classify something recognize something so we will see more examples even image processing we said our input is an image and our output is a processed image but could be this just some examples of image person algorithms it could be denoised results maybe input was a noisy image and the output is denoised image it's more smooth it can be a color adjustment application done by image posts and algorithm you can do image augmentation very interesting for uh generating test datasets for AI for instance or you can remove the background like Zoom meetings are doing now change in the background this is also an image process in algorithm you can do Edge detection Corner detection like Implement you can do such methods by using image person algorithm but assume that you've got an image of the edges at the end and also you can make old photos new by guessing what colors they they should get at the end and there is an example of uh pretty much like looking at old videos colored and this is a result of image processing algorithm where image person is applied to each frame of the video so I will show you some image person algorithms but uh maybe the most important thing to express is histogram bam we are talking about how to do image personal algorithms histogram is distribution of brightness values within the image and it's the most interesting thing interesting thing to look at in an image when you get it for instance here there's a scene that you see if we put the histogram we mean we are looking at the image values if we are talking about 8-bit image the values are from 0 to 2000 255 because we should have 256 values at the end for an 8-bit image each of each of these values are telling about brightness level of the image if it's very bright if it's very white the value is 255 if it's totally dark the value is zero and the other grade values are in between if you look at the distribution how much brightness bright pixels there are in this image if you look at the Statics you see that some accumulation is happening is close to zero and some accumulation is happening close to 255 higher values we can expect that these darker values this month a mountain is causing these accumulation of Darker videos here it gives us an idea about the values within the image and are they distributed nice or not and we can also do more processes to the image by just looking at this histogram uh the first important thing to know about how to do things with histogram is to do is to know how to do histogram Equalization most of the time when you've got input images for instance from a drone it's looking at a limited area where it's also projecting its own shadow we will have very dark values first of all we have to make sure that we have stretched these values good enough that we can see some texture some properties within otherwise we cannot do further computer vision applications to extract some features to recognize things the first thing interesting thing to look at is histogram Equalization and it's trying to stretch the histogram to make sure that the distribution is done between 0 and 255 and we can see some texture with them this histogram Equalization is easily done by looking at the cumulative values of the histogram that means if we've got the brightness values distribution we can count how many brightness values until this value in order to project this cumulative histogram then we can use this cumulative distribution in order to find new values of each brightness by just you using the cumulative distribution function as a projection function to find the new distribution new brightness values to project into stretched values so I hope you can find more at the references but we use these histograms for um making the image more visible the features within itself are more visible but we can also use it for segmentation for instance if we look at the histogram and put a threshold value like remember the mountain and the Sea background we can threshold the histogram in order to remove the background or to get the foreground only we can use the histogram for segmenting objects as well we can do this segmentation by looking at the histogram of the overall image or we can look at small windows and choose different thresholds if the uh brightness is fluctuating a lot within the image and the question is here is how to choose the uh threshold value automatically in order to find the to do the segmentation don't worry opencv comes with help because there's a method called otsus thresholding and opencv offers you just in one line what it does is to I draw it by hand here uh if you get this brightness values assume that they are distributed like in this uh video I have to go a little bit quicker if I can if you look at the histogram what we need to do is to fit two gaussians to this histogram I can't fit two gaussians to two peaks with two peak values that I found in this histogram like this and the intersection place gives me the threshold value to segment this image automatically I have to go a little bit faster we can discuss in the Q a we can look at more algorithms that what we can do in image in the area of image processing we can use the brightness values in order to extract lines uh how we can do that we can use hysteresis threshold in called method for instance what hysteresis threshold in does is instead of selecting one threshold like otsu did hearing we select two thresholds one is high threshold the other one is low threshold then we are looking at Connected pixels and the connected pixels let's say all of them are lower than maximum threshold that may say we remove these connected pixels they don't do not represent anything significant however if some of the parts of the connected pixels have brightness value higher than the high threshold can be assumed all connected segment even though some of them some of the places were darker we assume all the connected segment pixels as one Edge together so that gives us very robust algorithms because uh one threshold may not detect all the connected lines but two thresholds with hysteresis threshold in like methods help us to develop more robust approaches so I I've given you uh some algorithms to process images further but I have to go quicker and please look at the slides later in order to do uh image processing algorithms we can do metrics operations because image is a metrics we set we can make a lot of applications by making a convolution window and skin in the image with a convolution window like for instance if the convolution window has only values one that means we put the average of the value within the window to each pixel of ambier traveling and at the end we've got average average pixels which gives us a smooth image if we replace this convolutional window instead of once we put a graphs in shape that means the middle pick cell is making the most influence when we are having the average the weighted average now and the more far away pixel is making the less contribution when we make the average so that gives helps us to achieve two results which are smooth but the edges are not smooth as much edges are preserved at the end I advise you to look at my slides later I'm going to share but it's also possible to make this image processing algorithms in the frequency domain by converting the image into frequency spectrum and then process it there by applying low pass filter high pass filter and then convert it back to image domain to achieve results anything that we've done here in grayscale is also possible to do in red green blue bands of the image directly and maybe combine the results later but the most interesting thing for the people in AI is to look at augmentation methods that we can do with the image processing methods to generate a lot of data that we can use after processing the image then the computer vision comes computer vision tries to extract important features we call it local features to look at the image to recognize things in order to do so we try to develop some methods which can identify Corners sharp edges or specific texture properties like Harris did with with its algorithm compared each small window with its neighbor to see if the neighbor made a significant change or not here is the map of the Harris at the top right side he obtained if there is a significant change by comparing the neighbor it's highlighted that when it's thresholded he got Corners at the end this is Harris algorithm then later on other people proposed heuristic approach or a band pass filtering approach which are more robust to noise effects because Harris was not robust to noise effects if there's a noise picks up it's going to be highlighted at the end our features enough for recognition no I will answer for you we need to describe each feature with a feature Vector which we call descriptor Vector in order to be able to do machine learning algorithms to learn or classify things and please look at the descriptive Vector extraction and I give you within the source code box which is offered you by opencv directly and there are different methods to use descriptive vectors in order to recognize or match things like directly matching with a template that we already hold in the memory we can try to find the most similar descriptors within the image or we can try to look at the special distribution of these descriptor vectors with a graph Theory and I gave you more examples but before we end I want to discuss how it is different than the AI algorithms that we know of does it mean that if we develop AI with more sophisticated algorithms we achieve more not always sometimes computer vision and machine learning comes with more advantages because we hand tailor everything and AI comes with other advantages so I listed some comparison lists here and uh generally computer vision allows us to hand tailor what we are looking at however AI look at finds the good features by itself it makes things maybe more robust or generalized but it might think make things also more or vulnerable to adversarial attacks when we we cannot hand tail alert what to look at significantly AI always needs a large number of data sets to learn from however as you see if we are using computer vision we can use only one image template to find the similar thing uh in which competition Vision methods it's um it might be challenging to find Optimal model because we show only one template however AI learns from large data sets it might maybe find find generalization better and finally uh for competitive Vision methods you don't always need GPU but with AI methods to develop and to find the optimal solution you definitely need a GPU so please go ahead write the codes that will open up your camera and get grab an image that you can start using opencv algorithms and this is a very short time unfortunately I have to close it up now and I'm very happy to go through your questions and open up the topic more thank you very much for inviting me you can find more about how to access me at my website my first name surname.com and you can find me on Twitter please subscribe to my YouTube channel where I teach a lot and also my newsletter to get all the free content that I'm delivering every month thanks a lot [Music] thank you [Music] with Braille we have a polling question that comes from that comes from like the audience so if now here's the question let me just pull that up right away alrighty so if let's say that you have a company right and you need a software solution for some object recognition that you want to do so what source would you trust you have some options below and if you want to answer this question go to mentee.com and use the code 9471-503 so I repeat go to mentee.com if you want to answer this question you think you have a great idea just go to menti.com type in the code 9471-503 to submit your response all right and with that I'm going to welcome on Brill now for the question and answers hello bro Hi how are you doing today I'm fine thank you thank you very nice to be here very nice conference awesome well I'm we're all glad you're here and we want to start this conference well we've already started it but we want to continue and get the show on the road with some amazing questions from our audience right because they have so many questions for you okay so here's a good question Okay so um what are some examples of processing that can be done in the domain of like frequency and audio can you use opencv for that for example oh for audio processing opencv meant for open computer vision it stands for computer vision and doing image processing uh so it's not meant for uh audio processing directly but I'm assuming that you might find out maybe one or two functions just uh it ends up uh you can process 1D one-dimensional audio signal too as we've seen in slides all the images that we are looking are two dimensional matrices so they are little bit different than audio which is one dimension and comes in time so it's opencv it's not meant for Sound Processing I'm sure you can benefit uh some tensorflow libraries uh but other libraries maybe it's good to look up further it's not really my domain yeah that's actually true because I know that the operation of convolution although like you do tend to apply to images and it's like the fundamental for operation for computer vision we do have like one-dimensional conference like um we do have one dimensional convolutions for like signal processing as well but yeah that's true I also haven't heard too much about it from opencv but it's definitely like relevant in terms of fundamental operations yeah and okay so here's another question I know that you you've introduced like some math uh on your slides while also explaining opencv and I know for for many people mathematics could be very daunting and like because of this there are people who look at the math and they're like okay I don't want to get into this would you recommend people just starting directly with opencv and then do the math later or is there like some other approach to this computer vision that you would take yeah that's very good question uh first of all just in my mind I want to mention I shared my slides I think it will be almost on a website or Twitter that's the slides of this talk you can uh use it for finding the references I always put the link to read further so maybe you can click on the link and check the call check the source to learn further what you heard right now in this uh very short talk unfortunately I cannot explain find everything in detail but when you're programming you start doing things it's really good to check the maths otherwise you will end up maybe with a good result with your programming but is it really a robust algorithm which you can Implement and works in in real life or whenever there's an error there's a false recognition where does it come from it wouldn't be easy to debug if you you are not aware if what kind of algorithms behind or you need to know the parameters of this function to tune them but how do you choose the parameters if you don't know the match behind it I'm not telling that you go and take a master PhD degree like something I I not everybody needs to do that but I know very professional uh developers who really do robust good stuff just self-educated and nowadays everything is online I would recommend maybe you're writing a quote it works good but just go and check the match behind it on just on online reading some blogs forums and asking questions there to other people it would be beneficial yeah I completely agree that a good healthy balance between like the actual practical aspect with the theoretical aspect can do wonders in helping you understand exactly like what you're getting into yep thanks for that answer and we have some more questions here too and uh let me just pull some of them up okay so do you have any ideas or you know techniques of computer vision that you would use in healthcare or farming or any of the other social causes that you care about yes a lot and Healthcare Fields I used it for for cancer skin cancer and identifying whether the cancer area is malignant or benign so it and what stage of cancer could it be and for this purpose I've done feature extraction just inspired from uh doctors they explained me how they look at skin and how they uh classify the stage of the cancer by just looking with with their eye of course there must be biopsy but doctors have seen so many examples and they can just look and tell and I was really curious how do you look and tell explain it to me when doctors explained me I found a way to describe what they explained mathematically then it turned into a feature extraction algorithm which could be used for classification of cancer stage at the end it always inspires me to think of how human look at this and how human recognize and can I find a mathematical method to do something similar yep I think that yeah that type of meta understanding of like being in the mind of like how a human would do this and trying to instruct a machine to do something very similar is very important and sometimes it's also like very hard to get in terms of like health care because I know like if you if you make like for example in cancer detection like mistakes that you make can be more fatal like it's you're playing with human life so there kind of is that you know up and down of like okay is it okay to use artificial intelligence we don't really want to be the decision maker but want to be like a decision supporter to to doctor always that's a in those kind of critical areas we don't want to kick out the uh human being but just want to be a supporter and second thing uh if you put 10 doctors they will not end up with the same answer if you ask what cancer stage it is they've got also some fluctuations in their answer so maybe a machine Learning System could be a way to normalize their answers and um just to get a second opinion from the machine got it yep thanks for that answer and I think we have one more final question that we could get to um right now like maybe in your current work probably in your startup and uh create 4D how are we can how are you currently using computer vision and even opencv for that matter in this work in your current workspace yes it's very interesting to look at what's going on on the uh Earth especially in agriculture area since uh it's my passion to look at Food security and how plants are doing do we have enough food soon in the supermarket or not and uh the methods that I described in this talk like simple convolution help a lot to extract some features which might indicate a sudden change in a local area maybe something wrong in this area and I can go there visit uh and rescue the plants before they die or spread the disease to other plants I want awesome well thank you so much Farrell I think we're out of time right now for questions but I appreciate you coming here and just answering our questions and also giving us a good insight into computer vision yeah computer vision and everything I mean like it's amazing so everyone by the way if you wanted to go and talk to Brazil right now she will be available in her speaker room to answer your questions in in the event that you know you couldn't get your questions out at least in this q a session so thanks a lot for joining us for real it was great having you here and we will see you later in your speaker room thank you bye-bye for now bye-bye and now so before uh baril actually spoke about her introduction to opencv and computer vision we gave you all a question so that question was if you had a company and you needed a software engineering solution for like object recognition which source would you trust and pulling up some of those answers now it looks like it's well it's almost unanimous that we would trust machines over people because people have their flaws and clearly machines that don't well that's that's always encouraging to hear from an audience and well right now thank you all so much for answering that question we will be now taking about like a little 10 minute break um we hope you enjoyed this little this this little end screen this Splash animation that will come for the next few minutes and we will join you shortly for some more amazing talks again give our give our give our claps to our speakers give a clap to Brill give up to Thomas Wolfe for their amazing presentations and we hope you support them in the future too so we'll just take a few minutes of a break and we'll see you soon take care [Music] cheer meow s [Music] [Music] foreign [Music] [Applause] [Music] laughs [Music] when I'm down [Music] foreign [Music] [Music] [Applause] [Music] [Music] [Music] thank you [Music] [Music] foreign [Music] [Music] thank you [Music] thank you foreign [Music] [Music] foreign [Music] [Music] thank you [Music] thank you [Music] yeah I hope that you had a bit of break right maybe you do some stretching right because it definitely helps to stay alive like to this entire conference yeah so back to machine learning and back to even more amazing speakers so as you know as we mentioned before as well like in order to train machine learning right you need to have data some of you are happy because you already have a data set with all those things some of you have like a new problem right that you need to have data set collected for you and you can do it manually right you can pay like lots of money to lots of people or you could use some of the services one of the services is actually called toloka it's also uh happened to be our goal sponsor so this is like a great largest or it's actually the largest crowdsourcing platform globally designed by Engineers for engineers and it's allowing you to get a very good data labeling at scale I think there's also going to be one of the workshops happening on 11 so please join them and get to know how to label your data um if you're watching this conference alone right and you don't have anybody to discuss all what's happening here please remember what betin said in the beginning because it's like awesome tool called rainbow that allowing you to join some 3D virtual tables and talk to Thai Universal people no not really people actually real it's just that avatars are going to be tidy yes and with that we're gonna go to the next speaker and the next speaker is Robert Robert is actually coming from um software engineering background but you'll be like hey but it's a Machinery conference like what he's going to be talking about um and his talk is going to be is called The Evolution Revolution that sounds a bit of password but he actually gonna go all the way in and out how to build machine learning how to bring it to production and what software engineering practices you can actually learn from machine learning and apply basically uh from software engineering so please welcome and give some virtual clubs on Discord and Community Channel or here can also Club at home please welcome Robert [Music] foreign [Music] I probably am the least qualified to be here um but that aside uh I want to I want to set up a scenario and we'll walk through it and um and I want to propose something um so so here's a scenario there's Adam right for machine learning for um uh what do they call the trainer there's various words for it um you come up with this new idea trainer called adam2 and it's revolutionary it's going to change everything with machine learning and um and so you're like psyched about it you want to get it um implemented somewhere and it's amazing because it represents innovation and that's really what machine learning is is kind of all about like we're we're innovating our Breakneck speed and so we don't want things to hinder us um in the serene so here's our our elegant and simple solution that represents Adam and here's Adam too um so yeah it's it's amazing um you're going to see it everywhere yeah but there's a problem um even with just adding that one little number there and that's this uh it represents an engineering problem the implementations that we have to manage so I personally write a note um I'm in charge of a um sorry I'm rather the tech lead on a machine learning team and um for us to use this in node we would have to come up with translations in all these languages and potentially more if we wanted it to execute in these environments so we've got JavaScript webgl up and coming web GPU wasn't native bindings perhaps TPU and even new languages so yeah that's what you need to know to add that little two in there and also you don't want to forget about unit tests for each language because you want to make sure that your your stuff stays together and they don't tell you this when you're developing on the GPU or when you start getting to really close to the hardware ads or Precision differences that have to be managed and that each implementation has various needs and so they're going to need different abilities different capabilities and each one is going to eventually compound technical debt that's just kind of how development goes and that problem compounds because of those bugs you have to fix them um the underlying languages are constantly being refined and changed right and they have to be managing to add that to and that that atom um function you you have to go into those languages and actually alter them and then what if somebody creates atom 3 what does that do to atom two um did they um extend atom two with atom three if they did um and they use some sort of class structure or maybe they can't because they use functional depending on the environment and each one's different and so that leads to diverged code because each code is different and each code being different uh has to be tested differently and you have diverged Management on each one because each language requires different strategies different system to manage what you've implemented and then two there's how you arrive at the math that diverges and so each implementation think about this carefully each implementation that you add nearly squares the complexity of the original implementation and all you want to do is add a two and I haven't even mentioned the worst one which is abstractions abstractions take away our understanding of of what we're trying to actually get at the math right it makes it really hard to understand and that's why we're so afraid of it like we talk about uh you don't want to have to worry about this underlying maths because if you do you'll go nuts there's just there's so much to think about because there's so many different levels of abstractions and it all slows innovation so our elegant and simple atom 2 which you can see on YouTube it's not really on YouTube I made all that up uh it doesn't stay elegant and simple even though in theory in the math it is it doesn't stay that way and so we have to kind of change our understanding of things if we want to scale right and and in our elegant and our simple are our engineering right what was the the main objective we want to get at innovation we want it to be Innovative we want to be more Innovative than we currently are so Frank Zappa he mentioned this without deviation from the norm progress is not possible and he's kind of a interesting fellow but what he said was absolutely uh on par without thinking about things differently we're going to just keep going down this this road of Divergence and it becomes more and more difficult as time goes on so I'm going to change Pace here a little bit uh there's this movie I watched recently with my kids and I loved it it's it's such a great movie I won't spoil anything for you in it if you haven't seen it but Big Hero 6 . now in it there's this fantastic illustration uh hero the main character he builds a robot and that robot is it's really cool it's extremely capable of doing all these kinds of really gnarly things and um and he sets his mind to innovate to do something even more amazing than he's ever done before and all he does is simplify the design and make it smaller and more portable and when he presents it to everyone nobody's head even turns like it's not even that big of a deal and then he he shows what they're capable of by simplifying by innovating by using this little bitty bot he basically turns into a superhero and everybody they turn and they look at them and they're like what you know he's like floating from the ceiling and he's skating through the air because these little Bots are aiding him because they're simple because they're innovative and uh and all he did was make it simpler so there's a really amazing story to be expounded upon there so if if we were to kind of take some of that away from Big Hero 6 of all things what if we did something more radical uh implements at once and I mean truly once now think about that for a second implement it once there's currently being planned right by very large machine learning players huge entities out there that run the biggest search engines on the planet they run my phone right um they say we're going to do it we're going to implement it once so let's walk down that path what they're they're talking about they Implement a new language right at the start brand new language yay right amazing so now you don't have you know eight languages or six or five languages or three you've got two languages but you still have bindings and you have a new language now think about that we we have a new language guys we do not need another language and we do not need to implement it twice these are not we don't we we don't need to do that we we have we've had for 30 40 50 years low level math being handled on Hardware and we have a means of describing that in higher level languages such as for us JavaScript or python or whatever the case may be we have a means of of describing how to perform the math the problem is we don't have a means of talking directly to those Hardware devices or at least we haven't and this implemented they quote once but really it's twice it still leads to abstractions so that's that's still a problem so we have to reboot our way of thinking if we really want to get down to the metal and think of things and Implement them once now I got started in machine learning I did a lot of research prior but what really kind of sparked it for me was when I was looking at the internals of brain.js brainjs is a really nice JavaScript library and it's written in JavaScript and it is um at the time it was written just for the the CPU now it runs on both um but they did everything in one language and their back propagation used two forms of training rather than just one so two optimizers two trainers whatever you want to call it we call them uh Praxis and and brain and to me that was super Innovative I'd never seen anything anybody do that before back propagating using two different forms of training and I actually went ahead and named it it's the Arthur deviation um because it's it it deserved more credit than it was given in fact um this this fellow here um Jesus I think is his name I probably am pronouncing that incorrectly he went on and he took brain and he turned it into a natural language processor and his quote from this this URL we broke all the records and he was comparing brain.js this little Meek little library to Google diagflow Microsoft Luis and IBM Watson he outperformed those with this little bitty and I think in part is because of the the quick Innovative ability to edit in one language and that for me that turned brain.js into a superhero that that was amazing so if we were to truly implement it once we run some difficulties here uh the type inference transpiling error handling type handling language differences because you have multiple languages to kind of translate into but we really don't have time to expound on that because I'm building my argument and I'm over half my time has been an ate up so we're going to skip that but we're gonna we're gonna just tell you right now that gpu.js translates to GPU code for JavaScript and right now it does it for both node and in the browser and brainjs is written in JavaScript and it's currently being converted to typescript it implements gpu.js so I'm not just here to plug brain.js and gpu.s I'm really here talking about the principle I just want to make sure I convey that the idea is to write it once bugs are handled once there's no Divergence you embrace the math because the math when we write out the math for these operations the math is beautiful but as soon as we engineer it we take all that beauty away but I'm saying let's embrace it and get as close to that as possible and let that be a commanding Force by doing that you keep your sanity and you innovate faster you become like a superhero do you think about some superheroes that really set them apart the guys that were normal like Batman it was the car or the suit right it made him a superhero it was something I mean you could not maybe it wasn't very simple but um the car uh for Big Hero 6 it was that little bot so here's some practical examples to convey I guess kind of what I mean so our first example is is um was written by uh Gantt Laborde and he wrote colorblind uh what it's like to be colorblind in tensorflow and so this was the tensorflow a version I call this the canned approach Okay so we've got like these these functions that we get and there's nothing wrong with those functions in fact in parts of brain.js we use those functions just the same but um but just just bear with me you have to with tensorflow.js you'd have to be if to alter any one of those methods like um split mean stack uh or concat you'd have to be able to get at the code the underlying code and for each of the environments so that's really what I'm talking about here so that's the canned approach now this is the uh I I've got brain.js up here that's actually the gpu.js version um so we set up our environment you see settings we have the our output width and height and that we're graphical but really everything that is important is in the create kernel function so we have we're grabbing a pixel and we're just basically dialing in what the red green average is and then setting our color so there's really like three lines of operations there and that is for me the direct approach because you're dealing directly with the numbers and for me that's more innovative so that that's one version this is next one is a green screen uh we took that baboon I didn't even show off that picture of the baboon right there the top right um that's what it looks like uh the output right so it's kind of like with this color missing that's because that's what it looks like when you're colorblind uh the green screen I took that baboon I took a green screen from Wikipedia and I I merged them together just thinking hey what would that look like um and so this is the math behind that so you basically just take the color difference and you're just saying uh hey should I replace because you know they're within a Max difference or Min difference um I've I've gotten this one before mandelbrot set a lot of people look at that and they see the math and it's so simple and they're like what I can't understand it uh it's too complex well this is the Mana brought set beautiful beautiful and somebody even wrote Amanda bulb for gpu.js this is the code actually is Amanda Block Set just rerun it recurrently or um uh you know in a loop so it just kind of compounds on itself but that's what it looks like in JavaScript right at the math and and here's a more practical example in brain.js so in machine learning there are layers and there are four propagation and back propagation well just taking this one layer which is the add layer and we look at the predict or forward propagate function we can see there's really one line of code aside from [Music] you know setting up the actual function name and the the arguments we have a return of the input weights one we get the X and Y value and then input weights to X and Y value and add them together so super super simple now is there an end-to-end solution that's a kind of much bigger right because we're talking gpus all all across the board well I'm working on Welton I'll have it finished um maybe today or tomorrow uh and links to it that you can actually pick how you want a beer to taste and smell and how alcoholic you'd like it and it will tell you what type of beer that you're trying to brew what color it will be and even how to brew it and it's actually not that hard it's actually a standard classification problem it's just um we haven't really thought about the normalization denormalization of the values and um but I'll have links to that on the presentation and um as a more General use we have a practical example uh end to end for brain.js this is just xor or zor or however you want to say it um the the most important thing here is that the net is is built on this new architecture called feed forward which relies almost entirely on the graphics processor you can switch it I say almost entirely you can switch it between the CPU and the GPU um but here you sort of work with layers like you would with react components if you guys are familiar with you know setting up websites and that type of thing so if you can think of of brain sort of being like the react for for machine learning we have a input layer defined as an input hidden layers is an array and the input from the argument previous which is the actual layer instantiated is going to be fed in so these feed in recurrently from one to the next you can have as many hidden layers as you like each return value from one is going to be the input into the next and you have your output layer and then it goes to the output and then you basically net.train net.ron and now comes some values so uh that's not only just the end to end that's the entire training that's the entire running that's everything um all the layers and uh maximum terse uh readability right there for you so that's that's kind of like the proof behind this idea and um ultimately we just want to make sure that you know we're we're trying to make something that's super straightforward not just in engineering but all the way down to the math so the evolution Revolution that's the name of the the talk that I've got today um and we're not talking about like Evolution with animals or anything like that we're talking about evolution of ideas one idea Builds on another idea Builds on another for the longest time all we have done is abstract into these separate layers um it's been very hard to do it the other way which is to read the high level language and compose the lower level language well that's what I'm proposing and I've actually found out after you understand the in environment and it's really not that complex in fact everything is strongly typed when you're right at that layer right at the execution layer because you can detect what's coming in and you know what's going out and you're detecting where it's going and so if I can simplify it the high level language becomes the chalkboard the atom 2 implementation that we originally talked about becomes super simple and super elegant because the the IDE that you implement it on is exactly where you're testing it and you no longer have to worry about those underlying languages because they are composed for you so you write your Innovation once you embrace the math and that that leading on that previous Innovation that even leads to faster innovation so that makes you like a superhero that's my talk I want to thank these guys call Hammer Supply in addition our ml conf EU and call Hammer supply for the beer recipes gamble board I mentioned him briefly he's been a really nice colleague in driving machine learning and understanding of it and um as well the contributors to gpjs brain.js and then the Sligo from these slides thanks and um hope you guys enjoy the conference [Music] again Robert for this amazing Evolution Revolution talk was really interesting perspective right because usually if you're thinking about machine learning only from you know mathematical point of view and this brought a bit of also production and Engineering point of view so this is also helpful right diversity does help make our products better but before we jump into questions with Robert we have like another question for you from mentee so we can coach your audience right and your answers yeah so in order to answer this question please open your browser again you know the drill rate of inventment.com enter the code 7196411 you can still see this code on the screen and I will repeat it one more time for you seven one nine six four one one and the question for this slot for us is what is your favorite brain power snack or drink because you know it's sitting in this conference uh stream for quite some hours already so how do you renergize yourself please answer it on 20 and we're gonna gather answers just like in some 10 minutes basically with that I'm happy to uh welcome our speaker hey how's it going hey it's going pretty good I really did enjoy your talk um how's it going for you like I see that you still got a bit of Sun On Your Side oh yeah it's uh it's just a pleasure to be a part of of the whole operation you guys have really put together a tight uh set of sessions I guess yeah yeah it's funny that we have speakers like all around the club right and you never know what time is it right now right and uh it's good to see that you still have sun because for me it's already like night but let's jump to the questions um I've seen some questions already like on our Discord channel so people can ask it in ml q a and when it's like really open-ended but I guess it's like still helpful to answer since you're like also part of a team of maintainers of uh brain.js so the person is basically asking like hey um it sounds amazing right but do you want to copycat everything for neural networks training in JavaScript when we have powerful deep learning Frameworks like tensorflow by torch in Python what is innovation behind it and uh by the way how big is the audience like who is using where is the edge for this one so the question is like really open-ended right but if you can maybe slice a bit more on your developer audience right and why let's say there's a new grade right new graduate basically from University and he or she has a choice between you know tensorflow or brain.js and like what they should decide right and how this decision process going to look like well so tensorflow is a very open-ended tool right it's very it's very um each time that you use it you you end up starting you know connecting up everything uh layer by layer input and output and then explaining that and then and running it um whereas brain.js is not really designed that way it's it's more designed um to allow for those repeat steps to be done for you and so you can kind of get quickly to what you want to do uh without thinking about it a ton as far as re-implementing uh I mean languages change over time and there's new languages that come out web GPU for example is something that's that's brand new so it'll have to be re-implemented in there no matter where we're talking about at least for those that want to take advantage of the graphics processor and JavaScript so um there's work being done in tensorflow to do that we haven't yet started with brain.js but really all we have to do is change the underlying engine gpu.js and then all of a sudden brainjs will just speak that so it's it's more of the idea between a long-term and a short-term investment short-term investment is duplicate the code and and um you're you're building up technical debt whereas long-term investment is right at once and then create an engine that transpiles it to whatever environment that we need to run it in yeah no and there's a good I completely agree with you right because you can also ask like hey we had like C right so why do we need any programming languages right from one place to another right yeah it is helpful right as long as we understand what is um how is it different right because you know the trust is like memory safe right you know is that calling is like good for tiny things right and I guess you still have um your definition for brain.js maybe I mean it's like very impromptu question but how would you explain brain.js like in one sentence right what is this you know like rust is like for memory safe right and Bridge brain.js is like for okay well uh I would I would say it's a practical machine learning library that is data Centric it works it's really tailored to working with data first uh uh and I don't know there's a way to I would clean that up probably yeah I wouldn't broadcast it you know to thousands of people but yeah no I mean it's also a bit of like me asking it like on spot right so don't worry about having like separately answer and I guess like you know what is that practical it's something that sometimes you're also forgetting right because by you just using like out of the box perfect like working solution right you don't learn much right you cannot be creative right you cannot go extra mile and uh apply it in a different fashion right so it's definitely like a good USB right you just need to find what is a strong side around this one yeah I mean so an example um we started working with convolutions and um there's there's parts of the network part parts of brain.js the project not the network um where there's different networks that you can use different um and in researching for example convolutions and actually looking at the shape of them I found that they're really not well understood um there there are you can look at and there's an algorithm that's implemented but actually seeing what it does it's so much different than um rather how it does it is so much different than than the way that we have traditionally thought and that's why you'll you'll see these really like inefficient implementations of it's where they step through each cell and they basically just ask uh if something it relates to it whereas the really quick way is to bend it the other way around where you say from the output look exactly up your inputs and then like calculate but I found that there's not really any good examples for the back propagation of that and it really bothered me but anyway we ended up writing it after you see it though like in JavaScript and a language you don't understand it's like a light bulb moment oh it's so so simple you know you're just like uh looking up the locations and it's it becomes a whole lot more digestible um no it's a good point like I think it's like a good way to also I mean I don't want to say sell right because you don't have to sell anybody it's like open source product but a good way to explain what is it about right so this is a good point maybe a different kind of a question right I'm not sure if you did follow like for other bigger frame machine learning Frameworks right but right now I have a feeling also that who is using framework in uh research right it's like a good indicator who's going to be like what people are going to follow later right because like example Cutting Edge research right the more pre-trained models like the more exciting model zoo is like available for developers or Engineers right the more easy they transition from I don't know the tensorflow to pytharch to branches whatever else so the question I guess are going to be like in two parts like one part do you feel or do you see like a community of researchers like trying to use brain GS right because it's like so easy for them to you know convey what they want to try to implement and second part do you try this maybe table brain chairs try to make it easier for researchers right and do you feel that they're like also part of your audience right or you're mostly focusing on developers and and consumers basically how do you feel about this one yeah so this I'll try to answer and and try to remember the best I can but there are there teams of researchers that use it uh yes there are um one project that I saw was uh the ability to to look through a human body um um from different points of view so you could actually um uh pull out like um meaning from those those uh those slides but it having it basically on the graphics processor you can basically like um process that data very easily you can see it and then you can and actually you end up using gpu.js for that too um but I think uh I think it's more about like if you can see the results of something if it can prove that it is a valid tool by numbers alone and by the logic of it being simple then then it's a shoe-in for anybody to pick up and one of the examples that I highlighted in my talk was that of natural language processing with brain.js where it actually beat Watson and um like the biggest players machine learning players of the time um because there was a bit of innovation that happened within the net where you could use different types of activation which is now something that you can do but um it was I hadn't seen it at the time um and then what was the latter part of your question I didn't recall yeah do you try to explicitly Target researchers right I don't know like maybe some people are trying to no for them right or trying to just engage with problems they're having right because I guess what Engineers who are using already um you know like established models right and somebody who are trying to be flexible right those are different needs right you need different kind of like apis or whatever else yeah no so that another uh not targeting anybody just uh just targeting Simplicity and in JavaScript we don't have really mature tools yet in JavaScript uh outside of tensorflow and so um that are that are really efficient that really take advantage of the hardware of a computer and so brain.js is my my own selfish uh work to achieve that basically for myself um uh yeah and it's a cool hobby right as well so I completely agree and additionally there is also like quite a bunch of JavaScript developers all around the globe right so for example having one model is definitely something cool and exciting if I would be JavaScript developer I would be definitely open my console and just be like hey let's do some much yes but for now I'm like on a python side so let's see let's see how it's going to develop cool um it was awesome having you thank you again for your answers for your open source contribution and for your amazing talk thank you very much it was a pleasure cool and vishala please give visual Applause to Robert also on this card because I mean there is like a good way to see um appreciation for one person but if you have like a couple hundreds like doing that on Discord that's going to be like a big wave for appreciation and uh Applause obviously if you remember we started this conversation from another question from benty as well so the question was what is your favorite brain power snack or drink and let's see what are the answers um and it looks like 65 percent of you are coffee people interesting like I think we should have had also teeth here and while they're going to decide but let's figure out in the next one the next one is menti meti this is like amazing speakers yeah that's me I guess uh first once again I want to thank one of our sponsors Focus reactive Focus reactive is a London and of course remote based consultancy offering expert level JavaScript reacts and node.js software development services and also coaching so thanks a bunch Focus reactive now it's time for our next speaker which Shivani Buddha she's super active in the community if you look her up on LinkedIn she you can see that she's sharing all sorts of amazing stuff from machine learning she's an Amazon Alexa Prize winner and yeah she seems like an amazing person and I'm really looking forward to her q a session later on to get a bit of a chat going on um besides being a of course machine learning developer Shivani is also um uh well I lost my train of thought she loves to work out and paint and if you're gonna invite her for dinner she would love a nail but I can pronounce this which is your favorite while you can watch the Netflix show the Queen's Gambit that reminds me of the alphago movie like the where the Google deepmind beats the biggest chess players is amazing um maybe something like that will come up in her talk let's see how it goes give it up for shifani with some fragile collapse in the community Discord Channel foreign [Music] I'm Shivani I work as an ml engineer at Facebook and today I'm going going to share um with you how to mlfi almost any product um this talk is going to be a more practical talk where I'm going to walk you through when is it okay to use ml we are going to discuss a use case wherein I add Facebook used ML and successfully so when I'm going to walk you through the cycle of an ml models development cool so um the first question that we have to answer is that is your product really ready for ML um so this is one of the biggest mistakes that people do is thinking that anything can be plugged in uh with ML and any problem can be solved with ML so I think really there are two uh questions that you want to answer the first one is that can your problem be solved with simple rules can you um just think of a threshold or is it a binary flip decision of whether or not uh your problem can be solved with a simple rule the second one to think about is what is the scale of your problem do you need to generalize Your solution to uh a lot more people than a few hundreds one example is that say I want to classify apples from oranges and all of my product is just about classifying apples to oranges um having a small filter which would say orange is orange and red is Apple would be a reasonable approach if I have 20 users who are using my product and not yet justifying whether or not we need ml however if I was to classify different types of oranges which could also be reddish and different types of apples which could also be orange I would need more than just color as a rule I would probably need the shape I would probably want to employ some computer vision techniques and so on and so if you can answer both questions in affirmative that you do need more than just simple rules and your problem is ready to scale you need ml for your product let's now walk through a real life scenario of how we use this for Facebook portal so I was working in the Facebook football team and uh one of our hero features was calling so the user would come they would say hey portal call John and the idea would be for the device to understand who John is in your friend list and if there's multiple Johns then it should disambiguate who the right John is and then create a call to whoever this person is and note here that uh the the cost of failing is high because you're ending up calling the wrong person you leave a missed call and so really the option here is to be very precise and so when we started out the flow we had was that the user would initiate this command portal would um understand who the most possible John is and this was simply rule based we would pick the top contact that we got and then we would issue a confirmation prompt and so if a user uh said yes I confirm call them we would call them otherwise we wouldn't but this was a very uh tacky long process right the user had to come in and select confirm select who they were calling oftentimes engage in touch in the UI and would just take away from this uh experience of the user interacting hands-free with this device which is smart and so we had to uh tackle this problem of hey given John how can I um predict who the actual John is so that the user does not have to come in and um do all of this work themselves and so there are a lot of rules that we could have used um we could see the similarities for in whether the name reflected or matched the name of the person we could use confidence score whether the ASR confidence and ASR is a speech recognition system and so whether it understood John correctly we could also use the relationship of uh the user to the person they were calling so naturally if somebody was a family they would uh be more likely to call for some users and for some other users if someone was uh calling a John they messaged or called frequently it would be more likely that they're calling the same John and then it could also vary on time of day um whether they actually gave the command or not so for a lot of users um they would be talking to somebody else sometimes ASR picks up incorrectly whether or not the user is trying to call this person and so what is the probability of noise um what is the frequency in which that they're uh talking to this one person what is the score of these Upstream modules that essentially uh translate any speech to text as all of these were a barrier of rules right and they could not be abstracted in one single rule for our purposes um and so since these are not just one rule uh and we needed uh the model to actually learn not just from a single flip-flop rule but a data distribution we decided that we're going to be leveraging ml for this problem and so this is the problem I'm going to walk you through um through this life cycle of the ml model development um this is what it looks like right so whenever you've now decided you've answered this question for your products that you do need ml for your model development and so what does it look like it starts from data collection um data collection is setting up the right feature and label sets for your model training and so this can be organic or this can be artificial in our case uh for the example I just gave you since we were already using confirmation prompts for from the users to decide whether or not the contact was the right contact we already had organic labels collected for our features to solve this ml problem and so one can either use um the pre-ml error data which is anonymized from their pre-existing user interactions to train their model or what ends up happening in a scenario you can use um you can actually create tools to lack data to solve the problems that you're offered for the purposes of this talk we are going to be focusing on organic data collection and how can you um how can you get data from an already pre-existing product and how can you then use that data to inject ml in your product and so this is what our features look like um for our organic data collection our features where ASR confidence for which was um if uh if a user says something or ASR or speech recognition engine translates that to text converts a con converts it into text and generates a confidence score with it we then have the nlu engine or the intent detection engine that takes as input this text and outputs the intent of the user so the user could have wanted to create a call they could have wanted to message and so on we then have the entity recognition engine which takes the name of the user's friend here John and then predicts who the actual contact is and so this is entity recognition which is contact recognition once we had these we also have uh some other user context so how often does the user message this person or call this person this would give us context to know whether or not this is a new contact that the user is never messaged or this is somebody that the user often times messages once we have these features we then also curate labels from that confirmation prompt where if a user said yes to the confirmation prompt in our previous log data and ended up creating a call to this contact we say that the label is a positive label if not and if the user said no or cut the call right as a dialed we say that it is a negative label so now that we have our data set up the next step is how do we decide what ml model to use how do we um how do we decide whether to do neural networks deep learning or to stick with the simpler classical ml models use just regression or decision trees and so on so for this purpose um we knew that we wanted to learn non-linear Transformations we had a lot of rules and we wanted a model to be able to learn complex planes um in those roles we also knew that we have a mix of categorical features and discrete features and also data distributions um we knew that our categorical features could be sparse there could be some information available for some users while some users might not have given information like whether somebody is their friend or family and so on and so given these aspects and given the fact that we wanted to um use as a little compute as possible we decided to use gbdt gbdt are gradient boost decision trees these are more simplistically an ensemble of regression trees that are averaged together to perform a classification task so it basically iteratively learns um which features to inject at each of these nodes to Output the leaf label which is whether the uh so so so so so some way in which this will look like is um with the previously discussed rules the model will learn whether um is a user a family or not and then based on some threshold that the model has learned go to one leaf or the other in the next decision it could learn um whether or not the user called this friend uh in the last seven days or not and then again based on the threshold that it learned it can go to the left hand side or the right hand side and eventually predict the label uh learned by the model in saying that whether or not this is the right contact um so this was a model we used um and now that we have the data we had the model the next step is evaluation how do we evaluate or how do we understand that the model that we use actually performed well how do we train the model and what metrics do we use is the next question so there are two kinds of evaluation that I'm going to cover one is offline model evaluation and the second is online offline model evaluation is purely done at the development time so you do a quantitative and you do qualitative evaluation which is focusing on actually how the data looks like so for quantitative metrics uh an array of metrics can be used uh it could be Precision recall f-score it could be accuracy these are for more um classification kind of tasks for more natural language generation tasks one can use blue scores for more uh similarity detections tasks one can use similarity scores and so this depends on which model you use for our purposes um we used quantitative metrics of precision recall and we also use qualitative analysis so qualitative analysis is very important at development time so and and here's why your quantitative metrics will tell you whether the Precision or recall of the model is great however what if all the labels that you had were once so all of your data set you had 20 samples and all of your data set was one single label the model will very easily learn that I always have to predict one to do well on this data set and then when you actually test the model on more data which has other labels the model might start predicting goes wrong we'll start predicting those wrong um but in your test data set this label that you had in abundance is say 80 and this new label that the model has never seen before is say 20 your accuracy still stays 80 percent and so unless you went in and qualitatively analyzed how your model is doing and what it's predicting you would uh say your accuracy is 80 which is great quantitatively but does not scale well qualitatively and so for our purposes we did both quantitative and qualitative analysis um our model did really well and we were able to get a Precision a high Precision of over 95 percent um and qualitatively our models seem to do exactly what it what we wanted it to do another Point here is that for our purposes Precision was one of the most important metrics and here is why I've alluded to this point before but if we made a call to the wrong person and now remember we've gotten ridden of that confirmation prompt and so if he made a call to the wrong person that would mean that uh the user would have called somebody they didn't mean to and would have like this embarrassing situation ahead where they would have to explain how their smart device was not so smart anymore and ended up calling this person and so what we had to do and optimize for was reducing our recoil even to the point where Precision tended to almost 100 and so a lot of times the thresholding that you select for your model would depend on your end use case and you might have to make trade-offs where um Precision recall is a curve and so you might have to reduce your recall to improve your Precision or vice versa there might be other scenarios in which you want to fetch the maximum kind of things that you can increase recall while you might have a leeway to decrease precision and so great so now we have data is collected model is modeled um offline metrics look good what's next next is online evaluation so now that we are confident that our model is doing well um we are ready for deployment how do we know um how do we productionize this model how do we um know that it's actually working well for real users do we just launch it to everybody so no we don't uh do we have a second step before we can do that so for our purposes we productionize the model um we write a wrapper which essentially abstracts the model binary and makes it available in runtime uh with a predict function where we can derive the features that the model learned in real time so now that we have a request the user made a request we feature to write that request into all of those features that we discussed before and we call this predict function um in runtime with these features um we return a label and then in runtime we see whether or not the user actually ended up creating that call or went in and cancel that call and so that's when we come to online a value and we developed this proxy yellow facing so the Matrix can be something like call success rate is increasing and call success rate is defined as the call that the user initiated and then ended up completing um and not just dropping off and then we can also Define anti-metrics for example call cancels are increasing is an anti-metric we don't want that to happen um and then we essentially roll it out to a set of users a smaller set of users initially we call this a b testing we see how these metrics are trending and if these metrics are trending as we expected them to Trend then we go ahead and launch the model to a larger group of people and so on and so out and do a progressive rollout until we've launched it to everybody if not we go back to the drawing board we see what the issue is we see where the whether it's hyper parameter tuning that we need to change uh and so on to figure out uh exactly where the model is um not performing well finally one would think that you've done data collection you've done ml modeling you've done offline evaluation you deployed it you did Stellar online evaluation what next there's one more step which people oftentimes Miss which is um Active Learning or iterative learning or maintenance of the model so what is that Active Learning Active Learning can be in this context it means a lot of things but in this context uh this is purely from the standpoint of how will the model maintain its online metrics over time over different users and so on so we only uh in active learning we will trigger retraining of the model when um either the Upstream modules change so in our case we had an Lu module which was uh detecting intent we had ASR module which was converting speech to text and we were depending on the confidences from these modules as features so what if my nlu model changed and the meaning of its confidence uh changed what if the confidence that uh it was predicting is 0.5 before was 0.6 now my model will start making uh wrong predictions if it was still in the older distribution while the distribution from the newer model changed so if any of the Upstream models which inform the features of the models Downstream or your model change you need to trigger retraining if there is a change in significant number of users so oftentimes there will be different patterns that the model learns based on the current users that it is learning on so if there is a change in those then you will have to retrain your model and then lastly if you uh explicitly evolve your features uh right based on qualitative analysis so you realize that there's more features that you can induce in the model to make it better make the recall better Precision better you of course need to retrain the model um and yeah so all of these together will ensure that you have a robust ml model at the end of the day for your users and will ensure that your model is um adapting itself to the changing infrastructure as well as user needs thank you um I hope that you learned a little bit about the ml development cycle in a product company like mine and reach out for any questions that you might have or if you'd like to connect [Music] well that was mind-blowing as you may remember as that what I said in the introduction I'm a front-end developer and I think there were four words which I understand I'm literally just trying to move boxes around and trying to make a website look good on a mobile phone and this is mind-blowing stuff for me so thanks you finally for sharing your knowledge with us uh before we go to introduce Shivani to our q a session we will go to another poll so please get your phone or another browser tab or whatever you use to go to mentee.com and enter the code 5478-871 and answer her a question how experience are you in machine learning so you'll see the possible answers there so once again go to mentee.com enter the code 5478-871 in submit your reply and now I would like to introduce you uh introduce ask Giovanni to join me here on stage hey hey good to have you I think thanks for joining us all the way from the states uh it's a bit early for you probably I hope you have some breakfast it's 9 30 a.m it's uh not that early ah good good you didn't have to set your alarm for this nice um we're gonna go to the first question which is from our audience member sanjit and he's asking how's your AI methodology different than any standards like SEMA slash Crips Dash DM methodology um I have no idea what the acronyms meant uh maybe uh maybe if you can uh say them again um I'll I'll know better okay uh sanchit if you can explain further then we'll try to get back to your question later on we got to the next question which is from Camp dot a no at the start of your talk you said the problem had to be solvable with simple rules how to tell if this is true do you sometimes just go probably train the model and check if it solves a problem even though you're not sure what the rules are yeah I think that's a great question um and there and and you're right uh not always can you uh latently tell whether a problem can be solved with simple rules or not um there's there's two general categories of problems that you have one is classification problems and the second is generation problems um classification problems mean that you have a data set and you want to basically classify it into labels and the skin it sounds simple but this can have a array and gamut of uh ramifications or applications in the way that it's used generation problems say that the machine is given a data set and it has to come up with its own way of generating something so these cool things that you see where um a machine is writing poetry or a machine is um writing stories all of those are generation problems while when you see that a machine can detect objects like apples from oranges those are classification problems so oftentimes you will see that classification problems are the ones that can be solved with simpler rules while generation problems are more the ones where you want to rely on just any ml algorithm and then go from there and so that's usually my go-to mantra for deciding whether a problem is just solvable by rules or with ML or not um and uh yeah but this is this is basically what I use okay well I hope that Kent I know is happy with that answer uh just as a reminder this discussion room going on about computer vision on spatial chat so be sure to go there after we've done this q a um the next question is from my co-mc AJ you mentioned about gradient boost decision trees which is definitely good but isn't gradient boosting slow for practical applications with an algorithm like catboost or XG boost be better in general so I guess from uh from the perspective of any uh company like whenever you're building machine learning at a big company you ideally want to rely on algorithms which are tried and tested especially when you have the scale of uh people that you're deploying that uh model to as we do we have billions of people who use our models and so to that end a lot of times yes a bunch of these alternate methodologies can be better on paper but we like to default to the ones that have proven better in practicality or once we've deployed it and so we always have to see whether a model scales well or whether a model is able to be good around edge cases and gradient boost decision trees um from both qualitative and quantitative analysis over time at least from the perspective of the work that I do has been established to be one of the best approaches having said that um that is not the only approach we could have used any other approach and gone through the same through the same cycle that I Illustrated there and we would have had gradient boost decision tree is not performed as it did which was uh beating the expectations we had from the model we would have gone back to the growing board and tried for uh looking at a different uh model okay thank you uh we have another question from an audience member minhai what was your hardest challenge while working at Facebook oh that's a good one that's a great challenge there right no I think uh I really like working at features bottom driven as opposed to top down every day we decide what is it that we are excited to build and then we roadmap what we are building and so the only thing we have to see is whether that aligns with the goal that the team and the company is going in and so to that end it's a lot of freedom to actually build things you really care about um but then because of the scale of Facebook it's also a lot of responsibility so when I was at grad school uh many years ago I would just take any model I like take any data I like and throw in the model and try to publish a paper without thinking about whether this data is anonymized properly whether I'm protecting user privacy and whatnot so now I think since we've learned so much everything I have to do at Facebook has to be considering all of these things and their ramifications um before I just go and build a model and just deploy it so I think all of the good guard rails that we have in place to ensure that we protect the data we are using for machine learning appropriately that and how do you make a model work which depends on this data and balancing those has been one of the most challenging and exciting things to learn because now there's things like Federated learning which tell you how can you actually featurize data on client and then send it to the server so that you don't compromise any information that the user doesn't want to share and what not so there's it's a lot it's a lot to learn but I think it's both challenging and exciting all right so as I understand it's more about working at the scale of Facebook and actual technical challenge you have and they protect the privacy and stuff yeah well good to hear that that's uh that's high in the agenda next question from our audience member Reg how is GB DT different from reinforcement learning yeah um reinforcement learning is more it's defined very differently in terms of just the way the problem is uh mathematically represented where in reinforcement learning you basically have a reward function you define and then you basically try to make sure that every action and you learn a policy that every policy a user takes can um optimize that reward function and so or a user takes or a machine takes uh can optimize that reward function gbdt uh while this problem that we are tackling it's great that you ask this question can be very well modeled uh in reinforcement learning because to decide whether or not we call this contact or not is deciding a policy or an action for the assistant in this case we haven't yet used reinforcement learning to model it that way and we use gbdt which is predefined labels we don't have a reward structure defined which we try to maximize here while in reinforcement learning you would have the whole reward function you would probably do queue learning one of the uh reinforcement learning algorithms to optimize the same problem I hope that answers your question I'd have to go in a hotel uh I'll have to go in a whole steel and a lecture about deep learning reinforcement learning if I go any deeper well that's what we're here for today right but uh let's let's not forget after this q a session you're going to be in your spatial chat room right so people that have more questions or want to go even deeper on your answer they can go there and discuss further with you um one more question from our audience member unsol how do you handle handle label delay in example labels arriving one month later than the training data great question um label delay I and and I think I would presume that how do you handle any delay of uh features or data coming in there's multiple ways to make up for things like label delay one could be a data augmentation approach where you say this is what I hypothesize or uh Computing your expected labels and basically training your model from there and once you do have uh the labels that you're waiting for then uh doing the um self-learning or Active Learning thing that I mentioned which is anytime your labels change you've triggered a retraining of the model in this case you will compromise uh the performance of the model for the initial whatever time you have the expected labels the performance will be sub-optimal however this will ensure that you have a independent and self-learning pipeline of modeling developed um but there's I'll have to understand more about what this label delay looks like to give a better answer which might be more suited to this particular use case and once again we'll plug your speaker room where people can ask more questions or go deeper into your uh deeper into this and so so unsolve if you want to discuss with uh I'm really lost for words today start it's a good day to be in MC when you're lost so if you want to go deeper go to the speaker room and right now I just want to thank you for joining us and I'll beat you farewell to your speaker room thank you okay bye-bye so um as you may remember from the start we wanted to see some photos of how you are watching this conference so show me where you are show me your home office show me the picture at show me the top of the mountain where you're with your phone on the 4G connection maybe 5G even show me where you're watching tweets with the hashtag mlconf EU show us where you are it will be fun for us to see and of course we have the nice raffle where you can win the get Nation multi pass so next up is a break we have another 10 minute break and and um oh but before that we're gonna go to the results of the poll so everyone has faulted and how experienced are you in machine learning well most of you 80 48 are starting their machine learning Quest so it's good to see that people are here trying to learn more stuff so I'm really happy and oh 32 even is a student so there's only 90 percent of its expert level so really happy also of course to have the experts here that they can help the students and the starters in their machine Learning Journey so fun to see this okay so now it's back to break time hope to see you after the break in 10 minutes and after the break we'll have the next speaker which is Lawrence moroney really looking forward to his talk bye everyone see you in 10. [Music] [Music] foreign [Music] [Applause] [Music] When I'm Gone [Music] thank you [Music] [Music] when times [Music] [Music] [Music] thank you [Music] [Music] foreign [Music] [Music] laughs [Music] thank you thank you [Music] [Music] thank you welcome [Music] [Music] [Music] back [Music] you gotta be away foreign [Music] and we're back everyone hope you guys have have y'all had like a wonderful break if you went to the restroom you got a nice snack you got a little beer whatever it is hope you did well now before moving on to our next speaker let's just quickly mention our silver sponsors MathWorks and infinite red Academy once again this conference would not be possible without them so thanks a Time alrighty now on to our main speaker for now we have Mr Lawrence moroney so let's get let's let's start to get to know about him okay so he's a bestseller author of many Tech books a script writer for quite a few sci-fi scenes an instructor on Coursera and YouTube and AI advocate of Google an artist and all I don't even think I got even half of like what he's done now one of his major goals is to spread the knowledge of tensorflow and how we can leverage this for machine learning his time at Google was quite the adventure so we started out on Google cloud and then moved into Mobile development um working with Android and Firebase and he also dowledged a little bit in the making of maps and in 2018 he moved on to tensorflow so a couple of fun facts fun fact one his intriguing machine learning started during his graduate degree when he created a tic-tac-toe learning algorithm that would learn from past mistakes against humans fun fact number two he's hosted a series on Google developers that YouTube channel which is called coffee with googler so if you want to see a certain googler drink generous amount of coffee while spouting knowledge on Tech you totally know where to go and fun fact number three he's also really good with cgir like I mean really good you need to check out his website for some more cool out a cool app cool art and with that brief intro out of the way I'll pass the mic on to you Mr Lawrence Maroney and you can get us started but before that let's greet him with some virtual Applause in the community channel on Discord [Music] thank you [Music] hi everybody I'm Lawrence moroney and I lead AI advocacy here at Google and it's really my job to educate the world around Ai and to try and make the world a better place through AI a little bit about me is I'm also the author of this book Ai and machine learning for coders it was just released so it's quite a new release and it's uh it was actually the number one bestseller and a number of AI categories uh in its week of release on Amazon and if you're interested in the book on Amazon this is the URL and the book itself is actually based very much on the syllabus that I'm going to be talking about during this talk so first of all I want to talk a little bit about Ai and where AI is at and this curve I like to use and this is the Gartner's um lifecycle curve and the life cycle curve of any technology usually begins with the technology being introduced and then it reaches this peak of inflated expectations and the peak of inflated expectations is the kind of thing that when you see massive hype around the technology but that hype isn't really based around anything real on the technology it's based around speculation and the technology itself and then often the uh the the life cycle curve drops us into the trough of disillusionments and despite the negative sounding name it's actually a very positive thing because that's when we blow through the inflated expectations we blow through the hype and we understand what the product and what the technology is really all about and then once we reach that point from there on we can reach productivity unfortunately AI right now is probably somewhere about here on the curve there's still many inflated expectations and just to call this out sometimes so it's inflated expectations can be positive where everybody's thinking about like the amazing things that can be done with the technology and sometimes they can be negative where people are terrified and they're afraid of the technology but their expectations about its capabilities are what are inflated because of this hype cycle curve but my job is generally I'm trying to get people down here into the trough of disillusionments and I sometimes joke that I'm I'm a professional disillusioner but really with the idea of having the world understand the what AI really is all about what you can do with AI how you can build with AI and from there then you can grow up into the productivity so like I said we're here right now and my question then becomes why do you think we're here right now well there's a number of reasons behind this the first one I will show is um this number and this number is three hundred thousand and three hundred thousand is the number of AI practitioners in the world according to a survey done by a company in China and so they wanted to just take a look at why is there such a global skill shortage around AI this was about two and a half years ago why was there such a global skill shortage around them and they wanted to say well how many qualified people are out there and it was 300 000 AI practitioners now I like to compare this with this number which is 30 million and there were a number of estimates out there as to the number of software developers globally and they vary wildly I've seen some around 22 million I've seen some around 35 million I'm gonna like I'm just picking a number roughly in the middle of that and it was 30 million and I could argue actually that the number is far greater than this so for example at WWDC this year Tim Cook mentioned that there are 28 million developers in the Apple ecosystem alone so if I did a rule of thumb that half of the developers in the world or in the Apple ecosystem we could be actually closer to 60 million developers globally well let's let's work with this number of 30 million and remember there were 300 000 AI practitioners according to the survey there are 30 million software developers according to my estimate so I made it our vision at Google is that what if we could train 10 of the world's developers to be effective in machine learning and in artificial intelligence and if we did that would have three million um Ai and ml developers which is 10 times this number so we made that our goal can we increase the number of AI practitioners globally by 10 by a factor of 10 not by a number of 10. and uh so you know we said with sat down we'll make this our goal we started this journey about 18 months ago a little over 18 months ago and today I just want to share like the strategies that we use and the results that we've gotten but first of all when working with software developers and when I talked with them and when I would um look at how they were being trained I got a lot of feedback around why they thought AI was difficult and why AI was something that was uh while it was something that was of interest to them it was something that it was going to be too difficult for them to kind of give up a lot of their time and a lot of their study time to be able to pick it up and I started seeing like a lot of terminology like I've put in this chart here people saying it was difficult there was a lot of math there were a lot of terms that they weren't familiar with like unsupervised learning or supervised learning um they really like me they hadn't done things like calculus and probability in 25 years and as a result the number of Concepts that were being thrown at them just to get started uh made it like there was a massive Road bump that you'd have to get across to be able to get started to be able to transform your career and transform your skill set to be a machine learning or an AI developer so I saw that as a challenge and one of the things that we do with Google is that we give ourselves a mission and our mission is defined by three words and uh so the three words that I used were make AI easy and so I've made that my mission at Google is really it's all about making AI easy and I skipped one slide here for a moment and then that led me to this overall mission that I wanted to be able to train millions of developers to reach billions of people and going back to the life cycle curve and the hype the hype cycle curve that I spoke about earlier on if we can train millions of developers and I think millions of developers are the key to be able to reaching billions of people with AI Solutions and with AI applications that will burst us through the bubble that will burst us through the hype bubble so that we can see what's real in AI so we'll make AI easy so some of the strategy around this was that um the first goal was to come up with a syllabus of material and a set of content that was aimed at developers so it was by a developer me and it was aimed at developers who are coders uh you don't need a PhD didn't need math any of these kind of things I don't have a PhD and like I mentioned before I haven't done calculus and probability and statistics in 25 years so I wanted to come up with a syllabus that was buy a coder and it was four coders and then to build a strategy around this syllabus and the strategy is twofold first of all is the professionals strategy so this is the strategy aimed at educating people who are already developers people already in careers as coders and these and folks in that domain tend to be reached in three different ways there's massively online open courses or moocs does being able to do it direct and at scale and then there's managed training so let me talk a bit a little bit about each of these some moocs we partnered with deep learning.ai and Andrew Wang and the great Folks at deeplearning.ai to produce three specializations we've released two of them tensorflow in practice and tensorflow data and deployment and I'm currently hard at work on the third one which is going to be released in November and December and we're calling that tensorflow from Basics to Mastery and the idea behind this one was to produce 12 courses in three specializations that would take somebody from no knowledge at all except maybe a little bit of Python Programming to be able to intense flow and practice to be able to meet the common scenarios that are needed in machine learning careers and that's natural language processing computer vision and sequence modeling the second specialization data and deployment is then taking the work that you've done in building machine learning models and making them real by putting them into mobile apps by putting them into websites by using them in JavaScript and things like that and then the final one tensorflow from Basics to Mastery is kind of going back a little bit to the model creation from tensorflow in practice but going deeper learning how to extend your models learning how to create custom that we have things in models called layers learning how to create custom ones and then looking at some of the advanced algorithms and being able to break them down so that you can understand them a little bit better in that one I'm afraid I broke my promise of not doing math because we had to do math a little bit in that but we tried to make it as approachable as possible uh directed scale is then we launched a tensorflow YouTube channel and we wanted to make that the go-to place where people could have short quick lessons either about how to do something in tensorflow or what is happening in tensorflow new releases all of those kind of things we also put together a number of Hands-On code Labs again aimed at people you don't need a PhD on math you sit down you start coding and you start building Out Concepts and it's in line with the three pillars that I spoke about earlier on that we saw as what employers were looking for and that's computer vision natural language processing and sequence modeling and then we've also been working through a number of different Google training initiatives and I created a series for Google training and I created another series as free training of when people were at home in lockdown and those kind of things called ml foundations and all of that was being done direct at scale via YouTube and Via other Google channels and then oops and then finally sorry I just uh uh before I go into the next one and just as an example of how the model of being developer focused has been successful um uh Carmel Allison and I did a talk called machine learning Zero to Hero at Google I O 2019 to test it out with the developer audience uh so I I'm trying to break the machine learning audience into two there are the people who are expert at machine learning and Ai and then there are the software developers who want to use machine learning and Ai and my focus has been on the second group So Google I O is the audience for them and we did a talk at Google I O that we call machine learning Zero to Hero to take somebody in about 40 minutes to go from basic python knowledge to be able to do some pretty complex computer vision in this case in the screenshots I was teaching them how to do a computer vision exercise that recognizes rock paper or scissors and but again when it was done very developer focused one of the nice things we've learned about this one is the viewership of this video we're approaching one million views on it and when it comes to talks at Google I O this was Far and Away the most successful from the YouTube perspective so we realized that we have resonance with developers with this type of message and then the third one I mentioned is uh managed and so this is where we have high touch managed efforts to train developers and two of them that I show on the slide here one of them is an academy in Indonesia called banquet and uh bankit was an exclusive machine learning learning academy led by Google working with the government and working with various startups that we brought in 300 people it was application based we trained them in machine learning and the Indonesian government really wanted to be a part of this so that they could start seeding their startup ecosystem with people who are qualified and people who know the technology and then the one at the bottom as the Google developers machine learning bootcamp is running in Korea it actually started right at the end of October and the idea behind this one is that Google and a number of companies and many of whom are startups got together to sponsor this boot camp but to make it a very different boot camp and that is the cohort of people coming into this boot camp if they pass the content in the boot camp they have a guaranteed job I'm particularly excited about this one given the economic situation at the moment because of the pandemic that to be able to work together with industry to create a pipeline of people who are qualified to do a job but have the motivation for those folks to go through this boot camp to have a a guaranteed job or internship at the end of it I find wonderful and all of this leads to what we call the employment oriented developer certificate so we've created a rigorous certificate exam and this certificate exam is that once you pass that you're proving that you can be a developer with computer vision natural language processing and sequence modeling and this is also part of what we had with the Bankhead Academy and with the the academy that I mentioned in Korea the boot camp I mentioned in Korea the second half of this is to work with Academia and so to work with universities and so the number one thing that I when I speak with professors at universities that I hear is that they would love to teach new courses but every time you need to add a new undergraduate course you have to get rid of an old undergraduate course because there's only so many that they can teach and there's a great risk involved in teaching new courses there's um there's a lot of time involved in developing the curriculum in training teaching assistance and all of those kind of things and as a result there tends to be a lot of friction about adding new courses and those of us who have graduated with computer science I have half of a computer science degree and I did have physics half computer science but I recall when I graduated the skills that I learned to graduate where the skills of maybe four or five years prior to my graduation day and the languages and Frameworks that I needed to get a job on the date I graduated just weren't taught in Academia so we wanted to help try and solve that problem with machine learning so we created a program where we help universities in developing courses we provide them with the same curriculum that's used by the moocs the same curriculum code first oriented curriculum um that like we're using our own trainings and all of that kind of stuff with support and in some cases some financial assistance to for example higher Tas and that kind of thing and so we've launched that with a number of different universities and again the idea was to drive this towards an employment-oriented developer forget and the same one as the professional folks had so some of the results I'd like to share about this and the global impact of this so first of all the moocs Coursera and deep learning Ai and the two specialty that the two specializations that we launched um as of now when I'm recording this we're over 600 000 Learners and it's it's growing exponentially and this new specialization is launching next month we think we're going to be getting you know uh hopefully above a million Learners by sometime next year and one thing that the world economic Forum did a jobs report about the jobs of the future and the jobs of the fourth Industrial Revolution that AI has been called and they actually cited these moocs as a necessary skill uh they listed a number of them and the top two were Andrew Ang's deep learning specialization which I learned from with terrific specialization and um my specialization the tensorflow and practice one the ones that we created with each other so you know that it shows that it's Landing well from an employability perspective even recognized by the world economic Forum jobs report Udacity also a mooc they launched an introduction to tensorflow and some tensorflow like courses and they like coming close to 300 000 Learners in China the syllabus was used to create a um a course with company called netease and this launched just last week or middle of October by the time I'm filming this and they have around 3 000 Learners already so it's uh we see that it's really beginning to pick up and uh udemy uh launching some courses quite soon so again and the syllabus that's being used by multiple Folks by multiple providers and we're seeing really nice numbers with the universities uh we've had dozens of universities globally um that we've assisted in teaching the tens flow syllabus as I described earlier on but I want to call out a couple of highlights Imperial College in London um used this to create their teaching syllabus but they also use this to create an online massively online open course of their own that they teach on Coursera which I found really exciting um one of the things we're really committed to with this is widening access as much as possible so I've been talking about you don't just need a PhD and things like that but we also want to widen access to people who are traditionally not represented in Ai and KO University in Japan um have launched a course for women and we know that you know there is a there is a representation problem in Tech A diversity problem in Tech with having women in Tech and in Japan it's particularly sharp because of some historic issues uh there so KO University really wanted to see if they could bring in more women into Ai and they are just launching uh right around now they're launching this course for women being taught by women and again they became part of our University funding our the program that we set up and this is one of the results that came out of it and then given the issues uh globally and what we've been learning globally about racial issues and you know the fact that black lives really do matter uh one of the things that we've been trying to work very hard on is to work with historically black colleges and universities in the USA to help them you know gain the benefits of uh the uh the gifts that we can have with AI and to be able to take advantage of what the world's economic Forum was showing about the growth of jobs in Ai and to be able to train their students for that earlier I mentioned the certificate program we launched that in the middle of 2020 and uh sorry early 2020 spring of 2020 and it's an employment oriented certificate a rigorous exam covering NLP computer vision and sequence modeling and as of now we have about 1200 holders globally and it is increasing nicely and the vision here is that to really help people show their skills but to help employers know that these skills have been shown one massive part of the jobs Gap was we discovered that a lot of employers didn't know how to find Qualified AI people and didn't know the right questions to ask them so we created a certificate program with that in mind and then just to wrap up um I alluded to it a little bit earlier on but part of the impact on this one is you know we really want to help seed the economy we really want to help train developers to be successful in the new economy and I want to share some statistics about the state of employment globally first of all this chart uh this is a general nice upwards Trend chart but there's a big dip at the bottom and this is uh the chart of GDP of I averaged that across several countries UK USA and others and you can see the general Trend over about the past 35 years is that it's been mostly upwards there's been a couple of little dips but the problem we need to chart something like this is you see a general upward Trend and you think everything is fine and you might miss some of the important nuances in this so I re-shared it as a derivative chart where instead of just showing the overall growth I wanted to show the growth in any given year relative to the previous two years and as a result the the same 35-year chart showed this this to me is really interesting because there are a couple of big peaks in this chart one of them is here and this one I don't have the years on this one but this one was from about 1993 through about 2007 and this was the how the growth of the web and how the emergence of the web and the tech economy really changed the world economically and this massive period of sustained growth and it's no surprise what are now the biggest companies in the world by market cap are all companies that came as a result of this growth the microsofts the apples the Googles all of those tech companies that are now Giants came from this period of growth the second one here is the app economy this is really the growth that came about as a result of the emergence of the smartphone first engine Juiced I would say you could argue by Apple with the launch of the iPhone in 2007 I know there were other smartphones before that but this economy really picked up then then the emergence and growth of Android and other platforms and we've seen this as a really really huge growth area and while there's a big dip on the right because of covid and because of the pandemic we just want to realize that even like apple this this year released a statistic that said in 2020 the app echo system has grown by 300 000 jobs in the us alone so we can see that jobs and employment are still very very hot because of these two peaks and the the next Peak maybe in 10 years time if you look at a graph like this one there'll be a new Peak and that's the fourth Industrial Revolution and the economy that comes as a result of AI the world economic forum for example uh did a report where they wanted to look at the jobs of Tomorrow starting in 2020 and ending in 2022 and they had five categories uh data and data and AI engineering and Cloud people and culture product development sales and marketing and Far and Away the biggest as you can see here was about a 37 growth in jobs between 2020 and 2022 in data and Ai and Engineering cloud computing in C was plus 34 a lot of that was around ml stuff in clouds also so we can see it's a huge potential growth thing for jobs and unemployment there's also a Forbes report and I put the link to it here that showed the global ml Market will have a compound annual growth rate of about 44 between 2017 and 22 24 from 1.5 billion to 20.8 billion so it is a huge growth area and AI software Revenue they also reported between 2018 and 2025 from 10 to 126 billion which is a kegger of 43 so we can see here that there's definitely massive massive growth in this area coming you know they're all all the reports that I see tend to put out as a compound annual growth rate in the 40s both of these 43 and 44 support that and even they they mined LinkedIn to take a look at jobs and 44 000 jobs in the USA required some kind of ML and almost a hundred thousand globally required some kind of ml so we can see ml AI these kind of skills are massively massively important as we go forward and we wanted to be the people to help train millions of developers to reach billions of people and with that I just want to say thank you and I'll be able to answer any questions [Music] thank you [Music] that was an amazing presentation Lawrence I'm glad that we actually got a good educational perspective to the field too it's a very unique kind of talk and I'm glad you all actually got to experience this too so before we actually jump into the Q a session with Lawrence we've got a poll question for you so where would you like your ml models to be or hosted right and we have a couple of options here to answer you go to mentee.com and enter the code 9975-682 all of this system that you reply so I'm going to repeat the code again so that's go to mentee.com and to answer this question enter the code 9975-682 and we'll be looking forward to your responses now in the meantime though in yeah so in the meantime now we have a q a session with Lawrence everybody let's just say give it give a good clap and give a very warm welcome to Lawrence Maroney here yeah oh thank you so much thank you yeah can I answer that question with all of the above no yes please you you do have an answer right now let's do it yeah so like I I wherever I want to deploy my models out absolutely everywhere everywhere it's like all the options what's the correct option everything everything models everywhere that's awesome well right now let's actually start with some amazing questions because we kind of have some amazing questions right here so actually this is a by Jose I hope I'm pronouncing your name right but which do you think is a better option for the industry tensorflow or pytorch well first of all I'll start by saying I'm completely biased given that I work on the tensorflow team I'm wearing a tensorflow jacket and I'm drinking coffee from a tensorflow mug so I think my answer there is going to be tensorflow um and but so if I take off my jacket and take off my mug I think it's always good for everybody to learn as much as possible about everything um you know we welcome the great competition that we're getting with pytorch I think is brilliant for the industry to have competition amongst the vendors but I will answer that you know despite that I still think tensorflow just because of the number of different places that you can deploy tensorflow models right from the massive Cloud scale infrastructure with multi-node tpus all the way down to the tiny um like microcontrollers and embedded systems that your tensorflow skills regardless of what type of model you want to produce there's a place for you to put it and so like from the industry perspective I think you know I would strongly recommend despite my jacket I would still strongly recommend tensorflow yeah definitely I do know like that you know pytorch does abstract a lot of well hidden mechanisms that tensorflow does provide and like while on one it's kind of a trade-off because on one side tensorflow is definitely more adaptable and controllable and flexible whereas with pi torch some people just find it an easier interface to just use in general yep um I I won't claim any kind of expertise in pie torch it's it's something that I need to learn more of to be perfectly Frank um but yeah my focus has always been on enabling people to succeed with the technology and you know it's one level is being able to build models but the other level and more important level is having somewhere to deploy them and I think you know that deployment surface is a massive massive strength of tensorflow and it's you know where I think we distinguish ourselves across in ml and that's why I'm particularly passionate about it and also here's another question from Arjun Kumar do you recommend how do you recommend continuing to build your career after you know completing like horses like yours or engine's for example yeah good question I think um practice practice practice is very important um I think like with any kind of coding the more that you do it uh the better you get at it you begin to realize the pitfalls you begin to realize like those tips those tricks those things that make you better at doing it so ml is no exception in that regard it's really practice practice practice um it doesn't have to be public with competitions on kaggle or anything like that and one thing that I always recommend is you find something that you're passionate about and then there's got to be some data out there around something that you're passionate about and start building models in that regard and you know just continue to practice and become a better model Creator and then once you have those models that to practice practice then and deploying those models be it like a large-scale global scalable infrastructures or maybe just a single cell phone for yourself to use but you know the key with anything is you know constantly practice to improve yourself um yeah that's that's true you know also like about like practicing too and everything that that is associated with this too like there are also like a number of people I'm also like looking at a bunch of questions where there is that like mathematical aspect that is deep learning and machine learning whereas you know tensorflow is more of like okay let's kind of break things apart from a more practical aspect and then you know dig deep into that way so a question would be like you know how do you kind of balance these two you know these two completely different Notions of approaching machine learning and deep learning would you prefer one or the other or somehow like a symbiotic relationship between like Theory and practical applications when going on your like machine Learning Journey I I like the term that you just use symbiotic because I think that's exactly right but symbiotic like how do you start and so they should be meshed together but how do you get started and I mean I can say just for me personally I hadn't done calculus in 30 years when I started looking into modern machine learning and when the initial material that I was looking at was kind of teaching you how an Optimizer Works through gradient descent and tons of calculus or you know teaching you how a convolution Works using mathematical and Greek notation I'll admit I was lost um I it was harder for me to follow it but when it was like I started looking into this like using code and primarily code it's much easier for me to read and understand code so then my symbiotic relationship began definitely on the Practical coding side of it so that now I can understand what a model is now I can understand how a model learns from parameters now I can see how back propagation works and things like that and then with all of that context there around it then to start going into the math and then seeing ah now I see why I'm using calculus for gradient descent and all of those kind of things so to really build that context around it so that I could understand it better and that's why now it's very much that symbiotic relationship that you know I could start building models without knowing any math but once I started building models if I wanted to build better models then to start understanding what's going under the hood was was necessary yeah that's completely true because like I know there's a lot of people who are in school especially like based on our polls and you know maybe like in school you'll be like learning some very complex mathematics like kind of like calculus for example like that you mentioned and you're like well if you're just if it's just thrust at you it's like okay but where do I use this what am I going to do with it I don't care as long as I just pass this course I'm good but then when you're like but I am interested in machine learning and deep learning hmm what do I need to do for that incoming calculus just surreptitiously it's like hi again and then you're like Okay cool so this is how you can use calculus to do to create like all of these machine learning you know projects that you know you see everywhere just being deployed it's like oh my God there's self-driving cars there's like this system that can detect your fraud whether your activity is fraudulent I mean yeah just like the the applications are endless and all with Calculus so yeah that's definitely very interesting to hear yeah but it's like don't let that be they don't let that be the gatekeeper that prevents you from getting in that if you're like me that once you start reading calculus and your mind your brain goes to sleep uh you know that kind of thing you can really get started and start doing all of this stuff and be productive before you start getting into the math but it's like once you're at that productive level then it becomes easier to get into the math because you've built that context around it and it makes it much easier for you to do so sorry you said you had another question now we always have so many questions for you they just keep pouring in so you know we've seen 10 you've seen like tensorflow just like you know build just become this huge thing right in front of you but do you have any reservations about tensorflow like maybe somewhere along the way tensorflow could have done something better and it's probably still something that's a big problem is there anything like that I I don't think there's anything that I would consider to be a big problem now um I think one thing that would have been really nice to have earlier but we then eventually had with sense flow 2 was to think about it in a more pythonic way uh because like uh early versions of tensorflow were very much graph oriented so you'd load all your stuff into a graph and then you execute like a session within a graph and for me as a coder as opposed to an AI person that was a little bit alien to begin with um it was very hard like because you can't do step-by-step debugging and things like that when we release tensorflow 2 became like with eager by default it became more pythonic so it made it much easier for the people who are coders to get up and running and if there was any one thing that I had the time machine that I would change to do sooner it would be that Ah that's very fascinating and I think we have like a final question over here so I uh so there are people with like some experience in like JavaScript and Python and they they're always like asking like so which is the best language to start my machine Learning Journey do you have any any specific thoughts on that yeah I think the best language to start your machine Learning Journey of those two is the one that you're most passionate about using uh I know a lot of people who love JavaScript I know a lot of people who hate JavaScript I know a lot of people that love python I know a lot of people who hate it um if you don't have any particular Affinity to either of them my recommendation would be to start with python uh just because of the number of support libraries that are out there for the Beyond machine learning part of it the data science part of it they're drawing graphs and all of that kind of thing so um like I said if you were gonna pick one from scratch I would say definitely python but there's certainly nothing wrong with JavaScript the things you can do with tensorflow.js are fabulous and yeah I think Jason's going to be talking about that at this conference later today and he always shows amazing things exactly yeah I'm completely looking forward to that too um by the way Lawrence I think we are out of time for some questions but thank you so much for answering all of our questions until now and for our lovely audience here of course like we're always happy to have you and if you guys were you anyone here just did not get a chance to ask Lawrence your questions or they didn't come up right now feel free to reach out to him in his speaker room even starting like right now after this call so just be there Lawrence it was great having you right here and we look forward to hear from you soon yep sounds good thanks and I'll see you all in the speaker room sweet take care now for all now I asked you guys all a question right of we had a little poll that was going on of where you would run your machine learning model did you forget I hope you didn't forget don't forget because it's right on screen right now so now let's take a look at some of the results and oh wow looks like our the the answers are kind of split between different members of the audience but it's more or less unanimous among people with rather deploy models in the cloud over like any other platform either on mobile or web that's interesting because it actually does make sense quite a bit too because I I think that like all of your models for example you don't want to use you know your current computers capacity and be constrained by Harvard limits when you're actually you know creating these models so cool option I probably would have gone for the same who knows I don't know and okay um oh by the way we have some we we also have like some lightning talks and sorry we have a pretty cool surprise for you and so we have spatial chat and an after party and then there is like a quake tournament that's gonna be go on like we mentioned before and it's gonna be a free-for-all so there's you all just have to join in have fun and you can follow us on Twitter with mlconfiu just to get the access code and join us for the party and with that we're gonna have a couple of lightning talks today that I'm gonna pass the mic over to my co-mc Sergey now if you'll take it away yeah and uh as AJ said right we're definitely going to have a cool lighting talks right um I think we had already pretty cool speakers beforehand right but now we're gonna have it even more dense right even more exciting speakers in the question of like half an hour right so what I want to offer you is the four speakers three dogs right in half an hour so we're gonna go from very diverse topics starting from have to apply transfer learning and deep learning to such an industry as semiconductors right because everybody can distinguish cats from dogs right but can you use deep learning for semiconductors and the pajan and uh sorry like and session gonna be talking like how to do that and like what are implications on the next speaker like this is the lightning talks we're gonna go and use R and kubernetes for robots data how cool is that I mean we're all talking about python we all talking about using notebooks right but here we're going to use R and kubernetes as a cluster setup and robust data I think it's almost sounds like Christmas for me and to make it like even more exciting if you're gonna make it even wider and more diverse and we're gonna talk about optimal so how to make your machine learning learn its own representations right and like what structures and auto magical ml essentially right so please give some virtual clubs emojis in Community Channel and since we have four times more speakers you should set at this like four times more emojis as well so please welcome our speakers [Music] thank you [Music] hi everyone so today we will be talking about deep transfer learning for computer vision and we will be covering a couple of real world applications in the context of the semiconductor manufacturing industry at Nano scale a bit about ourselves so I'm the function I'm a data science leader at Applied Materials also Google developer expert in machine learning and an author over to you Sachin hey my name is Sachin I lead the data science competence from Bangalore in Applied Materials I have an experience of more than a decade uh handling multiple Enterprise white use cases and that's in this particular scenario we will be discussing about Sim conductor industry related AI use cases so let's jump into it dependent okay so today we are going to talk about uh defect classification so I think all of us know about Moore's law where it states like every two years the transistors in a chip gets doubled and how it is possible like it is possible by the atomic level engineering what companies like applied uh you know day in day out they do and for that they need to develop these recipes and these recipes are not easy to develop there the process Engineers who work on these recipes have to deal with lot of defects while refining these recipes there they deal with you know these high-end uh image capturing techniques which these tools like laser tag and AFM which is atomic force uh microscopic Tech tools are you know taken into account and in this particular uh you know cases some of these techniques are very destructive where the Wafers are destroyed that's where AI can help by you know figuring out these defects and uh have a lot of cost savings and Time Savings so I'll give you a quick overview of how we are leveraging AI techniques in defect detection for AI you know semiconductor manufacturing so if we talk about the defects like we we deal with multiple type of defects in this particular use case we have defects like particle large particle small fit and all you can see there is in a lot of noise and variations so it was not easy to uh have one single algorithm or one single model to deal and you know find the whole classification done what we have done here is we leverage multiple class classical as a stacking technique where we input the image then we do a lot of image processing through different techniques we do the denoising and then we feed that image into first level of classifier which figures out whether you know the there is a defect or not if there is defect then we try to figure out by second level of classifiers whether it's unknown or a large or a small defect even if find that then there are different type of you know image processing techniques we apply to bring that effect out of the background and then we try to do further analysis like whether there is noise or the threshold levels are you know the noise levels are below threshold so that we can directly go to the fourth level of classifiers which are to decide the ultimate class whether it's a pet or a particle in this way we try to leverage uh you know transfer learning based techniques to figure out the classification uh approach what we have got so I think dependent will get more deeper into it and I will over to you so here we try to apply a hybrid classification system where we fine-tune a pre-trained resonant model to extract the Deep features and we also use traditional image processing based features like gray level co-operants matrices and zernic moment features and we combine these to form a fusion feature vector and we pass it through a deep Learning Net to do the final level of classification and we also do some image augmentation especially in cases where we have less data just like in these cases because we can't afford to have a lot of data especially with regard to semiconductor level electron microscope level scans and we also leverage pre-trained models like resnet 50 in the back end of a object detection model like a faster rcnn to detect and count the number of defects because sometimes you also have to give the defect count besides the type of defects uh with regard to our evaluation performance as you can see the performance is 95 on a holistic overview in terms of the four types of classifiers and we do a classifier level performance also to understand the level of performance we are getting at each specific uh hierarchy in our overall hierarchical classification system uh now we'll discuss about uh unsupervised learning case in the context of deep transfer learning uh over to you Sachin thank you so as we saw like the last case we had we were you know we had prior knowledge of type of defects but that's not the case always like many of the times we are not aware of how the defect is getting induced in this whole recipe and there we need to do a lot of analysis like what is the effect count you know what are the morphology of the shape what are the elements like whether it's nitrogen aluminum like at different uh parts or different steps of a recipe we can encounter you know different foreign particles which could result into effect so because of lack of prior knowledge we go to the unsupervised based technique where we have built an interface where a user can browse the location where we have got all these defect images once the images are fed to the system we use the resnet 50 model to extract the features these feature vectors are then fed to algorithms the various clustering algorithms of what we have got you you know get the higher level of clustering which is then provided to user to decide how many clusters they want and then we go to the final level of clustering like admirative clustering are a fine uh you know Affinity propagation clustering and all based on these clustering results we figure out the right clusters as well as the outliers and that is presented with the user so this makes the whole process very reproducible and repeatable which is otherwise very subjective between different manual users so over to your dependent to get more into details of this whole level yeah so again here we leverage a retrained resnet model along with computer vision specific features like the heraldic features of the zernic orthogonal projection moment features and we get those Fusion vectors which we typically pass to our clustering models and that is where we use hierarchical clustering in the terms of forming that dendrogram getting that right level of cut off forming those clusters from the data itself and then we also isolate the Singleton clusters which are basically very different from the regular clusters in terms of similar kinds of defects and this is a view you can see in terms of the interface which can show the different clusters to the user and then they can save it and leverage it in the future to even feed it to a supervised learning problem and that brings an end to our Whirlwind tour of deep transfer learning for computer vision thank you thank you [Music] foreign [Music] thank you very much for your interest in this talk my name is Franz from the name I'm the chief data officer at expandia and we help organizations to boost their data analytics capacity and the stock was prepared together with Florian pistoni Florian is the co-founder and CEO of inorbit and in orbit is a cloud management a cloud robot management platform and they are taking robots developing that further to take robot operations for the global scheme robots are everywhere Now everywhere you look even more now with covid they're working alongside with humans they're working autonomously and it means that the volume of the data that we're Gathering From Dom from robots is increasing exponentially of course this offer is very interesting chances for us data scientists to work with the with the kind of data that has some some special characteristics and um if we look at what in orbit has been doing they have accumulated accumulated more than 3.8 million hours of robot data in in the last 12 months alone and they're growing they're growing rapidly right so right now they're adding about a Year's worth of data each and every day now for us to do analysis on that even though this is a lightning talk it's very briefly but I want to highlight some of the some of the issues we found and how we've solved it so the main the main problem we found is that um in orbit offers their services to any Fleet we have no control of how exactly those data are being gathered and how they're sent to the central service so in one of the pocs we did we were faced with many robots that were sending millions of files right one one file every five seconds for every agent and but each file contained data of multiple sources and each of those sources were in robots or in agent um operations they had their their own time stamps they were not directly related so in the first step that we needed to do was Data extraction and this was a little bit more complex than we expected especially because we needed to join observations by mayor steinstein I will highlight that a little bit in one slide further and then do the feature engineering on top of that so what I mean with the nearest time joining we have different different signals of admission localization and speed and we need to find an interval where we can join each and every one of those um of the signals to have a single observation and we worked out how that could be done um and then we needed to start the feature engineering so once we have one line one observation um for a protagonists um we wanted to look back and we wanted to look back at what happened before a failure so there's a failure right here and if we go back say 42 seconds that we need to do that for each and every one doing that for um and taking into account all the cases where we couldn't uh include the data and for instance when there was a failure uh within the 42 second time frame was absolutely possible but then we were faced with an enormous volume where our local computer simply said no this is not going to be possible so immediately thought about forming this out to kubernetes we set up a bucket with the data that was going in we packaged them in one day Zips we packed cheeseste Json gz um Zips to make it a little bit more workable and to be able to to transfer the data with Maurice and then form out with the form out the full data extraction of kubernetes a second bucket with the intermediate result form of the data engineering we have the result ready for analysis what we found is that it is much easier to get the help of something called packeter so Pachyderm is um is a product let's say it's a company and they have a they have an open source version of this which is what we used and what we have there is not a bucket what we have is a filing cabinet we have a repository where we conversion the data that is coming in and version the data that is coming out doing this kind of data Pipeline with version it means that if there's one change at any point in the pipeline the rest of the pipeline will respond and will update the data automatically so that prepares us to do all the um the to have all the heavy lifting ready once we bring it into production just a quick look what this looks like we create pipelines that are very similar to your to your kubernetes configuration files and the key thing here is that we can connect the data in the repository in the packeterm repository PFS is the packaging file system so what we're running in our RS3 because there are scripts they were already parallelized but parallelizing was not enough so we were now able to farm this out in making that connection uh setting whatever data preparation rules we're doing the next and for each datum that is sort of the Pachyderm term for what each unit of data in the pipeline is we can see whether it was successful or whether it failed right this is screen from Patrick Santa Maria I did most of this work with so being able to monitor um on that level uh what is happening with your data is in in practice it was absolutely great right so parallelizing our code it's it's easy there's a package called fur built on future by Andrew Benson and it's very easy to parallelize it out going from parallel article to a massively parallel pipeline is doable in things to compare to packaging for us it was much easier to work with kubernetes through that the other thing was that large clusters are surprisingly affordable right we worked on a 80 CPU 520 gigabyte RAM cluster for under ten dollars an hour um which was something we haven't experienced before and now we are using more and more in the work that we're doing for the team that we were working with for an orbit this also have huge implications because it means that as fleets grow they know that collecting and processing data is you know becomes critical for them you also now know that where they understand the scaling of their full platform it doesn't need to be hard but scaling up the analysis doesn't need to be hard either and using AI at the robot Fleet level you know unlocks many new opportunities that they are working hard on to continue to offer to their to their customers right thank you very much I hope that this was useful if you want any further information please visit our website expanded.com or inorbit.ai we both have blocks running and we like to write about this stuff so we hope to be in touch sometime in the future so thank you very much [Music] thank you [Music] hello everyone my name is Bernhard Zoom I'm product manager for machine learning with the math works let me motivate my topic with some questions to you where do you want to apply AI are you concerned with the lack of your experience in AI or with black box models the community widely recognizes these as challenges and barriers to Broad adoption of AI across many Industries today I will focus on automl automation that takes the routine iterative effort and most of the science out of building machine learning models so what exactly is automl to understand that you need to know a bit about the typical workflow for building machine learning models the focus of this talk but building deep neural networks isn't that different first you need to process your raw data deal with its messiness and get it into a shape that's suitable for later stages like dealing with missing data and outliers next you need to engineer features extract a few variables from your data that serve as input to your model and capture the majority of the variability that's fairly easy for numeric data but a lot harder for signals next you are faced with the choice of different machine learning models and even to experts it's not clear which model performs best On Any Given problem so you have to try multiple which leads to the model tuning stage where you assess the performance of some initial models optimize their hyper parameters maybe select a subset of features to avoid overfitting but that may not be enough to get really good performance you may have to go back replace some features with others and do this all over again if you're familiar with machine learning you will know the most difficult and time consuming stages are the feature engineering and the optimization if your head is spinning now don't despair because you don't have to know all this complexity the point of automl is to simplify it ideally to go directly from your initial data and your machine learning problem to a model you can deploy however really taking it uh seriously that is not a realistic expectation in a single step machine learning is not possible however what is realistic is freeing up Engineers like yourself to focus on the hard machine learning problems and on your application otherwise without automl you'll have to find that AI expertise either inside your team and organization or outside and those data scientists are hard to find and expensive so as the first barrier that automl removes it overcomes the lack of machine learning expertise but even if you have that expertise you you are increased in productivity because automl takes away those time consuming and iterative steps finally automa allows you to solve problems that otherwise wouldn't be feasible like use cases where you need to build many different models representing different variations for different environmental stages so how do we apply autumnal for engineering most of those engineering applications are based on Signal image data and that's where the feature engineering becomes critical for good performance and that's notoriously difficult we had MathWorks brought our signal processing knowledge to bear and came up with the following three-step automl first you apply wavelet scattering these wavelets are very suitable in their time-bounded shape to represent spikes and irregularities in your signal therefore you get very good features many engineering applications however require deployment to memory and power limited embedded systems for those you cannot deploy large models so second we apply automated feature selection to reduce the may be hundreds of wavelet features to just a few very performant features and reduce the model size finally and key is the model selection hyper parameter tuning step you have a choice of different models and for the model to perform well the hyper parameters need to be set just right let's look at that stage in a little more detail how would I say simultaneous optimization of model and Hyper parameters work well you can perform random search but that's not efficient either because the search space is very large we employ Bayesian optimization that builds a model of the search space and here you can see how that patient optimization switches between different types of models and optimizes the error over in the course of the iterations how do we know that automl works we compared automl to the traditional manual process on two classification process first we looked at human activity recognition where you take auxiliary meter data from mobile phones we have about 7K observations in the set we collected and we manually engineered 66 features using various signal processing functions second we look at hearts on Calif classification think about being in your doctor's office with a stethoscope and listening to your heart sound so those phonograms we could have say the set of 10K observations it's publicly available and engineered less than 30 features so what results did we get we can see here with the manual process we achieved accuracies in the high 90s as you would want to have for such an important application for automl for one application slightly lower but the point is without all that expertise and without the time consuming iterative process you get very good models uh in this in a few steps so automl empowers Engineers without AI expertise to build optimized models including for signal applications where the feature extraction is notoriously difficult you can apply automl to signal applications in a few steps automated feature generation with wavelets automated feature selection to reduce model size and make it fit on your Hardware and model selection along with Hyper parameter tuning in an efficient way using Bayesian optimization finally to deploy your AI model to the edge and embedded systems you need low level code like C Matlab you can translate automatically to CC plus plus code that can be deployed directly and thus another barrier to broader adoption of AI removed thank you for your attention and if you want to know more Monday afternoon or evening I'll have a longer session on automatic interpretability a seminar on those two topics one hour and two hours a Hands-On workshop on machine and deep learning using Matlab online and now I'll head back to the moderator for questions [Music] yeah that was like really dense and informative 30 minutes of our time right thank you again Deepa Jan sashin France and Bernhard for this very diverse and interesting talks and now I'm happy to actually welcome them to the stage so we can answer your questions that hopefully you are asking on this card so please get your round official Applause hey people welcome welcome how's it going for you I heard that you're also coming from very diverse and geographically far away country so maybe if you could just say like what time is it for you right and where are you located geographically yeah um I'm in Bangalore India it's just past midnight now definitely diverse locations India that's I'm in Costa Rica so it's a well-passed lunch now it's almost two almost one o'clock and I'm located on the east coast in the U.S so it's just two o'clock in the afternoon wow that sounds like a future right because usually you're having people I don't know maybe in three different rooms right then I'll be coming covering like three continents it depends how you count um anyway so let's try to get to some questions first of all thank you for delivering such amazing talk um and I will just start with some questions and hopefully we will have some other questions arriving meanwhile um so first question would be maybe to uh Independence um so you know people always trying to start with I mean these images right because it's like very representative right they can see they're like cats and dogs right and they start to train some convolutional neural networks right to distinguish them and now you're doing like semiconductors right what were the the biggest um you know like discoveri