IDEA Talk: Adrien Pavao (INRIA) on Machine Learning Challenges: Crowdsourcing Big Data Problems

Key Takeaways

why the soup so welcome everybody welcome to our first actual idea talk at the fall 2019 season as most of you know we try to intersperse on these Wednesday evenings 22 Enslow which are topics of interest to the teleportal constellation semantic web knowledge graphs these sorts of things idea talks which are grad students and visitors connected to what work that idea is doing more generally and they're happening here and also our pirates meetings which are approximately once a month our pirates the RPI are users with meaning um tonight we have a visitors come all the way from France all the way from Paris experience are wonderful whether it give us a talk on on Friday inside I've been advertising it as an insider's view to standing up machine learning challenges Adrian has been associated with the co2 lab or the challenge run by Isabel Dione who manages the co2 lab which was originally done by or created by Microsoft and is the management that was handed over to his Abele's organization Avery has been working with us for some time helping us understand how to set up challenges we also have a guest in the back kind of Thomas who is a there's a developer who has actually been supporting that the code of flat platform the long-term view they go this is so Adrian and Tyler right here this week is part of the United Health Foundation a health data UNICEF project which include all sorts of things over the past couple of years and we are the ultimate objective is that we will actually be standing up machine learning challenges on an internal platform for various courses so with that I'll turn it over to age and thank you thank you John for the introduction and thank you all for coming here so we are going to talk about machine all new challenges and if you not know anything about it don't worry because we are going to talk this year what it is so the the presentation is going to be divided into three parts so firstly what is the national challenge then we'll why we are organizing and participating in them like what are the advantages of it and we will see how to organize machine learning competition how so to make good machine learning competitions that are so more complete more concretely you'll really upload it and make the website to work so firstly what is machine learning challenge so Doug Lee this is a way of solving big AI problems so it can be sent to sequence in Austria but or so more and more societal and ethical problems by using a competitive framework so this is real competition so between humans but also a kind of competition between methods of machine learnings so every participant tried different methods and submit the results which are then automatically computed to get a score and so the golfer got people to get the highest possible score and to be well-run cat in the in the leader bounder and eventually when the Commission and if you are in the sub places the window the competition and you can win a cash prize from $100 to million of the graph so it treated under the challenge so yet so to give an example if you never heard about challenge easily let's dive into a simple example so firstly you need the task so in this case face detection which means automatically compute is the fatal left in a picture this is quite an interesting task now we have a rope task we need data so our data set will be pictures maybe thousands of pictures of face on that face and from each picture the the solution so this is a supervised learning framework inescapable now lot of other facilities and really importantly you have to divide your day time to sets at least to set you that can be morbid at least to because but seasons are going to train their model in the transit and we are going to test the scores under the perception the solution for the test set is only from organizer and either to box event so basically they have to predict the solution for the test set and then we are going to compare the proposed solution to the ground truth with a metric to give them a score and a rank in the repo you then need a metric so in this case you can choose accuracy which does the rate of good answers and basically that's it and so the letters use to tackle the problem is absolutely participants meaning that they can do whatever they want to use our best to win the competition so if you never have about personal challenge I think if you remember all this you have the reading so competition are in a platform so there are many platform for competition our disco Dan so the main advantages of code above our other platform is that this is a low process of open source sorry project and it is free to everyone to organize their own competition or participate in as many as competition they want so this is really what makes dispatch from unique so it will have gentle it was created roughly by Microsoft and as well stand from the 7 years ago and as well as some people from Shaolin like is a very UN Eric is the the main developer of the platform with Tyler which is a restroom so quickly some history of the platform so in 2013 there was the first competition that bloated so the venture mode segmentation challenge was the first one I guess and then the next chair collision challenge got some quite decent participation with more than 300 participants then new features got I did like code submission so we are going to talk more about that later basically it's instead of submitting your result you can submit directly your model which will be internally trained and tested in the server so this has many application available if you have private data you want to share with participants or if you want to be fair like having the same computational resources for every person and so two years from now the there were less than 500 challenges and a lot of users in the platform so it's really going fast so let's take some very fast challenges so you use here some example and how they are useful oh I think you may have already heard of image net data set it's a really famous computer vision data set with more than 14 million of images label so each gender is a challenge with this data set which is growing and this is what led to the the big explosion of deep learning methods and the use of big neural network this day another example is the learning torrent challenge at nips conference so I took this example to show you that not only classification and regression program file number this one is the reinforcement learning competition if you're not aware of what reinforcement learning is a different framework in machine learning well instead of having beta you have an agent that takes actions and get some feedback from its environment so in this case the agent is supposed to learn how to run in a 3d environment and the reward or our objective function is how far we can get and less time as possible yeah yeah that it's 3d modeling some basic physics law like gravity and a basically agent can control articulation and measure to try to move so maybe Robin is running try to find the good move go through this fake 3d one so that is another example do 2ml challenge series are challenges where the goal and the task is as you can guess to automate the national in process of the motor selection and the tuning of the model so in this case you there are many data sets and subsequent upload the our models and the data set are hidden and our various tasks such as for example images video can be from different domain like medical data or satellite images and the word is reached trying to find models that is able to handle really different data set and do the best cross possible in every task so there have been many - ever challenge and even this year we are still creating new so this way of watching the ultramel auto computer vision challenge an automatic natural language processing challenge and in December odd needs conference we are going to learn sure Auto deep learning challenge with really a lot of various tasks to be a very exciting challenge the last example wanted to take the games challenge which Christine used during this class I just wanted to take this egg over to show that combination are not only for big expert and big research team for high cash prize it cannot see be used in the classroom for students and at the it's really motivating to have an environment where you get a real-time feedback for your your submissions and as you can see there are some to learn with other countries people they are trying to change our model to to do better score and beat their classmates so it's really a good way to learn machine learning - and to try to go further and new method that hasn't been running in class so that was for the example of changing the challenges and I think you begin to see how useful it can be so let's see why we are using them and why we want to organize and participate in changing by the way if there are questions feel free to ask during the presentation or add remarks so for organizers changes are really a good thing because basically the participant are doing the hard job so you can get really good solutions for your problem thanks to the crowd wisdom which basically basically says that if a lot of people are trying to do something you haven't really get a good solution for your problem so this is really why as an organizer you would want to organize one and it's kind of rated with the second point which is kind of because you you put effort to create your challenge and maybe a huge cash prize which can seem pricey but you get you can get a hundred of team of expert trying how to solve your problem so this is prices prices in this case so you can really have good tarisha instead of just putting like when team trying to solve your problem you're really having a good version and how participants are a lot of difference we're out of course the cash price but it's really not the only reason to participate so of course we can have new lectures for the students or even researcher who can try out their method and know what works what does not work and so an access to a kind of playground with equal the organizer already to the boring part of gathering the data and making the occurring every Cinco can just go and try your things and check if your your method works in different problems so you have really a lot to to work from occupation as well as also sometimes jobs and paper opportunities if you for example issued well during the the competition and just to show you that cash prizes are sometimes not the biggest reward so this is examples of challenges and the Kaggle platforms see the other big famous challenge platform to get it out a lot of million price challenge but this one's the equivalent machine learning challenge had a small reward in comparison to a 0 for 13,000 if you can see and it has a really good participation with more than 1,000 teams and this was because at this time the higgs-boson problem was really hot in physics research and discharge gave access to really important data from research or more interested by the task itself than the cash prize so let's see how to organize the machine learning competition and so as I told you it's not really hard to to do because the participant do the hard work and as an organizer you just have to create the task and give them so it's kind of easy but not stronger than its weakest link because you have a lot of different things to do like choose the data to the scoring matrix a great documentation Rosen in any step if you if you fail you can really have a bad challenge let's say so what makes a good challenge I'll see you need an interesting turn so at Alden any scientific or industrial work edges the first thing is to travel with tasks having enough and good quality data so we are going to talk model data just after a clear objective and a good matrix so we are also going to to explain not this point just write that down and of course some other ingredients like the prices and the feedback and also a good starting kit is really important so what is a starting kit we may ask so the starting kit is a pack you give to newcomers for your competition which help them to type directly into your own problem without having to bother with boring stuff like you give them instructions that are so this is of code or notebook which already reads the data trainer basic model and the data record logistic regression let's say and then reflect a submission so they can really quickly go into the challenge and just replace the method by Amazon try quickly if they're interested in putting more effort in a challenge so this is really important part if you want to have a lot of occupation and on and I also I did fun because yeah I think it's something that really get people to do game and how you do the other things the five columns one yeah yeah indeed so talking about data so Peter Novick from Google said we don't have better algorithm we just have more data and I added the kind of rural sense because this ain't totally true so what he said is more data did better algorithm but better they tablets more data but basic is date already everything in data science so you really need to work on your data set so you have a lot of possible issue you need to be aware of with your data so firstly having enough data and module demonstrate is big more you need big number of examples of course better than anything but basically we can say that having a good train test and validation speed design I think the most important part is that is for example to have enough data in your test set because you really want to have good error bars when you're evaluating your your the different solution because it doesn't make sense if you don't score well the solution there are a lot of also possibilities of data leakage so you really need to be aware of that when preparing your data set for example a bad 20 split could be on medical data set if you have for example data from ten patients that are maybe 100 of second perfect patient you need to split and keep every patient in the same set to a date alligator the wrath of tiny paw severe ear also confirmed in fact hours and I'm going to show you sided of this and the resting the the bias and variance trade-off so the variation in data is genera when you doesn't have enough data nowadays not our biggest issue because we have a lot of really big data set but the bias issue in details emerging more in line there is a lot of possible bias so about confounding factors I I took this example of that convention well you were supposed to classify objects so planes and town like that and you may guess what the problem of design was on this one so the task was highly just to wriggle and so this is a big data register so the winners weren't learning to recognize the shape of the planes but just the color of the sky of the ground so this is yes this for real this is yes Orion is the problem is you're going to plant gravity yeah you're a neurotic way to the plane which is a part oh yeah I'm probably gonna happen which is the tail so you success in the having a good score but that's not what the design of the change wanted in the first place obviously so this is the note data set which is an example of good design this problem because they remove the color and the like the models are really looking at the shape of the objects in values hangars to avoid or so for example each plane on the same angle so the detail cake so you really to think about a lot of things with when designing your dataset to really tackle the task you want to stagger in the first place about bias so now really different possible bias in your data's like sample bias and it is really really merging program because you can really have like this WorkSafe discrimination prejudice and everything in your data so I don't have a like good example is let's imagine maybe if you try to classify between women and men pictures and know you have every time you have a woman GH and let's say and so you will have a real bias and misclassify a men because he's 11 X chatter okay that takes very century good to get you the idea of what can be a bias insulator because we want to talk about bias in national models the data itself can read attach them we can't really rely too much and the last point about how to design a good charity the metric so the metric is read the measure of your success in the competition for every participant flow is really important as an objective for example if you have a binary classification task with imbalance classes so meaning you have one class that is over-represented and you choose accuracy as a metric I think you can see the issue curving so just a classifier that will only send the over blended class will have a good call so we need to choose a metric that fits well your problem and whether sort of uncertain about that so in this diagram which come from the article competition you haven't bring the score of the so it's like a necrosis car maybe I may use Cisco of the baseline method so really simple method and in orange here the score of the best submission and basically you want your problem to have the biggest orange part as possible so this is what we call the high modeling difficulty and this is what makes your problem interesting and have wide variety of possible scores so that is also something you ready to look at all the it's just to say that the increasing difficulty is how much the scar is high unstack so for example this one the best possible score walls and the yo points also we can say that the interesting difficulty is really it's a high difficulty so this is just over I'd like to be the best possible score which is one I hope clear enough so that's for the part about health negative charge so just to finish we see more concretely how to create a charge on Colorado so we have this pack that we call a blender so you have everything that define your competition in it for the description I read the different web page in HTML format a program for the scoring as well as of course the daytime the starting kit so I think you will give your particle to help them and so you pack all of this in a deep fog lamp applause it on collar up and then holla you get your website with your challenge and you just have to go to participate and try to submit results or code submission and get it work so and after that as an organizer you have an editor that you can use directly to we change the data set and the title let's say the start date of the vision so it's really easy to use even without big background in computer science so let's make it easy to use a files many people as possible I just Samadhi title finish so here the link of the wiki of collab I think we we already talked about code submission she's real a useful feature and so we can add compute workers to compute computation is very demanding competition so for example in the automatic machine and conviction we have a lot GPUs every circuit submissions run on one GPU so you can get your your result quickly even if it's trained on five different data set and then tested on it and in the future we are going to add the collab benchmark features which basically competitions but without end date so you just keep the playground open and with the advancement of the machine learning techniques you'll see how how much you can go and program so that's it thank you question we have plenty of time if you like to do a live demo on you're creating you notice we only double one of the examples I'm just show our elevation you sure interested in you yeah we can do that you're cooking okay sorry I had one question yeah yeah if I set up in you a project on : who owns the code that the computers my comment on adidas I only give you seeing above the competition we're gonna submit models under the models of my computer I did it I'm actually feeling gin generally when in composition you have cuts addition there is clear rules about it for example in the automatic machine learning composition we said that if you want to underprice you have to make your cut public so and make it open source so it did you you choose in the world what they are going to do with the the code that these relief should send your card into a server you well that it is a really well by the process the IRS my question is is that from the code lab decides or someone the person running the it's not something the person only the competition decides is really the class or estimate code units to metric yeah this is a lot must come on use your resort to proposition for the testing set but dependent if so the point Alan based on the same question if the point that you made about the advantage of a code submission type contest of being able to apply that code to data which is not released is really interesting yeah yeah that has some good applications I want you don't want people to be able to tune by hand Damodara long as I was saying with private data that you want to more challenging with medical data which are is really private so this interesting literally didn't know how to to take in place but to play sorry that yes I'm very sorry to hear about you a little completion like you know how much liquid or please publish the work that you know the students love but by comebacks responses not allow them to have taken it is it always properly but then can come into contract on like what's the percentage or a crop idea so sorry we get a little a that students up in there for us to be their own so they want to write a paper you can you know opportunity to drop a preamble like you know from the past experience like you know is it a hydrate that people obviously work yeah it's a recommend lots of different conditioners I mean leads to say pearl and yackin first occurrence is really common after competitions because our condition is really prized Elaine such as biology physics when machine learning itself of course oh so there are a lot of opportunities to result our can a lot of competition say for example that they give the ability to participate in workshops or in covers for the part numbers of five rust on the reserve also she's quite commonness they're actually challenged have sold on a workshop at nips or whatever and no for IGN or whatever and you'll you know they have a challenge theory to some publisher and they eighteen we publish the work that we can is about she published edited lots of others because if you didn't check out the fifties red pepper run so better game increases publication I think in all didn't know because it happen the price all just like participating and submitting beautiful cake I don't think everyone wins but the top one is it's really no statistical difference between the top one generally means that they don't going down right anyway I open it at CDC are for the competition's that each of the competition positions like a brief competition in those chairs decide to publish and made its based on results but also based observation and in there are these company papers are much children really I think like TV PR is Maxim or page for the conferences like the competition paper for page - take this happy and so just being clear on because that was a little dim on mister some papers that were talking about our papers by the participants discussing their solution that's that's what the those ones are made hypothesis that as an organizer you usually also can publish analysis of the competition paper and comparing the different methods that got submitted and the result of hitch so it even has an organizer it's common to talk of a problem but everyone's win anyway okay so can you discuss a bit about the actual for conservation competitions of kind of the different languages that are used always in jail I thought so now it's not by them but it's not a future director of collab as an organizer you create your ingestion program which is the one that is going to load the code and train the model and everything so if tomorrow you want to organize a our compositions by example just to be doop about you to pipe that attention problem by the way and I'm going to create a our incision program but a basically not annotation of the platform itself but more like most composition use Python so that's why user it's a Python code submission which could be anything clear to the organizers to make that so a question that the we taskbar and yesterday I'm talking about this what sort of libraries are available like if you so if you want to click Python based models for your consideration what sort of libraries are generally available is our Kegel or code way up for actually what we have access to that you should enjoy very much you have like the big libraries and like s Callum and themselves iTouch seem like that but as an organism if many participants are asked for library you can admit to the video so it generates there are no big issue with that and even if you really need your library exhibition you can just add it as a package in your submission it depends on how much data you're allowed to upload you can usually add your package in it and to just be run as you are you're cutting yeah that's a big limitation those are questions all remark yeah ha magic you guys the usually accuracy only 3 depends on the task because I'm really very happy so I can say with an epic asking organization but there are thousands of different challenges so basically classification I you see is why we use that if you are in regression you're going to use something else and you can have also here two tasks challenges so you can use that the final metric there is your ranking in each tasks for example so 3 depends on the task personally I you see a lot I guess but it's so pretty values do you want to talk about how we're going to be using challenges to our guy we use challenges two ways one is we use the challenges that you can set in class recorder lab and we use it for class projects so continue making will open in analytics or stuff hope that we find very motivating right we also use it to on your students into our research program so we'll have them do a challenge is kind of a team exercise to getting into our environment and working together and you know they find a cool specific so that was that well yeah I think we did this year we actually entered a poet challenge which is called the run by hrq which is the health research quality agency for health secret quality I should get that right now to the United States and they offered a competition and they they give to ten thousand prize because you're spending like five seven times that entering it but we went to say once it's called mortality minor we're looking at that especially terms of mortality we've ever faced super prides at 50k it will only need a little one double chair in this room yes yes so that's exciting so I think there it's quite motivating to both be who's I guess for consciousness it seems were good to kind of get back that calendar will be different more of a qualitative challenge solve a problem we also run challenges that are positive on campus he had what we call state upon or fazenda problems typically we don't usually evaluate predictive accuracy so I think evolve in future so now we have our lab but we'll have people presented solution didn t have the problem and I will say uses data she's not something that we kind of deposited right see where she writes any other questions [Applause] the people at the back