New Approaches to Image and Video Reconstruction Using Deep Learning at Facebook at F8 2019

Key Takeaways

I thanks for coming everybody there's a few seats at the front still if you're standing at the back by the way some over here as well hi my name is Jeremy Howard and I am going to be talking to you about the exciting world of krappa fication and more importantly d krappa fication I'm from fast AI you may know of us from our software library called excitingly enough fast AI which sits on top of Pi torch and powers some of the stuff you'll be seeing here today you may also know of us from our course course dot fast on AI which has taught hundreds of thousands of people deep learning from scratch our first part of our course out now covers all of these techniques and I'll tell you more about the upcoming new course later this afternoon our course covers many topics each one there's a video there's lesson notes there's a community of tens of thousands of people learning together and what are the really interesting parts of the last course was lesson seven which amongst other things covered generative networks and in this lesson we showed this particular technique this here is a regular image and this here is a krappa five version of this regular image and in this technique that we taught in lesson seven of the course we showed how you can order you can generate this from this in some simple heuristic manner so in this case I down sampled the dog I added jpg artifacts and I added some random obscuring text so once you've done that it's then a piece of cake to use play torch and fast AI to create a model that goes in the other direction so you can go in this direction in a kind of deterministic heuristic manner but to go from me here to here is obviously harder because you have to like figure out what's actually being obscured here and what are the JPEG artifacts actually hiding and so forth so that's why we need to use a deep learning neural network to do that process so here's the cool thing you can come up with any heuristic krappa fication you like which is a deterministic process and then train a neural network to do the opposite which allows you to do almost anything you can imagine in terms of generative imaging so for example in the course we showed how that exact process when you then apply it to this crappy JPEG low resolution cat turn it into this beautiful high resolution image so this process does not require giant clusters of computers and huge amounts of labelled data in this case I used a small amount of data train for a couple of hours on a single gaming GPU to get a model that can do this and once you've got the model it takes a few milliseconds to convert an image so it's it's it's fast its success it's successful it's effective and so when I saw this come out of this model I was like I was blown away how many people have here have heard of Ganz or generative adversarial networks this has no games games are famously difficult to train they take a lot of time a lot of data a lot of compute this simple approach requires no games at all I was then amazed at about the time that I was building this lesson when I started seeing pictures like this online from something called do defy which takes black and white images and converts them into full-color images these are actual historical black-and-white images I'm gonna show you so this was built by a guy called Jason who you're about to hear from who built this thing called do defy and I reached out to him but I discovered he's actually a fast AI student himself he had been working through the course and he had kind of independently invented some of these dekappa fication approaches that I'd been working on but he could have gone in a slightly different direction as well and so we joined forces and so now we can show you the results of the upcoming new version of do defy to be today which includes all of the stuff from the course I just described and a bunch of other stuff as well and it can create amazing results like this now check this out here is an old black-and-white historical photo where you can kind of see this may be some wallpaper and there's some kind of odd-looking lamp and the deep learning neural network has figured out how to color in the plants and how to add all of the detail of the seat and maybe what the wallpaper actually looks like and so forth it's like that an extraordinary job because this is what a deep learning network can do it can actually use context and semantics and understanding of what are these images and what might they have looked like now that is not to say it knows really what they look like and sometimes you get really interesting situations like this here is the Golden Gate Bridge under construction in 1937 and it looks here like it might be what and the model laid it way now the truth is this might not be historically accurate or it might be we actually don't know right so actually Jason did some research into this and discovered at this time apparently they had put some kind of red primer to see what it would look like and it was a lead-based paint and we don't know in the sunlight did it look like this or not so historical accuracy is not something this gives us and sometimes historical accuracy is something we can't even tell because there aren't color pictures of this thing right so it's it's it's it's an artistic process it's not a historical reenactment process but the results are amazing like you look at like tiny little bits like this you see humanity realize it actually knows kind of like color in porcelain it's extraordinary what a deep Learning Network can do from the simple process of krappa fication and declassification so in this case the krappa fication was take a color image make it black and white randomly screw up the contrast randomly screw up the brightness then try and get it to undo it again just using standard image net images and then apply it through classic black and white photos so then one of my colleagues at the Wicklow a a medical research initiative guy named Fred Monroe said we should go visit the Salk Institute because at the Salk Institute they're doing amazing stuff with 1.5 million dollar electron microscopes that create stuff like this and these little dots are the things that your neurotransmitters flow through and so there's a crazy guy named Airy at SOG who's sitting over there who is trying to use these things to build a picture of your whole brain and this is not a particularly easy thing to do and so they tried taking this technique and krappa fiying high-resolution microscopy images and then turning them back again into the original high res and then applying it to this this is what comes out and so you're gonna hear from both of these guys about the extraordinary results because they've gone way beyond even what I'm showing you here but the basic approach is pretty simple you can use faster I built on top of Pi torch to grab pretty much any dataset with one line of code or four lines of code depending on how you look at it using our data blocks API which is by far the most flexible system for getting data to deep learning of any library in the world then you can use massive faster IDOT vision library of transforms to very quickly create all kinds of augmentations and this is where the crap application process can come in you can add your JPEG artifacts or your black and white or rotators or brightness or whatever and then on top of that you then can take that crap of a picture and put it through a you den who hears heard of a unit before so a unit is a classic neural network architecture which takes a high input image pulls out somatic features from and then up sizes.that back again generate some image and units were incredibly powerful in the bio medical bioimaging literature and a medical imaging they've changed the world they're rapidly moving to other areas as well but at the same time lots of other techniques have appeared in the broader computer vision literature that never what made their way back into units so what we did in fast day I was we actually incorporated all of the state-of-the-art techniques from upsampling there's something called pixel shuffle removing checkerboard artifacts there's something called learnable blur for normalization there's something called spectral norm there's a thing called a self attention layer we put it all together so in faster yeah if you say give me a unit motor it does the whole thing for you and you get back something that's actually not just a state-of-the-art Network but contains a bunch of stuff that have never been put together even in the academic literature and so the results of this are fantastic but of course the first thing you need to do is to train a model so with the faster a library when you train a model you say fit one cycle and it uses again the best state-of-the-art mechanism for training models in the world which it turns out is a very particular kind of learning rate annealing and a very particular kind of momentum annealing something called one cycle training from amazing researcher named Leslie Smith so you put all this stuff together you need one will step riches you need a loss function and so the key thing here that allows us to get these great results without again is that we've again stolen other stuff from the neural network literature which is there's something called gramm loss that was used for generating artistic pictures and we basically combined that with what's called feature loss together with some other cool techniques to create this really uniquely powerful loss function don't we show in the course so now we've got the crap of fire data we've got the loss function we've got architecture we've got the trading schedule and you put it all together and you get really beautiful results and then if you want to you can then add on at the very end again but we have a very cool way of doing gains which is we've actually figured out how to train again on a pre trained model so you can do all the stuff you've just seen and then add again just at the very end and it takes like an hour or two of extra training we actually found for the stuff we were doing for super-resolution we didn't really need again it actually didn't help so we just use an hour or two of regular unit training but what then Jason did was he said okay that's nice but what if we went way way way way way further and so I want to show you what Jason managed to build Jason thanks Jeremy so with the old apply success in Silva ninjas the next logical step in my mind was well what about moving images can we make that work as well aka video and the answer turns out to be a resounding yes so for the first time publicly I'm going to show you what the old fly video looks like you better get on the job some of the kids may be up this afternoon oh yes we can get along without dragging those young kids up here oh why don't you button up your lip you're always squawking about something get more static and radio [Music] so that's the old apply video so Nona oh thanks so now I'm going to tell you how that video was made so it turns out if you want great video you actually had to focus on great images and that is really composed of three key approaches the first is reliable feature detection the second is self attention and the third is a new gand training technique that we've been collaborating on with - di I'm really excited about so first is reliable feature detection so the most obvious thing to do with a generator that's unit based if you want to make it better is to use a bigger ResNet backbone so instead of using writing at 34 we use ResNet 101 in this case and that allows the generator to pick up on features better the second thing though and this is possibly even more important is you need to knowledge the fact that you're dealing with old and grainy film so what you need to do is you need to simulate that condition and so you do lots of augmentation with brightness and contrast and Gaussian noise to simulate film grain and if you get this right that means you're gonna have less problems with colorization because the features are gonna be detected correctly if you get it wrong I'll point this out you get things like this this zombie hand right here on the door frame it's pretty unsightly the second thing is self attention so not normal convolutional networks with their individual convolutions they're they're focusing on small areas of the image at any one time it's called their receptive field and that can be problematic if you're making a conversation model based on that alone because those convolutions are not going to know what's going on on the left side versus the right side of the image so as a result in this case you get a different color of the ocean here versus here versus here where's on the do defy model we use self attention self attention allows for features on a global scale to be taken into account and you can see that the ocean on the right side there the right side it render is consistently colored in I like that too so the next person X again so the original do defy uses scans and a reason why I went for games is because they're really good at generating realism they're uniquely good at generating realism really and my problem was I didn't know how to write a loss function that would properly evaluate whether or not something is realistically colored or not I tried you know tried to do not ganz because they're kind of pain but anyway so the reason why do defy has great colorization is because of ganz the original hag break authorization because of that but there's a big drawback in this first they're really slow I did the original the old apply took like three to five days to train on my home PC but the second thing is that they're really unstable so the original deal defy while had great renders if you quite frankly cherry-picked you know after a few trials overall you'd still get a lot of glitchy results and he'd have unsightly discoloration and that it knows because of the ganna instability for the most part and finally they're just really difficult to get right hyper parameters yet to tune and experiment after experiment I probably do it over a thousand experiments before I actually got it right so the solution we arrived at was actually to just almost avoid ganz entirely so we're calling that no again so there's three steps to this first you pre train the generator without gain and you're doing this vast majority of the time in this case we're using what Jeremy mentioned earlier which is feature lies for perceptual eyes that gets you really far but it's still a little dull up to that point the second step is you want to pre train the critic without gant again as a binary classifier so you're doing a binary classification on those generated images from that first step and the real images and finally the third step is you take those pre generating generated components and you're training them together as again finally when only briefly this is really brief it's only 30 90 minutes for deal to fly and put that in perspective you see this graph on the bottom here that that orange yellow part that's the actual began training part that's the rest of that is free training but the great thing about this is in the process you get essentially the benefits of Gans without the stability problems that I just just described so now I'm going to show you what this actually looks like starting from a completed pre-generated generator and then running that through the Gann portion of know again and I'm actually going to take you a little too far because I want I want you to see where it looks good and then where it gets a little too much gain so if this was before again this is where you pre pre train it it's a little dull at this point here you can already see colors being introduced and this is like minutes in the training right here is actually where you want to stop no it doesn't look too bad yet but you're going to start seeing their orange skin or it's going to go turn orange rather yeah right here so you don't want that you might be surprised so the stopping point of no Gann training at that gam part is right in the middle of the training loss going down which is kind of counterintuitive and I got to be clear on this we haven't put a paper out on this so I don't know why that is honestly I I think it might be because of stability issues what the bachelors I was using back there was only five which is kind of cool I mean it seems like new again accommodates low bed size really nicely it's really surprising it turns out as good as it does but yeah it's it's that's where I'm actually shopping it so no again really solved my problems here and I think this illustrates it pretty clearly so when we made video with the original deal if I it looked like this on the left and you can see like their clothing is flashing red and the guy's hand has got a fireball forming on it wasn't there in the original whereas on the right side with a no gun training you see all those problems just disappear it's just so much cleaner now you might be wondering if you can use no gain for anything else besides colorization it's a fair question and the answer is yes we actually tried it for a super resolution as well so I just took Jeremy's lesson seven lesson on this with the feature loss which guy had ups at that point on the left and I plot know again on the results and ran it for like 15 minutes and got those results on the right which are I think notice way sharper there's a few more things to talk about here there I think are interesting first is that super resolution result you saw in the previous slide that was produced by using a pre trained critic on colorization so I just reused the critic that I use for colorization and fine-tuned it which is a lot less effort and it worked that's really useful a second thing is there was absolutely no temporal modeling used for making these videos it's just image image on it's literally just what we do from normal photos so I'm not changing the model at all at all in that respect and then finally this can all be done on a PC a gaming PC just like Jeremy was talking about in my case I was just running all this stuff on a 1080 Ti for anyone run so I hope you guys find no gain useful we certainly did it solved so many of the problems we had and I'm really happy to now see you guys said it's up on github and you can try it out now and there's co-op notebooks so please enjoy and next up is dr. Manor he's going to talk about his awesome work it's all good Steve hi everyone my name is URI Manor I'm the director of the weight advanced bio photonic score at the Salk Institute for Biological studies how many of you know what the Salk Institute is all right for those of you don't know it's a relatively small biological research institute as per its name it was founded by Jonas Salk in 1960 he's the one who created the polio vaccine which fortunately we don't have to worry about today and it's relatively small it has about 50 labs which may sound like a lot to you but just to put things in perspective in comparison to a state university for example a single department might have that many labs so it's relatively small but it's incredibly mighty every single lab is run by a world-class leader in their field and not only is it cutting-edge and small but powerful it's also broad so we have people studying cancer research neuro degeneration and aging and even plant research and our slogan is where cures begin because we are not a clinical research institute we are interested in understanding the fundamental mechanisms that underlie life and the idea is that in order to fix something you have to know how it works so by understanding the basic mechanisms that drive life and therefore disease we have a chance at actually fixing it so we do research on cancer like I mentioned no degeneration and even plant research you could think of as a cure for example my colleagues have calculated that if we could increase the carbon capture capabilities of all the plants on the planet by 20% we could completely eradicate all of the carbon emissions of the human race so we have a study going on right now where we're classifying the carbon capturing capabilities of plants from all over the globe from many different climates to try to understand how we can potentially engineer or a breed plants that could do a better job of capturing carbon and that's just of course one small example of what we're doing at the Salk Institute now as a director of the Waite advanced bio photonics core my job is to make sure that all of the researchers at the Salk have the most cutting-edge highly capable imaging equipment microscopy which is pretty much synonymous with biology so I get to work with all of these amazing people and it's an amazing job and so I'm always looking for ways to improve our microscopes and improve our imaging as a microscopist I have been plagued by this so-called eternal triangle of compromise are any of you photographers have any of you ever worked with a manual SLR camera so you're familiar with this triangle of compromise if you want more light you have to use a slower shutter speed which means that you're going to get motion blur if you want higher resolution you have to make compromises and your depth of focus or you have to use a higher flash and all of these principles apply to microscopy as well except we don't use a flash we use electrons or we use photons and a lot of times we're trying to image live samples live cells and I'll show you soon what that looks like but you may be surprised but our cell did not evolve to have lasers shining on them so if you use too many photons if you use too high of a laser power too high of a flash you're gonna cook the cells and we're trying to study biology not culinary science so we need to use fewer photons when we're trying to image normal physiological processes if you're trying to image cells under stressed and maybe using more protons as a way to do that and of course we care about speed we want to capture the dynamic changes the whole point of imaging is that if we have the spatial temporal dynamics the ultra structure the architecture of the systems that underlie life that I just underlie our selves our tissues then we can really understand how they work so we want to be able to image all the detail we can with all the signal-to-noise that we can with minimal perturbations now one of the most popular kinds of microscopes in my lab and many others there's something called a point scanning microscope and the way it works is you scan pixel by pixel over the sample and you build up an image that way now you can imagine if you want a higher resolution image you need to use smaller pixels the smaller the pixel the fewer the photons and a few of the electrons or you can collect so it ends up being much slower and much more damaging to your sample the higher the resolution the more damage to your sample alright so here's a real-world example of what that might look like in terms of speed this is an actual presynaptic bouton and on the left you can see what happens with a low resolution scan and on the right you can see a high resolution scan you can see that on the Left were refreshing much faster on the right we're refreshing much slower but there's more detail on that image this is a direct example or a demonstration of the trade-off between pixel size and speed what you can't see here is this is a two-dimensional image this is a 40 nanometer section of brain tissue and what we really need if we want to image the entire brain which we do we're actually trying to image every single synapse every single connection in the entire brain so we can build up a wiring diagram of how this works so that maybe we can even build more efficient GPUs the brain is actually 50,000 times more efficient than a GPU so Google AI is actually investing in this technology just because they want to be able to build better computers and better software anyway I digress the brain is 3d not 2d so how do you do this one way is to serially section the brain at the 40 nanometer slices throughout the whole thing that is really laborious really hard and then you have to align everything so what we did was we invested in a one and a half million dollar microscope that has a built in knife that can image and cut an image and cut automatically and then we can go through the entire brain and a lot of people around the world are using this type of technology for that exact purpose it's not just brain we could also go through a tumor we can go to plant tissue so that brings me to my next problem sample damage if we want that resolution we have to use more electrons and when you start using more electrons in our serial sectioning device you start to melt the sample and the knife could no longer cut it try cutting melted butter you can't get clear sections and everything starts to fall apart or as Jeremy would say it starts to fall to crap whereas on the right with a lower pixel resolution with lower dose you can actually see that we can go and we can section through pretty well but you can see the image is much grainy er we no longer have the level of detail that we really want to be able to map all of the fine structure of every single connection in the brain but that is the state of connectomics research in the world today most people are imaging at that resolution but that's not satisfying for me so in my other life from a photographer and actually on Facebook I follow petaa pixel and I was browsing through Facebook you know stalking my friends and doing whatever you do on Facebook and I came across this article where they show that you can use deep lining to increase the resolution of compressed or low resolution photos aha what if we can use the same concept to increase the resolution of our microscope images so this is the strategy that we ended up on previously I used the same model that they used here which was based on Ganz which you've heard about now their amazing problem with Ganz is they can hallucinate they're hard to train it worked well for some data it didn't work well for other data then I was very lucky to meet Jeremy and Fred Monroe and together we came up with a better strategy which depends on what we call crap refine so we take our high resolution e/m images we crop it by them and then we run them through our dynamic res unit and then we tested it with actual high versus low resolution images that we took on our microscope to make sure that our model was fine-tuned to work the way we needed it to work and the results are spectacular so on the left you can see a low resolution image this is analogous to what we would be imaging when we're doing our 3d imaging and as you can see the vesicles which are about 35 nanometers across are barely detectable here you can kinda tell where one might be here but kind of hard to say and in the middle we have the output of our deep learning model and you can see now we can clearly see the vesicles and we can quantify them and on the right there's a high resolution image that's our ground truth in my opinion the model actually produces better looking data than the ground truth and there's a couple reasons for that which I'm not going to go into but one reason is that our training data was acquired on a better microscope than our testing data so now we can actually do that 3 view sectioning at the low resolution that were stuck with because of sample damage but we can reconstruct it and get high resolution data and this applies as it turns out the labs around the world this is data from another lab from a different organism different sample preparation and it still works and that's bonkers usually in microscopy and a lot of cases in deep learning if your training data is acquired in a certain way and then your testing data is something completely different it just doesn't work but in our case it works really well so we're super happy with that and what that means is that all the connect Thomas researchers in the world who have this low resolution resolution data exabytes of data where they can't see the synaptic vesicles the details that actually underlie learning and memory and fine tuning of these connections we can apply our model to their data and we can rescue all of that information that they have to throw away for the sake of throughput or sample damage or whatever the other thing I'll point out is because the data is so much cleaner autoencoders units they intrinsically denoise data so because the data is cleaner we can now sigmund out so much easier than we ever could before so in this segmentation we've identified the ER mitochondria pre synaptic vesicles and it was easier than it ever was before so not only do we have better data we have better analysis of that data than we ever hoped to have before of course even without Gans you have to worry about hallucinations false positives so we randomized a bunch of low versus high resolution versus deep learning output and we had two expert humans counting the synaptic vesicles that they saw in the images and then we compared to the ground truth and of course the biggest thing is that you see a lot less false negatives we're able to detect way more pre synaptic vesical than we could before we've gone from 46 percent to 70 percent that's awesome sorry we've gone from 43 percent to 13 percent that's huge even better than I just said but we also have a little bit of an increase in false positives we've gone from 11 percent to 17 percent I'm not actually sure all of those false positives are false it's just that we can we have a limit in what I ground truth data can show but the important thing is that the actual error between our ground truth data and our deep learning data is on the same order of magnitude on the same level as the error between two humans looking at the ground truth data but I can tell you that there is no software that can do better than humans right now for identifying pre synaptic vesicles so in other words our deep learning output is as accurate as you can hope to get anyway which means that we can and should be using it in the field right away so I mentioned live imaging and culinary versus biological science so here's a cancer cell and we're imaging mitochondria that have been labeled with a fluorescent dye and we're imaging it at maximum resolution and what you can see is that the image is becoming darker and darker which is a sign of something we call photobleaching that's a big problem you can also see that the mitochondria has swelling they're getting stressed are getting angry we want to study mitochondria when they're not stressed and angry we want to know what's normally happening to the mitochondria so this is a problem we can't actually image for a long period of time with high spatial temporal resolution so we decided to see if we could apply the same method for a live fluorescence imaging so we understand pilatus cope and then we use deep learning to restore the image and as you can see it works very very well so this methodology applies to electron microscopy for connectomics it applies to cancer research for live cell imaging it is a broad use approach that we're very excited about so in conclusion it works and I think there's a lot of exciting things to look forward to we didn't use the no Gans approach we didn't use some of the more advanced ResNet backbones there is so much we can do even better than this and that's just so exciting to me and I just want to point out quickly that our AI is dependent on ni natural intelligence and this would not have happened at all without Jeremy without Fred and without Lin Jingping who's our image analysis engineer at the Salk Institute and I also want to point out that all of these biological studies are massive efforts that require a whole lot of people a whole lot of equipment that contributed a lot to making this kind of stuff happen so with that I'll hand back over to Jeremy and thank you very much when we started fuste I a few years ago as a self-funded research and teaching lab you know we really hoped that by putting deep learning in the hands of domain experts like Erie that that they would create stuff that we had never even heard of solve problems we didn't even know existed and so it's like beyond thrilling for me to see now that not only is that happened but it's also helped us to help launch the WIC room a I medical research initiative and with these things all coming together these we're able to see these extraordinary results like like the ole defy and and like PSS our that's that's doing this this world-changing stuff which you know it blows my mind what folks like early and Jason have have have built so what I want to do now is to see more people building mind blowing stuff people like you and so I would love it if you go too fast dot a I as well like these guys did and start your journey towards learning about building deep learning neural networks with play torch and the FASTA a library and if you like what you see come back in June to the FASTA a website we'll be launching this which is deep learning from the foundations it's a new course which will teach you how to do everything from creating a high quality matrix multiplication implementation from scratch all the way up to doing your own version of do defy and so we'll be really excited to see that to show you how things are built inside the faster I library how things are built inside pi torch itself it'll be deviant digging into the source code and looking at the actual academic papers that those pieces of source code are based on and seeing how you can contribute to the libraries yourself do your own experiments and move from just being a practitioner to a researcher doing stuff like what Jason and we have shown you today doing stuff that no one has ever done before so to learn more about all of these decrepit fication techniques you've seen today also in the next day or two there will be a post with a lot more detail about it on the first day I blog check it out there check out the Olaf I on github where you can stop playing with a code today and you can even run it in free Kona lab notebooks that jason has set up for you so you can actually colorize your own movies colorize your own photos maybe if grandma has got a great black and white photo of Grandpa that she loves you could turn it into a color picture and and make her day so we've got three minutes for maybe one or two questions if anybody has any their questions okay one question yes okay thank you it looks like it works very reliably with the coloring of course like there could be mistakes happening and what about the ethical side of that or changing history kind of a thing and this is why we mentioned it's it's it's an artistic process it's not a recreation it's not a reenactment and so if if you give granny a picture of grandpa and she says oh no he was wearing a red shirt not a blue shirt so be it but the interesting thing is that as Jason mentioned there's no temporal component of a do defy model so like from frame to frame it's discovering again and again that's a blue shirt and so we don't understand the science behind this but we think there's something encoded in the luminance data of the black and white itself that's saying something about what the color actually is because otherwise it just wouldn't work so like there needs to be a lot more research done into this but it really seems like we might be surprised to discover how often the colors are actually correct because somehow it seems to be able to reverse-engineer what the colors probably would have been like even as it goes from like camera one the camera to of the same scene the actors are still wearing the same College pants and so somehow it it knows how to reverse engineer the color any more questions yes one over there I find that super-resolution image is really interesting so there was one that's showing with the cells and how we can make it more detailed with you know details on the cell so there it was side by side with the ground truth data and I'm seeing that the one that's created by the super resolution is actually it's interesting because obviously it's a lot better than the crappy five version but it's untrue on some levels in terms of the cell wall thickness compared to the ground truth so my question is given that's kind of AI is you know make a more detailed and making stuff up basically how can we use that right so that's why I'm airy and his team did this really fascinating process of actually checking it with with humans so as he said that they actually decreased the they decreased the error rate very significantly so that's that's the main thing but did you want to say more about that I mean I think you said basically what I were what I would have said which is that we tried to use real biological structures to quantify the error rate and we used an example of a really small biological structure that you wouldn't be able to see and low but you can't see in high resolution and we found that our error rates were within basically the gold standard of what we would be getting from our ground truth data anyway and partly what saying is perfect I think it will be come better especially working with Jeremy and it's already usable apparently this is coming out of the out of algorithmically speaking the the loss function can actually encode what the domain expert cares about so you pre train a network that has a loss function where that feature loss actually only recognizes something is there if the domain expert said it was there so there there are things we can do to actually stop it from hallucinating things we care about thanks everybody thanks very much [Applause]