Using PyTorch to Help Predict Wildfires

Key Takeaways

hi everybody happy Earth Day and welcome to the pie torch webinar using pytorch to help predict wildfires I am so glad to have you with us today my name is Susan Kaylor and I'm on the AI product marketing team at Intel now in this webinar we are going to walk through how to perform image analysis to predict potential forest fire likelihoods based on the regions of known forest fires acquired via the data set modus now modus stands for moderate resolution imaging spectr radiometer data set we're going to use the intal extension for pytorch powered by one API to optimize and accelerate pytorch based model fine-tuning now the pre-trained resnap model has been adapted to aerial photos we're also going to show you how to use synthesized data that is generated using stable diffusion now this is the last in the series of three webinars hosted by pytorch and Intel that are oriented around the UN sustainable development goals today's focus is on goal number 15 life on land as we go through the webinar please type your questions in the chat and when we finish up with the webinar feel free to go ahead and share your feedback also in the chat section our presenters today are Bob cheeseboro and Rahul n Bob's industry experience is in software development and AI solution engineering for Fortune 100 companies and National Laboratories and has been doing that for over three decades he's also a hobbyist who has logged over 800 miles and a thousand hours in the field finding dinosaur bones and you may remember that Bob actually delivered our first webinar on H dinosaur bones he and his sons discovered an important fossil of the only known crocodilian bone from the Jurassic period in New Mexico they've also discovered and logged over 2,000 bones and described a new bass Mass bone Med in New Mexico now Rahul in his current role for the liftoff program for startups brings his extensive experience in applied Ai and engineer ing to Mentor early stage AI startups his dedication lies in helping these startups transform their innovative ideas into fully fledged Market ready products with a strong emphasis on use case driven practical engineering and optimization I hope you enjoy the webinar over to you Bob thank you Susan I'll go ahead and present uh slides here see if we can get this uh roll just a check you can see my slide my title slide so this is uh we're going to be talking about using a p torch to predict wildfires uh I'm not a forestry person I have no background in forestry I have no background in predicting fires per se but I have quite a background in Ai and and doing Vision analysis and doing U uh classification and and object detection and so forth on on IM um this is the agenda that I'm proposing for today we're going to be discussing the just this intro to the topic of Forestry and fire prediction forest fire prediction and then I'll tell you a little bit about how what you can do if you wanted to uh get set up you know we've got a a free um web uh location you can go and a sandbox that has some very powerful Hardware that you could actually try this stuff out I'll provide you GitHub link so you can actually try the code and then we'll be just talking about some of the conceptual processes what's this life cycle of doing this Force fire prediction including getting the data labeling it and doing so forth doing the training and so forth uh just so you know we're going to be talking about something called fine-tuning I'll Define that a little bit later but uh we're going to be fine-tuning a model with pytorch on Resident 18 it's a very simple uh image classification model and then U I'll be telling you about how we can access the intel gpus on our server and to do that we need uh an Intel extension of a pytorch uh to be able to express and communicate with that GPU and then uh additionally I want to show you how we use synthetic data while we waited for the uh permissions and the the funding to go through from our team to the various providers of data so we were waiting uh a weit period to get the data and be able to use it so we use stable diffusion to get our our um models and our Pipeline and get our all of our code running in anticipation when we finally get the real data so we'll be telling you how we did that and then uh we use stable diffusion uh with that um GPU as well using the Intel extension for pytorch so we'll tell you a little bit about that uh just to talk a bit of background of the forest fires so I know today is Earth Day if you remember there's uh even I think it was last year there was a devastating fire in Hawaii um a lot of people have been impacted uh lost their lives lost lost loved ones in both the um fires in California over the last um decade few years Hawai various places throughout the world it's a very serious topic um so I I want us all to kind of be thinking you know with a proper degree of of uh uh sobriety you know when we're when we're discussing this and so I will be talking about technical details of image analysis and and different things but um really this is um uh something that that we need to pay due um homage to those those people so uh in terms of loss of human lives uh and damage to ecosystems and Wildlife uh this has been a a a an enormous uh period that we're living in you know according to uh Dr NL um Meyers uh he had a quote this was back you know even in 2020 I think it was that he said that just as a prediction that the total damage and cumulative economic loss for the 2021 Wildfire season then was expected to be between 70 billion and9 billion dollar in the US and 45 to 55 billion of those damages would occur in California alone so uh what can we do as community members and programmers and AI folks what could we do to throw our hat in the ring to help push a technology that may be able to solve this so early identification of fire likelihoods is um in my view key you know so it gives you time for remediation steps so uh with that I just wanted to kind of show you just a little bit of a image of an impact I didn't drill in too deeply because it it it can be a bit disturbing but this is just from the paradise fire uh image on the left is from before the 2018 fire Paradise California um and those neighborhoods were lush green you know trees all over and um then after the fire in 2018 the aerial photo aerial photos and particularly when you zoom in um you know it's just many of those houses vast stretches of them were reduced to just foundations and and it was just devastating and so this is just zooming back out uh you know kind of showing the the degree of loss and even the change in the foliage patterns and so forth as a result of that fire so uh if you wanted to play along this is not a a workshop per se but if you you know we'll be recording this uh session uh the slides will be available the um uh data sets and the code is available you can get on our free access to our Intel developer Cloud to try all this pytorch code out yourself so uh I just wanted to give a a nod there that if at some point you want to play along you know do this over the weekend or do some when you're free uh you could play this and you could actually try your handed improving the models trying different regions and seeing if in your estimation if this technique holds water okay so in predict predicting Force fires um all comes down to the data and one of the things I just I did want to say is that using aerial photos is interesting to be able to predict forest fires but um when you use the word prediction you're uh automatic Ally introducing a temporal component there's a component of time here so when you're going to get the data uh one of the things you need to do is find a data source where you can filter on time as well as location and so I've listed uh a number of just possible sources for the data for you to grab I've experimented personally with the Google Earth engine and also with the Earth Explorer from USGS us Geological Survey and the data sets that I've used most recently are the free ones from the uh Earth Explorer but uh the ones I liked better was when we anned up and paid money as a corporation to access the Google Earth engine uh technique I I I like their images better but uh you can get the ones for free and so our our when we did the workshop we we did it really with the free USGS uh images and then uh you know here's just a snippet from the JavaScript code that I used in in Google Earth Editor to uh grab the data and so um this was uh the the uh niop data is basically the um uh agricultural data set that shows um regions of forest and farmland and so forth and so with with this one you can specify a date range to grab images and you can specify a location then the location you can specify uh very precisely and you can grab images from that area but the uh collection of the the aerial photos are from whenever they had the appropriate plane flights and at some time those plane flights may have had cloud cover or whatever so they may um uh do Cloud removal they may take several different flights to get the the correct coverage and then do a mosaic so so uh uh this is the the the problem with getting the data the the better that you can get a really good temporal resolution of your data the better your model would be nevertheless this is what I did and I had good results in being able to predict uh these fires in in in California and I'm going to talk about that so uh as uh Susan mentioned uh we started by using the modus data set and uh this is key the modus data set is a very cool data set I think it comes from um Noah the National Organization uh atmosphere and and whatever you know Noah and and um uh so you can grab this data set now this data set spends more than just the United States it's a a global burn area data set the one that we're grabbing here and so I focused it down to Regions kind of centered around Chico California zoomed in to about the level that you see here on the map and then I specified uh a Time range and so what I wanted to look at was that known fire time span from two 2018 to about 2020 three years worth of of burn area and I just wanted to see the extent to which uh Burns have occurred and that's what you see there in those regions that are are red and so then the idea is that we want to go and Sample randomly sample uh locations within those burn areas and then from those locations we can then grab aerial photos for each location and so what we would want to do is have enough um samples that we have non-burn areas that we know in the future you know look at it say from 2016 to 2017 we would take areas that have not yet been burned but that from the motus stus that we know would be burned in the 2018 to 2020 time frame and so then what we can do is we can go back to the 2016 and 17 uh time range grab images from before the burn AC uh occurred from all over this region just randomly sampling it and then we can use this modus data set effectively to label where we know Burns would then later occur and so this is um just human random sampling so it's not truly random I I was just doing this this is the slides here was when I was just doing my very first experiments with it so I just went in and and in the Google Earth uh engine I just uh put way points down where within wherever the boundaries were for the motus data set and so so you'll see that the orange ones that I sampled there would be sampling burn regions obviously those little orange pins and on the right side the blue pins were where I was sampling non-burn areas all around where the burn areas were now subsequently in the notebooks that I provided on GitHub we do a more true random sampling of these things so it's not human random sampling but I I I express those boundaries and then we do a random sampling uh all throughout but this is just to give you a picture an idea of what you do you sample in those burn areas and you sample in the non-burn areas and now those samples are basically just a GPS location but then what I can do is go back and grab the images for each of those locations and I know in the future you know let's say from the 201617 standpoint future meaning 2018 to 2020 I know what areas were going to be burned okay and so then what we can do is we can use the um uh P to and the use the torch Vision uh libraries to um use say resonet 18 as a model to uh predict and to to basically classify so I'm going to have two classes I'm going to have burn and no burn or I think in the latest rendition I said fire and no fire okay and so um it's a binary classifier so it's very simple so we can get by with using resnet 18 and uh reset is just one of those those um uh sort of famous models that was um uh created several years ago you know I don't know five six seven years ago that um did very well at at doing classification of data and it's this is one of the smaller models uh it has only 18 layers and so but it worked fine and so one of the things that um uh our code does here is that we use something called fine-tuning really what fine-tuning is is that you you you take this model that has 18 layers in it and uh the pre-trained version that you would download from uh say even from the torch Vision if you look at that um uh tutorial that they have uh when you download that you would be downloading a pre-trained version of resnet that was pre-trained on the image net data sets so it has images of pedestrians and cats and apples and whatever in it but it has nowhere in it does it have an aerial photo anywhere in that uh data set that you trained the original pre-trained resinet model and so what you can do though is you can do what's called fine tuning where we we freeze we lock down all those early layers uh so we don't we're not going to be propagating uh gradients or anything back through those lower layers we're just going to train maybe the last two layers uh to fine-tune it for aerial photos specific to our problem and so that's the the fire no fire data image data and so we we can do this both on the CPU or on the on the what we call the xpu xpu is our uh Intel 1100 series Data Center gpus and so uh this is really fast it just takes a few minutes uh to fine-tune our model on those gpus and they're very powerful gpus we call that an xpu on ours Intel has many different kinds of accelerators and so our nomenclature tends to be xpu so uh the title got messed up on this slide a little bit I apologize for that but uh the the what I wanted to do here our goal was to use the 1100 series uh data center GPU on the Intel developer Cloud so what I wanted to do is introduce and the title says uh though you can't see it here this is an introduction to the Intel extension for pytorch and this is uh just some of the information about the Intel extension for p torch that um uh are key so think of it as sort of a wrapper or a name space around a py torch that just makes that namespace now is aware of the term xpu for example and it has a few extra methods to optimize uh for the xpu or even optimize for the Zeon uh CPUs itself also so uh Intel works very closely with py tor.org to um Upstream these technologies that we create for accessing our hardware and Upstream them to the mainline version of py torch and that's our goal and so many of the things that even listed here are now now in the Upstream version but there usually remains like I think the xpu is still one of those ones that we we um use our Intel extension for for pytorch to express and to be able to control those those gpus but a lot of these Technologies for using uh the vnn instructions and AMX instructions we we're constantly working with py tor.org to to push those uh back into the main main line of pytorch that's our goal so this is just a little cheat sheet so uh if you're familiar with using pytorch I assume everybody is you we're here on the py torch. org you know uh discussion uh then you know however you do torch Vision models however you do resnet you just do it exactly the same way uh there's really no difference other than what I've highlighted or I've circled here in blue in order to use our uh uh on a CPU in order to do inference against uh these models with an extra degree of optimization you would add these two lines that I've circled here we would import ipex which is our Intel extension for p torch and then we just say hey take the original model that we have and just optimize it uh you know just call ipex optimize give it a model and now we have a new optimized model that may do do things like reorder ing the the or order of channels uh you know color channels and different things to to really get an Optimum performance uh and then uh on this xpu side when we're T touching the 1100 series uh gpus from Intel then it would be these four lines on the right and this is how we do inference on What xpu GPU for from Intel uh so you would just uh convert your model to be xpu aware you would convert your data to be xpu aware and then you call this optimize step so these are the only uh big things that you have to do to your code it's it's really not that much uh to be able to then communicate from pytorch through our uh gpus our 1100 Series gpus so let me uh just talk about fine tuning just a little bit you know fine-tuning is really where you're going to retrain just the last few layers uh of your model so on the left we have sort of the pre trained um uh version maybe from from um image or from pytorch uh you know the resonet 18 model that we may download and um originally the resonet 18 model had just all random numbers in it and somebody took the the step to go through the image net data set with thousands of images and they pre-trained they trained this thing even from random numbers all the weights and biases in every layer were originally random and then some said well let's train it on the imag net data set and so they trained thousands and thousands of images it took probably I don't know how hours and days probably to do that back in the day and so they um created what we call a pre-trained model so now those pre-trained models all the weights and biases in every layer have um values that when you feed it an image of a cat or a dog or pedestrian or a giraffe it knows what those things are it's able to classify them because it's been pre-trained on a wide variety of images but those images didn't include anything like an aerial photo okay so um I don't want to train it from scratch I don't want to have to go back and spend hours and days you know um maybe reclassifying it and training a model from scratch all the way from random numbers up to a a current model because I've learned that and many of you have learned too that if we use transfer learning or sometimes we refer to this as fine tuning a model uh we can take a pre-trained model like that one that was already pre-trained with with uh image and then we can just freeze all of those bottom layers in other words by freezing I mean we're not going to um allow the the gradients to back propagate we're not going to update any of those values for the the majority of the layers it's just the very last layer the output layer and maybe one layer prior to that that we would unfreeze and allow us to train those weights and biases by allowing the the GR to be back propagated and allowing those view values for those last two layers to be updated and so by that when we're feeding at these um aerial photo images we're training a model just fine-tuning it and it doesn't take very much time to do uh against a data set that I care about and so you could do the same thing you can take a pre-trained imag net model and train it against a a widely disperate uh character characterization or character of an image than anything that was previously even in imag net and when you think of how an aerial photo might look different from an image of a person or a cat or a dog or a bicycle you can see that that uh uh maybe even for your application uh hey I don't have image as anything like what's an image net well neither did I and neither did we when we did this image Nets when we did these Areo photos and yet by just doing fine tuning on the last two layers we had success so at a high level uh the one thing that we need I need you to be aware of is that if you're going to use our uh gpus on our Intel developer Cloud for example uh for free uh what you could do is uh just make sure that your torch version your Pi torch version is um highlevel uh dot compatible you know the same version level as our Intel extension for pytorch version so they they're like twins so you you know keep the version numbers roughly the same you'll see that in the very fine tuned part of the numbers they they deviate a little bit but I've highlighted in green the parts you really want to pay attention to make sure they're the same model and then it's just a simple matter of doing the import for the ipex and then uh what we do in a object-oriented sense uh we we take the uh uh we create a method called two ipex where we do these uh conversions for the model and the uh call the optimizer to um in this case we're we're we're taking the model and we're putting the channels last as just an optimization that we're doing and then we we uh call that optimize step so we take a the original resonet model of the pre-trained version and then we uh pass it through the optimizer step of the ipex and so it's it's fairly simple and it's it's basically reflective of what the simple cheat sheet that I showed you before and then um really from there uh you know it's just matter of doing the the the train pretty much as you would notice that we are saying look if if intel extension for pie torch is involved if it's imported we we got it available go ahead and call this function that says to ipex and this will automatically convert your your version over uh Rahul am I missing anything is there anything you want to add color-wise to this or basically what what you mentioned so just just remember that um we Upstream almost all contributions to PTO mainstream like uh Bob just suggested uh there's around 90 to 99 percentage of optimizations for Intel is already in the pyos mainstream and um what we provide with Intel extensions of pyos is that first the support for um Intel accelerators specifically our gpus today um so if you uh want to use Intel gpus you would use um Intel extensions P to and it will add a new uh name space um into uh pyos then you can use pto. xpu um the other thing is that we do have some specific optimizations for models that might not be generally applicable um those uh elements also we add to uh int extens of pyos for now eventually this would be in the main line also so if you want to get Best of Both Worlds uh the pyos and uh specific optimizations that might be there in uh ipex is int extensive Vos um import it and just call the uh like you could see here in the two ipex function um call the optimizer uh ipex do optimize pass in the model and Optimizer if you're training the model uh if you're just doing inference just pass in the model that's just one line change it would be model equal to uh ipex do optimize model and the data type you want that's basically it and then I'm seeing that there was a question about the the size of the images we use I I answered it so we we use uniform sizes uh 512 x 512 uh the one thing could be an improvement right um for um for an extra thing that you could try out with pyos um it is really it is challenging to do distribution to make essentially match the distribution of the real data and synthetic data uh especially if that data is not part of the training set um so if you want and um uh to generate some image that was not originally used um in the um you know for forest fire for example I don't know how many of forest fires were used for stable diffusion um data so sometimes it could be challenging for the distribution changes if the distribution changes your model fitting would be uh problematic so there is still work going on on making sure uh customizing stable diffusion so not only from A visual representation these images are same but in a probability wise also um whatever image you generate has the same probability distribution as the real image so this could be something you could work on uh to create maybe a tiny phyto space Library um uh for synthetic data Generation Um a lot of people would love that well and two with the real data I know when I was downloading the USGS uh images they tend to be fairly large SAS and so I would um but I've specified kind of a zoom level so you know kind of an altitude effectively and that turns out to be fairly key like you want to get that somewhere near where I I put mine uh in terms of that zoom level when you're grabbing those images but uh uh then what happens is you have a map that's you know really big and so I would go in and I would uh I wrote just a simple little utility to cut out these you know 512 x 512 I tried different sizes and really the size that I chose didn't matter that much um surprisingly but you just need to be consistent so you need to be uniform in in cutting out those images and then there is another question on how many days or weeks were you able to predict Forest FES I think we will go into that detail uh uh on some of those things yeah um yeah that's that's end last going out about uh two to three years uh but uh I'm going to talk about some caveats I'm going to talk about it at the very end um there's uh there are folks that say oh this can't possibly work and so I will address some of those those challenges and bear in mind I'm not a forestry guy and neither is Rahul and we're neither one we've never won either one of us done um forest fire gear and gone out and and fought forest fires uh and we don't predict them for a living but what we do is we're experimenting with uh very powerful uh classification techniques pattern matching pattern pattern finders and when you apply these pattern finders to aerial photos we're finding some very interesting and intriguing results so really what my challenge to you all is you guys go play and maybe some of you bring expertise to this subject that um you could improve it and um so that's kind of the challenge there for you guys this is that's incredibly important Bob um especially with pitor and all the community around it is domain experts can now really easily build uh Solutions where we leverage machine learning to solve problems so we uh we do have internal experts who have uh uh expertise in forestry and we did show some of this code and data generated to them they were really surprised but what what I would love is for the community to go and create a solid solution for open science um it could be forest fire prediction there are many challenges where uh we are data poor uh especially in making sure that probability matches so this would be a big contribution to the community as a whole uh if you create a really fast pyto spased diffusion model uh for solving forest fire issues uh water scarcity issues um even in astrophysics there are uh use cases for this um so yeah I would like to see uh you develop a solution like that and maybe invite us to watch your Workshop yes and if you attended my first workshop on predicting fire I mean predicting dinosaur bones this is almost the exact same model uh it's using different image sizes it's using different um aerial photos with different classifications that we've labeled differently but uh the technique is very powerful and portable across domains but again we're we're limited to the data that we that we get and I'll be talking about that somewhat at the end but awesome there so P Patrick mentioned that um um he um they did a uh similar competition and one in uh they um worked on similar thing for forest fire prediction one a national competition so we would like we'd love to learn more about that and you know if there are any details you could share get a reer or anything that you worked on on that uh that would be awesome for the community yes and if we could collaborate maybe on a on a way to further this you know that would be really cool too Patrick so uh this is just some of the results so um I'm showing both the curves and the accuracy curves that I generated and this only took once we got the gpus cooking on this thing and we we rul did a good job of of doing a fast API sort of approach to uh doing learning rate finder you find a really good learning rate to use uh for this resinet 18 model and so once we did that you know within probably six or eight um uh iterations through the training Loop uh this was converging incredibly fast and so we we were getting these um uh accuracies uh approaching that 90% level now subsequently uh in what we did when I started really randomizing and collecting a little bit more data we got north of the 90 uh% Mark for for accuracy but this is one of the preliminary ones and you can see with the confusion Matrix that you know the the main diagonals were were were um uh High and the off diagonals were low so you know this was just our very first preliminary model uh but uh the subsequent ones the ones that I've posted to the GitHub uh they've improved you know quite a bit but I had this slide from from U our one of our earlier discussions our earlier presentations and so I kept this one but but uh it even from the very Inception from the very beginning we were seeing uh large amounts of success with predicting the accuracy of this and so um I did want to just say something about the stable diffusion thing um turns out that when we were doing this I grabbed those images from uh USGS I grabbed some images uh just as an individual from Google Earth engine and then when in talking to our legal team they said well as an individual you can grab those you they say you can use those as an individual you can use those images for free or whatever but since I'm a member of a corporation um I can't use those for free I have to pay for it and so I I had to talk to our legal team we had to talk to our finance team and so in that whole bu bureaucratic thing that you have to go through it took time to officially get the data to officially be able to use the data and so one of the things that rul and I uh talked about was could we use staple diffusion in the meantime let's create the model let's let's put all the software do all the things all the image slicing getting the uniform sizes all that preo processing stuff done get a training Loop kind of going on synthesized data and so uh we use stable diffusion to generate um aerial photol like images and then what I did is I sweetened it a little bit because obviously uh stable diffusion uh aerial photos don't exist anywhere on the planet okay so they're not going to be predictive of any kind of a fire or any kind of a thing but I just wanted to generate some images that would be aerial photo like but this set predict a different class than this set okay just to get our whole training pipeline and get everything ready to go and so yeah and one thing there what Bob was mentioning so um it's especially important what data you use to train your model or or create and there are um lot of concerns about data how it's being used uh so we go through stringent process on making sure the data is legally um you know we get legal permission uh if there's Finance involved all those things are done um you know clear early and it's it's incredibly important for us as a community as a company and and as individuals so this is one way to while that process takes its time to get the data how to accelerate through an engineering approach where you could bootstrap and try out some of this approaches so you don't you don't start building your model once you get the real data but you could start building the model while you're um trying to get the real data and really see if this would work so that's where it really helped us me and Bob like the we just tried it and we had to tweak some things to make sure the probability distribution uh aligns but just uh accelerated maybe uh a few weeks uh for us yeah so we were actually make real progress while we're waiting for the approval to usual data and so uh what I did is I took these synthetic data images that look like aerial photos but they don't exist on the planet I sweeten them by coloring this one these just show you over here just sweeten these by these are just slight 2% Browner you know a little bit redder a little less green and these were a little bit Greener 2% Greener a little bit less red and so uh for our stable diffusion ones our synthetic data while we're waiting for real data right to just get our all the other software ready to go um we did that and that worked out actually extremely well uh also so I thought I'd throw that out there as as a uh something for you to do uh so that when you're waiting to get your data approvals you you're you're not just waiting and and uh making PowerPoint and there was there was a question on could you provide a sample text prompt to generate synthesized data yeah we would definitely provide the sample text and also the code on what we used to do there are some code that uh we use to um sort of understand the distribution pattern we essentially use tne and some principal component analysis to plot uh and Visually understand you know if this data is in uh the same distribution but that's like generic code you could easily do once you create the synthetic images that particular code we are we don't have it in the repo but we have code for generating the images and also the prompts that we use to generate the images you'll always would like to tweak it and we would really like to see that automated approach even remove that human in the loop were to solve the distribution challenge as well if one of you could do that that would be great we we would love to see your PRS to the repo that Bob would be showing on how to improve this um you know please please go ahead and and we would like to see interesting ideas yep the the code for the stable diffusion model has all the prompts that we used and the uh it was a it was a uh we tried both image text and what uh text image and imaged image we we settled I think on image to image but um uh you know we've tried both so all that stuff is kind of in there and then uh in the GitHub and I'll be sharing the link here directly um so what data are you using in the workshop Okay so in this particular case uh even though I said there were multiple different sources and I I even like the Google Earth engine one better because it had a trer color to my eye it just looked better um we wound up using the USGS Earth Explorer data and we used about 100 images in total and then uh uh I've just explained what we tried with stable diffusion to kind of get our pipeline set up but once we got the real images we were using the data from Earth Explorer and that was some of the results I was I was showing you uh earli with the with the uh confusion Matrix and with our accuracies getting up to about 90 we've since actually pushed that into I think the last time I ran it was either 94 or 95% so uh anyway so this this is uh uh some of the details if you want a little bit more detail I I wrote an article predict Force fires using pytorch you can click on that hyperlink when you get the slides and go go look at that and uh you kind of read it more at your leisure so you don't have a temporal uh video thing that you have to slide back and forth you can just look at an article uh so just what is stable diffusion so you we should probably talk a little bit about that so um Rahul you you want to take a spin at a definition of diffusion or sure these are um stable diffusion generally it comes from latent diffusion models so what we essentially do is that we um in the text to image um we have we start from noise a Goan noise and we condition that that's what you see in step one we condition that noise um using a text um prompt so the text prompt would be converted to the same Dimension as an image using something like a click model if you are familiar with embeddings like clip clip uh what we essentially do is that these these particular models are trained on both images and text so if you convert that to bunch of vectors where you pass in a text to A click model and image to a text uh image to a click model both of these vectors are in the same space so you know okay uh give me an image of a dog and image of a dog both of this would be closer to each other than an image of a cat so essentially the models that we give these vectors will have an understanding of what are related to each other and what are not related to each other so we use a clip model get an output from a text prompt to a vector um use the Goan noise and we schedule each uh time gradually improving essentially we are you could think of diffusion models as um a sculpture creating art where we Chip Away things that's not really needed and get the actual image we don't add add things we essentially remove things slowly uh when it comes to image to image um instead of go no is it will be an image and we'll start with one image uh use a text prompt convert to a uh clip embedding and we schedule maybe 50 or 60 steps there are there are stable diffusion model there are diffusion models stable diffusion is only one kind of diffusion models that you could use to maybe in 10 steps or 20 steps even in one step create an image uh that matches very well with your text uh there's a high level idea there are many articles detailing how it how it works we are not going to specifically into detail of table if you should if anyone wants the details let us know we can add relevant links yep and I see the time we're starting to run a little late so I'm going to kind of go through yep stable diffusion wasn't really the the central thrust we're doing we just thought it was like an interesting bone to throw you guys if if you're waiting for your data um so I'm going to go past some of these and start talking about the uh optimizations so um I mentioned something about how to do the um op in general how to use in extens for pie torch with the GPU specifically what we did in the stable diffusion uh code was basically these four steps and I'm going to keep going through because we spent a bit of time uh just with respect to the synthetic data this is uh basically and it gives you an overview of what you do uh in synthetic Land There is no modus data set uh the the images don't really exist on the planet so when you're synthesizing your data uh you create a random sampling and you you look within a boundary and this is a good explanation of what what I did with the modus as well but this is for the synthetic data um you can sprinkle the the data around randomly and then you can uh count which ones are within the boundary which ones are not you can you know sample those images that you've generated here and the ones that would be generated in the star pattern here I would have salted to be slightly Browner like I described and the ones out here would have been slightly Greener uh just by 2% or so and then we're getting similar accuracy curves from doing this on just the synthetic data and now going back to the results from the real data uh this is what the ultimate map looked like so um uh it was actually very good reproduction here of the the true data the motus data set you can kind of see still underneath as a as a placement guide what I did is I I took every image from every pen whether they're red or green and I fed it to our trained model now okay so now I'm just going to score I'm going to inference with these randomly selected images here and so I take an image I actually literally grab the real image this is trained this is run against the the the model that was trained with the real data right and so then we get a score and so it's basically the score was basically pass through soft Max so it's either going to be class one or class two so the the the greens were the non-fire regions so what these are the pred prics this is what the model predicted it said anything in the Coastal Range over there on the left no fire danger any of the things north of the Coastal Range and between kind of up and the upper left you know those two fire ranges it said no fire danger here with a couple of Exceptions there were a couple of little um pins that it said well maybe that's a fire right there right and so it said that was a fire same thing over by the paradise area and over here in the Sierra Nevadas over by Chico and Paradise um almost anything that was truly later on within the the motus uh range it predicted all those things yeah this looks like a fire to me looks like fire fire fire it mispredicted right here it mispredicted in a few of these spots saying hey I think there's fire here but if you got outside of that region it was predicting yeah there's no fire here at all there's no fire down here so this is just an intriguing result okay it's not definitive proof it's an intriguing result that that's why I want to lay it at your feet and have you guys go and experiment more I did the same thing with some of the fires we had here in New Mexico we had one of the largest for the largest forest fire in our state history I think it was last year or the year before and uh I was able to do the same kind of thing I can't use the California model this model on my New Mexico data the foliage types are different the elevation topography stuff is different but when I sampled those areas around our Las Vegas of fire or cow fire whatever it was uh I sampled the areas that were were not burned the areas that were burned uh from a previous year right previous two years and then I predicted okay this is what the foliage pattern looked like two years before the fire now let's train the model into Mexico and then I was able to predict the fires here in New Mexico as well so it's worked in two locations okay um to about that 90% level so um the New Mexico model probably wouldn't work in Colorado probably wouldn't work in Arizona or California same thing the California model it's kind of specific to this General North North uh Northern California area but uh that model um worked to the degree that I've shown you and somewhere on that uh 90 95% accuracy level so this is uh some of the stuff now I did want to just highlight some of the potential criticisms of the method I presented this Workshop um um a few months ago and somebody in the fire industry they predict forest fires they say oh this this can't work um you know really uh what they do they're in the business and what they do is that they focus on ignition they say there's too much uh variability and Fage you can't possibly look at a at an aerial photo and kind of predict what a force fire is going to do there's just too many variables that you look at and so um I think to a human looking at those patterns that's probably true um and have to take the word for it they have experience in that area their primary focus is on ignition which means that they look at weather data which is out to about two weeks they're predicting essentially lightning strikes you know um thunderstorms um and so that's how they predict um the data sources here uh that I'm talking about can be unreliable when you're using the nyop data uh you know they don't collect that data every single day you know so there's and when they're flying over there may be cloud cover there could be smoke cover there could be whatever there is uh and so they may do a composite um they may collect the data in September in one location in OCT cober in another location uh so the foliage differences are you know there's different foliage and and trees that are grow in California versus New Mexico versus Hawaii versus wherever so these are all known issues okay I get it I get it I I I agree okay um this uh model appears to key in on the colors and the contrasts uh you know basically those Co contrast patterns are going to be indicative of a topographic range the the elevation differences between highs and lows and how frequently that that occurs and so U you know maybe in Iowa uh this isn't going to work you know because uh what we're doing both in New Mexico and in California there's you know large uh elevation gains and losses in a given range however all that said I want you to remember the story of the Bumblebee see according to the theory of aerodynamics and as may be demonstrated through laboratory tests and and Windon experiments the Bumblebee is an able to fly this is because the size weight and shape of his body in relation to Total Wings spread makes flying impossible but the Bumblebee being ignorant of these profound scientific truths goes ahead and flies anyway and manages to make a little honey every day so which is it um is it are are we just uh in the industry are we just um we found a way that we can predict out to two weeks and that's all you can do and there's all these Theory reasons why it it can't work um for predicting later you know two three years in advance or is the Bumblebee really flying right and so that's what I'd like to leave you guys with is you guys go out and catch us some bumblebees and you tell us whether these models are working for you too okay so question Bob uh Patrick had a question how many days in the past did you label the picture as fire uh for example if there was a fire at on 66 2022 did you take the photo of 56 2022 no the the trouble with the data sets that I'm using and this is one of the things I kind of alluded to on this on this previous slide on the caveats when you're using the um nyop dqq Q or even the the the uh um data that comes from the US Geological Survey um you when I'm looking at those agricultural type images it's um I would I would have to specify a date range so I can't I can't just say you know give me the image from July 6th you know 20 16 or whatever uh there weren't there weren't any flights that day or the image that they have there was cloudy or it's just not available and so you have to give it a range of data and this is one of the valid criticisms of the thing is that I can't predict right now I I just can't get the data for every single day to be able to predict with that degree of temporal resolution so all I can do is go from like a two-year period and say well grab me some images in there generally and then grab me some images temporally from somewhere in that two or three year span and some regions that you would go to uh they won't have those aerial photos from a given range you know you might specify a six-month period and there would be no images so you have to open that temporal window up and that is one of the problems right and so um the one of the things that I did not do was kind as you're saying like to be very much more precise about the time frame like what season what what exact date what month right and maybe even other conditions you know where the recent uh uh rainstorms you know whatever it was so I did not do that I just blindly grabbed that time slice from 2016 to 2017 any image that they had from nyop from that range and that was the images I trained on uh and then the way I labeled the images was by looking at the the motus data set from years later 2018 to 2020 and so I would grab those as the labeling mechanism to say okay I have these 20167 images I have a bunch of them which ones were in the future going to be Burns which ones were not and so that's the way I did it and it's it is a limitation that I I can't be as precise as what you're articulating um so if you want get started go ahead yeah here the instruction for the developer Cloud that Bob has given um we do have Fe free Jupiter notebooks available here in the standard account anyone can create an account there and each user gets a 48 gig GPU um and a 112 core CPU with um half a terab of ram um you could try this same Workshop there or there are many examples of gen using pyto uh there as well and then I did leave a a reference here uh for the code and for my media article so if you want to reproduce What I Did you can go here and so um it's this is all in the slides will'll be provided for you the video will be provided and then I just want to leave you with this call to action that uh go try it you know be the Bumblebee go try and you know just throw caution to the wind just go see if it works because I got it to work in two different states and maybe that's a fluke or or maybe there's some Merit so uh you know I'd appreciate it the broader Community going and trying these experiments yourselves so this highly educational webinar and wow it was just so appropriate for Earth Day thank you to the pytorch foundation for hosting us today I hope that after watching this webinar that you're inspired to use pytorch to help solve challenging issues that impact our Earth look for the summary blog of all three webinars in this un sustainable development goals series with pytorch and Intel AI which will be available soon on py torch. org take care and enjoy our beautiful planet thank you