WebWorkers: Code Session - Supercharged

Key Takeaways

it is thursday it is 2 30 p.m utc it is time for supercharged hi i'm sturma hi i'm madiko and we are going to do some web workers let's go [Music] so web workers usually on supercharge we do more like leading edge technology things that only work in chrome canary with like the web platforms flag and able but today with webworkers we're actually using something that has been around for years yeah and many browsers i think even internet explorer multiple versions have support for web workers pretty much so you can follow along in whatever browser you are today and that's new so mariko welcome i'm very happy to have you here thank you before we dive in into code i think we should make sure that everybody like i'm not sure everybody knows what a web worker really is yes and even i'm not sure if i know the right definition of it why can you maybe bring us all oh okay cool sure um should i do the picture if you want to so what would you describe with volcker if you if somebody asked you like one line i would basically say it's a a actual second thread for javascript like proper parallelism in javascript and then you have some very simple ways to communicate with your original main thread as it's called right so uh oops this one um i had that um explanation given to me and i was like what do you mean thread javascript is a single thread exactly and that's what already confuses people i think yeah yeah so when i so i discovered webworker when i was trying to make the very similar app that we're gonna try to make today uh and i wanted to make it performant and i discovered booker and i was like what does that even mean it was like really confusing so i pictured it for you i drew a picture you are known on the internet for drawing explanations of things so i'm very happy that supercharged gets to enjoy these drawings of yours as well so imagine that your browser is a planet earth something you might know when the garbage collection is you know the the garbage day schedule because garbage collection you know all the elements on the true planet because html over there and the web worker you can think of it as something like internet international space station okay it's something that you create from the browser the planet browser you launch it i like planet browser out of the world it's outside of the planet browser but then from planet browser you send the data to do all kinds of scientific explanation exploration that happens so they can do a lot of tasks but because it is outside of the planet browser they cannot touch the elements on the thing so in the web worker that makes a lot of sense it does not have a lot of uh it does not have access to window objects it does not have access to document objects basically you cannot do anything that touches what's on the screen but you can do a lot of javascript true and that's what we're going to use today yes and so this was the first explanation probably of many to come because we are introducing some new concepts but if you have any questions you people watching us live in the chat um ask them because i'll be only throwing the chat or the hashtag supercharge on twitter or my mentions on twitter and i'll try to forward them to mariko because she's the expert in all these things and she will hopefully answer your questions even if they are about coconuts jake thank you for making sure that people believe us that this is actually live so um coconut i'm sorry he jake said in the chat that i should say something about coconuts to prove that this is life oh yeah by the way somebody who doesn't know jake is our colleague who's literally you did a live stream with me two months ago three months ago about service worker who would have thought okay um so as i explained in order to like utilize webworker you kind of have to have like a lot of javascript to do like something that is not changing the visual or like making a dom element it's a lot of javascript so i have a plan for today that um kind of thought of so like what if we so i know matt did last supercharged like image processing yeah we did webgl but um i don't know how to speak webgl but i know how to do 2d canvas so what if we do a lot of image processing in 2d canvas right and you do basically like image filters do a lot of um using the bits thing and then that involves a lot of the bits thing nice thing yeah the thing the picture if you don't know what you're talking about i recommend you watch the previous live stream after we're done like watch us first then watch the other one because we have many live streams on this channel which you also should subscribe to so here is here is the very simple diagram what's in my head okay this is what we're gonna build yeah so i'm gonna have an input button that opens up the file dialog so it's like i want to edit this image right okay that is loaded into the screen and then when that gets loaded into screen it does something let's say like it's kind of like an instagram filter right like yeah you know let's make it like a black and white or um like you know anything normal processing yeah and then displayed it on the screen because i suspect that doing those filter things involves a lot of computation and maybe it's a good example to put it into what workers so let's get into first making that application before even we talk about yeah let's go for it all right so i have index.html and i started a simple server that is serving that file so let's okay just let's just see like you know i'm not lying uh let's say hi and then if i reload it it just shows up as a high so that's working so for now let's just say we're done see you next time supercharged oh you're even on brand that's so good emoji and then no it's one word thank you very much for reminding me so as i mentioned i need to have input button so input we have the first question for you on the chat what do you use to draw your diagrams i use ipad there you go apple pencil okay do they do they care about the app that i use there's like a bunch of apps that i can recommend later um if you like wants to do it yeah speaking of follow morico on twitter as well yeah if you get one or more have more drawings that's the place to go we have the link in the description cool so when you have an input element you can specify what kind of input it is and in this case we're going to use file and then when you were doing the file input you can also specify what kind of file this input accepts so accept and this time we want to only deal with images like you don't want a user to accidentally json file take some file or something and just like or everywhere like let's like you know do that so image alps image and then let's say star so that like in png jpeg bitmap anything accept it and just have it also use the mime type not the file extension right okay um id yeah input because people someone brought up that uh we are missing some meta tags in the header it that is a good opportunity to mention this is not production-ready code this is only supposed to work on marico's screen just now it's more about the concept that she's going to use yeah and explain to you you can then later on go to our repository on github grab the code and add all the meta headers that you want prs welcome that's that's a good that's a good uh pr for pr exactly so uh choose file or something right so if i have that well i should have four inputs and then let's read all this file and then i have file input it's right now it's very small so make it bigger yeah if i click it the file dialog opens up and then i can select it image okay and then open it but and then right now nothing happens right now why would it you haven't written anything nothing you're in it so let's before dive into it so do you have this thing where like you start doing this and then like unless you style it somewhat nicely you can't really move because like your motivation is not no i'm the exact other way around like i hate making my things pretty i just want to get the core functionality done and then hand it off to someone else so as you can see i'm a visual person you don't say the person who draws on twitter i mean i'm not like no way like a designer or anything but like i'm definitely drawn to like visual like uh shiny things so let me just style it quick yeah go for it i mean um i'm just going to style it bluntly um you can take questions or i guess i can actually first of all i have a question for you oh a question for me from i want to have a id selector on my css yes this keyboard does not seem to have a ah so you are actually usually in the new york office yes not here in london with me she just came over to visit specifically for this live stream specifically for this live stream um so she's not used to a uk keyboard shift n3 is the combination for wait oh that was caps there we go i'm sure what is it oh on the american keyboard you have a separate key for the hashtag right uh well the where the british pound signed is it is already listed as a um i mean pound but like pound meaning like a hashtag not our currency words are confusing all right keep styling styling um i wanted to quickly answer tim's questions on the chat on the video as well because it is interesting we have been talking about modules on the chrome dev summit pretty much recently it's been on the top of the topic list for many frameworks switching over to modules or supporting modules and since we're usually going to use workers today it's probably important to mention that no browser yet has support for workers for modules in workers the syntax has been specified how to load a module in a worker but no browser has actually implemented it that's something that i guess all buzz have on the roadmap but i'm also assuming it's not a very high priority for most browsers because very few people actually use workers which is what i'm hoping to change with this live stream so um maybe after this we can exert some more pressure on the browser people to speed the process up a little bit ipx so we have some more colors yes let's make it font size bigger i never use the em type maybe i should use it more so yeah that looks good so what i did was i hid in the like a default input button ah and then i made the label part as a clickable oh that's a good old trick it ha you have to be careful with accessibility on this one so i i prepared for this i care about this thing i do front end development you go and explain this so uh right now if i tap it i can't get to the input element which should be able to because it is input element yeah and but the one that is focusing input element is basically there was still is overflow here and it's completely hidden and it's just a display is you know taking over practically invisible right so i need to put a focus uh to this element so again uh also when the input element is focused you're going to change the style of the label tricky because what a lot of people do is actually display none the input field and that is very bad for accessibility because then it becomes non-focusable for people who use screen readers so this one is a different trick where you make it invisible by giving it a zero by zero size and then making sure the styles are propagated from the input field to the actual so i kind of cheated because i couldn't remember the order of like pixel order or style and thing but let's see if this works so if i tap it yes nice and then tab out so now i can select it so now i can see this is selected enter and then file okay and then select it so cool now we have something something to start with and that's pretty much the only ui element we will have on this sounds good uh now we have to do something with the image yeah let me let me just change the color this is not color uh let's say uh f3 s3 you're just making those colors up aren't you yeah blue is not cool yeah let's go with that okay so now we get into javascript bar so the input element the native element will let us have the model for the file and then let us you know give the information about the file that we selected but we need to access that and we need to write a javascript for it so let's write a script and apologize i'm going to put everything on index html yeah that's how we wrote here no worries about that so let's do the chocolate that's autocomplete don't you think my sublime setting is superb we haven't had sublime on this team in a long time have you heard of vs code yes just to be clear before a flamewar starts in the chat again your editor is fine as long as you press a key and that character appears on screen you're set for web development i don't care about anything else so if you use webstorm if you use sublime if you use notepad plus plus it all works for web development so don't ever get feel discriminate if you use a different editor from what you see other people using yep so while someone is putting the disclaimer about the editor i just put the um i like the shout out to the old jquery syntax oh so this is what i do so whenever i do um reference to the dom element or html that is already accessing the screen then i even though i don't use jquery i just put dollar signs so that like mental notes i like it you know i can see like which element still which element object still lives in the memory and then which one is actually like displayed on the screen so in this case input is already on the screen so i just put the dollar sign and i add event listener so change event is fired whenever uh user selects different files yeah so another apology i'm gonna console.log the hell out of this code because that's how i code that's how the professionals do that i think that's perfectly fair so let's see what kind of event data that we get so let's tab select picture oh oh oh oh i need to do that dev tool don't i inspect console something is wrong oh yeah oh capitalization of that function call is always weird consistency something that we don't have on the web all right so then usually you know usually it's like in the target where you want the data to be right yeah and i just happen to know that if it's file uh in the target there will be files ah so you actually get and that's where all and that's an actual file type so something you can yep so the name of the file that's actually interesting because um by the user selecting a file is implicitly granting you ask the website the permission to read that file so we are already done we can now read this file and work on it which is what we want so that's cool so let's say the file that we want to access will be const target dot files and then the first one in there now now we have access to all the information but we have to actually load it in the browser yes we only got the metadata yeah it's like object representing the file not the actual contents browser hasn't figured that out so in order to do that we make a file leader so oh i have touched on the file reader api not on stream but i have used before it's not not the nicest api to use but um i guess you'll see now so file leader is name of the as it suggests it just leaves the file and it comes with a lot of method to read so one way delete a file is um bead as data url which basically returns a basic if it's image basically 64 representation of the file okay so i created the file leader here i guess it's interesting to note that like most of you probably have seen data urls somewhere uh like when you inline a little icon in css i think in most browsers these are limited to 2 megabytes so if you want to use in this case we might want to load a bigger image later on so that would probably not be a good choice for us but we're going to start with it yep so and then once you leave the file you have to do something once the leading is done because it is you know done what do you call it like asynchronously yeah you have basically have to listen to onload event so yeah i would add a file leader dot event listener oops and then [Music] load so somebody asked why are you using const is it necessary that's a good question i don't know i started using const i started basically ditch every var and exactly so i think on some stream previews we talked about this there used to be everything in java used to be var and now there is let and const with es2015 um and i mostly use cons because it signifies to the next person who reads my code that the value of that variable is not going to change anymore it's just something that makes like you have to juggle less state in your head when you read the code so i try to only use constantly and also the advantage is they're block scoped rather than function scope which in javascript is the default and very confusing at times yep so i just created a file leader read it as a data url pass the file and the file leader i added an event listener for load event so now if i load it i get the event back and inside of the event again within the target i get the result here which is a basically for the presentation of the image that we look beautiful i can practically see the image so let's say constant image data well that's going to be confusing so ing is e dot target dot result right so now we need to somehow put this data into the browser so that yeah i mean browser shows an image right easier way to do it is making an image object so yes again image new image ooh you made a comment recently about my code i did i personally am a big uh proponent of using semicolons in javascript mariko's not i disapprove i don't have any uh strong philosophical um belief or anything i just worked in the company for two years that and i think that's the thing it's mostly required the way you start coding yeah is the way you're going to keep coming if somebody makes me like you know you should like let's say like our team decides like everybody should write like semicolon like i would happily adopt it but like there's nothing no mandate here so i mean we have we have a google eslint configuration which will demand semicolons but you know i mean i i have esl on most of my projects but um for this supercharged it's a production ready code so who cares yep so i created an image object on 31 line 31 and then now i assign this the b64 representation of that oh let's say let's see this base 64. and then src much like the attribute on the image image tag 64. and now once this operation is done then the load event will be fired for this image object so once that's done i need to append that image object into the screen right so the let's check if this works so document dot body or append child image and let's see if this works click click it to cook yay we just loaded an image with a file selector yeah that's pretty damn cool yep for the reason that comes in later i'm just gonna wrap this in div so that the again you can always add this i just i just need to like kind of like i'm not a designer but i need to style something you know okay oops all right so now that we have a user select image data image is you know grabbed from the memory put it into browser let's put it into you know so that we can apply the filter yeah the original plan right so i guess you can't do that on the image element but you need a canvas yes so um image element is great but it doesn't have a final way of like tweaking things i i know you can use a css to put like a filter but that's not the point today but like you know yeah that that is like very specified thing and that's not the point like you know that is like offloading that's our math problem to css like we want to do the math on the javascript so that we can use webwalker so let's make a canvas where we just recently published an article about how expensive a blur is and that if if you animate a blur you will probably see a lot of jank um i'll link to it in the description later on if you want to read that and you should because i wrote it and i would appreciate you reading my articles i did it was it was wonderful article um yeah so that's and the point is today we want to use a web worker eventually to offload heavy work but that article basically explains why a blur is so computationally intensive even though it is like a simple effect in terms of what it looks like context 2d so i i suppose i should explain this part if like somebody have not done canvas work so canvas much like any other htm element you just create a canvas tag right an element yeah and then you can reference it by id or class or however you you type with the id oops what the v is missing what ndi there you go so so that's what's happening here at dollar sign preview however in order to do a lot of like computational work on canvas you have to tell kind of like um define the language that they are speaking so going with like a planet explanation let's say you decided to i don't know you're a master creator so you decided to create this planet right and then you decided as you're known to do on friday you decided to speak um i'm going to speak this one language which is webgl and then last time yep so you would say you know the webgl use specific thing called shader and then that's how you communicate with this canvas planet in this in my case i want to use 2d which is another set of language and that's called context so whenever you define that language part of the canvas you you call get context and if you're trying to do the shaders you would say webgl but this time i'm getting the 2d context and basically preview con ctx context context is the one you interact that's the actual manipulation api yeah um so like you know the canvas you can maybe set the width and height of the canvas and that's about it that's pretty much it like if you do webgl canvas you get you know the all the functions to load triangles to the gpu to upload textures and in this case you get functions to draw lines draw circles whatever you want to do as a 2d operation yep so instead of loading image as in child element now i put a image into the canvas and call the method the help method that canvas has which is draw image which basically takes an image element and put it into the canvas so let's do the thing and see if it shows up oop hello oh okay when you're drawing the image you first argument is the image object that you're putting to and then you need to give the location where you want to draw i wish that was just half sensible defaults but sadly this function does not so i want to draw it four sides so i would say the top top left corner has to be zero zero x y so zero and zero and hopefully this should draw a picture into canvas so i mean part of the image is missing but you know that's good so it's inside of the canvas right it's not an image element anymore um so i know why this is missing because we only created canvas and the default size for canvas i think it's 200 pixels by 400. if you thought it was 300 by 150. oh maybe like an iphone i only remember one by two yeah that might be true yeah so we just need to set the size so in here i will say preview with the name of the compass so ah so you're going to resize the canvas before drawing the image which makes sense um because a lot of people apparently are very interested in the soldering iron it is real it's just not switched on and why would it be it would just be dangerous you wouldn't wha so i mean why the soldering iron oh yeah yeah it is real i'll tell you that there we go so now shows up on the great screen all right um let's just make maybe i'm going to do really bad technique of putting vr i am disgusted because all i care is how it looks and that's how bad websites are born welcome to supercharge does that even work yeah of course later if i was pushing this i will clean up with nice putting and everything and check with this designer but for now right uh what i need to do oh yes so now that the data is in canvas we can also get the data out of canvas we can read back the pixels yep so let's just um switch up this thing and make it like a function so apply filter as a function and then we will have a separate function i was about to say it's a function that you're going to write it's not a function yet right now it is oh i can type [Music] you have the excuse of having to type on a foreign keyboard not only in terms of layout but also on on our streaming laptop which you probably see this is the thing it's not showing up but the enter key is so small on the british thing it's bigger than on them i guess it's it's taller but but but narrower yes yes while on the american it's like wider but and whatever you're used to you will totally mess up once you're on the other keyboard it's horrible when i go to america i just i can't type anymore and for you it's basically the same just now so canvas has this um very handy method called get image data and put image data you wrote with here i think oh how did this work just by coincidence probably love it maybe yeah all right okay all right apply filter in order to update thanks to the chat in order to apply the filter you need to get the chunk of pixel data and then you need to analyze those pixel data and then change something about that pixel data right now we're only interacting with the html element we don't have that data element right so in order to do those canvas has a method called get image data and put image data yeah so the easier way to explain to me is like this so imagine this is a canvas right you have a canvas you go through context right context dot call get image data yeah and then you get on that uh parameter you can specify where that should start and how big so you can like you can get like a sub rectangle of the entire so you have like a cool four arguments and then you get data and then make it pretty the data pretty and then you pass that data into put image data method and then put it back into the screen okay so that's what we're going to do so you're going to take the image out do some processing and put it back in and that's how you do effects yeah so let's see what the data that we've been talking about look like so say i already have a context so play view context dot get image data and zero zero and it would be image.width oh we're using that that makes sense i hate yeah yep and let's see um so what exactly is like what is the image data is it like a special type or is it just an array or so image data is a special um i guess oops it's an error is not a function i think the eye needs to be oh yeah so it's a special what do you call it object data type type so you can see um okay keep going yeah they have three data in there width and height and then inside of that data data is a array and the array typed array rate yup array is this is the feature i love devtools where you can like how many 360 000 element array and you can inspect every single one of those cells so you can see here is the data about a whole bunch of numbers okay but those are just numbers not really like how do what what do those numbers represent so yes this represents the image that's on the screen here it's like already on the canvas but it is a challenge in the certain way and it's a one-dimensional whole bunch of numbers yeah and the way i think about it um is so the the it comes in the set of four so from zero to three is a data you need to represent one pixel ah okay and you might have guessed it it's r g b and alpha that's why every fourth value just now is two five five because we don't have any transparency going on yes okay so every group of four forms one pixel and then these four elements in individual groups from the r value the g value the b value and the alpha value as you might know from css so you can probably handle that one thing that changed from the css is in the css if you do lgba to set the color the a the last value would be zero to one but in the canvas case it'll be at zero to 255. ah okay yep because the the thing that we're using it is 255. so um you may ask like but it's flat how do you know where in position that that sounds like located right so when you are writing letter to somebody and i don't know if you do this in german i suppose you grew up in germany so in japan if you're writing a paper or a letter you would get this graph paper it's like so each square is for one letter right yeah so in my head i'm thinking in the one dimensional right way of like hi sarama it's like one dimensional i wouldn't think about like here's gonna be line break or anything right that makes sense i am going to write it from the top uh left corner say hi sermon and then once i run out of the space i go to next flow and then fill in what's in my head as a one-dimensional data right it's the same thing so from here the first four is the top left corner and then next one is the the first low but the second from the left so it goes to the right until we reach the width which we have in our image data object so we know once we have 300 times four yeah right we go back to the first pixel the next line yeah all right yep that makes sense because we have the width and height and all whole bunch of data we can deconstruct this okay we know where exactly which which number is four which pixel or which color channel and now we can probably jumble around those numbers and make the image look different yep so let's see um in order to do this ah this is very tedious part so as i mentioned we were just like explaining this we have to do this in code right so i'm smelling for loops yes yes so for loops so for everyone who has joined just a couple of minutes late then we started uh welcome this is mariko hello i'm surma we are i mean we thought we're going to do web workers we haven't even touched the web workers yet so far we have been loading images from the hard disk and drawing them onto a canvas so we then can process them and show i don't know mariko's coding not me how would i know what we're doing um i'm monitoring the chat so if you have questions just put them in the chat and i'll like i'll try to follow along we're very very many people today so it's kind of hard actually to keep track of what where where i stopped reading where i have to catch up um but yeah keep keep sending those questions and mariko will tell you how to do image processing in javascript and then later on what webrockers have to do with it yeah so so okay we're doing the classical yeah if you if you've done dealt with uh converse this is like you know english into your heart that you like ah yeah yeah yeah you don't do those things but i will explain what's happening in a minute there should be apprentices and there should be d so are you sure oh index um and that's probably four right should this be should the that also be a times another plus yes sorry no stay image processing we have done this before i mean not not us but i have done pixel traversion on images before right so what's happening is outer for loop is x so going through this direction and inner for loop is y so going through this direction yes so in combination of those you get to do every pixel every pixel however every pixel remember has a four values so index has to be padded with four yes so if we increase the index by one we have to jump four fields and that's why it's times four yeah okay can i show off my drawing because i'm leaving i would love for you to show off your drawings so here's the visual explanation of what i just said right so the outer loop is the green oh in this case i actually did y sorry it doesn't matter really yeah yeah so the green one goes down from top to bottom and then orange one is the x and go from left to right and then inside of that you have index and you get to access each pixel by index plus one index passed to index for three and yes to make something cool thing happen once you have access to those things we're gonna do the cool thing as well don't you worry so let's see uh let's say um changing opacity maybe yeah that sounds good the last one is the opacity so alpha is image data that we have inside of image data we access data and then inside of that index plus three is where all of the alpha channel lives and then right now it's all 255 so i'm just going to set it to 20 72 or 27 and then let's see if i actually get to oh i need to put back into the data so [Music] okay so now we have changed the fourth value in each pixel pack the a value 247 which is basically 50 opacity and it's plus three because as always in program we started zero so zero is the first pixel and so yep plus three is the fourth value so i just changed all of the value which is in the imagedata.data right and then i put that data back here on this line so hopefully this picture will come as a slightly opaque nice we just have done image processing yep so we can do [Laughter] we can totally do the thing where um let's see ins was changing the thing i think matt did like a warm yeah he did a warm filter or something so r is a red value so i think what he did was like increase the value a little bit to make it like i think he had like a factor and he would go a little bit add to the red and subtract from the blue depending on how high the warmth value was supposed to be okay well let's just do the simple thing of like red is where the first one so index right basically you're going to make jake's face more red which is what he looks like once he has seen the sun true oh that's a good one and then literally 1.1 or something well 1.2 let's see right so because the first is the red value you're increasing the red value okay so and interesting fun fact because the image data.data is a clamped u and 8 array if we go above 255 it will automatically stick to 255. i love this feature and the same with zero you can't go negative with a clamped you indicate array which makes it super handy for this kind of image data so choose jake's image definitely more definitely more land you seem like nicely tanned color i never do in real life though so that's good yes and this seems working fine yeah but you know in real life the cam the photo we take on the phone is like much much bigger that's true i mean that this picture i think we saw in depth was 300 by 300 uh yeah nobody takes those phones no nobody does and i want to keep like i don't want to like resize it to make it like fast right i want to keep the original size i basically went on the equation quality just because you did some you know just some touch-ups yeah yeah yeah so i have a big file the cat jpeg and once i open it visibly because it's very easy nothing happened oh it just oh so like it didn't not work it just took that long to draw and process the image okay that is a little bit long i have to say and also we wrote this on the main thread javascript so probably the browser just froze yeah so let's like investigate right so uh you might need to help me here because i'm not a expert on the thing but as i heard you just hit record that is correct do something that you want to record and wait for it to okay so devtools is still working it still hasn't appeared on screen there it is stop it and let's see what devtools said death tool says there is one massive red bar at the top you never want to see those red bars because that's what we call jank that is when the main thread is blocked and basically cannot react to user inputs like scrolling or clicking a button um you never want to see those if you do you have a giant problem most likely and have to investigate how to fix them yeah so that screen so i think if you remember paul lewis he always said you have a frame budget of 16 milliseconds you cannot do you shouldn't be doing any more calculation in your javascript per frame than six milliseconds this was six seconds that's too much morico yeah and i uh to be fair like i doubt it will go down to 60 frames per second no it's 2d canvas the image is just that big there's not much we can do about it but we can minimize it so that the user can have a kind of bleak point so that they can interact with oh okay it's like a perception of the speed is faster okay fake it till you make it that's right that's that's what we call performance yes so let's do this the first thing that is very blocking and it's it help me if you can point it the loading the image part um i mean there's the load event oh that's when it's already loaded right yeah a function call that event listener it's real quiet what are you looking for the image loading part i think i suspect this chunk is where we yeah file leader yeah we were leading the file so i happen to know the it's not that new but it's newer than web worker true that i mean pretty much anything is newer than web worker almost so uh do you know what it is that i'm going to use is is it create image bitmap yes yes it is so there are the what do you call an api yes or quit image bitmap which allow you to pass a file data and magically creates an image for you so this will allow us hope also to not only make the image decoding uh be a janky process but it will also avoid the file reader api yep so i do not need a file leader api anymore i don't listen to the thing i don't need to listen to the image uh i still do maybe yeah you still want to know when the image but i can delete the file leader and that's good for sure create bitmap clear image bitmap is a promise based so nice one i'm a big fan of promises uh you call it with then and then function and we can use then it's completely valid saying do you want to do you want to do it this is your code the if i would take over right now the first thing i would have to do is add semicolons to every line we're not doing that you keep coding so bitmap now the return value from this create image bitmap promise should be the image right so you can append that to image.s soc wait no no no you directly go to the camera so let's let's just console like this i don't know what it was yeah so maybe you want to just see what kind of type you get back so let's just do the small one before we go into the thing okay so maze bitmap just has height and width yes done and then i remember this api only exposed width and height to developers right but they can be methods yeah it is a representation of the um data right so i can directly draw that into canvas i think because it's already image and canvas already have a draw image method okay so let's see i take this code here and then move it to inside of them so actually don't need the image event listener yeah i think we can delete it's cool so play with would be the bitmap width okay and then the preview height will be bitmap height and then preview ctx draw image will be the bitmap i don't need a console log and fingers closed this is no needed fingers closed this image will load i have the greatest of faith in you still works you're good to go yep all right so we now have avoided the image element we have avoided the file reader api which is good so create image bitmap um i think the reason it got introduced is exactly for this reason that you can decode an image get get the image data without having to jank the main thread it's in chrome it's in safari it's in firefox it's in firefox but not supported in edge or safari but they have it on their radar as far as i know i'm not sure if they have it under consideration or in development but they definitely know it's a thing and and no and they know it's a thing that developer want so as far as i know in cloam they do this cool thing where the decoding dom image which is the most expensive part they do it off thread off the ui thread so you can see those in a network record start recording do this thing stop and then you can see like less um usually yeah like much smaller thing however this year the task scheduler render a foreground worker now that's a name that's great so that's basically there you go it means decoder so the image decode is happening happening in a worker that chrome spun up which means it's on a different thread so it doesn't block the main thread and that's what we want yeah that's good for bigger images yeah i think as of like a year ago it was um decoding image on the context that called so if you called it from ui thread then it will decode it in the ui thread yeah if you call it from webworker thread then it will record in walker thread and i think paul lewis wrote a helper library to make it easier for you so that of course it takes a little bit fast and i think i'm sure you can find it on our github thing but chrome since then changed it so that like any image will be off the thread okay which is amazing so that was the one half part of it but then we haven't figured out the apply filter part so yeah so the apply filler part is the one that still makes it super slow yeah and not nice yep so apply filter uh we can delete this part and this is when we get to use finally after a good amount of work we now have reached the point yeah where the workers come into play yeah so what we're going to do is as we saw it this part of going through each pixel and doing something was really expensive yeah if you have like the other image is like what 4000 by 3000 or something so it's like 12 million pixels that's gonna take a while in javascript so um so but this part the lucky part is that this doesn't touch the dom this doesn't touch the ui this doesn't have to do anything it just needs to do the math it just has to have the array right right yeah so we can load these to the worker thread so reflector of how worker works more drawings love it so again there's a two threads that we're dealing with one is ui thread or main thread that you call it which is where you deal with the dom and ui and then you spawn up a web worker thread let me go dive into the code yeah i really like the style of annotating the code and kind of like picture the code so let me explain what's going to happen so inside of your application we are going to create a walker which is the newer broker and then you pass a name of the javascript file that you want to execute in the wordpress thread that is kind of like launching a international space station now you are starting we are launching international space stations something you can uh communicate with right so now go back to the walker threat side if you want to use external library it's kind of i think it's similar to a service worker right import script yes it's exactly the same thing the award you use too so just just to be sure service worker not supported in many bro in all browsers like edge is just now working i think we're close to launching who knows uh apple has announced they're working it firefox has it chrome has it workers without the service bit every browser even old internet explorer has these yup just just to be very clear about this this is not like bleeding edge technology this is 2001 technology i think or something so yeah you create a walkerjs with whatever that you want to do you launch it from your application side and then now it lives in a different thread of the memory how do you communicate those data right like in i'm sure we use satellite to send the data to um well international space station in real life i don't know what kind of technology they use do they use internet it's probably lasers anyway so that communication is done by post message yeah and all message event so from the application side you call post message with you know the data you want to pass and then on the worker side you listen to all message event and i kind of like drew it like you do all the clicky thing while expensive stuff happens on the worker side and once the worker is done then they use the same thing post message to send the new data or new information that they calculated back to the main side and now main site can listen to all messages and like if you need to terminate it you can terminate it and i kind of like put it into like guiding that's horrible we have this man-made glorious object the international space station and you want to throw it into a black hole i could have come up with good ending for web workers well i mean the other would have been just sent into the earth atmosphere and let it burn yeah i mean that's sad it's just a black hole let's go with black holes like new frontiers kind of approach to things so let's go with that yeah so let's create a web worker ball car new uh walker and then i need to pass the file so i need to make new file what's the shift key for this every key is different here yeah okay yes so now i can reference this with so that will basically just download the browser will go down on the file create a new javascript thread load that file in the thread and then you have control of a separate thread just for yourself yep so as i mentioned worker takes a listen to on uh let's see the post message first so i created the worker i want to pass the data to worker so that working you can deal with it so walker dot post is message m capitalized again api is so hard for me so if you haven't used post message it's just what it sounds like it posts a message over to whatever you're post messaging to and post message can send javascript objects but only objects so you will lose as it's called the prototype chain so if you have like a class instance it will not arrive as a class instance on the other side but as a simple javascript object but it's usually good enough if you have like just do a little like json style object be like this is what i want to do this is the my parameters and which is you're probably going to do right now um so i created a post message send the image data as i mentioned this for loop needs to go to worker thread so let's go to worker thread local file and this one will come into play later but let's create a message handler how do you do would you do the self dot or message or what do you usually do i would usually just say and that's and i have to blame jake for this because he basically said self is unnecessary so i stopped using self i would just say add event listener message but if you want to use the on message i would probably use self on method do i need to reference anything here or no you can just oh interesting yeah i know right because i think i on the drawing i still use the self dot on message i think i just like use whatever the google developer site explain it to me or maybe html5 locks explain it there's probably an html for frogs out there for there so message event will be fired and let's see so this it's called d data should be the data we passed yeah and i want to see if what what this looks like actually so again console.log the greatest tool let's see that's also something that is good to know if you want to work with workers the debugging story around workers is actually fairly good so console logs from your worker will just appear in your devtools just as well so you will be able to use this approach for debugging index 44 something is not working oh oh oh oh oh because we are not using that image anymore we're directly loading that correct so this should be the size of preview now because we set the width and height there so i can reference it with canvas okay so let's hope this works do the thing there message event and then inside of that i have a data where data is exactly the representation of the image data we dealt with yeah so inside of the worker i get to access data.data.data so this was a big image but transferring it was surprisingly fast why oh well supposed to be this it shouldn't be fast true why was it fast either way so we're so the image data contains an array buffer and array typed array buffer and typed array buffers are javascript version of actual chunks of memory rather than you know abstract objects you can pass around and postman has the ability to transfer those which means instead of copying the entire image it will actually transfer them so no copying necessary the main thread will lose access to this type of buffer and the worker will get access to it right yep so basically like let's say you and i have memory yeah um field right i have this image data here and then i want to pass it to worker worker if i don't on this line 46 if i don't have anything specified this is like literally copied into your field however if i specify um uh argument which we'll get to then it will basically says just to move the pointer it moves the friendship yeah like you lose ownership i get ownership and no copying needs to happen so that's good and it's the weird api where like you have to pass an array it's a bit weird and like specify things so in this case image data dot data is the that's the action dot buffer right because typed array buffer like there's array buffers which are the chunks of memory and then there's the typed arrays which are views onto these buffers because you can interpret the same chunk of memory as like a series of floats or a series of eight bit integers and we want to transfer the buffer not the interpretation of the buffer which is i guess the distinction which it's still a little bit weird but maybe that's a good way to think about it yeah um this is like a little quirky bit that when you're dealing with worker that you have to encounter i guess unfortunately so inside of the worker the image data is the one that we want to access so let's say means data is the data right and then we can basically access uh same thing so let's do the same computation of making a face red inside of the face red operation well every warm filter not the face red filter i like i like face red operation much better as in it seems like better branding to me okay so i have a whole bunch of weird love friends that i need to so let's say width is inside of image data so with w is image data dot width just make it look more readable that's good dot and then data the one that we are actually dealing with is image data okay so now i need to tweak this a little bit ah enter key backslash w height index yes bw [Music] all right it's just data now yeah all right and now so now we copied so we sent the buffer to the worker the worker did the same thing as before like went through all the pixels did all the right manipulation now we need to send it back yep so we need to send it back on a post message so wait so if you are in the other event listener how do i send it to pause message uh isn't that global just post i think in workers it's just a global software possibilities just go post message okay it gets more complicated if you talk about service worker because everybody can talk to the service worker but the server can talk to all the open websites you actually need to define which window to talk to and that's a whole different story in this case we have it nice and simple we just called postman yeah it's it's weird for me to not like live from self but i guess if jake's self is not necessary then it's not necessary okay so supposedly we do all the things and then send the po use a pulse message to send the data that it was passed because we manipulated inside of the data in this is d the data or is g the event because you're not sending back the event and you want to oh that's true so i have to do the image data yes and also have to transfer a bit again right yes sorry this this is where you can tell that the postmatch api is really old because it feels pretty inconsistent with the rest of the web but that is also you know the charm of the webcam like that's how it landed and that's what we got now and it still works so we'll make it work yep so now send it but then we need to listen to it so we can ah this is what inside of doing it would you do it outside of this function yeah you've done it so far why not yeah just do that [Music] listener message and let's see in consistency let's make a d and uh if this pattern is repeating then d data should be image ah we're copying pasting lines now this is proper code reuse and then put that data into the canvas all right i'm nervous let's hope this works yes so it looks the same but something a whole lot different just happened i actually yeah let's let's do the performance check right away so basically what should what is going on now is that we're doing the same thing as before but all the expensive processing is happening in a different thread meaning there shouldn't be any red bars if we do a performance analysis on this code set it done how's that okay that looks pretty short and we can see the dedicated worker thread is what we are using has some work to do yep so function called the worker.js happened in dedicated worker thread okay so previously all of this right this part was clogged inside you can mark regions with shift by the way shift and click shift and click what shift click and hold it you can mark regions and you can measure how white they are oh til performance pro tip so previously this whole chunk of yellow was living in here which means it was blocking users interaction if they want to click on something or like they you want to do another javascript location it was just blocking yeah and now you put it uh done into put it put it into worker thread i mean the ultimate test is now load the big image right yep and see if we have a six second frame or if the frame rate keeps going better right yeah so let's do the recording big image get that cat in there that was also quicker yeah visually quicker so here's an interesting fun fact something i recently learned if you look at the depth at the the bar graph at the top the you know the little mountains thing when something is striped it means it's off main thread so it means something is happening but it's not blocking dimension it's good as you can see there is no red bars because we are never blocking the main thread with this version because we're using decode image get image bitmap create image bitmap create image bitmap and the worker yeah so good yeah so pretty that's pretty much it right do you want to do anything to other things no i mean we're perfectly on time once again as unsupercharged it is common um thank you so much for traveling all the way just for this uh we're gonna walk as miracle promised she is of course gonna clean up the code or just give it to me i'm gonna put it on the github repository which i'm gonna put in the video description once the video is up on youtube um thanks everybody for coming and talking to us in the chat been super active i was really really happy to see that as i already announced on twitter this is the last live stream for the 2017 season it's time for a little christmas break but i'm going to be back we're going to be back in 2018 with many more things i want to try uh i hope you enjoyed this i hope you learned something if you have more questions you can ask mariko not me or me it's okay you can ask me too um yeah twitter or like ask the column dip twitter also go to chrome dev we're pretty responsive there yeah and actually we are the to active person who's you're mostly going to get replies from either mariko or me if you tweeted chromium death if you want to you know stay up to date with supercharged live streams microtips maybe even totally tuning tips who knows subscribe to this channel uh so you get like little notifications when we put out a new video or we go live the next time thanks for watching and i will see you all in 2018 [Music]