Progressive Performance (Chrome Dev Summit 2016)

[Music] hi good afternoon I'm Alex Russell I'm an engineer on the Chrome team and to set expectations this isn't the talk I usually give so I hope you'll bear with me through some difficult content uh these days I usually put this slide up to start a talk about Progressive web apps which Francis bman and I named last year um Progressive web apps are the culmination of multiple years of my team's work I've been working with Jake on service workers for like guess four years now um and the team that I work on has been designing and building the core technology for Progressive web apps the stuff that you've been hearing about for the last couple of chrome Dev Summits and all day today and probably all day tomorrow and I apologize a little bit uh but not really but as you can imagine sort of building and maintaining all of that stuff and working on the standards for it is a full-time job uh but I'm not going to talk about Progressive web apps today uh at least not directly you see for the past year I've also been working with ta and her team to partner with folks who are about to launch their Progressive web apps to make sure that they're really high quality and this sort of incidental Consulting work has given me a broad view into the practices of many of the teams that are building for the mobile web today what I can say with only a few exceptions exceptions like booking.com and the great work that the flip card team did is that most of us don't really understand how hard mobile actually is I haven't exactly been making friends in the Java script framework Community by saying that sort of thing out loud turns out but as the polymer team will tell you this is basically the PG version of what I've been saying to them for something like a year and a half we had some really tense meetings I kept saying things like it needs to be more asynchronous or look at this Trace you really need to load a lot less JavaScript and 12K sounds good to me actually um or you really need to break up your script so you're not ex executing this entire long block what the heck is going on here and at some point they said we get it we get it just stop telling us what to do and start telling us what goal to hit and this was kind of a breakthrough in the conversation we've been sort of at loggerheads for a while um and so I kind of put my finger in the air and said it would be really great if you could get me something that's interactive in about 3 seconds on a 3g connection on first load and interactive in about a second when I launch it from the home screen uh so the polymers uh went off and they went through the stages of grief we had some denial I'll admit there was some denial uh anger yes bargaining absolutely uh depression uh luckily they sit far enough away from me that I couldn't actually see them sobbing in their cubes um but it finally accepted the challenge and they came back with a purple pattern which Sam is going to talk a lot more about uh meanwhile the rest of the JavaScript Community kind of hasn't internalized the same message to the same degree so I'm here to let you down uh not easily this may be a little bit hard and I apologize in advance but we need to get to the bottom of what mobile actually means and when you see me tweeting things like this this is actually kind of desperate um I have spent years of my life working in tc39 to make JavaScript a better language I have spent countless hours persistently advocating for extensibility to give you more power when you're writing JavaScript in the web platform I've designed features like service workers with Jake and a load of other folks that are entirely predicated on JavaScript in the first place back in the day I used to work on on a JavaScript toolkit with Scott and Steve from the polymer team it's not like I hate JavaScript or I don't like it or I think you shouldn't be using Frameworks I don't hate them it's just that we're really in the midst of a crisis and collectively we don't understand how bad that crisis is if we did we would have already modulated our Behavior so what I'm seeing when I do reviews is almost universally bad news specifically in the context of the rail performance model a quick recap of rail uh basically our stand stands for responding to input in under 100 milliseconds animating at 60 frames a second which means that because the browser has to apply the things that we hand it on every frame we probably only have about eight milliseconds to get to do our work when you think about web VR it gets 60 frames 60 frames a second is hard 120 o um when we're doing background work we need to make sure that we're breaking up that background work into 50 millisecond chunks so that we can respond to subsequent input and stay under that 100 millisecond budget and we really want to complete actions for the user in under a second because over a second research suggests that users lose focus on the task that're they're trying to accomplish if we can't actually complete it we definitely need to acknowledge that work in under a second give users the sense that we're doing something for them so you've heard this number all day today Darren had it in the keynote uh but the double click folks went and did a bunch of work to find out uh what are the bounce rates for sites and does performance matter and the answer of course is yes 53% of users Bounce from mobile sites that take more than three seconds to load you leave real money on the table when your site is slow yet most of the apps that I've traced over the last year have been performance travesties my experience isn't an outlier the last that same report noted that the average mobile site takes 19 seconds to load 19 seconds collectively we're failing Sam's got more data on this uh and I actually don't have time to go into it because we're going to talk about some of the reasons why but I think one of the key reasons that we are not succeeding today to put it kindly uh is a lack of understanding and respect for how hard mobile is I have faith that as a community if we understood and respected the limits we'd be doing much better nobody here wants to make bad user experiences I think part of this is because we're the only platform in the world that tried to take all of our desktop stuff with us you don't take a Java jar that was a swing application and run it on your Android phone you don't take a Mac universal binary and run it on your iPhone right you don't take a win32 app and run it on your Windows Phone everybody else switched their tools when their form factor and their constraints changed we didn't we didn't make that switch and the proof is in the pudding it's in the traces that I'm looking at every day but that's not the only reason why are most of the popular Frameworks and the tools that we're using using the tool chains that we all wind up setting ourselves up with unacceptably slow by default why are our tools producing such bismal results it's not like we're bad people who want bad things for users I think in part it's that you all aren't actually developing on mobile phones Paul showed you some really great Dev tools they're awesome uh so but I want to understand who uses Chrome Dev tools to sort of get that respons of view and understand uh how things will look on a phone okay uh now keep your hands up if you're using webpage test for testing on real devices oh I like you some of you are liars um and keep your hands up if you use Chrome colon inspect over USB to do real onice debugging okay who's done it more than once okay that's what I thought um it turns out the emulation is nothing like real devices Network throttling CPU throttling they're all kind of a fudge they're better than nothing please use them please set them on by default please use them all the time but they're not the real thing not even close let me give you a quick example this is the io 2015 website which was a polymer 0.5 Progressive web app it was super bleeding edge it had push notifications which we I think we had launched in Chrome like the week or two before there were bugs um it was amazing uh on desktop it felt super fast on this Wi-Fi connection we get Dom content loaded and it a meaningful paint at about 700 milliseconds onload isn't far behind and that starts a nice SW swooping out animation which is smooth most of the time uh overall we spend about a half a second in script which is well under our budget for actually getting a good responsive experience and we get interactive content in about 4 seconds including that long animation this is a really great experience on a desktop class device which is to say my MacBook Pro this is the same site running on the same Wi-Fi network on a Nexus 5x Dom content loaded doesn't show up until two seconds onload triggers at the six second Mark which is where that animation starts part of that delay is down to this huge honking script ofal that's locked up the main thread for nearly two full seconds script execution Bloons up to four seconds in total and for all of that work we still don't even get smooth animations look at all those long frames content doesn't become interactive until seven seconds this is what TTI will tell you in Lighthouse and this is not acceptable ouch so what have we learned traces from real mobile devices are a harsh master um and when I show folks their apps on real devices most have the same reaction and in fact I do this a lot they're shocked at how slow the median mobile mobile CPU actually is not the iPhones in their pockets they don't really understand the difference between desktop and mobile discs in storage and most of us are super ignorant about how crazy mobile networks are in the real world I think we theoretically know at some level that they're bad doesn't begin to cover it they're so bad it's important to understand the depth of the deficit though so we can start to adapt I've been letting out down a bunch of Engineers for the last year hard soft I've just sort of been like eing them into this this sort of you know grief curve and then hopefully we get out the other side and it it goes well but you have to put in the work but unless we do that work unless we change the way we're working the web won't work for the next billion users not practically speaking so something I say in basically every meeting is that the truth is in the trace and by that I mean Dev tools and chrome tracing attached to real devices nothing else cuts the mustard it doesn't get you there so this is what's sitting on my desk on a typical day and aside from the pixel XL that's on the right hand side all those phones are less than $300 new the folks at Conga report that the most recent commonly used phones in their Market are in the sub $100 range new I carry most of these phones around uh in this bag this bag is in my bag my other big bag uh with me all the time um and these are some slow phones um and I carry them because I have zero faith that any I trace in Chrome on the desktop is going to be like the real world unless I've put it on a phone and emulated 3G and we'll talk about 3G I don't trust that unless I've done it on real hardware and you know what else I don't trust is the marketing numbers from phone vendors so here are some of the headline specs for the devices that are in my bag all these devices have flash based storage naely there's no reason to think that script should run 10 times slower on my Nexus 5x than it did in my MacBook Pro if especially if I looked at these headline numbers 2.8 GHz versus 1.8 gz that's not a 10x difference now eely there's no reason for that right if we just looked at these numbers we wouldn't really understand what's going on it Bears repeating if you think the $700 iPhone in your pocket is what people are going to be adopting in the next couple of years at the median you're delusional the average seller selling price of smartphones is going down not up because the next set of people who are going to buy phones are not replacing their current phone they're buying a new phone for the first time and all the rich people already have smartphones the next billion users aren't buying high-end devices they're buying at the margin and that margin is a cheap device worldwide phones are getting slower so is the average network connection your test device needs to represent that reality so that we don't wind up building what Bruce Lawson has called The Wealthy Western web so this is motion Mark it's a benchmark that Apple put together earlier this year um and uh it tests a bunch of graphics performance but it's very JavaScript Bound in many cases uh on Apple's to Apple's uh Hardware with Safari on and Chrome on running on the same version of os 10 um Chrome ties or beat Safari in most cases uh basically this is not something that we're actually slow at so here is the same Benchmark on the same version of Chrome on an exus 5x the desktop St version is 25 times faster for as slow as that xus 5x is I was able to change just one thing and get the motion Mark Benchmark running 15% faster 15% for one change what the heck did I do is it magic we're all adults here so I think it's safe to admit that magic isn't actually a thing instead I used a little bit of science U and I added this makeshift ice pack to the bottom of the phone I got this idea from my colleague Victor who's been looking into optimizing this Benchmark earlier in the year and we're seeing massive variant across runs what the heck is going on here going back to basics recall the computers are basically just a bunch of wires those wires have resistance and voltage and power are running through them which means they dissipate heat and we dissipate some more heat every time a transistor flips from an on state to an off State or vice versa that same process generates computation but it also generates excess wattage in the form of BTUs so a chip built on the same process with roughly the same architecture with the same number of transistors that dissipates more power and turns it into heat is the chip that does more math and doing more math is how you go faster when it comes to computing power is literally power and power equals heat so these are the guts of a regular desktop class machine the chip in there a slower version of the one in my laptop the square fins on the top of that thing um they're the heat sink and the job of the heat sink is to evaporate or sorry dissipate uh the heat coming off of the chip now that heat sink is Seated on top of a metallic top of the assembly for the chip with a little layer of of thermal paste between it so there's no air gap that air gap would like cause a thing that would explode and break the chip because it gets so hot um there's a fan that's running over the entire box extracting all that heat that's getting dissipated out um and the result is that a desktop class or high-end laptop chip like this can dissipate something like 60 watts under load this is what 60 watts looks like I don't know about you uh but I haven't chosen to hold 60 watts in my hand dissipating uh more than once and the key this is the key reason that mobile phones don't run as fast as desktops or laptops even if they can include as many transistors or scale up to the same frequen ques these chips in these packages just can't dissipate 60 watts without burning your hand as Taylor said earlier so let's look at inside the guts of one of these phones this is the remains of the Nexus 5x that I used as my daily phone for a couple of years it gave up the magical blue smoke and stopped booting a couple of weeks ago so now I get to dissect it um and unlike desktop class machines where the GPU and the memory might be on different sections of the board or different boards entirely the whole system on a chip lives under the other side of this and that's that thing there is the um the power supply on the flip side of the same PCB is the entire system on a chip it's got an aluminum cover like this sort of a heat spreader uh so when we flip it over this is what you see there's no thermal paste no fans I took the shield off but that's all in fact the CPU module isn't even visible on this board because it's sitting underneath that Samsung made Ram chip think about that to get heat off of this CPU it has to go through another chip and then through the casing of that chip then to air then to a thin aluminum thing to maybe spread some of that and then out what the screen the two layers of polycarbonate plastic in the back two layers separate layers aren't even connected remember that polycarbonate plastic dissipates heat a thousand times less efficiently than aluminum if you can't Evacuate the heat then you can't really generate a lot of it without the core temperature rising to levels that damage the circuitry and then the magic blue escapes smoke escapes and phones stop spoting I wonder what I did to mine so Chip designers saw this coming they've been putting Dynamic voltage and frequency scaling into chips for more than a decade and more recently they've started enabling features that allow os's to turn off cores entirely all this reminds me of this paper that I read a few years back from 2011 if you have some spare time I recommend it while perhaps not intended to be it reads like a prophecy from half a decade ago about the experience that we're all carrying around in our phones where a huge percentage of the Silicon in our devices isn't actually available to be used thanks to the power and thermal constraints and the power thing is real there aren't any heat sinks in your phone although I'm pretty sure that you wouldn't want a heat sink and a fan in your pocket um but imagine if you could have one why why don't those exist why can't I get a bulky phone the basic reason is that this battery only contains 10 hours of power think about that in terms of the light bulb you'd only be able ble to keep it lit for a couple of minutes if you had to if you could drain the battery that quickly which you don't because doing that causes batteries to explode so just don't try just FYI not not a fun experiment at home um this is why mobile phones are slow we can't dissipate power because we can't carry power that battery has to deal with all sorts of stuff it has to deal with the CPU and the GPU and the Wi-Fi radio and the Bluetooth radio and the NFC radio and the cell radio and the screen and the touch digitizer it has to power all that stuff and keep you satisfied for a day worth of use on a single charge on something that can't keep that light bulb lit for more than a couple of minutes so to keep from wasting power there's a lot more complexity in modern phones uh most of them today use what's called a big little architecture and that means that they try to move work from high power cores to low power very aggressively the systems that move that work around are called schedulers all Kernels have schedulers and the Android ecosystem schedulers are a bit all over the map the big thing to understand about them though is that your phone probably isn't using what's called symmetric multiprocessing that is to say not all of the cores respond up to the same voltage and frequency running at the same rate all the time they're at different levels most of the phones that you probably have have something called a global task Schuler which moves workaround between those things the systems in Linux that wind up doing that management That Work Management are notoriously hard to tune and they use all sorts of fistic to do it some do something called touch boosting and what that means is when you put your finger down on the glass they power up the big CPUs in anticipation of you doing some work like animating something or flinging it around some of them have specialistic for launching applications so that when you launch something from the home screen they power up the big core so that that thing launches very quickly now that looks nothing like the web workload the web workload today looks like tapping on the URL and then waiting for the network and and maybe those cores got scaled down again and then your content comes in and then we start processing it basically the web is fundamentally not aligned with the way mobile phones have been optimized to work because our workloads don't look like they're workloads lastly remember that light bulb the light bulb is why you shouldn't believe any of the numbers that really you see in bench marketing right the idea that your mobile phone CPU is as fast as a desktop CPU may be true in the limited case but not in the common case you're going to get Scaled you're going to get throttled things are going to move to a low power State as aggressively as possible that's not how the world works we don't keep things spun up the way we do on desktop so here's that chart again but I added the details of the CPUs it looks a lot different now doesn't it I'm not even going to get into the details of the huge differences that come from caches and pipeline depth and in order versus out of order dispatch and memory bandwidth but all of it matters the tldr is that you actually get what what you pay for on a mobile phone importantly the MacBook Pro packs 100w hour battery that's the FAA maximum limit that you're allowed to carry in a single battery onto a plane design constraint but as a result of having that much power available and because the heat sink and the fan and my MacBook Pro I can keep those four cores spun up under load and they can dissipate something between 40 and 60 watts all the phones in the chart are big little devices and that means that many of the cores are powered down most of the time that Moto 4G at the bottom has has eight cores but if you get more than three of them working for you at any point in time you are really lucky right so mobile CPUs aren't exactly what you thought they were uh I mean since when would a hardware vendor ever use an opaque number to mask a major difference in performance right and okay maybe memory pressure and smaller memory footprints on mobile devices don't allow us to on the browser side trade away space for Speed as aggressively as we do on desktop and maybe this storage systems are roughly as fast though so maybe we could get something back there I mean they're just solid state flash devices on a Linux OS if you're running Android right who has heard of the term mlc flash okay I think it's like 10 people um that's about what I expected uh mlc flashes multi-level chips basically they are chips on top of chips inside the same chip package and that is how you make storage cheaper um and it's a primary reason why my Nexus 5x gets 400 megabytes a second of read throughput and my MacBook Pro gets 2 gigabytes a second of read through put in ssds the way you get better performance is parallelism in order to read or write a block you want to distribute reads and wres for large reads and large wrs to as many different chips as possible because the latency for getting data to and from each of those chips is constant so you have a controller in front it's got some memory and it distributes that workout to those chips now space physical space is at a premium on mobile devices but so is power and so vendors have tried to consolidate those chips as far as possible and that means that they're using fewer and fewer usually one chip for all that reading and writing and that means low parallelism which means low performance we don't get that benefit that my MacBook Pro does of having many chips in a row on a mobile device mlc flash is also just out slower and file systems haven't really caught up basically what you should think about the median mobile phone having is spinning disc from 2008 think of it that way that's probably a pretty good parallel okay that's kind of a bummer right spinning metal um and if mobile discs make you sad the state of mobile phone networks will make you wish that mobile discs were the problem you actually had if you hadn't I recommend checking out IL gregoric highperformance browser networking it's free you can read it online and it goes over a huge amount of the total end end Network stack that gets bits from your server to your phone highly recommended if you spend some time with it um you'll get to where I got to which is that mobile networks hate you sell networks are basically cryptonite to the protocols and assumptions that the web was built on where TCP in the web was built around the assumptions of a relatively stable underlying transport condition cell networks gyate wildly and from millisecond to millisecond where TCP assumes relatively constant packet loss and constant rtt times cell networks deliver anything but transitioning from one network type to another or one subtype to another in real time and where the web's model of hot linking sub resources assumes reliable networks for the duration of the page well we kind of know how badly that breaks down in practice don't we at least those of us who use public transport or riding cars or you know basically use phones uh so this paper is actually a pretty good entry point and especially its references into um why why these networks hate you so much and they do they they hate you a lot uh when you dig in it turns out that what's really killing you is the variance and the volatility and the underlying Network substrate now some of you might be thinking isn't LTE going to save us well yes and maybe um here's last year's performance for us LTE users uh compared to the year before these networks are actually getting slower in fact the variance in mobile networks is so massive that it feels like a farce to call something 2g or LTE some of the largest Emerging Markets have median rtt times north of 400 milliseconds when you open Dev tools and you do Network throttling and you put it in the regular 3G mode it says the rtt to 100 milliseconds which is part for the course in the US but wildly wrong in other markets especially when you think about how many carriers wind up throttling things even further down the same network type May mean dozens of different things for for your users for the TCP gigs in the house thinking about what that sort of round chip time does to the bandwidth delay product can really bring you down Channel capacity be damned that sort of latency eats your transfer speed for breakfast but of course this is mobile so it's worse than that uh as Ilia said to me recently a 4G user isn't even a 4G user most of the time cell radios are magical things sure they try to preserve power too though and they seamlessly transition between their high power states and their low power States across different radio types different cell locations they do a ton of work to make sure that we never see what's happening under the under the covers but that creates variance when users try to connect their phones might be in a low connectivity or low power State when they weren't just a minute ago in those cases the radio resource control protocol that ilas book goes into some detail on determines how the connection gets made for users in varo power States on 3G connections it can take seconds to just start the radio handshake at the physical layer so that you can start transmitting data if you want to get bits on screen in three seconds you're in a really tough spot you can't do DNS TCP TLS or even start sending those HTTP headers down the wire until all of that is complete now consider adding hundreds of kilobytes of JavaScript to the mix that's not theoretical the HTTP archive is showing that the top 10,000 sites put almost a megabyte of uncompressed script on their pages today on those networks on these CPUs this is a recipe for for disaster no wonder users have the pervasive feeling that the mobile web is slow I think it's only reasonable to be sad about all this uh the tools and techniques that we've brought over from the desktop era really aren't serving us to make great Progressive web apps we need to do things differently we need to load less code we need to load it at the right times and we need to let the browser do work for us whenever possible use the platform isn't a nice to have on mobile it's the only way to go Sam's going to go into a lot more detail about the depth of the crisis that we're in but make no mistake if you're using one of today's more popular JavaScript Frameworks in the most naive way you are failing by default there is no sugar coating this except for the tiny Club of fast enough by default tools like the polymer app toolbox and preact with some good web Pac Fu today's Frameworks are mostly a sign of ignorance or privilege or both the good news is that we can fix the ignorance so when we're armed with data we can make better choices and avoid those slow by default tools now I talked to a lot of teams who've gotten a long way into their pwa development story and they've got very heavy client side JavaScript apps their apps feel like Gmail basically they get a loading bar or something like it while a ton of script starts to execute and then they get a fast UI all the subsequent interactions wind up being fast because they've paid all that cost up front but many developers find that this is kind of slow it feels slow to use once everything is loaded it's great but as we saw earlier JavaScript execution on phones makes this strategy kind of a loser JavaScript execution is single threaded sure we can parse and compile off thread but we can't use the preload scanner to grab sub resources if they're embedded in that script we can't speculatively build Dom or parse CSS or apply it when you go with one of these tools that doesn't use the platform well you bet the farm on a single core on a phone that might be thermally throttled or in a low power State good luck so what we're seeing now is something called server side rendering AKA isomorphic rendering AKA Universal JavaScript um I haven't looked in a couple of months uh is there a new term for this I'll take that as a no okay um so the idea is to run JavaScript on the server the same JavaScript on the server that you run on the client and then send down a a pre-computed snapshot of the HTML that you were going to send and then you load the gargantuan JavaScript bundle and hope that it all works out on my MacBook Pro or an iPhone with the fast connection or my pixel something like that this works out pretty well but for the vast majority of users with less expensive phones or less good connections you get this crazy uncanny valley when the JavaScript arrives the main thread locks up all the same until it starts executing and finishes the content might be displayed but it isn't meaningfully interactive now there's a debate that starts some folks think that because maybe you can start scrolling this stuff because browsers are magic and they do threaded scrolling and don't scrolling actually is an interaction if I can't put my finger down and tap on your UI and have it start responding and doing work for me under 100 milliseconds it isn't loaded it's broken okay all right what we really want is Progressive interactivity and this is what the purple pattern that s go into delivers the Insight here is that you should only load the code that you need right now if possible for the views that you're actually sending to users and combined with service workers and http2 push it's possible to achieve this without over bundling purple and the poly app toolbox represent what's possible once we take that Mobile by default thing seriously and it's night and day from where most popular tools are so this is the shop app that you've seen before the polymer team released at IO you can visit it right now at shop. polymer project.org um and here it is running on a desktop browser we get to interactivity very quickly on Wi-Fi and it only takes a few hundred milliseconds of script overall you know so what we've seen this story before but what about mobile again Nexus 5x same Wi-Fi connection despite the lower CPU the app sends down an appropriate amount of script so that we get interactive performance at under two seconds there's nearly a second and a half of script execution overall but thanks to the granular use of HTML Imports in http2 push in contrast to Major bundling most of the components load with tiny execution slices which means that the content that's already on screen stays interactive this is what mobile first looks like and it's radically different in a good way the other thing that the purple pattern ads is a service worker you might be thinking that service workers are about handling offline and while it does allow you to do that that's not the primary benefit for most end users service workers matter because they let you deliver reliable performance because they can handle the top level resource and always return something from the cache you can dramatically improve the performance of your apps when you use service workers this way that huge variability in networking conditions it evaporates when you've done this so this is a chart that Eric bman gathered from this year's IO site it's also a progressive web app but with polymer 1.0 but as you can see from the giant dark green Spike when the service worker is active the distribution of load times moves hard to the left and that's a good thing this is very literally what faster looks like I'm seeing a lot of teams try to add service workers as some sort of transparent pass through thing using a network first approach don't do that please don't do that use the purple pattern and make sure that your top level app shell never depends on the network if you do that you can compete with Native apps on on the experience that you deliver if you saw Darren's keynote this morning you saw exactly that with the keynote Tech the CET Tech today pwa if you don't do that though you will never match their performance until recently it's sort of been difficult to verify in an automated way that your service workers installing and the rest of your Progressive web app properties are actually met correctly you've heard a lot about Lighthouse but please put it in your CLI uh put it in your continuous integration system and let it tell you how you're doing so I think it's safe to say that mobile is much much harder than we've collectively understood it to be to make good apps in this environment we need to change our Outlook our tools and most of all our priorities and the fastest way I know to get in touch with that that ground truth is to test on real Hardware so please if you don't already have a cira 2014 is Android phone go out and buy something like a Moto G4 if you can use one of these and chrome inspect on dev tools you'll find yourself in touch with how it really feels to be at the median if you can get something worse like this is an Android one from last year um you probably can't buy one but get something worse if you can um and if you can't afford any of those please use webpage test.org to select from the list of real mobile devices that are sitting in a rack at your disposal to test your urls and whatever you do Implement as much of the purple pattern as you can Sam will fill you in the details so stay tuned for that um and lastly Lighthouse and chrome Telemetry are potent weapons in your ability to not regress on keeping sure that you're doing the right thing I want to apologize for being a bit of a downer today usually I'm telling you about how good PW are and how great the experiences that you can deliver and that's all true um and there is good news and that it's that modern web technology makes it possible to build truly amazing experiences but it will require ditching or radically reworking the way that we're using these slow by default tools add's got a whole talk tomorrow about what kind of elbow grease you really do have to put in if you've bought into one of today's Major Frameworks but no I think now that the challenge is much larger than you probably thought it was now that you know I'm actually very confident that the folks in this room and on the live stream are going to internalize this and use it to make really great experiences so thank you [Music] up