Practical on-device AI for web developers

Key Takeaways

hey everyone I'm Kenji product manager in Chrome and I lead our AI ml efforts on the web platform hi I'm Mo developer relation engineer in Chrome now I know you are probably up to your eyeballs in mind-blowing AI demos either the sky seems to be the limit or there is hypogen in the AI work f s driving cars that double as therapist maybe someday but today let's cut to the bir instead let's talk about practical AI the kind that may even run on your devices and to be clear we're not talking about yet another chatbot we won't solve existential question like what makes the perfect back perfect although maybe we are talking about practical solutions to help your users and boost your business in fact let's make it again see if we can say practical more times than AI ready let's dive in before we dive in just a notes no ml expertise is required for you to follow this talk and you'll find all the links we mention in the video description below now ai can amaz in demos but there are also successful real world practical applications for example nearly half of you are using AI Tools in their development workflow now what about user-facing features well we went straight to the source web developers like you to uncover your biggest pin points for building user faing AI features and here is what you told us first we're in the early days teams are exploring how to augment their product with AI in ways that are business posting second especially with large language models L&M for shorts developers worry about controlling non-deterministic outputs and finally for now most of you prefer pre-trained models and readymade because your focus is building product features so for now you'd rather leave training and fine-tuning to ml Specialists I can totally relate with the desire to build valuable feature without having to be an AI expert a feature's value isn't defined by the fancy tag behind it so let's hit pae on the high and focus on what really makes a difference right mode exactly we'll show you how to augment existing features in your web application with a touch of AI we won't get into training or fine tuning instead we'll really focus on ready to use Solutions and we'll also share engineering challenges we encountered on the way okay let's show some practical demos whoa not so fast what about all the Spain points you mentioned earlier we got to take a step back and before the demos and talk about the why so let's see why on device AI ml can be your superpower and how it fits with server side AI options now we know on server AI is a Workhorse it's familiar it's reable and it does a fantastic job so we're not here to say ditch the server instead think of on device AI as a powerful new tool in your toolbox one that can unlock some amazing possibilities alongside any existing server side setup on device AI lets you handle data locally this can greatly simplify your privacy story if you have users with sensitive data for instance you can still Delight them with AI features while also giving them an end to end encrypted mode for some use case ditting the round trip to the server means near instant results and so on device AI can be the difference between a viable feature and a janky user experience on device AI can help you manage your server cost your user's device can shoulder some of the processing load in exchange for more access to AI features you may even consider a free tier with on device AI features to help your customers get a glimpse of what you would get with a Premium plan one last thing before we jump into the demos practical tips about on device AI first not every device is going to be an AI Powerhouse and so design your feature with Quest full fallbacks when the device can handle the advanced stuff also run Benchmark on your your target devices second our device AI works best in a focus wall so with the right script even the smallest model can deliver a step stealing performance within the user Journey also you can always fix it in post in fact both pre and postprocessing this can help simplify the Step at hand AI models can be Hefty so be smart or transparent about downloads especially on mobile so that you don't blow up your user data plans in addition hosting these models can be demanding so be sure to have a good serving and caching strategy all right with this background let's start with the Practical AI figuration of hypothetical website mode and kji practical bags shopping site so I'm a product manager mod is an engineer and we both work in Tech which means that we have a backpack obession because we don't just carry gear no no no we carry our work lives and so every pocket every scrap has to deliver for us choosing a backpack is like designing a website it needs the perfect balance of functionality Aesthetics and a grid of pockets for all our gadgets basically the S perfect ux for our life and so we built a website for folks like us Hing to crowdsource the perfectly practical bag and judging by about the volume of product reviews it seems like there are dozens of us and we are all incredibly oped about seers sadly all these reviews make it harder to debug The Good the Bad and um Can it practically fit my absolutely large laptop bits and so many folx just give up and end up with another sidef free conference. bag I wish we could summarize all these reviews into a tldr with the pros and the cons I think this would work best on the server side that way we can do it once for everyone we can also summarize our existing reviews and store the reserve and then when new reviews cross a threshold we can refresh the summary to capture the latest inside so mod can you walk us through the technical [Music] details sure to create summaries a synchronously we can use a job queue but let's focus on the gni part what we want is something like this a summary of all reviews along with a short list of most common pros and cons of the product so to build this feature we can use an llm API like the open AI API or Google's jni API Gemini is Google's largest and most capable AI model and here we'll access the Gemini API with the Google AI JavaScript SDK for node.js applications but sdks are also available for other languages like python or go and and there's also rest API so here's how to build our feature first I create a key for the Gemini API the API offers a free tier and a pay tier for higher volume and I Define my key in my environment file remember to treat your API key the secrets then I npm install the Google AI JS SDK I require it in my node application and I'm ready to use it so I first instantiate a gen object and I get a model here Gemini Pro here I could also tweak the model safety settings or pick a different model if I needed Vision features for example then I call generate content on that model passing my prompt as an argument and don't worry we'll dive into the prompt in a minute and this is here an example output a summary of all reviews alongside a list of common pros and cons this summary is llm generated so run this through checks before displaying it to your users and also allow your users to share some feedback in case something is off now let's take a closer look at my prompt I'm being specific about the output format and I'm also providing an example this technique is called oneshot prompting and this will help me get consistent summaries and by the way for designing your prompts check out Google AI Studio it's a great webbased tool for fast prompt engineering and interation now one heads up many reviews can hit the token limit remember a token isn't always a single word it can be parts of a word or multiple words together and Gemini Pros 30,000 token limit means that my prompt can include at most 600 average 30w reviews in English minus the rest of my prompt instructions so I could take for example the 600 most recent reviews and use the count tokens method to double check and by the way I wouldn't need to handle this if I was using a model like Gemini 1.5 which has a 1 million token liit one privacy tip I'm stripping usernames from my reviews there's no need to risk personal info which doesn't help summarize the reviews anyway so that was the gini API but if you're on Google cloud or need Enterprise grade support check out vertex AI you'll get Gemini Pro and more like anthropics Cloud models in the cloud console what I really like is the model Garden View because it's a great way to explore and match models to your specific use case okay chy what should we look into next let's stay with the product reviews for a moment the summarization feature we'll have but the quality of the reviews matters too and sure we are already running checks on the service side to eliminate toxic reviews um in in fact let's have a look at some of the recent example that got fled out H that's a bit of a mixed bag this reviews are toxic but a few actually contain useful observations this is unfortunate because according to external data 82% of online Shopper actively think negative reviews before making a purchase and these can also help reduce return rates so could we gently nurge users away from writing reviews like this back sucks and toward you know actual details about sticky zippers it will result in more high quality reviews and save every one time mod here are some ideas explore let's check the review before it gets submitted so for example this backpack is bad because something is totally fine but we don't want reviews with square words also Let's help our customer focus on writing a helpful review by maybe suggesting the associated rating and how about checking if a review is negative or positive although that might be a bit redundant with the star rating that said I would like us to explore different approaches finally some user experience consideration ultimately users should decide and so let's give them the choice to modify the suggested rating let's also clearly explain to our users that the suggestions are autogenerated on the technical front we could do it all server side but from a cost and latency angle I think it makes more sense to try on device after all we want to speed up the process for the user and so it would be great to autogenerate all of these as soon as user stops typing sure and I like your approach we take features we have today in our product and we just end them with a touch of AI so I built a little prototype let me show you when the user stops typing a review We automatically suggest a rating and a sentiment and in case the review is toxic we display a hint nudging the user to rephrase the review in a more constru itive way all of this runs on device in the browser there's no server round trips no API key and here for the sake of the demo I'm using a mix of techniques to show you what's in your toolbox for on device AI for toxicity analysis I'm using tensorflow JS which is a widely used open source ml library for inference and training on the web for sentiment analysis I'm using Transformers GS which is a web AI library from hugging face it supports inference only for models that have been converted specifically and Transformers GS gives you a developer friendly use case focused API for store ratings I'm using Gemma Tob it's the smallest version of Google's open weight model Gemma and I'm running it with media pipes experimental llm inference API this API opens the door to running massive language models fully on device across platforms and with state-of-the-art latency and that's a big deal because l LS have memory and compute demands which are over a 100 times larger than Classic on device models okay let's take a look at some codes um here's my code for toxicity I import the library I load the model and I run my classification function asynchronously and then I check if for any of the toxicity categories displayed here the prediction is above my threshold which I set to 0.9 now let's look at my sentiment analysis code with Transformers JS I import the library and I do a on line setup for the whole sentiment analysis process this is called a pipeline and with this I don't need to wrestle manually with raw model loading when I use a pipeline for the first time the model gets downloaded and then cached but after that it's much faster I then await my classifier and I pass it the text to analyze and I get the sentiment label outputed by the model my code only uses a sentiment label if it's sure over 90% confidence now let's bring on star ratings so I downloaded Gemma 2B from kaggle in case you're curious the it here in the model name stands for instruction tuned which means that I can interact with the model by giving it instructions as you would with Gemini or CH GPT and here's a quick look at my code I import the media pipe Library I npm installed then I do my file setup which means I point the code to the right model files and this is important here because gni models may have a specific directory structure for the assets I then load the model which basically means that I get the llm interface ready and finally I fit it my prompt more on my prompt later and I get the text outputs and then I'm ready to PST this for the actual store rating I extract the store rating from the from the response as a number Okay cool so all little prototype works but but what have we learned and what do we observe so observation number one none of the code I wrote requires ml expertise and sure I did spend some time crafting my prompts but the rest of the code is really standard web development stuff so on device AI can fit right into your existing skill set observation number two whoa this is fast ji rating and transform as sentiment analysis beat server round trips for Speed so what's great is we get instant feedback as the user stops typing I'm using an 800 millisecond timeout here tensor fles toxicity is a little bit slower so that model needs optimization love so to boost it have turned on web GPU remember can just tip make sure to run benchmarks on your target device because inference speed can vary a lot what's awesome is that we expect on device inference to keep getting faster with web GPU web assembly and Library updates great news for example Transformers JS added web GPU support in their version three which is a huge Win For Real Time on device AI observation number three I said this was fast right but wait a minute inference is fast but what about loading that's the challenge our toxicity model is a few kilobytes plus the T of ljs Library which is a few kilobytes manageable but Transformers JS sentiment model is 60 megab Gemma 1.3 GB I know this will take Serious download time and impact user experience hosting and serving these models will also have a cost so it's time for a reality check these models dwarf the median web page size which is 2 megabytes so is on device AI realistic well it depends first as Kenji explained the business value is key is my 1 GB Stone load Justified for B star rating probably not but it may be a great solution for privacy sensitive features enhancing your free tier without service side costs or long living use cases like browser extensions where the model is downloaded just once until the next update also there are mitigation strategies for performance remember KES tips use caching provide clear loading indicators don't load smartly don't block the whole experience on model loading try and design on device gni features as enhancements not core requirements finally and that's the most exciting bit genni is a nent field so we expect smaller web optimized models to emerge over time observation number four well that's pretty accurate both the sentiment and the toxicity analysis are really accurate now what about my on device star ratings with Gemma while to assess accuracy here I use the Gemini model server side ratings as a reference so for this prototype I was really only checking manually with a few reference reviews for a real application you need the proper accuracy testing system so you see my Gemma ratings mostly match the Gemini models ratings but this took work for example initially GMA was shy at outputting extreme reviews so one star or five star reviews and that's likely because these are less frequent than mid-range reviews so it took quite some prompt tweaking to fix that while Gemini on server Nails it with a basic prompt like this one while Gemma on device needs more handh holding and that's expected because it's a much smaller model so it's less capable here's my final prompt for Gemma I'm using a few techniques here first one Chain of Thought make the model think first have it analyze the review and then give the rating and for me asking for the reasoning before the rating improve GS accuracy second technique few shot prompts use examples but watch out don't test with the same examples you used in The Prompt because that will inflate the results so here's a key takeaway for us don't expect small offthe shelf on device gni models to do exactly what you can do on the chat UI like gini or chat GPT because these are very powerful huge llms running on servers so remember Ki tip understand your use case to decide whether on device AI may be a fit let's talk a little bit about use cases of the Shelf models that are specialized in a classification task like toxicity or sentiment analysis these work great we've seen this with the Transformers JS llm model and even with the tiny tensorflow JS model which is not an llm it's not a gni model that's just a classic NLP model for custom classification like our star rating task GMA Tob did a good job with the right prompt because it's a versatile llm but for a more complex custom classification task for example assigning multiple Texs to a review well then it may require some fine-tuning and jma's open weight architecture allows for funing but for more generic gen tasks like summarization while GMA to be on device can do a great job out of the box W nice bag of tricks I wasn't expecting that an llm would run on a laptop without a feny GPU card that said I think there are few glitches for one it's not practical to have every site download their own llm also you mention that you have to do prompt engineering that's really unfortunate given that most developer would prefer to focus on building features not spending time finding the right incantation but the use case are totally of interest for instance cyber agent a leading internet company in Japan is one of the partners who share their excitement they would like to use onice AI to help creators with suggestions for the final steps before hitting the publish button this could be suggestion for a title with different styles like in this prototype or an absp for the article it's also important for cyber agent that this is done on device because the content isn't yet published and so it shouldn't be sent to another server so it's time for another PM move learn what if step what if the model was already there for any site to use that way you wouldn't have to download your own llm what if you could get the model to do what you need like summarization without having to master prompt engineering skills here's how this could look like your brother could have a foundation model and some smaller expert models or fine tunings of the bigger model there could be a prompt API mostly for exploring the potential of the on device llm and apis for specific use case using the llm with fine tunings or the expert models in fact we felt so compelled by this opportunity that we've been working on a proof of concept with Gemini Nano in Chrome a prompt API and some purpose built API or the Cyber agent demo you saw earlier it was us this experimental build so while we made a lot of progress It's still very early days and so we love to hear from the community get back on your approach key use case you would like us to consider and much more more about this at the end of the talk for now let's not get carried away by this whole new bag of opportunities instead let's see if we can find one last pocket of pain that we could improve okay so perhaps a bit of a stretch but I was wondering if AI could help our site be more successful abroad the top feedback we get from International customers has to do with after sale support and by unpacking the issue we quickly find out that the boot cause appears to be language barriers since we only have english- speaking support agent it would be so much nicer if we could help our customers and agent communicate in their prefer language without any need for external translation tools and sure some users try to overcome the language barrier with their brers building Pat translation feature or third party translation tools but user experience is subar with interactive feature especially like our after sale Super Chat and so for a chat with integrated translation I think it's best to minimize any delays so we should definitely process the information on device it will also allow us to translate in real time even before the user decide to submit their message without having to send anything to a server on the user experience side I believe that transparency is crucial when we try to bridge language gaps with automated means so before the conversation starts let's make sure that everyone understand that they are using this awesome Tech this will set expectation and help avoid awkward moments if the translation isn't perfect plus this will likely guide folks away from complex eums which might leave the AI scrambling for meing sadly I couldn't find a small enough enough model that would work for our use case and this is very unfortunate because we've heard interest in translate use case from many partners for instance the insurance broker polic Baza will love to do exactly what we are talking about their primary Market is inja which officially recognizes 22 languages so translation will have customers and super agent communicate more easily and our device will help provide a real-time user experience while minimizing the costs associated with frequent API calls so let me wear my crpm hat for a moment sensing an opportunity for a purpose built API we started working on an experimental translate API with an expert model built into Chrome and as it happens I have another proof of concept build so mode can you ziers through sure time for another demo here we have it a customer Super Chat where I can type in my native language and get real time translation the support agent replies in English and I see it translated back on the Fly this prototype here is possible thanks to a translation model that's built right into Chrome so let's peek under the hood here Chrome exposes a neat API with a window. model object two key methods here can translate this one checks if a translation model for your language pair is ready it returns readily if the model is already available on device after download if the browser first needs to download the model and no if translation is not possible second method is create translator and this sets up your translator object asynchronously in case the model needs downloading first it will wait until it's ready and then the Translate object itself has a single powerful method called translate you fit it to Source text and it outputs the translated version now remember this is experimental and chrome specific from now so wrap all your code in feature detection so you see the model might need time to be available to the user and you have two approaches to handle this one is you could only enable your transation powered UI elements once the model is good to go and second is you could start with cloud-based translation and then seamlessly switch to the on device model once it's [Music] downloaded as mode demonstrated there are lots of practical use case for AI and recent Innovation will unlock even more opportunities we've provided practical advice about when to choose on device or server side as well as how to combine on device and server side for Greatful back and Progressive enhancement we are eager to see what you create with these IDs use # webi to share your discoveries and challenges this feedback will help us make AI a powerful yet practical tool for everyone talking about challenges we did hit a few rough patches in our journey yes and we Shar a few work in progress IDs that would like feedback on speaking of which if you are interested please engage with us on this journey feedback about your approach key use case that interest you the most as well as signing up to receive key updates for our progress in addition please consider signing up to our early preview program to eventually test drive these early IDs through local prototyping before we leave um did we manage to say practical motion than the other world what other words do you mean AI H please don't say the word AI a PR [Applause]