Spotting and solving everyday problems with machine learning | Session

Key Takeaways

[Music] hi my name is dale an applied ai engineer at google and as part of my job i'm constantly learning new technologies now i don't know about you but when i'm learning a new tech i always like to do it in the most applied way possible so if i'm going to learn kubernetes i want it to be because i'm working on a project that actually requires i use kubernetes now machine learning is this completely new way of getting computers to do things and most of us don't have a lot of experience with it so it can be kind of tricky to even identify those projects for which ml is a good solution so that's the goal of this talk i want to give you a sort of a framework for spotting and then solving problems with ml so let's take a look i build apps powered by ml all day and i find that they usually fall into three buckets so the first one is being able to handle and search and deal with complicated data types multimedia data types what do i mean by this i'm talking about images pdfs videos audio files anything that's more complicated than like a simple row in a spreadsheet a couple of months ago i realized that i was actually sitting on a perfect example of this type of problem that could be perfectly solved with ml when i was born my dad went out and he bought a video camera and he came to the hospital and he filmed me and then uh he didn't stop filming for like 27 years so he collected hundreds of hours of family videos and i knew that there were lots of moments in those videos that i wanted to relive but the problem was all of the things i wanted to see were buried beneath hours of like friends i didn't even know anymore opening christmas gifts and this is like a perfect example of why video is so difficult to work with without machine learning how do you find a certain moment in a video you have to have a human sit there and watch the whole thing unless you have machine learning so let me show you what i built i built an ai powered family video archive so i uploaded all of my family videos to google cloud storage and then i used a machine learning model powered by the video intelligence api that's a google cloud tool for analyzing videos and that tool does a bunch of different things so it takes your video first it uh transcribes it so anything anybody said in the video gets transcribed to text then it recognizes any text that might be on screen like the date here june 15th and finally it uses a computer vision model to see what's going on in a video so if i show you like this one i'm on the swing set and the vision model says oh this is a playground there's outdoor play equipment it's a public space and so on so as soon as i built this and i tagged all my videos i you know i had some memories in mind i wanted to find so the first one was i knew that when i was somewhere between three and six my parents surprised gifted me this sparkly barbie bike best day of my life and i wanted to find it so that was the first thing i searched for here was i found it i found it somewhere do you want it yeah oh okay for this archive finding this barbie bike it actually wasn't that difficult because there are lots of different things the model could key off of it recognizes to be a vision that there's a bike in the frame my parents are talking about the bike that problem is easy so i wanted to try something a little more complicated could i find my first birthday party so the archive could find not only my first birthday but also my brother's first birthday and i was really kind of delighted by this because what i realized is the way it recognized this was by actually reading the text off my cake written in icing happy first birthday dale same thing with my brother's cake so okay so i found my bike and i found my first birthday but what if i tried something more abstract like what if i wanted to find my very first steps so i think it's this one right here [Music] that's dale walking this is the first time she's taking major steps on this floor so it worked but what was interesting is how did my query connect to this video it had to be what my dad was saying so he said oh dale's walking this is the first time she's taken major steps on this floor but i searched first steps so it wasn't a perfect match and this is another really important part of what made this work the text matching wasn't exact it was this sort of fuzzy machine learning a powered text search more on that in a second first let me tell you how i built this so first i took all my videos and i uploaded them to google cloud storage then via cloud function i triggered a machine learning model to run in this case the video intelligence api which takes as input a video and then spits out a bunch of text the transcripts for the video the tags that were identified by vision take all that text put in the database of your choosing maybe bigquery in this case and then you want to implement some sort of smart text search on top so that you're able to find things without having to know exactly how people said that which brings us to use case number two for machine learning that i use all the time it is understanding language specifically through something called semantic search so human beings right we have many different ways of expressing the same idea i could tell you that i'm tired that i'm sleepy that i need to hit the hay i could yawn these all express the same sentiment even though i use different words and the idea of using machine learning to understand language is that computers should be able to get this as well you should be able to get at what you want to say even if you don't use a specific set of words the way that we solve this with ml is with the technique that's used all over the place called embeddings embeddings are a way of taking a type of data and finding a way to map them to space maybe a two-dimensional space maybe a dimensional space such that points that are close to each other are similar so in this case we have a word embedding and the way that these words are plotted in this two-dimensional plot is that similar words are near each other so jetpack doesn't have anything to do with apples so they're far apart but a jetpack it's kind of like a flying broomstick and it's kind of like a cloud so they're all close together in space and then we can use an algorithm to say okay if these things are have a close distance maybe they're related and how do you calculate distance by the way pythagorean's theorem euclidean distance there are lots of ways that you can do it now let me show you how you might actually use this in an app so here i have a spreadsheet full of uh news article headlines so imagine that i'm i'm a news company and i want you to be able to search for a headline and then i match it intelligently i'm going to use a tool here which is a sheets extension an add-on called semantic reactor this is a tool built by google research to sort of help you prototype language apps and what it does is when i load it it takes all of the headlines and it embeds them it turns them into points in space and then i'll type a query and it'll compare all these headlines in space and find the closest points so let's say i want to find an article about fashion i hit react and the first headline that comes back is an article about fashion week okay so the words overlapped that's simple but what about the next one a visual evolution of what we wear on planes from lapels to leggings the word fashion isn't in there at all but because of the way embeddings work and they're able to almost pick up on cultural associations it understands that lapels and leggings and wearing all have to do with fashion same thing with the next article which is prince harry really knows how to rock a casual look it recognizes that this has something to do with fashion so you can use semantic search uh for this sort of like uh fuzzy matching this demo in built-in google sheets actually on the back end it's powered by javascript it's powered by a javascript tensorflow model which you can download on the tensorflow website called the universal sentence encoder module it's actually one of the most popular models in all of tensorflow and the way that you use it is that you load in your javascript app you call model.load you take your sentences and then you call model.embed sentences and what you get back is something like this it's a set of vectors or you can think of these as coordinates in space so that you can then go and see how similar sentences or words are and build your own smart matching system okay third category and this is a big one because there are so many different use cases for ml you can use machine learning to convert between different data types and of course i don't mean string to int i mean image to audio perhaps so for example i read a lot of research papers which are usually pdfs but sometimes i wish i could you know take a stroll and be listening to a research paper so this is kind of right like an image to audio problem first you need machine learning to analyze what are the different sections of the document this is kind of like a vision problem so what is the heading of the pdf well that's usually a block of text that's a larger font size and it's kind of near the top of the page versus the body text which is smaller near the bottom of the page so you could use a vision model to identify these sections if i were doing it on google cloud i'd probably use automl vision which is a tool for building custom vision models without having to know how to code next after i have those sections i want to convert the image to text you can do this with a google vision api and finally you might want to take that text and then speak it out and you can do that with the text to speech api and here's what the podcast sounds like a promising path towards auto formalization and general artificial intelligence christian segetti today ai systems are able to learn solving tasks that used to be thought of taking uniquely human capabilities until recently okay here's another idea uh rather than going from pdf to audio why don't you stay within a video but just translate the language you could almost use ai to do an automatic dub right so these pictures are all from youtube whenever i do a machine learning project i usually make a youtube video that goes along with it and i noticed that a lot of my viewers were in india but i don't speak a look of hindi so i wondered could i use machine learning to automatically translate the videos first step transcribe the video get what i'm saying in the form of text then use the google translation api to translate that text to hindi and then use the text-to-speech api to play it back and then you just have to take the audio file and overlay it onto the original video here's what that sounded like software developers i have no idea how good that was but i've heard it's not bad now i'll just give you one last use case for how you might convert data types using machine learning and this one does require that you stretch your mind a bit but sometimes it's not that you just want to go from one format to another but some formats of data are just easier for programmers and computers and data analysts to work within others like i mentioned that if you're looking through videos it's easier to search through text than it is through video right and the same thing is true like for example a couple of months ago i wanted to see if i could improve my tennis serve with ml uh so i went to the court and i set up a camera on a tripod and i took you know an hour of me hitting the ball but the thing is if i want to do any numerical analysis on my serve it's very hard to do that on a video file right it's just like tons of pixels over time it's very cumbersome what would be much easier is if instead of a video i had maybe uh the position of my skeleton of my joints over time because if i knew for example where my wrist was and where my elbow was and my shoulder i could sort of calculate the angle of my serve and then say oh my arm should be straight and it's not and in fact that's exactly what i did and i also use the video intelligence api from the first project and i use this pose tracking feature that you see the person in a video it converts it into their pose so we covered three use cases that i use all the time for machine learning uh dealing with custom data types to make them searchable understanding language and converting between data types i hope now that you have these frames in your mind you see ml problems everywhere and you decide to start solving them yourself and if you do i sure hope that you use google cloud ai tools i've listed a bunch that i used in my presentation here today and these are for the most part really user friendly they don't take any data science expertise and they're sort of plug and play that said if you wanted to go a step further and start developing your own custom machine learning models i would highly uh suggest you check out vertex which is google's new platform just released for training building and deploying custom machine learning models it's also a great place to prototype building ml models and if you want to connect with other machine learning enthusiasts check out the tensorflow forum which is just launched where you can chat with other ml practitioners and the tensorflow team that's all for now if you want to know how to build any of these projects that i talked about they're on dale on ai.com and on the google cloud blog and thanks for watching [Music] bye [Music] you