ml5.js - Friendly machine learning for the web

Key Takeaways

[Music] hello hello uh my name is Aiden Nelson I'm a visiting faculty member at New York University's interactive telecommunications program and I'm here today with you Lee hi I'm you I'm currently a resident researcher at nyu's interactive telecommunications program and we're presenting on ML 5js which is friendly machine learning for the web but first I think I need to say a little bit about why two people from an art school are presenting at a web AI conference uh ITP was founded as a graduate program in the 1970s with the goal of democratizing access to Media Mak uh here you can see a photo of our founder red burns with the Sony Porta Pac which was one of the earliest portable consumer video cameras uh red recognized that these Technologies had an incredible power to change who could tell their story and how and even as over the decades these technologies have shifted from videotape in this case to personal Computing to the internet to AI to web AI the mission has remained consistent which is getting students to ask how they could enrich their lives the lives of ordinary people and improve Society so that brings us to ml5 and why is it friendly so ml5 is an open source Java JavaScript library with the goal of making machine learning approachable and accessible to a broad audience of artists of creative coders and of students it is built on top of tensorflow.js and it integrates closely with p5js which is a widely used um creative coding framework it's built and maintained as an open source project through a collective effort from nyu's campuses in New York and Shang high and from creatives all around the world so a bit of history about ml5 so in 2017 deep learn JS a predecessor to tensorflow JS was first released and an ITP named chrisal Valenzuela worked to integrate this library with P5 and created some of the first examples as ml as tensorflow.js was released ml5 grew alongside it so as examples and models were added to tensorflow ml5 took them and wrapped them in a friendly and Incredibly accessible API here you can see a gif from poset a human pose detection model running in the browser and throughout this process ml5 has been taught alongside existing creative coding Frameworks like p5js and through these existing channels um so it's been taught in both graduate and undergraduate courses at several University it's been taught at workshops in art spaces in New York in Berlin and elsewhere and it's even been taught in primary and secondary school workshops and this is all because we have a focus on providing this initial entry point to working with AI and working with machine learning as a creative platform in 2019 and 2020 with the release of media pipe models supporting face keypoint detection hand pose and object detection we worked to explore creative use cases of these models and making them available to our students and artists in 2021 we worked with uh colleagues at the NYU Law School to develop an experimental open- Source license and code of conduct focused on the ethical use of AI uh building a neighborly approach we used a modified version of the blue Oak model license and most recently we've been in collaboration with NYU Shanghai me NYU community members in New York and Shanghai we've updated for compatibility with the latest versions of tensorflow and its pre-trained models and we've improved our a API and documentation so what does ml5 look like in action so machine learning Frameworks are typically geared towards people with somewhat Advanced knowledge of linear algebra statistics data science and years of programming experience in a language like python or C++ obviously I don't need to speak to you all about how accessible the web can be um so we wanted to present a way for folks to think creatively about how to use machine learning as a creative platform so mlv's examples are all built using p5js and the P5 web-based editor which means that all dependencies all model weights are available using a single script tag uh in this example you can see uh face keypoint detection which the model is instantiated the webcam feed is uh grabbed and the output of the model is visualized all using about 35 lines of code so in practice what this allows is for students to very very quickly get up and running in jumping to the creative use cases and these might be in building interactive installations in building web art projects and they might integrate with other libraries like 3js or in developing games design so next we'll hear a little bit about ml5 today let's take a look at today's ml5 how is used by our students and educators to make creative applications of machine learning and how it teaches machine learning in a friendly approach ml5 encourage friendly and creative applications of machine learning and invites the community to explore alternative and experimental interfaces one of our students create this unique project that reimagines bird watching as an interactive web interface in this artwork she uses ml F hand post to that user engage with the interface through handpoint hand key points detections and then the Contour of the hand key Point simulates the bird watching gestures and the point of view of the binocular this example is built with p5js and ml 5js directly on the P5 web editor this shows that how they together make it simple to get started with building interactive web art and experiences with all minimal set [Music] [Applause] up here are other examples of of using ml5 to create experimental web interfaces the example on the left which we just saw is a sound synthesizer controlled by the user's mouth the opening and closing of the mouth triggers the start and end of the synthesized sound once the mouth is open the opening size of the mouth controls the sound volume the sound of the pitch of the sound depends on the tilt of the head and turning the head to the left and right will adjust the harmonicity of the sound the example on the right is an experimental drawing applications that use users face as a pen brush to draw on the web canvas both examples utilize ml5 face Mash along with p5js ml5 body pose model is commonly used to create interactive screen based art and installation that responds to body movement such as in dance performances or live audience interactions traditionally creating these type of installation Arts often require extra Hardware like KCK or specialized software like M mapper touch designer or other programming tools like C++ open Frameworks however with ml5 everything can run in the browser using just a web camera this makes the seup process for arts and exhibition much easier and more access accessible it also reduces the need to learn new hardware software or other programming languages the creator of this artwork mentioned that he has always wanted to create realtime body movement based installation art but hadn't pursue it due to the inconvenience of requiring extra Hardware software however with ml5 he was able to achieve this easily with just using a web camera and he was also able to enhance the artwork by combining other webbased Sound Library such as tjs or P5 sound ml5 also has body segmentation ability including selfie segmentations and body pics they enable students to easily Swap and style their backgrounds without a green screen when drawing and making creative applications in p5js when using ml F image classifier and sound classifiers they are often paired with the teacher B Machine machine this is because working with the teacher B Machine allows beginners to get started with transfer learning with very little data shortterm training times and entirely on the web locally without uploading data to a server this makes it a friendly and approachable way for beginners to play and experiment with pre-trend models this integration also helps our students quick L prototype their projects and simplifies the proc creative process for both students artists Educators to experiment and develop their ideas besides using the pre Trend models you can also Trend your own machine learning models in the browser with MLV neural network the example we are looking at here is a simple gesture classifier Trend with mlis neural network it is featured on D shiftman's nature of code website where it is used for educational purposes training machine learning models can sometimes sound intimidating for beginners and artists however mlv's simple web-based approach allows beginners to explore machine learning without needing Advanced technical skills and extensive data uh so we're excited about what's next for ml5 we have ongoing research projects in collaboration with uh our friends in NYU Shanghai uh some of those researches involve uh as you can see on the top right here this is an image segmentation model that's backed by Transformers JS um so we've been excited to to look into new backends uh and what models might be allowed uh by integrating them into ml5 uh on the bottom right you can see an example of a a project that extends ml5 web-based neural network class to support time uh time series data which allows for gesture recognition and has a lot of creative use cases and finally we've been inspired here today there's there's a lot happening in web AI we're excited to explore other collaborations Integrations and expand ing the pedagogy around AI for creatives and creative use cases uh thank you so much and we look forward to talking to you at the break [Music]