October AI - Video Calling with One-Tenth of Internet Bandwidth

Key Takeaways

hello everyone what's up harshitayagi here and this video is part of the new series called data science monthly where i will be reacting demonstrating or simply sharing cool new and sometimes old projects in the world of data science and ai and this is kind of like a monthly news in the data science base but if you guys enjoy it do let me know in the comments and we can make it a weekly thing so this is just one of the ways i can be more consistent with my output here on youtube and so how should we do this why don't you guys just watch this and i'll see you at the end of the video [Music] so the first update of the month is the stable release of python 3.9.0 now the release was announced on october 5 uh 2020 uh with the python community uh announcing it on twitter and various other platforms uh through some blog posts as well as tweets by the creator himself gato and rossum and a few others from the core development team uh the major new features of the 3.9 series uh are basically about uh 573 584 all of these pep codes are listed over here the major ones that i feel are type hinting genetics in standard collection uh the other one is support for iana time zone database in the standard library through the zone info module then we have relaxing grammar restrictions and there's new tech partial for c python and there are a bunch of other things now i have written a blog post uh covering all the major updates in python 3.9 along with some cool highlights as well so this is basically like a step a side-by-side comparison with the python 3.8 so i'm covering type hinting generics and flexible function and variable annotations so annotations were already available but now we can add data type to those annotations union operators and dictionaries then two new operators that have been introduced for dictionary merging process and then we have zone info module that's been introduced uh so working with different time zones is no more uh hassle than string methods to remove prefixes and suffixes so two new functions remove prefix and remove suffix uh they help you strip off all the prefix and suffix in your strings then other cool releases uh in the other release highlight section so this is going to be a side by side comparison as you can see python 3.8 and python 3.9 how you did annotations earlier in python 3.8 there are no data types here now you can use the annotated module and add the data types to your functions in your annotations i have added screenshots how it works how you can work or use different attributes to get all those information then merging and updating dictionaries so this was an ugly looking syntax for merging two dictionaries in earlier versions but now 3.9 has this uh two updates uh union and update operators so the union operator basically merges two dictionaries but it preserves the original dictionaries and the update operator is for in place merging of two dictionaries again i have added this code snippets along with their screenshots now the zone info module was introduced so this has included the entire uh internet assigned the numbers authority ina timezone database that you can access and work with different time zones all over the world then uh we have remove suffix and remove prefix two new functions uh uh earlier we had l strip and r ship methods but there uh we've been stripping off characters from like either ends and uh of the string using these strip methods but the remove prefix and remove suffix or these two functions uh can now use a bunch of permutation combination and you can strip off your strings using these uh new methods so uh other cool release highlights include see python the new parser that has been introduced so this is all based on peg we can still use the old parser which is ll1 now guido who is the creator of python he said that based on his research peg parsers would be more robust one thing that you should do is check out if your code runs smoothly with the new parcel has the command that you can find in the blog itself then multiphase initialization is now available python community has adopted this new release cycle that they would be coming up with every new release a new version every october so this was under pep 602 then pep 614 is for relaxed grammar extinctions uh on decorator so uh this was all about python 3.9 uh you can check out uh more details from the blog post so the next project that uh we have on the list is cnn explainer now this is a very fascinating and very cool project that i came across while scrolling twitter now cnn explainer is basically a browser-based visual illustration of a convolutional neural network and it is fascinating uh to see you know how we can just simply write a few lines of code using libraries like keras or fast ai and we can create state of the art cnn models to classify objects but understanding the mechanics of how cnn processes an image to obtain the final output is actually very crucial now this is where this project comes in and it explains uh it actually gives you a behind the scenes uh of how a cnn model actually trains and classifies an object so uh this entire project was uh built on tensorflow.js which is an in browser gpu accelerated deep learning library to load the pre-trained model for visualization and the entire system uh is actually written in javascript using swelt as a framework and d3.js for all the visualizations that you're looking at now you only need a web browser to get started with learning a cnn here using this particular uh blog post uh and and this uh very cool uh visualization so you can actually click on uh any of the neuron that you see and it'll actually show you what's actually happening so i have an input layer and then we have you can see this is a convolutional uh layer first convolutional layer you can actually click on these and it'll show you how it is training so you can see uh the convolutions and the filter this is basically the kernel it's a three by three kernel the input size of the image was actually 64 by 64 and after uh the convolutions and the kernel ran over this image uh we got an output uh after the computations were done so it's basically just a dot product uh element wise dot product that was performed and you can see all the values uh the kernel rates and the biases that were being added so very cool project uh to actually learn and understand how every detail is working you can press on play and it'll keep on running and show you uh how the processing is being done then you get a list of all the layers so this particular model this particular system cnn explainer uses a very simple cnn which is called tiny vgg which comprises just three convolutional layers a couple of max pool layers as you can see here and a flattened layer to classify an image into one of the 10 classes which is basically lifeboat ladybug pizza bell paper school bus so on and so forth so it's just the tip of the iceberg you can actually use these tools to understand you know what's a tensor neuron kernel weights and biases you can actually understand what's happening behind each layer input layer convolutional layer max pool layer what do they do how do they reduce the size or what's what are these hyper parameters that we have so you can actually run so they have given some really cool visualizations to understand kernels padding kernel size stride in a cnn and they have also talked about activation functions like relu softmax that we keep on encountering uh in our day-to-day deep learning projects so uh do go through this entire blog post as well as uh this entire project is on github itself and and see if you can collaborate in some way really cool uh project that i came across and the credit goes to the team so cnn x this was created by jay wang robert turco omar sheikh uh haikyuu park nilak thus fred hoffman and minsook kang and so this was a result of a research collaboration between georgia tech and oregon state very cool uh kudos to the guys who uh you know delivered this really cool article uh which helps you learn the mechanics of convolutional neural networks so another really cool project that i came across was it called tensor sensor so there's this guy named terence per so he has developed this library uh which is basically uh you know created with the purpose of handling exceptions when building neural networks when you're working with neural networks and you're working with different matrices or tensors of different dimensions and you're using libraries like numpy tensorflow or pytorch and you know that you would be working with different dimensions of tensors so one of the biggest pain points is to get all of these tensors of different dimensions uh to line up correctly so usually when you are using or any of those three libraries numpy tensorflow pytorch and you encounter an error which is related to the dimension of the matrix uh let's say computing matrix multiple multiplication so the interpreter basically throws an error which is you know uh without any details without any specifics it's hard to debug so what you do is you actually go and write print statements so that basically gives you uh what the dimension of that matrix was so i have created this i was trying out this library so it's uh you can install it like simply like any other python library uh pip install uh tensor sensor uh so i've imported this it's imported as the sensor so uh run this now i have created some a sample code i was trying this out so here's this rows which is let's say called the number of instances columns is number of features and then there's let's say 100 neurons so i've created like weights uh biases and this is my x the feature is it and the y uh the target variable so weights let's say i have created a using uh creating some random numbers now for the features for each of those neurons so this would be my weight matrix this would be my biases uh biases again uh one dimensional and then i have created my x which is my feature set matrix so here uh rose is thousand four hundred and neurons so uh you know that this uh is uh so when i'll be computing this dot product uh it would be uh difficult it would be it would basically throw an error so let's see so this is the error which is thrown by the interpreter now if we had to find out like what uh where it went wrong we'll have to type shape and similarly you will do that for all of these and find out like where it went wrong so it gave you that this is the line where the error occurred but you'll have to figure out by looking at the dimension of each of these so let's change it uh okay so now you see we have 400 and 200 a hundred into one and we are trying to multiply w with the x so you can see uh there's uh the dimensions don't match basically uh get this right you have to do all of this debugging now with tensor sensor you can actually write this code as with the sensor dot clarify and it will give you this exception uh message which is very easy to understand and gets you up to speed like tells you like where it went wrong so you see it tells you that the w matrix that you have is 400 by 100 whereas the x matrix that you have is 400 by 1k so the dimensions don't match and that's why you're getting the these errors now there are bunch of other fact functions that this guy has written so there's extract syntax free that you can develop there there's explain function that you can use so do check this out do try this out uh here i have added the link again in the description so he has provided snippets for all of these different libraries pytorch tensorflow and a bunch of other cool things that you can do in case you want to customize it to your project or as per your need so next up we have this news from nvidia where they have launched their cloud ai video streaming platform which will use gans generative adversarial neural networks to boost bandwidth performance and as you all know that during these covet times uh we have been facing a lot of bandwidth issue everyone wants to get on a video call uh be it business be it personal relation be it uh anything so video calling has become uh an integral part of almost all the major industries so uh what nvidia has proposed is that they've come up there with this solution where they're using gans in place of the software called video codec which was typically used to compress and decompress video for transmission over the internet now their work basically enables a video call with the one tenth of the network bandwidth which users typically need to conduct a video call now how this works is that a sender first transmits a reference image of the caller now rather than sending a fat stream of pixel-packed images now this system sends data of locations of a few key points around the user's eyes nose and mouth there is again which is generative adversarial neural network on the receiver side that uses the initial image and the facial key points to reconstruct subsequent images on a local gpu and this is again using nvidia gpu so as a result you would be sending much less data and the bandwidth requirements will definitely uh become more efficient so the technique works even when you with the callers are wearing hat or a glo or glasses or mask or headphones and they are still working on a more serious thing which is uh called gans recruits deconstructed free view so here what they're doing is they're using neural networks to align the position of users faces for a more natural experience so that the collars would look like as if they are looking into the camera because most of us have this uh habit of looking into the video feed so we're basically looking at the other person but whereas actually should be looking into the camera so this would give you that free view uh where you would get uh you know a feeling of face to face connection so uh that's what they're working on so the max and video calling is something that we should be looking out for in the near future in almost all of our uh video calling uh platforms and softwares now the last project of this october series is the coolest one this is called google magenta tone transfer now this lets you transform everyday sounds like birds chirping you know pants clicking or any or your your voice your singing everyday sounds uh into musical instruments so the stone transfer is basically transferring one tone once some type of sound into a very melodious musical instrument sound and this whole project was a collaboration uh between two teams within uh google research so this was magenta dean and the aiux so there were a few ux engineers who collaborated and designed this whole flow and it transforms everyday sound into musical instruments and this whole system is basically uh built on a technology that the google magenta team open source earlier this year that was called ddsp which stands for differentiable digital signal processing and they early used to have this only colab notebook which they used to uh use in order to showcase what they have been working on but now they have created this entire platform that you can use uh and and play around with so here's what i have for you from don't transfer if you [Music] could have stayed [Music] [Applause] [Music] [Music] [Music] hahaha now they do this by training machine learning models to distill what makes an instrument sound like that particular instrument and it picks up edge octaves loudness etc and removals everyday sounds into musical instruments now to extract patch from audio another model developed by google research which was called spice that was used and upcoming releases will enable you to easily tr train your own uh ddsp models and deploy them on a phone or an audio plugin or a website you can actually check out their github repository you can contribute to it you can learn how you can use these this particular technology and create something cooler so do check out this very cool project on audio so i hope you found these projects cool or useful they can be new business ideas new project ideas or maybe you would want to collaborate or contribute to any one of these so if you enjoyed this video do give this video a thumbs up and do let me know in the comments below what else should i add while reacting to these projects or while demonstrating them or how else can i make it more entertaining and interesting for you and do subscribe to the channel help us grow and share it with your friends and i'll catch you guys in the next one until then keep learning data science