Enhance insights and efficiency with Google Cloud object tracking

Key Takeaways

currently I will be speaking about unveiling the power of Google Cloud object tracking and henching the uh enhanching insight and efficiency so I'm a software engineer at moal right now um and this is more about me so currently I'm working in moal it's a driverless vehicle company based in Singapore uh and previously I work in spoify and travala currently I'm also a master student in data science at University of edin bra um aside for um Google Cloud other things I'm also active in the cloud native foundations thingy for example I kubernetes and I'm also active in the uh woman techmakers and woman in data science previously I'm also a mentor for uh grasshopper open source day in 20122 and become for track shares for machine learning and data for cube Cod and cl dative con EU for 2021 and 2022 so this is the today agenda we will be having why object tracking matters in Data Drive and work and next we will be having Google Cloud video AI transforming video intelligence and next we will uh be talking about uh object tracking specifically and next we will be having the case study for mity manageable Transformations with Google Cloud so why object tracking matters in Data Drive from word so you can see on the screen that there is um a road where there is like a lot of vehicles and um there's like a bounding box for each of the objects that we have um and what is actually object tracking so object tracking is a process of identifying following objects like people Vehicles items over time using data capture from various sources such as camera sensor and GPS so imagine that you have like rifless vehicle or yeah maybe some kind of like poish um uh CCTV you can actually identify what kind of like object that can be seen through the sensor or like cameras that actually available and what is the functionality for it so it is actually not being used to um identify the object it also track the locations for the objects in the space and also predicting their future movements and behavior based on uh accumulated data so uh this is um the graphic on uh the increasing number for Publications in object detections from 1998 to 2018 we can see how how much it already increased uh over 3 years and mostly it's actually about the real time inside meaning that for example like for for my case before there is um a real-time object detections for example for prous vehicle um or maybe like inventory related details for retails um and there is actually the emphasiz of the value of immediate data processing analytics for timely decision making um and there is also like Predictive Analytics realtime tracking feeds into Predictive Analytics allowing business to forast Future Trends and behavior with higher accuracy what is like the correlations between object tracking in business we want to gain more granular insights for example like we have a customer Behavior analytics where we are eventually tracking customer movements and interactions and we will be able to kind of like gain patterns preferences thus leading to personalized marketing and improve customer experience another example will be like asset managements imagine that you are in the logistic and man manufacturing um sectors you want to kind of like making sure that the streamlining process between uh providing the insights uh into the operation workflow making sure that all is in place and then this is like uh real work examples we want to kind of like enable automated object tracking system to process and analyze data much faster than the humans which is crucial in environments where real real time tracking and immediate data processing our Essentials such as a logistic manufacturing security survillance and we also want to make sure that it can be easily scaled to monitor large number of objects simultaneously because there is like um uh we we we understand that human is BR to error when we need kind of like focus on multiple details for surf aliance related matters so yeah and then uh how Google Cloud uh fi actually transforming the uh video intelligence Technologies so uh this is um how uh video analysis can be used um Sorry video analysis um will that we will have with the Google Cloud videoi so we have like precise video analytics we can recognize of than two 20,000 objects places and actions uh in start streaming video and then we can extract uh the frames and metadata for each of the layers for example and we can create our own custom entity labels with auto m video intelligence so we can use like the existing models and then playing around with um adding more annotations for each of the layers that we have on the video and then we have simplify media management meaning that we can search uh our own video catalog the same way as uh we actually search uh documents and then we can we will be able to kind of like extract the metadata and this can be used to index organize and search the video content as well as control and filter content for what's most important and we can easily create intelligence video apps where we can gain Insight from video in near real time using streaming video annotations and Trigger events based on object detected um and we can also engage in customer experience with highlight reals recommendations and more so this is the difference between the automl video intelligence and video intelligence API for automl it provides us a graphical interface for easily for easy custom model training and for the video intelligence API we will have uh the pre-trained models for recogn recognizing numerous objects and actions with continual improvements so if you are still new probably you can try the automl video intelligence and use like the existing graphical interface but if you want kind of like do some kind of like iterations then you probably want to kind of like use the video intelligence API more so this is the key use cases for Google Cloud videoi uh this is like content moderations so we can kind of like instantly moderates content across extensive data and we can provide content recommendations builds recommendations engines using label and user preferences uh based on like video layers for example and we can also add like a media archive create indexes archive for easily for easy accessibility and then we will have like contextual adverstisement by identifying optimal and placement based on the video content next um we will talking more about object tracking itself so what's the difference between object tracking and label detections because eventually with using label detections we will be able to kind of like um know what kind of like object it is so the distinct features will be unlike label detections uh that only profile label without special context object rackings offers labels for individual objects with bounding boxes to see um the object the multiple objects that we have on the uh video for example at every time step so uh for example imagine that we have like multiple cars inside of of like a roads then if we we are only using the label detections we can only identify that all of those are cars but using uh object tracking we will have the locations for example the special contacts and then we will have like the object ID and we will also having like the label for example like pedestrian or maybe like U vehicles or um other things and this will this will be processed on like time step basis for each of the time frame for for the videos so this is more about like object tracking we will be having like multiple object tracking uh we can detect and track multiple objects in the video and returning the levels or TXS associated with each detected entity including the locations for each entity within the video frames and the limitations will be very small object within the video might not be detected and the the difference from the label detections we already like talk about this before and we will be having more detailed labeling and time segment informations because of like the object metadata for each of the object uh track entities and we will having like uh additional entity informations uh based on um the Google Knowledge grab search API So based on like the entity ID and then the locations we will be having like more informations regarding that so understanding object tracking and the context of Google CLA video intelligence it involves analyzing video content to identify and track objects throughout the video durations meaning that it will um analyze uh per frames entities and then we'll track each of the objects available at a given time stamp and to initiate object tracking the annoted method is called by object tracking specifi in the features field and then we have the key features of object tracking that is the entity and loc locations annotations the system annotates video with labels for detected entities and their special locations for instance a traffic video might generate labels like car trucks bike Etc and we have bonding box and time segment uh where each levels actually coming with bonding boxes and each associated with a Time segment indicated through of offset from the video start so we can ID uh like trying to kind of like search based on uh the offset and look at the uh motions of the each of the track IDs that we are having on on the videos and then we will having like the entity informations so The annotation itself will include additional entity information such as the entity IDs and this can be referenced in Google Knowledge Graph Search API for more details so this is actually one of like case studies uh for um a smart City management Transformations that actually using Google Cloud so imagine that we have a cities that contains like 63 portoles There is lot of holes on it and uh we are wondering if there is like a way to make um the government efficiently um kind of like repair this spot holes um and then this is the fact that approximately 32 3,200 work hours were dedicated to annually to port hole repairs which is like a lot of manual works and it might not be as efficient as we want it to be and then residents used mempis 311 and and app to report the issues meaning that this is basically based on like the users like um contri contributions to like the apps and then uh we will try to kind of like GA where the issues is mostly located at and this unfortunately only capture about 20% of the problems and it's still not that efficient so what can we do kind of like improve these conditions given that we already tried to kind of like Outsourcing the informations from the users um but it's still not covering that much of like our problems so fortunately we have uh the city bus where the city bus itself having like more data for example like video uh using like a camera and it actually kind of like track down uh how the road conditions and then uh in sense based on like this informations we can actually store this data and then um process the data letter and evaluate the video video with AI and decide which course of actions to take if there's like a p hose on the roads and based on like this recommendations we can dispatch Crews only on like the uh necessary place right so imagine this is like data that we we get from like the cameras for from the bus there is like utility poles um and then like wire Loop and everything is actually kind of like tra and we can see that there like a bonding boxes and then track IDs for each of the objects um and it's movement so we can easily like uh identify which uh utility poles that we want to track for example and where's the locations and the time stamp where the events is actually happening so we can kind of like sort out the priority of which one that we want to fix first right so uh this is like um the flow on how it works actually so imagine that we have like the cameras and sensor datas from the city bus and we we want to kind of like ingest that into cloud storage and we will have like data pipelines using data flow and we will store it in cloud storage and the query and doing some kind of like analytics first to understand how the pattern is and after that we already kind of like understand what uh kind of like technologies that we want to go which is kind of like machine learnings uh by using like object tracking and we will be using AI platform which API and then surf flow for example and the output can be um seen by the maybe the government or other stuffs right and it can be deployed on like app engine for example or uh bigquery which allows like people to kind of like get a recommendations on which ones that kind of like being Argent to be fixed first so this is like an example how like me CS actually shows the uh recommendations for um the government to dispatch the crews for example here is like uh an unreviewed issues and there's like detected date based on like the informations on um informations from the bus cameras we we we understand that object tracking will provide like the time stamp right and we know like when this video is actually being taken and we will have uh this checkbox of being unreviewed meaning that this is like a new case that needs like uh attentions from government and the government can decide whether we want to like fix these issues first or not if let's say there's like a lot of data that report there is like um same problems uh for specific track ID which is like the track ID for the roads for example then it might be like something something that um kind of like urgent to be fixed and this interns actually saving quite a lot of like cod it it it saves around like 10K until 20K in year uh in related to claims and enhance the prioritizations and discovery of issues like incorrectly paed areas which is like quite nice because we don't need to kind of like do the job manually but we still be able to kind of like prioritize um the like fixing of the issues and yeah we will be working more efficiently with dispatching the uh C crew to like the uh the area that we want to fix [Applause] yeah