Media Transcoding for 10 Million users and beyond with Metaflow at Epignosis

Key Takeaways

our next speaker is um chrysostomos galatos chos joins us from Greece uh he's a devops engineer at epignosis learning Technologies uh they are using metaflow for some amazing use cases for transcoding media files serving their vast user base of over 10 million users uh this is really really awesome so chrisos thank you so much for joining us and and over to you let us know how you use metaflow hello uh thank you first of all Ryland great job um that's fantastic um it's the first meeting I join here for meta uh of sant um even even the first use case I is fantastic so let me let me start by sharing my screen um um I may have to relaunch Zoom so I I might take 10 seconds I'm sorry yeah yeah no way all right here he is we can see your screen chrus yeah we can see your screen being shared okay that's perfect uh so um first of all good morning or good afternoon depending on your time zone um I'm here to share how metaflow has held our engineer team at eposis um so a few things about eposis first uh osis uh is a leader in the learning technology sector uh we develop a suit of products designed for corporate training and uh workforce management uh we have a global footprint uh we have more than 11 million Learners worldwide using our platforms and the product Spectrum ranges from microlearning mobile apps to to large scale fully customizable Enterprise Solutions um um in short we have uh we have four products five for or five products mainly um Talent LMS is our Cloud hosted uh learning management system uh it's the main product of the company and it's suited to small to mediumsized and businesses uh Talent HR is our solution for Human Resource Management uh e front is tailored for larger Enterprises and can be either be hosted by us or can be hosted by the customer on their on their own premises and talent card is a mobile lab uh for micro learning through uh flash cards small learning cards and lastly Talent Library complements our our platforms with a collection of some pre-made courses uh so a few things about me and the team uh my name is chrisos as thank you for introducing me to three uh I work as a devops engineer at the pignosis in January um I I first started to code a couple of years ago uh now at the pignosis uh I've been I've been there for two years and I've been part of the devops team for the last nine or 10 months um there I'm working alongside a small yet highly proficient team of Engineers uh we are Curr are currently for engineers and we are responsible for S reliability architecting Solutions U mainly everything everything that has to do with cloud or AWS so we're a pretty small team um so what led us to meta flow um for us media transponding is a pivotal operation due to the nature of our learning Solutions uh we deal with a lot of video and uh and audio every day uh and to be precise we we Pro we get about 5,000 jobs a day that's files that a customer may have uploaded so these files are mainly course video videos so they tend to be lengthy and sizable and um another aspect another aspect you have to consider regarding media transcoding at the pnos is that uh we offer unlimited storage to customers so they are free to upload as much content as they want um the content that they upload uh may be consumed across our our offerings and and products and not only different products but different platforms because for each product we have a web version and we also have mobile apps um and uh we also and and and and all this uh together amounts to to a great cost for us regarding storage um so um our existing solution um was developed in 2013 uh it was uh it was an app you built with Python 2.7 and Ruby um although it sh do well initially with time it started to show signs of strain and uh and that was to be expected because osis grew almost has grown almost tenfold in the last three years um so um it was also supported by a contractor which in itself posed a dependency risk for us um and they abused the custom pre- Cloud era autoscaling mechanism and it was becoming difficult to maintain that extent um so Qui hey quick quick question so just to kind of you know understand the setup here so eposis sells uh solutions for companies to kind of sort of create their own training uh training material exactly so if if I'm a retail store and I'm going to have let's say you know 50 different employees at 50 different locations I can create the material for uh how do I do checkout and how do I do whatever how do I stock things or how do I plan for buying new things so all of this material I can create tutorials for this as videos and whatever text and whatnot and I can upload it to eposis platform is that right yes that's exactly right so we offer the the platform and and the learning tools a learning manager may need to set up training for their for their corporation beat uh video courses beat quiz and test um or or chain chain training material um we offer offer exam functionalities um so um and and this these these files that are getting uploaded the audio video uh files they need to be processed yes so that they can be like like you mentioned like you can serve it on a web browser you can serve it in an app you can serve it on desktop computers maybe with like different types of screens and whatnot so to be able to do that you need the transcoding and I'm guessing that's where probably metaflow will come into picture is that right exactly um okay cool our customers you know um they uh we have almost no limitation to the to the kind of media file they they may want to to use on the platform um but uh we have to to to re-encode everything uh because uh we have to to make it as um as accessible to anyone as possible and right now I'll be touching on this on the next slide so um I'll continue there um let okay so that goes that some parts of the new solution that we wanted to have we wanted to have a new solution um that uh that would be autoscaling so um it would be cost effective and the performance would be there we had to develop to develop it ourselves in house uh so in order to reduce dependency on external contractors uh it had to be cost effective of course uh it had to be extendable and that was uh and that was uh one of our main targets because um uh because of the of the small capacity of of our team and and this and the low number of our members um you know we we wanted to to make sure that if we commit time to a solution it would be a solution that we would be able to extend it in the future and not only that but we would also gain knowledge that would be uh beneficial for other projects in the future um H also it had to be feasible and that and that's um that it may sound kind of weird but um it had to be something that we could realistically implement it uh within with our capacity and and lastly it had to be maintainable uh not only by um the our team the devops team that is only for members but uh it had to be easily maintainable by developers from from our products as well so these requirements kind of steered us towards meta flow um so um uh we we got together in Athens uh and we started an internal h let me say it and our inspiration for using metaflow was the diffusion metaflow project because we are only already experimenting with stable diffusion image creation and we came across diffusion metaflow and uh that's when it clicked that metaflow can can also be used for task like that and not only um Mach not it's not only a machine learning uh framework um so how we we use metaflow now um it's not a great it's not a greatly complicated solution it's pretty simple actually uh we have a simple task we want each file that is uploaded to be re encoded to h26 264 so it's compatible with everything and so and also we wanted to downscale a video if needed if the source resolution is too high and we don't support it and we also want to be able to apply watermarks all these are very easy to do with fmeg so our stock is based on the cloud formation uh outer bound is uh is um uh maintaining we have made some slight modifications but they're not not worthy we use the simplest Docker image uh with FFM binaries uh it's just plain python 3.9 slim uh for production we export workflows as the function as the functions and uh for consumption by our products we use uh we use Lambda triggered we trigger those functions through Lambda functions uh and that's all behind an AWS AP Gateway uh we also use spot instance pricing um because uh its job is very short and it's 100% tolerant to to fults we have uh we don't really care if if if a if a machine gets reclaimed by WS because go uh with the functions it would get retried and that's fine for us and we have also done a canary launch and right now we have almost 25,000 uh Warf free conversions um so uh why met flow is here to stay at AWS um f first of all uh repeatability um our workflows have become very easy to replicate and our conventions are predictable the um um we have after developing the solution we have had no no instance of of something unexpected happening um what was profound for us was how quick uh how Qui we went from idea to to to an alpha version we could test on the products uh that was two weeks and that's huge for us uh remember we are small team so um when we um when we uh allocate time for for this not only by our team but by also by product teams uh it's it's easy it's EAS not to find that time to never find that time so having such a quick turnaround was a huge win for us um another great benefit that we had was that the whole project was handed over in two days uh meta flow you know the workflow is pretty simple uh we don't have a very s phisticated workflow uh so it was very simple to onboard new people on how they will develop using metaflow um as everyone knows we have no no maintenance with infrastructure it just works great with AWS resources um development is really really fast and uh what we really liked was the the the fact that you can export workflows uh as as step functions because um our all our products have have had to have a a transition a transition towards Cloud because they are they are products that have been in the market for more than 10 years and in the last years um we have we are we're actively pushing developers to start utilizing Cloud resources more so this uh makes it very uh very easy for us to uh to uh help developers consume uh our workflows within our products and due to the to the easiness of developing of on meta flow um anyone can help chip in and experiment uh the enthusiasm among the developers has been infectious and uh we are hoping to lay like to lay down the groundwork for for data and machine learning talent that is bound to come to to to the company we are still in early stages uh in those sectors so um make it by chance I don't know but you know um coming into metaflow was was a huge step for us on that regard um some future ideas and projects uh we we started discussing is uh extending our media transcoding uh to redubbing in in other languages uh for video localization this will make it easy for our clients to have content accessible to uh to to many different linguistic groups um regarding stable diffusion we have already a project that uses it and we are actually now exploring doing that through metaflow um up until now we haven't Venture into hosting our own inference we are using um uh hanging pH inference end points for that and uh last but certainly don't least we are right now developing a new a data warehouse within within a pignosis and we're pretty sure that metaflow will play pivot role in our in our ideal tasks um um when it come and also one last thing that we are actively thinking about using metaflow is a large scale report creation we have customers with more than 50,000 Learners that that required that required huge reports very detailed reports on the progress of each of each learner so we are actively uh exploring Outsourcing all of these uh to to to metaflow um so um that was it um I hope I gave you um a quick glimpse of how how metaflow has enabled a small team like us to to to iterate quickly and to uh to to bring prod projects to production that are truly wor free for us that's the greatest benefit metaflow has given us uh thank you this is this is awesome Christos this is really good thank you so much for sharing this one few points that quickly came to mind by the way is you mentioned about like uh wanting to um run the the metaflow jobs um today they are run periodically but you could imagine kind of sort of running them uh based on events things like when a user uploaded new files into S3 or wherever uh you want to trigger a job which immediately starts the transcoding or the downsizing or adding The Watermark or things like that so some of those things are also available in metaflow if you just look at like metaflow event triggering you'll find some of the documentation for it uh on the open source metaflow you know docks uh so you can imagine kind of s making these things also event driven rather than having to I mean of course depends on different use cases but if you want to do things where a customer is already kind of you know onboarded and you know they add three new videos today uh then automatically triggering a pipeline based on that event and then making all the transitions from there happening it's it's possible through metaflow so that is really uh would be something interesting for you to look into that's great you for sharing