Data Version Control in Python and Real Python Video Transcripts | Real Python Podcast #25

Key Takeaways

welcome to the real python podcast this is episode 25. wouldn't it be nice to use a form of version control but for data something that would allow you to track and version your data sets and models well that's what the tool called dvc is designed to do this week on the show david amis is here and he's brought another batch of pi coders weekly articles and projects david starts with a real python article titled data version control with python and dvc we also cover several other articles and projects from the python community including where to get exposure to well-written code delegation composition and inheritance good python project ideas for high school students never running python in your downloads folder and more we also have a special guest this week i talked to sadie parker who recently joined the real python team to help create and edit transcripts for all the real python video courses talk about how you can take advantage of all the features this new resource provides city also discusses how she uses python to speed up and simplify the editing process the transcripts and closed captions are now live on the website for all new courses and we're working our way through the back catalog alright let's get started [Music] the real python podcast is a weekly conversation about using python in the real world my name is christopher bailey your host each week we feature interviews with experts in the community and discussions about the topics articles and courses found at realpython.com after the podcast join us and learn real world python skills with the community of experts at realpython.com hey david welcome back hey good to be back all right so you cool with starting off this week with your first topic yeah let's do it okay so i've got a real python article by christiane evanczyk hopefully pronounced his name correctly christiane has done a couple of articles i'm not sure if there's a second article or third article for us now but uh he had about a year or so since the last one so this is he's come back and written this excellent article for us on data version control with python and dvc and it's a really cool project this dvc which stands for data version control so data version control is a concept right controlling version control for your data dvc is the project name okay it's written in python it's a python it's sort of like git for data and it's really really cool i think it solves a really a really interesting problem that i think a lot of people run into i think that it's not uncommon for people data scientists people working in academia to have your spreadsheets and your data just kind of all over the place and you know you have some weird version control of you know maybe underscore date whatever some number at the end of your files and you just kind of keep track of them that way and it's a difficult way to sort of keep track of versions of your data set you could use git already to version control things especially if you're working with like text files and things that are small right and things that are small yeah if it's a small data set then you know git can kind of take care of that for you but as you start getting into the realm of larger and larger data sets git becomes troublesome for that and it can be difficult to to deal with that so what dvc does is solve that problem if you have a large data set that you need to keep track of then dbc is an excellent choice for that so christian walks you through setting up a working environment using dvc and everything and and he walks through the workflow and kind of compares it to git and there's a lot of i guess cross over there there's there's a lot of similarities between dvc and and git so for example you can initialize a repository for dvc using dvc init sort of like the git init and that you know sets a few things uh sets a few things up for you so it creates a dvc folder that holds all the configuration information so this is just like the dot get folder that git does and it sets up the repository for using the dvc you can add data to a cache so it works on like a like you have a local cache basically and then you would have like a remote file storage system and the remote could be could be another folder on the same machine or it could be an s3 bucket or it could be just you know anywhere really i mean it's it's compatible with almost anything you have some remote storage and a local cache so again that's very similar to the way git works but uh you've got a dvc add command that adds things to your local cache sort of like git add it's kind of analogous uh to that although it does it's not quite the same that dbc add just a little bit more it not only says hey we need to track changes on on this file you you add some file using you know dvc ad and then file name or folder name or whatever it says okay we're going to track changes than this but it also puts it into that that local cache for you already so that's a little bit more than what git add uh might do then there's a dvc commit command which is like git commit if you have new changes to a tracked file you commit those to your local cache and then there's a dvc push to push whatever is in your cache to the to the remote storage and then on the you know on the pooling side getting data from remote storage you've got a checkout command a fetch command a pool command that does like a fetch and then a checkout so there's a lot of similarities between dvc and git yeah it's really targeted towards large data sets that need to have the data version controlled and it just integrates really nicely with with existing kind of git workflows and everything so christian walks you through this whole workflow using sort of a pet project so you build a machine learning model it's nothing fancy honestly you know he's kind of got the the code there for you it's not you know the model itself is not the point of the article so it's really not trying to do anything fancy or get great accuracy or anything like that it's just you know kind of a quick and dirty thing to kind of get you going uh but you've got an actual file that you can work with and or an actual model that you can work with you work on training it and then evaluating the performance and then getting that model and the data into a version control and he goes through some interesting ideas on you know how you how you might work this in with with a git workflow so for example when you have a machine learning model you do a bunch of like experiments and the experiments might use different training data or maybe a different split or maybe just you know different things that you want on the experiment maybe a whole different data set so he talks about one way you might do this is have different branches and get per experiment so he kind of walks you through a a way to integrate all this with your your git workflow and have something that is easy to reproduce and share with people he takes a deep dive into something called dbc files which those are the text files that actually get committed to your git repository so your github or gitlab or bitbucket whatever your whatever you're using but these little dvc files contain like a pointer to the remote storage uh for your data so instead of having to commit that large data you're just committing this little tiny little tiny text file that has like a hash in it and you know some other information and then another really cool thing about dvc really goes beyond the get analogy is you can create these reproducible pipelines with it so you can set up like the what needs to happen for fetching the data what needs to happen for preparing the data what needs to happen for running the training and then evaluating that training run and you can stage all this stuff in a pipeline and use a dbc run command to run that pipeline and and then it's reproducible the whole workflow is reproducible for everyone on your team or whoever you're you're sharing this with so that's a really powerful feature that uh that's really cool so the article just yeah it walks you through all of this christian is a talented writer i think he has a gift for explaining things really well it's just a really enjoyable article to read if you're in the data space i definitely recommend checking this out i'm really kind of excited by the idea of it especially the branching for your data that sounds like that would be really helpful yeah especially you know with a lot of the data science kind of things where you are trying different experiments or you're shifting from training data to other data and so forth yeah exactly all right so my first one is a reddit thread it's titled expanding exposure to good code and it's sort of this idea of like you know where can you go out there to find well-written code and kind of learn from reading code and this has been an ongoing topic that i've asked as a repeating question on the on the podcast but this kind of goes a little bit further beyond necessarily going into reading git repositories um which we've suggested in the past it has a bunch of other resources that people suggested that kind of went in slightly different directions instead of saying oh go read the flask documentation or go read you know the source code off github and they mentioned some some resources that i wasn't familiar with one is called python module of the week which i thought was kind of neat which oh that's a good one yeah yeah it goes into depth on you know individual modules and kind of lets you kind of focus your attention like okay this you know and again you can kind of pick and choose because it's been been going on for quite a while you know subjects that you're interested in and you can kind of use it as a checklist to kind of continue to build on your learning there's a an algorithms implemented in python get repository that i think is really kind of neat too one that i think i've heard before mentioned on a different podcast probably python bytes but with something called awesome python applications oh yeah another github repository and this one is is talking about actual applications that are out there in the world where the source code is something where you can go and read it and they're organized by the type of application so it could be you know an audio player or a game or some kind of other web resource and so forth where you can kind of learn you know how did they structure this type of code for an existing you know thing that's out there and and it's a really neat way to see not only you know i guess one of those awesome lists you know of like okay this is these are created in python and kind of give you an idea of what what's happening with it yeah that's a really cool repo i think the talk python to me podcast actually had the author of that repo okay on to discuss that so there's a there's a whole podcast episode that goes into it talks about some of some of the different projects yeah but uh yeah it's probably grown since then so that's a fantastic resource for sure yeah so again wanting to go and read and learn from them and then somebody else mentioned a youtube video which i started watching this morning just to kind of give heads up as i'm getting prepared for us to talk today and the youtube video is by james powell it's from 2017 um pi data conference and it's titled so you want to be a python expert and it starts off in a really interesting way diving into sort of explaining dunder methods and you know that kind of functionality and it's done in a really well structured way he doesn't have any slides or you know anything he's kind of working from memory as he's going through this list of okay as you create a class you know what does this class need to do and and so forth and it it's really well presented and it's almost two hours long i didn't go all the way through it but i'm excited to watch the rest of it so i think it'll be a good resource to kind of again kind of take a different angle on the way uh of thinking of you know how to learn python he was he initially complained that he feels like a lot of books that are out there uh say oh you know here's here's something that you know that is a feature of python you know and then they go and here's another you know feature of python here's another one and so forth but it doesn't ever really get into the thinking that goes behind in this case this idea of data classes and and how that relates to all these dunder methods and you know some people think of like magical methods for these things but if you're creating and using object-oriented programming and creating these things it's kind of vital to kind of think in this different way as you're creating them and i felt like even within 20 minutes of me diving into it that i was like oh okay this is really tying together a lot of different thought processes you know as i've been sort of immersing myself further and further into python it was i think it'll be really useful for for a lot of people that are in that kind of intermediate stage yeah definitely cool so what's your next one yeah the next article i've got is by leonardo giordani and it's about object oriented programming in python specifically the topic of delegation before i talk about though i want to kind of go back this he's got this whole series on object-oriented programming in python that i think is is really well done it goes into a lot of depth and it's also very approachable i think he's got really good examples that he uses in the articles they're they're very short to the point and sort of easy to digest which is enormously helpful when you're trying to learn something like object-oriented programming because it it can be it can be a lot to take in at first if you're kind of new to this whole concept so having very concise short examples is really uh helpful but there's three pillars to object oriented programming that he's gonna expand on on this whole series and that is data encapsulation delegation and polymorphism the article that we featured in in pi coders recently and that we're talking about today is on on delegation and it specifically talks about how you sort of assign tasks you mean i mean that's kind of what the word delegation is right like delegating something to uh something else so it deals with these things two concepts called composition and inheritance and inheritance is when you have like a parent class and another child class that inherits from that so you could have something like i don't know a vehicle class and then a car class that inherits from the vehicle class and maybe you know all vehicles have things like i don't know a make and a model and uh maybe a like a type of engine or some sort of propulsion system whatever it is that makes it go right that makes it a sure a vehicle and then the child class a car is going to have like a you know very specific kind of engine which could i guess be now like an electronic engine or it could be a combustion engine or something like that but but the the parent class is like a more general concept usually of of the child class that's one type of of a way to sort of delegate things that the child class can use things that are defined in the parent class to work yeah the other type of delegation would be composition which is where rather than inheriting from something you assign another object uh to like maybe an attribute on the on a class so maybe you have some class that needs to incorporate logging and it needs some sort of logger well rather than implementing the whole logger in this class you can implement a logging class and then use a logger instance in the in that class to to handle the logging so it's then delegating the task of logging to this other logging class or the instance of the logger that you've uh attributed to it when you create your instance of the of the class but so this is the idea of delegation how are you delegating different tasks around and he talks about sort of four ways to to look at delegation and i'll just go through them fairly quickly so one way to look at delegation is the how you're sharing state so an inheritance a child class has access the child class shares its whole state with the with the parent class so it's like everything is just getting absorbed completely so that's one one way in in uh composition it's you're like picking and choosing which pieces of state kind of get shared like you're passing something into a method on the class that's being composed so it's you're not sharing all of your state you're just like whatever you are comfortable giving to it basically so this is the idea of sharing state in these the other thing is about control so he says another way to look at the dichotomy between inheritance composition is that of the control we have over the process so uh there's an idea of of control there's also relationship between two things so the way he describes this i think is really good is it to be versus to have so in the inheritance type of delegation it's the 2b kind of relationship a car is a type of vehicle right but in the composition it's a to have relationship right a car has an engine so maybe the right the you have a combustion engine class or a uh electric engine uh class so that's another way to think of the difference between these two 2b versus to have kind of composing of parts and exactly functionality okay and then also the different kinds of entities that are involved in the different kind of delegation so in inheritance you're dealing mainly with classes you're inheriting from some parent class whereas in uh composition you're dealing with like instances of other kinds of classes so you're assigning some instance to to that that class that you're you're creating so he just really goes into the the different kinds of delegation inheritance versus composition i think it's a really critical part of understanding object-oriented programming and sort of getting your head around like when should you use inheritance versus composition and most programmers are gonna most experienced programmers are going to prefer composition over inheritance whenever possible it's sort of the safest route it kind of makes the most sense in most cases but you know you need to know when to when to use it when to not so he gives some pointers in this uh he calls the section bad signs but he says you're incorrectly using inheritance when there's a clash between attributes with the same name and different meanings or you feel the need to remove methods from the child class you're incorrectly using composition when you have to map too many methods from the container class to the contained one or you're composing instances but creating many class methods so that the container can access them so it gives you some idea of when you might be doing it wrong i think that actually is kind of a good approach to teaching it because it's sort of like it's hard to know when to do it so it's good to have an idea of when not to do it right i guess right it's kind of easier to know i know you're traveling down the wrong path yeah exactly like it's it's sort of easier to spot like oh you start doing it oh no this this is not looking good i probably need to rethink uh the way that i'm i'm doing this but anyways it's a really in-depth article with lots of examples and then this whole series is quite long he's got uh several articles in the in the series about object-oriented programming and even some uh some videos that he's published to youtube so he talks about types and classes and how to create a class classes and instances those are all the parts before this part 5 delegation so if you're new to oop you might want to check that out before you get into this one but honestly i i feel like if you've already got the basics of oop down you understand the difference between classes and instances and things like that then jumping into this article is not going to be too difficult sounds like it's trying to give you lots of examples like i like the example they have that he created in there the one with the windows sort of like it kind of goes to like gui stuff if you ever used a gui framework right and how very often are inheriting a lot of things like okay this is a button okay but this one inherits from this general idea of button but it's it has a text field inside of it too it's like a submission button or some kind of other thing that's kind of building on top of it and so that makes sense to inherit it that way or he uses the example of a window versus like a transparent window it's still going to have you know resizing methods and you know right information methods and all these things that you feel like every instance of a window should have these things and that that makes sense for that type of path but i can see how you can really start down the wrong path pretty quickly you know if you kind of jump down one of these methodologies and you can kind of tie yourself in knots trying to fix it all and overriding everything and yeah it's you know inheritance is one of those things especially that can get real tricky the the you you develop this hierarchy when you create you know parent classes and child classes and child classes of other child classes and you can get really deep and nested and then it becomes really difficult to reason about what your code is doing and where things are actually happening and what's going on so it's it's good to know when it makes the most sense to use it but you also want to know when when you might be uh might be time to sort of rethink how you're how you're doing stuff so definitely my next one is is another reddit thread and i thought it was kind of interesting because we've been focusing a lot on our episodes of having projects at the end of them yeah and the idea of this thread was okay what are some good python project ideas for high school students now there's a caveat because it's got a long kind of initial introduction to it it's like okay well who are we talking about okay well generally high school students might think one way or another but in this particular case he's saying okay i have you know going to divide them up into teams and they're going to have you know i have an overall number of 10 individuals but mostly i think the core thing that a lot of people missed in it is that these are students that have expressed interest in continuing on in their programming education at a university potentially in cs or a related field and so he wants some additional skills to be gathered through doing this so uh sending them off to you know do something that doesn't really kind of he wants it to i mean i think any kind of project is going to help you build your skills but i think there's different types of projects that will help you kind of think about okay well do they need to learn about git and and doing source control do they need to learn a little bit more about working with an api um is it something that would be useful to kind of dive into and so i thought a lot of them one of them that i always have liked is a text adventure yeah because it kind of builds on top of the idea of object-oriented programming in kind of a fun way you are sort of integrating functions and loops and conditionals and and those sort of things and it doesn't always have to go down the route of it being like some kind of you know d and d kind of style thing i've heard text adventures be everything from like something that's happening with spies or could be you know something that's happening with the wild west or you know whatever you want to think of like just kind of think of some kind of adventure that somebody could go down the lines of and then modeling the real world you know it could be anything you want to create so you have to kind of maybe give some extra parameters there but and also you know i felt that way about creating games you know having to manage all the different assets and so forth yeah getting into a visual one kind of gets a little more advanced something like pie game or something like that you definitely need to think beyond like simple like characters and so forth but in general i think it's good to start there one of the main other suggestions was about visualizations and and working with data again if you have some people that express interest in in that kind of thing like they're interested in science and not necessarily purely cs then i definitely see that you know creating projects around that and there's so many great data sets we've talked about that before but you know things like kaggle that can provide really good resources to get people started on that but then it got into like you know creating a jukebox um you know some kind of audio player and that could lead them to learning about gooeys to learning about handling files and kind of all the different sort of interactions and potentially you know thinking about user interaction and having to work from there i think things like circuit python are great for beginners and the thing that it's sort of missing is potentially the whole source control kind of thing you do get this sort of great interaction in in working with hardware and building physical projects and i think that's truly exciting maybe tying that in to you know an api or working with like a website or kind of like you know having that physical thing that's out there and having it interact and seeing its actions but maybe having some kind of way of it tying into data might be even more useful that somebody else could like oh i want to look at this website and see you know the results of you know what's going on with you know this automation system i set up or this you know sensor i have for soil or light or moisture or something like that yeah and so i think all that's really useful and then you know just building a website these days like if you look at the tutorials for things like django girls you're diving pretty deep in building a website with like a framework like that you know building like a blog engine with something like django and it leads even to like you know testing and stuff like that so yeah it has a lot it's a huge thread yeah it is and i think it has a lot of really good things that are not only useful for high school students but you know again for those those of us who are intermediate developers that want to have projects to kind of mess with and you know web scrapers bots um things like that and you can find a lot of additional fodder for your own projects there hopefully yeah exactly that's why i decided to include it in in pi coders is i've just felt like when i was reading through that thread i was like this is not just good for high school students like this is good for anyone that's a beginner to you know early intermediate python developer so yeah there's just there's lots of interesting ideas there uh so if you're looking for a project and need inspiration yeah it's a good a good thread to peruse definitely [Music] this week i want to shine a spotlight on another real python video course it covers a function that can help when you're working with multiple iterables in python it's titled parallel iteration with python zip function the course is based on real python article by leodonis pozo ramos in the course instructor liam pulcifer takes you through how to use the python zip function for parallel iteration how to create dictionaries on the fly using zip how to sort multiple intervals and more i think it's a worthy investment of your time to learn the zip function and start wrangling multiple iterables to solve real world problems and like most of the video courses on real python the course is broken into easily consumable sections and you get code examples for the techniques shown check out the video course you can find a link in the show notes or you can find it using the newly enhanced search tool on realpython.com [Music] all right what's up next uh the next one i got is called never run python in your downloads folder and this comes from glyph lefkowitz who's uh the creator of the twisted framework this article on yeah what what happens or what bad things can happen when you run python in your downloads folder i'm going to just go ahead and say i've never run python in my downloads folder and i'm not i don't know if that's something that some people do or not but the i think the point of the article though is not so much that it's like focusing on the downloads folder specifically although he does mention a like an exploit that can happen that specifically targets the downloads folder and i guess what prompted this but really it's about understanding what happens when you execute python yeah and what things are trusted and what things aren't trusted and this could happen you know it pertains not just to the downloads folder wherever you're running python it's good to know what really really happens and so he starts off saying python needs a safe space to load code from yeah and there are three critical assumptions embedded in python security model every entry on sys.path is assumed to be a secure location from which it is safe to execute arbitrary code the directory where the main script is located is always on sys.path that's kind of a key thing there when invoking python directly the current directory is treated as the main script location even when passing the minus c or minus m options so these are the three key things to keep in mind and again it's not just about the downloads folders about any folder you're in when you when you run python and then execute some script whatever the current working directory is it anything in there is fair game it's just gonna say if if you're trying to run some other file in here i'm just gonna trust that it's safe and i'm just gonna do it yeah and that's really the key thing to take away from from this article and i think that it's something that for some reason gets overlooked when people are learning python it's something that i have to admit it took me a while to sort of understand how this actually worked in in python or what was what was going on honestly the idea of running arbitrary code felt a little odd to me at first and was like wait a second it does what like it's just gonna run that code in there like okay um well it's good to know now that i need to to take those things into into consideration but yeah it's something that i feel like it's overlooked a lot to for beginners and this also plays into something that i've i've made the mistake of doing it's actually how i ended up learning that this was a thing and i've seen other people have this issue as well and it it's when you are trying to use say a module in the standard library but you have in your current working directory or maybe even the file you're trying to run is named has the same name as the module you're trying to uh to import so like at collisions and stuff like that well what happens like let's say you have a a file called uh math.py in your okay in your directory and you've defined a couple of little math functions in there or something or and in another script you you open up you say i need to use i don't know the pi constant from for math so you import math and then try to use math.pi and you might get an error saying there is like there's no you get a name error there's nothing right in math called math.pi because it's importing your math module that's in your directory and not the one from the standard library so there's like this precedence right of what what happens where and what gets loaded first so that can be surprising i think another thing that's surprising is when people you know they download i don't know maybe they have a like a well we could we could use the same math example there you have a math.p wifi it's your main script and then you import math into it from the standard library well now you're going to get import errors because it's like just all sorts of weird stuff is is going on so you get some some strange stuff happening and when when people run into this it's like it's super confusing but it all boils down to the way that where where python is looking for things and the fact that hey like there's an order in that current working directory or the directory where your main script is located those are going to get put onto path and it's going to have a higher precedence than say something in the in the standard library and so one of the i guess bad things that can happen with this that can actually be bad is if you there's this uh category of attacks called dll planting and that's where certain browsers can be tricked into putting files with arbitrary file names into the downloads folder without any interaction from you while you're you're browsing and say for example you're on a website and you there's this dll planting attack occurs and it puts a pip.py file in your downloads folder and that pip.py file does something really bad i don't know it steals some data or it deletes your hard drive or i don't know it does something terrible well if you're in your downloads folder and you go to check something and you go okay python dash m pip install whatever well you're just telling python to run the pip dot py file that's in your downloads folder because it's gonna find that first because it's gonna find that first it's in the current working directory and it's you're running it python dash m pip you're running it as a module so it's gonna say okay oh look i found this pip.py before it ever even tries to go to the pip the actual pip is oh i found this pip.py and it's just gonna execute it because it's gonna assume that it's that it's safe and then whatever is in that pip.py it's gonna happen in the you know you're gonna your hard drive's gonna wipe down your date is gonna get stolen something something bad is gonna happen so that's the that's the article never run python in your downloads folder it's a catchy title but really the important thing to in this article is just knowing what happens when you run when you run python and how it treats code that you know if it's in your current working directory it's fair game whatever it is it's going to run it's going to assume that it's safe kind of goes back to that idea of like names are hard but you know using these really generic names uh for things again that's a theme that i've heard you know the guys on python bites talk about not using utils um right you know this whole idea of like you know creating a proper name and thinking about the the namespaces of okay what exists you know it even goes back to keywords right you know like this idea that understanding structurally as a you know beginner or an intermediate python person like these are things that are pretty big pitfalls that you can fall in with not realizing it i've you know i've seen people create a you know a a request.pi file or something like that exactly because that's what they're doing and it's like well okay but yeah and now you've renamed something that you're importing and you're you know and so you're getting this whole like collision and so forth and so it i haven't seen anybody say okay this is the best solution but the idea is like okay come up with unique names for your code that is you know going to be run first right it has that higher priority so that definitely what's it say in the zenon python here it is namespace is our one honking great idea let's do more of those exactly yeah cool so i guess that leads us to talking a little bit about projects so the project i want to talk about is called present and it's a presentation tool kind of like allowing you to do slides inside of the terminal so it's a terminal based presentation tool with colors and effects you can play these pre-recorded code blocks called a codio and that allows you to show off you know you as you're working through steps inside of programming if you wanted to it uses markdown so you just say the simple dot md markdown file so that includes you know the standard markdown things for headers the normal things for inline code links quoting lists but you can also just indicate the change to another slide instead of having a separate file you just put three dashes in a row and those three dashes indicate moving on to the next slide and then along with the uh kodios and the other things i mentioned you can have styling for the background and the foreground colors and then there's also the ability to put these uh slides with effects on them and the effects are all done in ascii which is really kind of cool looking of fireworks or explosions or a version of like the matrix kind of coming down it's pretty slick and so it's a neat way just to if you wanted to do a talk and do a quick set of slides for it you could do that all inside of this terminal tool all written in python uses another python library called ask schematics so check it out so what do you got for a project yeah this is a fun one it's a mini raspberry pi boston dynamics inspired robot the creator calls this the spot micro so i don't know if if you're listening if you've seen this boston dynamics they've created this like robot dog that i guess has applications and things like security and maybe even like military or things like that but but it's this i don't know if you really call it cute the actual like robot dog the micro dog is actually kind of kind of cute um the the big one is yeah sort of terrifying actually in a lot of ways but uh depending on depending on what they have it running out there and doing yeah yeah exactly but cool nonetheless i mean it's uh it's a huge feat in in robotics so this project is taking a raspberry pi 3 some servos an lcd panel and thingiverse 3d printed mic spot micro frame so that's you can actually buy this 3d printed frame that like all this goes into and building a controllable robot out of this that mimics the spot boston dynamic spot robot dog so the code is written in c plus and python and there's a github repository that's got all the code in it as well as like very detailed instructions on setting everything up as well as some ideas for like future work to expand on the on the project yeah like incorporating lidar developing autonomous motion incorporating a camera or webcam uh so they could do like basic image classification and things like that so yeah there's i mean there's a lot to a lot of things to expand on on here and it just looks like it'd be a really fun project to build you know over a i don't know if it'd be a weekend project for you know maybe if you're really good with hardware and really for me this would probably be like you know a month i've probably flown around every weekend to get something but there's a cool video it was featured in the raspberry pi blog okay and that's the article that we've linked to in in pi coders but uh there's a cool youtube video they've got in that article that's got you know the demo um he's in his uh living room and he's got it walking around and he's controlling it from his from his laptop so i took a little bit of a browse through the code and it looks like most of the control api is in python okay whereas a lot of the like the actual i guess would like the hardware control like what's actually controlling the hardware most of that is in nc plus but then the actual you know where you're orchestrating things and sending it commands a lot of that is uh is in python so it's also a cool project then i guess to see those two languages integrated with each other right like you've got those things uh talking and everything so yeah just a really fun fun project if i had a lot of free time i would i would love to tackle this one yeah i wonder what the price of the parts and stuff um the raspberry pi i can imagine is pretty inexpensive but i've never bought any robot other parts and so forth motors and servos and things like that yeah that's a good question my guess is most of it is pretty reasonable okay because of its size because of the size and everything and the the 3d printed thing you you down i guess it's free to download yeah it's free to download the all the files but then you would need access to a 3d printer and the materials to actually go okay print this which i think you know a lot of public libraries nowadays have some stuff like that yeah that's true yeah and there i think there's like maker spaces and if you're in a larger city you can usually find somewhere that you can they'll have the material and you just pay for the material and some time to go use the machine so depending on where you are i guess that's a big part of it like if you're located in a big city then you probably actually have fairly easy access to a 3d printer although i don't know how that works in the time of uh coronavirus maybe there's some that makes it a little bit more difficult but um yeah but yeah it just looks like a really fun project yeah all right well thanks for coming on the show again yeah thanks for having me all right talk to you soon see you later next up it's my conversation with sadie parker about creating transcripts for real python's video courses we discussed how you can take advantage of all the features and functions these transcripts and closed captions provide and we also talk a little bit about how she's using python in the process of editing and creating these transcripts so here's that conversation hi sadie i wanted to welcome you to the podcast hi chris thanks happy to be here it's really cool to have you coming on not only the show but also to come on the team to help us with you know creating these transcripts for real python maybe you could give us a little idea of what the project entails and what you've been doing yeah sure so it's honestly been like quite an honor to work on these transcripts and captions i mean real python is like a really great resource for me i do some learning of python on the side of my other projects and so it's always been something in my peripheral vision for the past couple years and it's really fun to work on some of this video content yeah and yeah i started with you guys about is it like a month and a half ago now yeah yeah we've been plugging away at the existing video courses there's a lot and we've been [Laughter] about a year and a half of them so yeah we've been i think handling transcripts for new courses really well and everything's going smoothly it's yeah it's fun yeah so i wanted to maybe go through questions people might have about them or like reasons like i guess maybe the first one to start with would be why is real python adding transcripts to the video yeah so i think first off it just adds a lot of a lot of value and it makes the video courses a ton more accessible it definitely helps with searchability and findability for the courses and for specific course content inside each course so that's really cool i don't know but what i heard was that from you guys is that there was quite a lot of requests for transcripts and captions just because of how much it aids with learning yeah when watching video courses yeah i wanted to kind of dive into a couple of those things there so the accessibility part again i think kind of getting this feedback through channels from dan he was saying that multiple people had requested partly it could be they wanted to use this visual medium but they couldn't hear the instructor speaking but now having the words there along with it as closed captions and then also it's really kind of neat the whole transcript is contained below each of the videos there that somebody can kind of go through and just read through what's happening so there's an accessibility thing for again somebody who can't necessarily hear those things but also if somebody wanted to go back and what did that person say it can actually go and look through all the stuff that's happening there or if your english isn't your first language somebody can kind of look through it and i think that is huge for the accessibility yeah yeah and then one of the other things i was thinking about that that you said there is you said searchability how has that changed like with having the transcripts um so now that there's like transcripts below each video course or the ones that are coming out it makes it quite possible i believe to search the site with the keywords of what you're hoping to learn about and videos that specifically mention that specific topic will show up in the search results and it'll also more easily show up in google search results and another part just within the courses themselves like if you'd like to skip forward to a particular part in a video if there's just one topic you're hoping to zoom in on right something you're stuck on right yeah you can you can just um head over to that particular sub section on that video course and like what i would use would be ctrl f and just look for those words and you can click on the transcript and it'll bring you right to that that time in the video so that's really helpful yeah i think that kind of brings us to that idea of the navigation part of it that is kind of a neat feature inside of it it's not a static set of you know words that's on the page that actually interact with the video itself acting as a way that you can if you use that search functionality if you see a sentence that you know say maybe you're looking at something about lists or something like that and there's a particular method that was mentioned and you search for that method you'll see the full sentence and as you click on it it jumps to that section of the video and repeats that whole sentence and so it's a way of navigating very quickly yeah really helpful yeah i think it's going to be really useful for learning and then the other feature that's added with that is that that's a link right so that you can actually share because you've actually done it for me i've asked you to since i'm working on the video courses you've been able to say okay this person's mentioning a link here and so i've been adding in this other section of the text that goes below the videos this area called the description if there are links that the video instructor mentions i can then you're like giving me this highlight and it's like literally like time code almost like exactly where in the video i need to go look for it yeah those links are really helpful they've been super helpful for me like in trying to standardize kind of the syntax and the formatting for these transcripts like in asking for feedback from the rest of the team like those links are very helpful for just getting everybody's eyes on one portion and trying to decide together what's both the most readable and the most accurate way to to transcribe this particular idea so that it's you know both reflecting the instructor's intent and accurate to the course content and emphasizing you know the main ideas behind the course yeah i love those links they're they're really nice yeah i thought about maybe we could talk a little bit about the process that that you're using there's this whole extra layer and partly of what your work is of taking what's being automatically generated and then making sure that it's correct yeah is that a big part of what you're doing yeah so we get these auto-generated transcripts from rev ai and they're they're really good you know especially especially non-technical speech they're quite accurate as far as word content yeah and then what comes after that is just going through and making sure that for the non-technical bits just making sure that the punctuation accurately represents what's being said so that if you if you don't hear the audio you're getting the right meaning from the words that are there yeah and then for the more technical parts we're going through and making sure that when when a particular verb is a function that we're making sure to represent that it's a function when a object is an object we're going with the the real python formatting standards and making sure it's being represented as an object because the words behind behind python and behind programming of objects and methods and variable names and everything they exist within like english syntax but they also touch on ideas specific to programming so it's pretty important to make sure that that we're representing that in the transcript so that they are helpful and like as good a learning resource as possible so yeah a lot of work goes into that just making sure that that that's accurate yeah i can tell like the amount of kind of back and forth over the last you know month or so of like okay well this is how i think it should be represented you know as code this is what i think will make the most sense as far as like what the instructor is trying to the pinpoint and having to kind of even bring those people in to say okay yes is this what you meant and again most of the video courses are built based upon articles there's there's some research that can happen there but very often the instructor's adding new content or adding new concepts to it so it's kind of you know a whole team of people that you have to kind of rope and wrangle to to make sure that this is you know coming across as the point that it's going to make the most sense to the person reading those transcripts as they're moving along so as you start this process what are the tools that you're using and what's the process that you're taking to to create these yeah so uh we start off with these vtt files and um basically we're just comparing it against the video and audio that they were generated against and to speed up the editing process and keep everything consistent i've got some different little macros and workflows for the most commonly replaced words or the most commonly misrepresented words from the