Detecting Deforestation With Python & Using GraphQL With Django and Vue | Real Python Podcast #64

Key Takeaways

welcome to the real python podcast this is episode 64. are you looking for an in-depth data science project to practice your skills on or perhaps you would like to add new tools to your python web development projects instead this week on the show david amos is back and he's brought in another batch of pie coders weekly articles and projects david shares an article about how to go about detecting deforestation from satellite images it covers how a data science team built a machine learning solution to do just that using fast ai for the modeling and streamlit to create a dashboard we also discuss a real python article about building a blog using django vujs and graphql graphql is a great tool to enhance your api to make it more flexible the step-by-step project walks you through turning your djangoblog data models into a graphql api we cover several other articles and projects from the python community including the tools and tech used to run a one-woman hardware company visualizing data in python using plot.scatter why the sad face when using black how to iterate over data frame rows and should you pipex is now a pi pa member project and real-time lossless audio compression in python with pi flac this episode is brought to you by digitalocean's app platform all right let's get started [Music] the real python podcast is a weekly conversation about using python in the real world my name is christopher bailey your host each week we feature interviews with experts in the community and discussions about the topics articles and courses found at realpython.com after the podcast join us and learn real-world python skills with a community of experts at realpython.com hey david welcome back hey chris thanks for having me back all right we got a whole bunch of pie coders goodness here yeah i'm going to start off mine this week talking about real python article from previous guest dane hillard dane hillard was on in episode 49 he was talking about his book practices for the python pro and kind of what makes a the term professional developer and near the end of the episode he dove into something he was interested in learning about and working with which was graphql and i guess he was probably already starting to work on this article at that point and this is another step-by-step article on real python so it's not a super deep dive into the technology like describing it in in sort of like uh research on that it's much more of an implementation and learning how to use this and set it up and kind of get your chance to play with a lot of this technology and so it's called build a blog using django which we've covered multiple times but then the next two commas are vue which is a javascript library kind of similar to react in some ways and then graph ql which is what he was talking about briefly at the end of the episode he was mentioning that he was interested in this query language and i was not that familiar with it and i've learned a lot more about it and i'll include some additional resources there's a javascript sort of centric podcast web development podcast called syntax and they have an episode where they talk about graphql in a lot more detail there's a lot of information out there it sort of originally developed in the facebook camp and it's open source and you know since 2018 has become a pretty popular way of sharing apis and those of you who've maybe played around with apis a little bit there's sort of this problem of having to set up an endpoint for everything you kind of want to access or the reverse of it is you set up an api and then it dumps like this huge amount of data back at you and it would be nice to be able to kind of you know sort through that and sort of query that and narrow it down and that's really the idea behind graphql and it's hard to really explain in this format it very much looks like json it's it's a the querying is very json-like in the sense that you're specifying the attributes and things that you want to to pull out of this information but it's it's pretty neat and it allows you to move beyond like having all these separate endpoints and and the pile of information you get back and you can kind of specifically ask for what you're looking for and then you can use it on the client side requesting it but you can also so like if you know you want to talk to other tools that are out there using graphql or you can use it and set it up on your own on the server side and kind of interestingly this tutorial takes you through kind of setting up both sides it starts out with really just getting going in django and you know setting up the architecture there and and using that as your initial like layout you not only install django and get it set up you create some simple models for a blog that go a little further in the sense that they now have like okay there's an author and some additional fields that that makes sense inside their publishing dates and things like that kind of you know fairly standard but you know more than the usual and you can kind of see why why you may want to to do that to be able to see all this extra information you might want to look through once you get the django blog set up you set up the admin and we have some other additional courses on this that kind of dive into the admin side where you can kind of customize you know how it looks and feels and what kinds of things that you want people to be able to enter from the admin side and then jumps right in and step three setting up this thing called graphing django and so that's the library to sort of do the you know requesting and talking between django and the graphql services and then eventually you get into the step where you're going to be installing vue js and setting up a view router to help with this and that's where you're kind of delving much deeper into javascript and working with it and again like all our tutorials the the code's all there so you can either as you work along copy and paste the code from the article as you're working but this it's all also available in the github repository so you can kind of compare what you've created and and check it all out so anyway so eventually you're kind of building up this router and then you're creating the separate view what are called components and they're very similar in my opinion to like you know the models and things like that but once you kind of get it set up there's this there's like these two different sort of endpoints that you're setting up your server will have a slightly different port than what's happening from the django side so like 8000 versus 8080 and so you're kind of talking back and forth on your local machine kind of fetching data and pushing it back and forth it's a really fun kind of way to kind of explore and learn a little bit more about this stuff and see how not only how you might implement vue but this very advanced you know way of working with apis that really looks like you know kind of the future of how you could work with apis in a much more elegant fashion of like getting to the specific things that you want and then also from your side as a developer it's much easier to sort of update and change your api just by sort of restructuring the code that's kind of inside there in the graphql language stuff that you're working with so i think it's a neat one to check out yeah i really like this step-by-step project format and you know we've got more of these coming down the pipeline and you're right this really isn't you know these this style of article is not really about teaching you about the different technologies as much as it is showing you how to combine things you might already know about into an actual project right which i think you know fulfills a need that kind of goes beyond just like a tutorial on you know just view itself or just graphql you really integrate those things and see how they all fit together in an example project and it could be a launching point for you to then go on and create uh your own thing so right so yeah it's definitely fulfilling a different need than like you know it kind of assumes for example you know something about django you know something about view maybe you know something about graphql it's not really necessary because you can kind of copy and paste and follow along and dane does a great job of sort of showing you at each step like what your project structure should look like like what folders and files you should have and where they are so you can if you don't really know too much about vue or javascript you can still make it through the end of the project and have something that's working but yeah it's just a great way to to see how different things integrate with each other and get an example project going yeah i really like the idea of like standing stuff up and then being able to sort of poke at it you know so right if you were to then go and look at other resources to learn more about vue or to learn more about graphql now you have this working project and you're now able to try those techniques out on your own services that you've set up yeah and can sort of customize them and so i think that's a nice practice for sure more than just sort of following the step-by-step you know which is always you know good to kind of get the muscle memory of typing things in yeah for sure yeah so what's your first one you got first one i got is called detecting deforestation from satellite images this one comes from andre ferreira it's a full stack deep learning project and it's sort of similar to what we just talked about with uh dane's django view graphql it's kind of a end-to-end like we're building this project together it's not quite the same kind of step-by-step format that we have on real python this this one is a medium article on the towards data science blog yep but it's it's a really good example of again like how you would approach putting together a data science machine learning project end-to-end like starting with they even start with for example like okay what data set are we going to use they investigate a couple of different data sets and they make a decision and they explain why they picked that particular data set then what uh what model they're going to use like what what package are they going to use for the for the model for the machine learning how are they going to present the information and the results what kind of dashboard are they going to have it just is the whole nine yards the article is not incredibly technical in terms of there's not a lot of code in the article they've got links to the project repository on github so you can go and you can look at all the code but it really focuses more on the decision making process in this and sort of like what they were thinking about they're trying to build sort of a minimum viable product of something that could be used to detect where deforestation was occurring from satellite imagery and they start by looking at kind of a plan for the project it's okay so what what what does this thing need to do and of course that you know is a great first step for any project you know sit down and sort of map out like okay what is it that we actually have to get done here what are kind of the key points so they talk about you know for this for this mvp the minimum viable project what uh you know what what do we need to have well we need to have a model that's trained and has acceptable performance on detecting deforestation we need to have a dashboard to show the predictions on the data set we want to have a github package we want to do some hyper parameter tuning that is you know optimization to improve the model and we want to have a robust data storage solution for the project so these are kind of like the minimal goals they set out ahead of time they talk about some like things that would be nice to have things like what they call out of domain or distribution shift analysis like they you know if they have the time it'd be nice to study the model's performance on data that's different from what it was trained on to kind of see how it works there they'd like to have tests for their package so yeah they talk about you know kind of these nice to haves and really lay out like a road map for like okay this is what we want to get done for the project so it's really cool to see that you know the things they're thinking about what they're looking for like i said it's really kind of a guide to like how would you approach building a project so then they go into okay how are we going to do this like what are we what are we going to look at um where is our data going to come from so they talk about you know doing a bunch of research looking at a bunch of different data sets and this is not you know exploratory data analysis this is just like just researching what data sets are available maybe looking at kind of the raw data just to kind of see like what's in there and it might it work they narrow it down to three data sets there's a data set about of images from the amazon from space there's one called the wids datathon 2019 which has got oil palm plantation detection in borneo and they've got another data set called towards detecting deforestation which is a data set for detecting coffee plantations in the amazon rainforest after they they look at this they narrow it down to this amazon data set which is looking at the human footprint in the amazon rainforest and they decided to go with that data set for training the model then they get into the exploratory data analysis and data processing phase so they they kind of talk a little bit about what they were looking for in this data set what was available some of the plots they were making to sort of understand what's there things like tagging stuff like that so they really get into how they were approaching the exploratory data analysis and they do some of that on some of the other data sets too and they kind of compare notes and they sort of explain why they picked the amazon data set then they get into the model how you know what sort of model are they actually going to use here and do they need to build this model from scratch is there something available they decide to go with fast ai which is a really cool project from there they they kind of talk about okay you know what kind of how are we going to set this up like what fast ai model are we actually going to use they use something called the resnet 50 model which is a convolutional neural network with 50 layers they show a little bit of code on you know how they're kind of setting this up and working with the fast ai and kind of showing how fast ai takes care of a lot of things for you so nice little code snippet just showing how they can kind of quickly get this set up then they get into okay we've got our model we've got our data how are we going to display the results right so what about this dashboard and what are some of the things we want to include they end up deciding to go with streamlit which i believe we've talked about on the podcast in the past yeah really cool library for making dashboards again they're just discussing this whole decision making process in all of this then they get into the data storage they end up going with some solutions on a google cloud and talk about how that's integrated and everything yeah it's just a kind of a walk through end to end like you know here's what we want to do and here's how we're going to go about building it and and why we made the decisions we made so it's really you know if you've been studying data science and machine learning and you're still kind of maybe stuck on like okay how do i actually put all this together into a full project well this is a really great example of how you do that and the kinds of questions you should be asking and the thought process behind making some of those decisions so yeah hands off to andre this is a fantastic overview and article and a really great resource nice [Music] digitalocean's app platform is a new platform as a service solution to build modern cloud native apps with app platform you can build deploy and scale apps and static websites quickly and easily simply point to your github repository and let app platform do all the heavy lifting related to infrastructure get started on digitalocean's app platform for free at do.co realpython that's d o dot c o slash real python [Music] this week i have articles and and features that are all for previous guests yeah it seems like so um this next one this next one is from stargirl flowers who was on episode five we were talking about circuit python at the time and she has created quite the little hardware company in the last year it's kind of amazing yeah and it's getting she's getting her stuff distributed in multiple places now and so she decided to create a post sort of explaining and the title of it is the tools and tech i used to run a one-woman hardware company so she's been making these boutique synthesizers and eurorack modules and there's quite a bit that kind of goes into sort of turning it into a company and not only just like it being like just a small project of designing hardware but also you know in a lot of cases those pieces of hardware need microcontrollers and the microcontrollers need firmware then there's the whole levels of like documentation and you know building and things and so forth and how can a single person do all that and so it's a really neat article kind of diving into it it's it touches on python quite a bit but it also just sort of kind of looks at like as a technological thing like kind of thought processes that would go into it so it it sort of starts off with like kind of an overall philosophy and she has a quote from gunpei yokoi and his quote is lateral thinking with withered technology and that's kind of her philosophy is sort of following that same idea is to leverage things that she already knows and is familiar with and can be fluid with but also look at things that are not necessarily the cutting edge all the time right and what helps with that is not only can they mainly be more understood because they've been used in other projects or they've been documented and then also in a lot of cases they're you know already kind of being maintained by others exactly and so that really led to the types of micro controllers that she's using and the main ones that she's been focusing on and literally you know choosing the chips to do this they're from a company called microchip and the series is the sam samd series there's like three that she focuses on there's like a nice table kind of explaining you know everything from like the processor speed to like sort of ins and outs that they can handle in communications and then also like how much ram they can have which is really small so thinking about like you know trying to program these things yeah measured in kilobytes yeah you're very very small yeah so these are similar microcontrollers to what's in the arduino zero which is a very popular kind of like you know prototype board and stuff to play around and then the adafruit boards which i'm gonna my project that i'm kind of messing around with uh uses this m4 style board um they also have an m0 board and there's a few others that she kind of leads out to so goes into that whole reasons for choosing those and and why working with them and then she talks about hardware design which she did go into in our our episode talking about this thing called kicad k i cad and it's an open source design tool for designing circuit boards and i got into electronics years ago and i was playing around with the idea of building like guitar pedals and and things like that and and i had this ancient book from a guy who's really famous in the you know diy electronics space back when i got into it and his name is craig anderton and he had this great guitar projects book and unfortunately most of the kits that he's talking about there you know would use parts that would be probably hard to find now and things that have been replaced and so forth but if i wanted to take some of his designs and i wanted my own you know board design with certain layouts and so forth this thing called kicad is really neat for doing that and she's had experience creating this uh genesis uh synth called the genosynth and then it led to her two other projects and now she's got like two a synthesizer one called caster and pollux and then uh another new one that she's working on anyway so keycad's a nice tool for that if you're interested in you know designing actual circuit boards and hardware for that and then surprising that with the resources now getting prototype boards or actually getting actual printed boards is doesn't require you to make thousands of them which is really nice too then it goes into firmware and she is using c which she has a background in not you know offshoots of c but actual just pure c and then um talks about the compiler that she's using there goes into firmware libraries and again this goes into the withered idea the idea that using some existing firmware libraries will be helpful for documentation and building upon she has links to all the repositories i don't know if i mentioned it but she considers the whole company an open source company so the the designs are open source the firmware is open source like she wants to you know get back to the community in the same way that you know she's getting in this way which is really neat so you can kind of you know a lot of people ask about like checking out code and so forth this is a really neat one especially it gets into this idea of build systems and running those compilers and you know outputting firmware she uses python and a build system called ninja and then it kind of wraps up kind of talking about python based tooling that she also uses and then kind of dives into circuit python and her first two projects the big honking button and seoul are both these circuit python platforms yeah which is it's a really neat platform and i hope to be talking about it a lot more we talked about it in that episode but the idea that you can just plug in a usb port of your computer and you know up pops up you know a little dot pi file that you could just start writing python into and then send it over and it it starts working on it it's just so cool you know and then when you unplug it you know if you have a power source or what have you you're you're running that code it's it's really slick then it kind of wraps up on documentation and creating user guides and she's a fan of markdown and using this tool called mk docs to convert the markdown into web pages and she's hosting them up on github pages because it's all just static files so her stuff she hasn't dived into the latest javascript stuff that we were talking about earlier she's trying to avoid some of the node.js ecosystem it's just yeah huge and hard to stay on top of and if that's not really your focus then that makes sense and so she's created her own little javascript library for working with things that that are useful for what she's trying to document and show and and you know create in there and that involves uh working with audio elements you know to actually play little audio snippets creating forms and then another tool which allows there to be uh web midi playback yeah and i'm i'm interested in that one to see kind of what's there but so uh her winter js uh javascript library is up on her github stuff too so it's just a neat article kind of diving into all that sort of stuff if you're interested in the electronic sides of things but also the idea of like okay if you're gonna you know create sort of an open source not only project or company like some good best practices there and she's been a maintainer of projects for a long time and was a psf fellow the previous year and so anyway need article check it out one thing i want to point out on uh on star girl's article is that and she kind of explains this kind of throughout there's little mentions of this you know all the decisions that she's making are coming from the standpoint of i'm working on this by myself i have to do i have to do everything yeah and there's a really i think interesting comment here at the end of the javascript section of of her article that says you know if you're a javascript developer and you go digging into my javascript code you might find some of my style choices offensive i use four spaces and uh snake case right so like right and you know and she says i do this so that i'm consistent with the c in python code which makes context switching much easier for me and to me that is like it's really great advice right if if you're a single developer and you have to deal with multiple languages and you've got common style that is valid in those different languages why not use it so that when you're context switching like it's just easier to deal with like the the the formatting looks similar just easier to digest and yeah you know if you've got a team of people that are working on this and they're kind of isolated from the people that are working on the other components of the ecosystem then sure you know they can follow the style guides and you know whatever best practices they they decide on and everything but uh i don't know i just thought that was kind of you know it sounds sort of in a sense defensive kind of like she's you know she's defending her choices from you know potentially people saying look this is not the way you should write javascript but i also feel like it's a really powerful lesson there for for folks that are in a position of look you're trying to do something by yourself you need to maximize your productivity and to do that you have to make decisions that make it easier for you to to context switch and to maintain things yourself and all of that so anyways just thought that was a really kind of powerful lesson and something that uh you know there might be kind of a mentality of like oh i'm learning javascript now so i need to like i need to write my javascript according to whatever the the current standards are and everything it's like well you know if you're on a team that's doing that or you're trying to get a job at a company like yeah you probably want to be aware of that stuff but you know if you're just trying to like get a job done right like focus on what maximizes your productivity that's that's the most important thing yeah i think i missed one of the other main themes that i should have included also is the idea that across the board thematically the idea is again for a single person trying to do everything is to you know literally not use up as much energy and resources and be as you know all those things you only have so much time you only have so much energy exactly but in the same way like materials wise if you can keep reusing the same materials that's great but she wants her projects and the things that she builds to be you know like with this dealing with this global chip shortage right yeah so if you were to use a project like the soul project that she has that is this reprogrammable sort of control voltage to midi tool but if you can go in and make it do your own thing by reprogramming it then you're not buying additional hardware not you know like throwing something else into the dump you know you're actually looking at ways that you could own this hardware a little more and and reuse it and modify it and that's something that she kind of really focuses a lot on throughout the article too which i kind of missed earlier sustainability yeah totally like just you know there's a sustainability as a human being as far as your own life and time and energy but also just you know from hardware also which i think is a really kind of neat focus too yeah cool for sure so what do you got next next up on my list is another real python article it's called visualizing data in python using uh plt.scatter this comes from stephen grupetta and this is kind of an introduction to creating scatter plots uh it actually goes a little bit beyond just an introduction and you learn a lot about making and configuring and customizing scatter plots in matplotlib it starts with you know you know how do you create a scatter plot what is a scatter plot there's this function called scatter that you use to make scatter plots and it gives a very basic example of how you do that here's some data points and uh you know let's let's plot them so he's got basically an example where you've got a cafe that's selling six different types of bottled orange drinks and the owner wants to understand the relationship between the price of the drinks and how many of each one that uh that they sell so you you plot price against sales per day and you get the scatter plot you can kind of see a relationship there so you get this basic example one of the things that i learned working uh with steven on this this article and in the outline and technical review phases was that there's a difference in performance between using the plot function which is kind of the general plotting tool in matplotlib versus the scatter function uh so you can create this exactly the same scatter plot as you know as far as the final product goes it looks exactly the same using either the plot function or the scatter function but there is a performance difference and plot is actually more performant so that that was something i wasn't aware of and hadn't even really thought about it just seemed like okay if i'm making a scatter plot i want to use the scatter function right well in probably 90 percent of the cases you do you want to use scatter but if if you really need the performance then you want to use a plot if you if the performance is important and you don't need all the customization that scatter provides then the plot function is going to be what you you want to use it's it's something on the order of like seven times faster than uh than scatter so that was something that was really interesting and something new that i learned from the article from there stephen goes into talking about how you can customize things how you can you know change the size of the the dots in your scatter plot the color the shape and the transparency and the transparency aspect is something that is really neat because in a scatter plot you have these dots plotted all over the place and there's in in you know unless your data set is like really sparse you're probably going to have a lot of overlap and you might not know like what the density of like like how many dots are there really in this region of the scatter plot if you adjust your transparency a little bit and make the dots semi-transparent then all of a sudden you you gain like an extra dimension almost into your uh data where you get to see like oh there's actually there's a lot like of points in this region and it looks really dense and that's something an important piece of information that you can get there so adjusting the transparency is one of those neat little tips that you can you can get with the scatter plots he talks about color maps in in matplotlib and and how you can use them with scatter plots and customize all that kind of stuff and so yeah it really just just uh really dives into you know how to use this the scatter function what all the different options are available to you and how you can customize it so it's a really nice it's it's not too long it's also not super short either it's kind of a nice medium length article it's got i think a really appropriate level of depth where it's like if you're if you're brand new to matplotlib and you're looking for some you know information about scatter plots this is a fantastic resource if you're looking to dive a little bit deeper into scatter plots then this is also a fantastic resource so it's really i think something you know if this is something you're doing day in and day out you'd probably want to bookmark this and just kind of keep coming back to it as a as a nice uh reference yeah it's kind of amazing with data visualization you know you learn these sort of like thematic elements but going into each style of plotting there's so many ways that you can apply it and you know like things like you're mentioning of graphing across different colors to kind of show another dimension there the opacity and you know the you know how transparent things are and and how you can make it seem like things are stacking up and adding these additional dimensions and then also just you know ways of like interacting and having the legends and other things kind of work with what you're trying to do it's uh it's definitely a deep subject so that's kind of nice to see this sort of deep dive into one specific area and i'm sure it kind of maps across additional the ones that you might want to play with in your data visualizations yeah my next one is again from previous guest this is from uh lucas langa and we talked about his project black on episode seven he actually mentioned this briefly in our conversation but this is a nice article really distilling his thoughts on it and he was kind of mentioning in sort of an offhand way that some people were unhappy with the way sometimes black would look for what they were trying to do and one of the main ones they were complaining about this idea of this they call it the sad face d dent yeah and it's pretty common if you have a lot of parameters or you're setting up a list or you're setting up a dictionary to format things where you sort of open up you know the in this case an opening of the parentheses and then closing the parentheses and having those on like a separate line so that all this individual elements that you're you're working with parameters or you know what have you are on a separate line with a comma and so forth and then you know one line down at the very end of it you'd be closing that off and in this case it would look like an unhappy face because it's a closed parenthesis and a colon right and so yeah for a function or a class or something like that yeah so when you're defining those things or like an if statement or some other kinds of things where you might see that and so yeah some people don't like that style but he explains why it's actually kind of a best practice in some ways and so there's some things that he says that might surprise you it kind of follows this sort of uh he sums it up with the the consistency that all the bracket pairs are treated equal so it could be you know curly brackets in the case of a dictionary or you know square brackets for a list what have you but treating them all kind of the same in that style his other one is readability um the function signature or if statement or wild test and so on is going to be clearly delimited from the body of it yeah which is really much easier to kind of read and look at and then he goes into developer efficiency which this is the one that i think was the most surprising to me but made the most sense to me of like oh yeah this is gonna save a lot it has to do with diffs or you know seeing the differences between two two different files yeah and where you would see this is like if you have all those elements you know maybe their parameters or whatever that you're sort of explaining in there and let's say you don't use the practice of putting a comma on the last one and then you make a change to that last line just adding the comma makes that a diff and so it flags that entire line as a change then the new line that you've added and then you know and and so forth so like if you were just going to change a single thing like you thought i was just changing this one line it looks like you're changing like three lines and the article kind of shows that it's a very short article but it you know kind of graphically shows that really well to kind of explain it and yeah so kind of diving deeper into the idea of the bracket pairs treated equals like you know you can do multi-line lists and dictionary literals when you're calling you know functions and and seeing those and importing maybe seeing importing multiple names from a particular library you know from this import and then like the parentheses and then have like multiple things that you're bringing in again stylistically just kind of keeping it going through everything the one also i've been doing a bunch of courses about boolean expressions lately reviewing them and kind of going into them and the idea of like those really complex sort of compound boolean expressions sometimes you might have and describing something of like and this or that or whatever and having them on separate lines makes it way easier to parse in a lot of cases and so and then just blocks of code in the end what do you get you get clearly delimited block header from the body it's going to minimize the diffs it neatly leaves a space also for like return type annotations which again if you're doing annotations it then shows the little sad face and then like your arrow and like quick this is what it's actually you know going to return here and you can annotate that very clearly outside of the confusion of the whole body of things it's a nice way explaining like why the auto format or black is doing this but also why it's a good practice right yeah yeah i you know something i just want to add about uh this isn't necessarily about the the sad face deden as he calls it okay but you know putting like if you have a long list yeah you know putting each item on its own line has a lot of advantages and one of the nice things is you can sort of like quickly like comment out yeah try things portions of your list right and like so yeah there's just a lot of i think advantages to doing that and yeah if you measure things in terms of lines of code you know this kind of style adds a lot of lines to your code right but you know i i think the advantages sort of outweigh that in a lot of cases so so yeah it's it's a nice little explainer of why why there are actually some like objective benefits to this and it's not purely like a subjective style choice yeah kind of going back to the the data visualization thing you know when you're creating these plots and graphs very often there's like you know a mass list of parameters that you're putting into it right and that's a really common thing which you mentioned of commenting them out it's like okay well let me try this without these things and just to be able to throw a comment or yeah like you know like we were mentioning the idea of like you go into do a diff to kind of see what's changed between these things it's it's much easier to see that in red than it across like the whole body of the thing and you're having to parse yourself where it is so exactly [Music] i want to shine a spotlight on another real python video course it dives into the topic of working with data frames in python it's titled explore your data set with pandas the course is based on an article by rika horvath and in the course douglas starns takes you through how to calculate metrics about your data perform basic queries and aggregations discover and handle incorrect data inconsistencies and missing values and then how to visualize your data with plots i think it's a worthy investment in your time to learn how to wrangle data using pandas whether you're calculating grades managing a data science project calculating statistics or building dashboards our video courses are broken into easily consumable sections where needed we include code examples for the techniques shown and all the lessons have a transcript including closed captions check out the video course you can find a link in the show notes or you can find it using the newly enhanced search tool on realpython.com [Music] so what do you got next next one i have comes from someone we featured a couple times now i think on the on the podcast this is matt wright he's got an article over on his blog writers.io called how to iterate over data frame rows and should you so this is an interesting article because it kind of ties into something i can't remember if we talked about this on the podcast maybe i mentioned it in in one of the news articles or something like that but stack overflow recently did kind of a some data analysis on data they'd collected about what are some of like the top voted questions and uh in particular what's the most copied answer within a code block and it turns right it turns out that the most copied answer it within a code block on the entire site is about iterating over rows of a data frame of a pandas data frame yeah so this is kind of addressing that you know it's it's such a common question it and you know the answer to that question is one of like is the most copied answer on the entire stack overflow site at least you know at this moment it kind of bears looking into you know well should you actually do this like is this actually a good thing and you you know you think of a data frame in some sense as like a two-dimensional or multi-dimensional array and a lot of times in programming you do a lot of iterating over arrays well data frames are not really i mean they're more than just an array that is sort of the underlying data structure of them but they're sort of more than that it's it's a way of working with data that doesn't require in most cases i'd say in in the vast majority of cases any kind of you know direct iteration so this article talks about like okay if you need to iterate over rose you know here's how to do it you know here's some ways to to do that and he and he talks about a couple of different ways to do that and i'm not going to talk about how they work i mean you can you can read the article you can see the stack overflow answer and uh and see that but he says okay is this really your best choice and you know it when it when it comes down to it like the the final solution that he comes up with for iterating over then i think i think this goes back to the uh stack overflow answer is using something called it arose and says it seems really easy and you can see why it's the top voted answer because it does exactly what you want why would there be any controversy and it comes down to some you know speed and memory yeah when you're working with really large data frames you know speed is is very important and numpy which is sort of the underlying library and pandas all the data frame and everything they're all implemented as numpy arrays kind of at the base level has some techniques that they're not specific to numpy it's really just a general array technique something called vectorization and there's also a kind of a related uh concept called broadcasting and this is a way that you can look at things in you know aggregate or sort of perform operations on multiple rows without having to do any kind of looping or iteration and it's extremely powerful it's orders of magnitude faster than just doing a simple you know for loop or something like that over this like it arose so he gives an example of kind of he's got this example that he's using throughout the article of scoring things like giving like letter grades to to things and he runs a vectorized example of doing the same operation assigning these letter scores and the vectorized thing runs on his computer in about 2.75 milliseconds whereas the itero's implementation of that takes 13.5 milliseconds which is quite a big difference yeah so you're almost five times faster and that's just with this you know tiny data set i think it's into things like you know cython and list comprehensions and some other techniques there's uh the apply method on data frames can be used to apply things to multiple rows and it talks about some different techniques but but really it's you know i think kind of the point of this article is if you find yourself wanting to iterate over rows in a data frame you probably need to stop and ask yourself do i really need to do this or is there some better way that's more performant and really more i don't know what the the corresponding term for pythonic would would would be for like pandas or something but but you know like i guess just more like a a better practice you know things like that so so yeah it's it's a really neat article it's really in-depth it does cover a lot of ways to iterate over rows because you know there might be times you really do need to do that it's i think it's going to be in rare cases but but overall you know there's you have multiple choices and doing something like vectorization which pandas and numpy together make that pretty simple is often going to be a better choice and he does give kind of an overview of you know how do you decide and when should you use these different techniques so he's got a chart of uh exploring kind of the run time of all these different ways of of doing things and when it might make sense when it might actually be more performant to do something like you know it arose or iterating over something and it really comes down to you know how big is your data set and uh what can you write and maintain easily so those are kind of the two things he he suggests to think about when uh when looking at this so yeah if you use panda's data frames a lot if you found yourself wanting to iterate over rows in a data frame highly recommend checking this article out and getting getting a good look at what all the different options are and how vectorization comes into play what that is what that means for performance and how you might go about choosing to do that yeah we have a really good course i'll probably feature in this week's uh spotlight about working with data frames and sort of exploring yeah the data inside them and it kind of talks about a lot of those kind of ideas too and awesome the idea of like the series inside there and and lock and high lock and these other kinds of tools that yeah are kind of important to just be familiar with um as opposed to just brute force applying things yeah right cool so this brings us to projects and the one that i have is really kind of interesting it's from sonos um the company that makes a lot of these nice wi-fi speakers and other kinds of technological audio projects and they are working with this codec called flac the free lossless audio codec which you know if you're an audiophile you might be familiar with this already because of the idea of like you know not working with mp3s and these other lossy compression things i don't want to get into that debate here this is more talking about using flac very specifically for their own tools and so they created their own library called pi flac to be able to use flac in their sort of data science and measurement processes and so forth and they you know they pick flack partly because of the idea that it will reduce the size of things and the library that they're using that's inside there is written in c and by converting it to python allows them to do what we've covered many times in this this show is the idea of like rapidly developing solutions and trying things out and experimenting you know the way that data science really you know is all about python in that way the idea like quickly trying things out and modifying stuff yeah so why do they need flack in this particular case why they need it to be lossless audio is a lot of the tools that they're designing are for within these speakers the speakers can determine the room that you're in they the speakers have their own microphones they can actually analyze their own playback and sort of tune the sound of the speaker to the acoustics of the room and so that goes beyond like the whole argument of like human hearing and so forth and what gets thrown out that supposedly can't be heard here you're really wanting to measure the full spectrum really get an idea of measurements across all the different surfaces of what's being reflected what's not yeah and so forth and so it's very important that it is uh really included in this whole thing even though they you might get into the whole perceptually redundant information idea but in this case for the machines measuring it they they need it and their technology is called true play if somebody has like a apple product you might have heard of this thing called the the homepod it had a similar kind of thing where like if you picked it up it could sense that you've moved it and it would then recalibrate itself to you know where you're moving in the room and so their technology is very similar in that idea so the library by them open sourcing it uh allows you to kind of play around with you know not only doing real-time analysis with flac in your own python projects but also has a command line uh interface actually a couple of things about it number one iflack is pip installable so you can get it actually from pi pi i'm going to include the github repository and then one other kind of nice thing is just right out of the box it has a cli command line interface that you can use with the library to do conversions from a wave to a black file and back and forth and it runs on you know all the main operating systems mac linux and windows but also the raspberry pi and raspberry pi zero cool so if you're looking at using it in you know kind of projects and stuff like that it might be a handy library there which i know a lot of a lot of people are messing around with that kind of stuff and if you're interested you know having audio playback and have a fancy player and working with lock files here's a nice tool that got open source from sonos so yeah and actually i think that it's you know if you want to work like if you just want to read and write flac files there's already an existing solution for that and i was just kind of browsing through their announcement article and it looks like they're using that it's called pi sound file yeah but what pi flac i think where it really shines and they mention this they say uh there are so many different existing python implications for flack encoding decoding however these tend to operate on files rather than real-time streams which is no good for continuous processing so that's really where pipeline is focused on that processing continuous streams of of uh of data so nice yeah and keeping it so that you know this data can be huge depending on the oh yeah the sample rates and the bit depth so um it kind of gets up there and so the idea of reducing it is really going to help as far as working with the data so yeah and sonos you know is a really interesting company just to kind of throw this out this is not python related but you know the the first real programming job that i had like professional programming job was building these interfaces for audio visual systems and it was not using python actually most of the coding was done in lua but we actually worked with uh with sonos and you know they're they're expensive they're pretty expensive speakers but i was always pretty impressed with what they were able to achieve in their uh in their their devices they're pretty neat little speakers so yeah cool stuff they got a lot of cool stuff going on there nice so what's uh what's your project mine is not necessarily a new project it's been around for a few years now i think it's called pip x and but i wanted to feature it because they just kind of had a cool news item related to them that is they became a member project of the pi pa which is the python packaging authority and pipex is a tool for installing and running applications in isolated environments so you can think of it kind of like pip you use it to install packages from pipi or other sources basically anywhere that pip can install from pipex can also install from that but in pip it's just installing into say like if you're just running it sort of globally uh it's installing it into like your global environment there's no isolation or anything if you're using virtual environments that's a little bit different pipex automatically installs packages into an isolated environment so where this really shines is let's say you want to install a command line tool from pipi you can install that tool in isolation but still have access to it from the command line and be able to run it like you would uh normally which is really cool that opens up you know being able to have potentially different versions of of that tool installed and have access to those and it makes sure that you know any dependencies of that project aren't going to conflict with any dependencies of another package that you've installed which is a it can be another thing that can be kind of a a hassle to deal with when you're installing tools in kind of a global context so that's a really neat feature of pipex another really awesome feature that i think is is kind of maybe the the coolest part of pipex i don't know i mean it's all really kind of a neat neat project but uh they have a run command so to install something from pipex it just works just like pip you know you do pip install something here you would do pip x install something there's an additional command called pip x run and that allows you to do a one-off installation to run something so say you want to try a package out right there's some cool new package on pipeyi and you think well i want to try this out see how it works you know see if i like it well normally to do that you probably create a new virtual environment use pip to install it and then run it well with pip x you can use this run command it will install it in a temporary isolated environment and run whatever you tell it to run so you do pipex run and then you know the package name and then whatever like if you were running it from the command line like you would just pass the options and arguments and everything to it straight away it would run it with all those options and everything that you have and then you would see you know see the execution see it work and then when it's done running it gets rid of everything for you and leaves the system back the way it was when it started so which which i think is really awesome that's like that's a lot of automation going on there and really makes it a nice way to kind of try things out before you decide to install them and use them so that's that's really cool and the deal about becoming a member of the python package authority is is pretty cool so i didn't i know of the python packaging authority they're kind of responsible for maintaining kind of the standards and best practices of packaging things in python so you know if if you're curious to know how you would create your own package you know the pipea has a bunch of tutorials and and things like that you can look at they talk about you know the different peps involved and and what the best practices are what tools to use things like that but they they host these member projects and when when that happens they actually become a part of the pipea github organization and get hosted there and i didn't know when i went into this i didn't really know like okay what else happens like what's the advantage of becoming a pipa member project other than maybe getting like a little boost in visibility or things like that well there's a pep called pep 609 that sort of lays out the government governance model for the pipa and they mention pipea member projects and one of the things that they provide one of the goals of the pipea is that in the event that a project needs additional support or no longer has active maintainers then the pipa will ensure that the given project continues to be supported for users to whatever extent necessary so it kind of provides some added security and uh to that project you know for people who are using that they'll know that like okay if if if something happens to the maintainers they can't maintain it anymore then it's not going to just die out completely there's still going to be someone there working on it and providing support as needed so so that's really kind of a neat deal and and it just i kind of you know i guess it goes to show that pip x is a valued uh valued project in the in the community and something that uh they want to ensure you know it has uh some longevity behind it yeah it's nice i'm looking through the other projects that are that are in there in the ipa collection on github and beyond pip and uh virtual m and a handful of other ones they're all these sort of tools for packaging which makes sense and yeah a lot of familiar names in the people list there yeah so there's stargoll and brett and a couple other people i'm hoping to have on the show soon so great yeah all right well that's a great list of stuff from pi coders this week yeah yeah thanks for coming on the show again and i'll talk to you soon absolutely yeah see you around and don't forget you can get started on digitalocean's app platform for free at do dot co slash real python that's d o dot co slash real python i want to thank david amos for joining me again this week and i want to thank you for listening to the real python podcast make sure that you click that follow button in your podcast player and if you see a subscribe button somewhere remember that the real python podcast is free if you like the show please leave us a review you can find show notes with links to all the topics we spoke about inside your podcast player or at realpython.compodcast and while you're there you can leave us a question or a topic idea i've been your host christopher bailey and look forward to talking to you soon