Preparing Data Science Projects for Production | Real Python Podcast #274

Key Takeaways

Welcome to the Real Python podcast. This is episode 274. How do you prepare your Python data science projects for production? What are the essential tools and techniques to make your code reproducible, organized, and testable? This week on the show, Quinn Tron from Codecut discusses her new book, Production Ready Data Science. Quinn shares how she got into blogging and what motivated her to write a book. She shares tips on how to create repeatable workflows and we delve into modern Python tools that'll help you bring your projects to production. All right, let's get started. [music] [music] The Real Python podcast is a weekly conversation about using Python in the real world. My name is Christopher Bailey, your host. Each week, we feature interviews with experts in the community and discussions about the topics, articles, and courses found at realpython.com. After the podcast, join us and learn real world Python skills with a community of experts at realpython.com. Quinn, welcome to the show. >> Hi, Christopher. Uh, thank you for having me here. >> Yeah, I was excited when your team reached out to have you come on the show. I just featured one of your articles on our other type of show called Pyoders, the top six Python libraries for visualizations. I really like how you put that together. >> Thank you so much. Yeah, you you do a good job of providing not just a set of features and comparison, but like a way for people to get an idea of how it might fit into the different ways that they work because there are so many different approaches to data science. >> Yeah. Yeah. When I write articles, I imagine my audience are data scientists who are, you know, very busy. Yeah. To like written article and they want just want to scan the articles, have an idea of what the tool does and what are the use cases. So every article I try to keep that in mind. >> Yeah. Yeah. I really appreciate it. I I I think your audience probably appreciates it too. So how long have you been blogging and doing the website now? >> I have been blogging for quite a while. I doing it when I was in college actually. So five, six years. >> Okay, great. >> Yeah. And so I it's not necessary. So initially I write a lot of medium and I grow like a big followers on medium. I think at the time I write and I try to commit to writing around like at least one article per week and it's always end up like three articles per week. like I wake up, I do something and then I write any free time I have I write and it's I guess it's like a fun things for me but it's also like a commitment but at the time I try to write to like you know build my portfolio but then after I have a job and everything it's it's still like a passion of mine to share my knowledge. Yeah. So I've been uh doing it continuously since then. But later I decided to create my own website to you know send out newsletter and stuff to like just kind of have me more control over. >> Yes. >> Um you know what I sent out because you know how is I really don't like medium it have like a whole pay wall and [snorts] >> to me I I like to be open to everybody to read. >> Yeah. as a a person who shares a lot of those links, it always is a little awkward like are they going to be able to get through that wall to be able to see the stuff that I'm talking about that I'm sort of summarizing it. Yeah, >> it's a little frustrating that way. So, yeah, that's good. That sounds great. And your site is codecut.ai. I'll share links in in the show notes obviously. >> Thank you. >> One of the things I wanted to talk about is you've kind of decided to grow into writing a whole book and you've written a book production ready data science. tell us about it. What's what's in the book? >> Yeah, so it's a kind of like a handbook for data scientists who want to learn how to turning their prototyping into code that is ready for production. And it's not a book about you know how to containerize your machine learning models or how to deploy to cloud but it's more about it. This come from my experience. So I was a data scientist in a startup. So it was a very small data science team. So my responsibility it you know like I do a lot of end to end uh like you know creating model analyzing data and then also ship it give it to machine learning engineer right yeah >> and I saw the pen of machine learning engineer so my my manager he's he's a very good data scientist but he's not very good at coding so a lot of time he would the machine learning engineer need to like write rewrite a lot of his code >> so I want to like you know how do I write the fold in a way that when I give it to him, he can immediately like just minimal edit and then use it for production. That's one thing. Another thing I found is it's a pain for me to you know write a lot of code in a notebook and then later try to understand what I wrote. Sure. >> And also you know if I run my notebook in different orders then it will show different output. So it's very messy at the same time it's not reusable. I cannot it is really hard for me to reuse it for a different project that use similar pieces of code like how do I extract it out so that I can reuse different components and another thing that go with it is you know unique testing right how do I make sure that this code works >> for different circumstances. So the book is about kind of good Python practices you can say for data scientists including writing Python variables, Python classes, Python functions like when should you even use Python classes because sometime you know you don't know it's not really necessary. >> Yeah. for data science project also unit testing also like hard coding right like a lot of time like in Python projects in a data science project you see a lot of global variables being used hardcoded variables and um I offer the alternative which is using a configuration so you have a configuration file where you put all the value so you can very easily change it right but at the same time you have your all your code logic in a different place so that it look very clean and also you can unit test independent of the hardcoded values. >> Yeah. Yeah. Those that's huge. Uh there's so many topics inside there that we could dive deep into. >> Yeah. >> Maybe I start in reverse. I feel that's a common thing with notebooks generally is they aren't very reusable >> in kind of a strange way. They I've reported multiple times on the show about this Jupiter notebook study that Jet Brains did where they worked with another team to look at hundreds of notebooks out there and >> try to run them from top to bottom and they just wouldn't run and it's just >> that's funny. >> Yeah, it's just so weird. This is like the idea is you are using it almost like a not necessarily a student but just as a notebook as a researcher and somebody taking your notebook and you say they stole it from you and ran off and they're going through your written notebook would they be able to put together your experiment anyway and it feels that way you know. Yeah. And so I definitely want to dive into that. I like that importing idea. Uh, that's something that I've seen happening more often where >> are you doing that from another notebook file or from a a like a standard packaged like Python file? >> Yeah, I like that idea. I think you know there's not like the right or wrong way, right? Sure. The thing that uh notebook does very well is that it is interactive and you can see the output as well as you can like you know turn it into like a nice report right with all the graph that cannot be done very easily with Python script. So I think it's not about like niching the notebook but like know when to use it for yeah >> I do think that it's very like for example you can start with a notebook >> but then like if you start seeing like yourself using like creating Python classes and use it for different cells maybe it's time for you to move it to like a Python script and just import it in and it look very clean right imagine you do like I don't know processing I don't know like some sort of processing function that you keep ping yourself reuse then I I feel like it's very clean just to import it and use it as long as you keep the Python function name uh readable as well. >> Yeah. Yeah. I I agree that the modularizing of your code is something that's I think this computer sciency kind of education part that is sort of stressed early of compartmentalizing your code and breaking it down and making it reproducible but also reusable. >> And so yeah, I agree with that. I think that's really fantastic. And then like you said, it leads to testing. One thing that I feel like you're doing it throughout the book that sounds really interesting is almost building what I'm getting back into creating a bunch of music. I kind of dropped out for a while. That's why I have all the guitars and stuff behind me because I >> wanted always to inspire myself whenever I would look at the camera and go, I should be playing guitar now. [laughter] But I use a lot of software and sometimes it's painful to start with all the technology that's available now. And often it's nice to be able to have a template. >> Mhm. >> To have like, okay, I want to do this type of music today. And in your particular case, you could do that with a notebook. You could have, okay, these are the types of libraries that I like to use. I know that I'm going to want to have a dashboard or something like that. Is that something similar to what you're talking about? Like being able to create uh reusable, if you will, partially filled out notebook page that you're kind of filling in the blanks on to get projects started. >> That sounds interesting. I haven't tried that before using a notebook as a template. >> Okay. >> Uh but um can you tell me more about it actually like about the idea? I think what you could do is have maybe with this new addition of lazy imports, uh, we could look at it too where you know you're not necessarily going to import them until you start using them. Again, this is a Python 314 further thing. Yeah, >> you could depending on the type of project have a certain, you know, not only the libraries that you might use, but you could have some initial code that sets up like the dashboard the way that you like to usually use it. Maybe there's styling that you have that has been templated. That's something that I always liked about Mattplot Lib because it is kind of painful to get MattPot liib to look attractive. >> Yeah. and the tools are there, but you just need to know about them and be able to bring them in. I don't know. I'm just sort of bouncing it off of you, I guess. I didn't know if that was kind of some of the ideas that might be part of the book. >> Yeah, that is interesting. Is I we I did talk about a template, but so I created a template for the book, but it's not necessary uh on notebook focus on notebook, but it's more about the entire project structure. >> Okay. [snorts] So let's because in my opinion there's different ways for you to structure a data science project. Yes. But there should be a standard meaning that the source code often put in the SRC folder. The notebooks should probably put in like notebooks folder. The test should put in the test folder was like you know similar like test. >> Yeah. processing function and in the source you will have processing function and there will be a config folder with all of the config files and also another thing I I guess it's similar to what you said about some of the stuff if you keep reusing it and it will be nice if you spin it up and then like you have all it in one place. Another thing that I found very redundant when I create a new project is pride.com. So I like to use right now I'm using UV and rough. >> Yeah. >> Right. And and I like to configure it in and for people who not aware UV is um kind of see alternative to poultry and pip and rough is a python formatter. >> Yeah. Llinter formatter allin one kind of toy. >> All in one toy. >> Yeah. And I like to like ignore some rules. So like you can set some ignoring rules and like you know like how many lies uh how many words in the lie. So those thing I like to keep it con like because that's not my preference but every time copy and paste that and among everything else including the structure is quite a bit of work. So I created a template that you can just say uh cookie cutter and then the the name of the >> Yeah, that's what I was going to ask if you're using that tool. >> Yeah. And then boom you have everything and it going to ask you some initial question like okay which Python version are you using what's your name you know just for the project if you decide to package in the package like some basic stuff and then it customize and you have something to start off with instead of like just empty project. Yeah, cookie cutter is I feel like it comes from the web development community >> and the idea of building sites, they usually reuse a lot of uh elements and uh doing things like you talked about of building a directory structure and having it be consistent. Uh UV does that too to a point which is also nice. you know, building the initial TOML file and a lot of these reproducible things. So, it it's nice to see the tooling kind of levels up and becoming more standardized. Definitely in the five years that I've been working more and more, even doing the show, I've liked how this has moved forward and become more standardized. But the trick is just finding the best navigational path because often it's a problem of There's so much overhead to just begin doing work. [laughter] >> Yeah. And there shouldn't be. >> And so, yeah, anything to help people start that. Yeah. >> Yeah. >> Well, maybe we could talk about that a little bit. You mentioned in there a few things about reproducibility >> and you are saying that a lot of the data scientists that you were working with some of the team leads you would try to share I guess within the team say a project that's you know living in a notebook what were some of the struggles of that other person trying to reproduce it what what was standing in their way >> I guess as as more of a struggle. It's both. So I I think what I mentioned earlier was more of a struggle between giving the notebook to a machine learning engineer. >> Okay. I guess it's a reproducible was a problem uh like if a data scientist give me a notebook and like often time their problems when it comes to dependencies management and it it's different problem can go with notebook but I guess like and that I don't think is specific to notebook but I would just say like dependencies in general like often time data scientists also like to use conduct and I'm I'm not a fan conduct really but I I so I would install so data scientist let's say they was using a Microsoft right another person using a windows sorry Windows I'm using a Mac >> sure >> so uh I try to you know uh cond install from a dependency management file then what happened is yeah I install successfully but like there there will be like some packages like some system packages that requires some installation differently on Mac I think and then like I couldn't run it. So it's a lot of troubleshooting that comes with dependencies and um so that's okay because raise an error but what I think is more dangerous is a silent error like when it doesn't raise an error but because of the dependencies differences you know then like you will it will produces different output let's say I'm trying to think of example for example pandas I I I will just throw it out there but uh let's say in the older version It doesn't require you to let's say default the default of the old pandas data frame for a specific value is a and then in the new version is b. So uh because you don't specify that default value. So when you go to a new dependencies it show like a different output but it's not necess so you don't know because it doesn't throw any error but then like you see a performance degrade and you don't know why. >> Yeah. >> So that that would be like one of the thing I think is a problem. >> Yeah. Yeah. UV is the tool that you're using of choice now. Mhm. >> I didn't think about the barrier of moving across the different sort of computer platform. Like if you're again on Mac and Linux, your environments are a little more similar. A lot of the packaging of the tools that we use, especially across Python, are similar, but I could see how the Honda stuff could be a little different in that sense. Is that kind of where that was breaking down? like there's like a a Windows version of the way it was sort of prepared in Has UV solved that for you? >> I think so. Yeah, I I think it's been solve it very reliably because I I I say I think so because for all the projects I've been working on is good but I didn't go back to my previous project and try to use it for that, >> right? >> But I really like UV because of the fact that it have two files. One is a lock files that you know that lock the dependencies right >> which is like if you want to reduce reproduce the exact dependencies you can do it but at the same time it had the pi project doctom with a more flexible dependencies so yeah that's become that's why I really enjoy UV is also very fast >> yeah yeah the lock files would solve that say potentially pandas issue you were mentioning before yeah now you know you're going to get the exact version >> there which is good. >> Yeah. And also like I would say another problem and I I wrote it in the book but like with pip right like compare UV with pip for example a lot of time you know when you run pip install something and then like there's a dependency issue but it's install anyway the dependency conflict that is not like something to be ignored and if that like there could be like potential downstream issues that come with like different that like you don't match dependencies that the package require. >> Yeah. [snorts] >> But with UV like if that happens it's going to stop you right. So you can try to find a way to match the dependencies of the packages that use together. >> You feel like it it presents you with information that you can kind of work off of what was compared to pip. >> Yes. >> In that sense. Okay. >> Yeah. >> Yeah. Yeah. One of the things you mentioned again kind of going back to the dataf frame area is that you're liking to use you're using polars outside of pandas. What are some of the reasons that you've decided to do it? I I I know one of the ones right away is that it's much faster, but are there other advantages you've seen with switching to polers? >> Yeah, I also like the syntax. Like I feel it's more uh expressive compared to like you like you can do a lot of chain. >> Yes. >> Uh with polars. I also like Yeah. I like the syntax is very expressive like you can say polar doc something something. It's similar to pispar in my opinion. Okay. >> And another thing is lazy execution. I think it's uh very cool that I think it's very efficient in the way that you know like it doesn't make sense for you to let's say load a 1,00 I don't know 1 million rows and then you only use 1,000 or you the same with columns or you use only five columns but you you import 100 columns. So, so it's very inefficient and the fact that >> there's something that like >> that just load what is necessary or like process what that you request for. I really like that about polars. >> Yeah. Yeah. That's something I' I've had a few people on to talk about and it definitely seems to be one of the biggest benefits is just the idea that you as a data scientist let's say you know a generalized you person >> don't have you know maybe you're working on a laptop and it has a limited amount of RAM and you still want to do data exploration and you want to try things out the laser execution lets you really do a lot of it uh without having to lift the entire [laughter] set of data into your, you know, RAM or whatever and have to work with it. And also the lazy execution lets it work across larger amounts you kind of after the fact, which is kind of nice, too. >> Yeah. >> Yeah. I've been impressed with it. I were you you sound like you worked with Pispar quite a bit. Have you used other things? Like I I have a background of using R and I felt Polers kind of was a little more like R's tidyverse kind of >> the way it's written as far as chaining things and or piping as you called it. >> Piping is good. Chaining. Yeah. Either way works. >> Yeah, they're similar. [laughter] >> Yeah. Yeah. But it's nice that it's it also not a lot of setup to use polars, you know, like you can leverage polars without the pain of setting up like pispar. >> Yeah. Yeah. Yeah. Yeah. I know that's a whole other [laughter] lift, if you will. Mhm. >> I've heard that some people switching from pandas to polers >> have had a complaint that it no longer has an index and they're accustomed to using it or using indexing as a as a technique inside of it. Have you noticed that or have you noticed any other things that have kept people potentially away from switching? Yeah, I I haven't heard of that one, but I think the one that I heard of is just the fact that Polar syntax is very different from pandas and uh for companies that have >> to learn the new syntax. Yeah, >> learn new tint stack and also like they need to do a lot of migration uh in ter like you know like let's say they have a large code base then they need to rewrite quite a bit of them in order to to use polars and so I I think the complaint was just to like the migration and the uh syntax learning curve. >> Yeah. Yeah. Makes sense. Yeah. >> Yeah. And also you need to onboard your teammates, right? Like yeah, you prefer polars, but your teammates prefer to stick with pandas. Then >> yeah, >> you cannot you cannot just switch everything to polers. [laughter] I think it's easier for the new project, but also another thing about polars is it's not like it's relatively new framework. So it's not every new all the framework like other frameworks might not support it >> right. >> So you you want to find something that support it. I do think that there's a lot of movement towards like supporting polers and it it's getting very easy now if you have a Python library that support pandas it becomes very easy now to also support polers. There's a package called now wells. >> Yeah. >> Yeah. So I have the person who created that write one of the blog for us as like a guest blog. >> Oh, that's great. >> And uh >> uh Marco Gorelli, right? >> Yes. And he uh it was it was great. >> Yeah. Yeah, he's great. He came on the show I don't know year and a half ago. It's it's been a while, but uh yeah, he's still pounding away at that project and it's great to see that somebody working on >> Yeah, he's very passionate about >> Yeah. interoperability uh not only of the data frame libraries but all the other pieces of data science that kind of go with it the you know the things that >> that I'm excited about of like doing data visualizations or doing other kinds of ways of manipulating >> data inside there so it's nice to see that work continuing that's cool I'll have to check out the article that he wrote with you >> going back to notebooks one of the things that you are championing that I've also been a big fan of this year is Mimo Mhm. >> I guess how how is that going and uh what are some of the advantages that you see of of people switching from Jupiter to using Marimo notebooks? >> Yeah. So I guess like Marim Mo when I saw it I like when I was writing the book Maro like when I started writing the book >> Yeah. Marim Mo hasn't come out yet, but I kind of blessed the chapter of Jupyter Notebooks at the end because this is this is the thing like I am not a big fan of Jupyter notebooks, but I know that read a lot my readers are using Jupyter notebooks. So the solution shouldn't be like hey don't use notebook right I want to find a common ground and some the issues I identify with the notebooks are the fact that if you run it differently in different orders then it can so common right and it it can um produce different output just one thing >> and and like you can do trick of that like you can try to I don't know like every time you run you run all sales right then but it's like a a lot of work take a lot of time a lot of computation >> yes >> and also other things I don't remember but maro is solve that exact problems right and oh also another thing Juda notebook is uh a JSON based and it's like yes oh uh Marimo it allows you to write it as a notebook but then at the same time it's a py file like that is amazing Amazing. It's it's like you have the best of both world interactivity and you have a Python script and if it is a Python script under the hood then you can do whatever you want with it just like a Python script you can do unit testing very easily you you can just import it the functions or whatever from that notebook right like the doc notebook. Yeah, exactly. Good practice also like another thing I really like maro like if you change you know like in notebook if you change something in one cell you you need to remember like which one depend on that one so that it of course you can run everything but you probably don't want to run everything if your notebook is very heavy so you run the dependent sales and to do it manually it's very difficult >> right >> to remember that but with marim mode it automatically run the dependent sales I also like the fact that it like keep track of which variable like the dependencies between cells and oh another thing is it integrated very well with UV right so like if you have UV and you have maro it's like very nice yeah >> um because one problem with notebook is you would like pip install something right I see people do that like in the notebook like pip install something >> yeah and they might even do it like deeper into the notebook like in cell 25 or something like that, you know, >> and they don't really specify the version of the package, right? or like there's no >> what I try to say is it says like let's say you pip install pandas and then later I don't know one year later maybe you run that again uh with pip install pandas but it will be a different pandas version right if it install from scratch so there's a dependencies issues there with a notebook but with marible what happen is you don't even do pip install so let's say you import something like you let's say you import plot proply right and then it will like detect it and it say oh hey you're missing probably dependencies like you want to install it and then when you install it is is trapped so >> okay >> what happens is the next sam it's trapped in like the top of the note which is apy file so the next sam you can do it it used with UV so UV I don't map the exact command but like UV run something something but like it reproduce exactly because you know if the codes stay the same if the dependencies are exactly the same you will have the same outputs >> as you've shared these techniques with other people what has been the reaction of it like what have people thought of you know when you you you do a lot of writing you get a lot of comments on your your site and maybe even back with your other data science comrades or colleagues. What would they say about >> MIMO and and the changes there? >> They I think they like it. It does take some time like when I heard I told them about the features like oh this is amazing I I want to try it. Right. What surprised me though is it seems like it takes some learning curve for them. So there's some restrictions in maro for example variable it doesn't allow you to repeat the same variable name >> gives you like a warning right as you type it out >> it's not even a warning it like you cannot do it it's an error >> yeah okay >> so for example data frame you do like data frame equals something something right then later you're like data frame equals something something it will raise an error and if you think about it makes sense because if you want to be rep first of all it track dependencies by the name so it it cannot repeat but at the same time you also think of like if it all in a python script then like you know it's it's a bad practices for you to >> right >> you re use the same variable names for different things and I do think that restriction is good but I also think that it's like kind of you know make it because a lot time I feel like data scientists use their notebook like a scratch pad like that they just put whatever code. Yes. So it's like a new to them it's like oh there's more restrictions in the notebook. Right. >> Yeah. It's this whole paradigm shift where they have to like really think differently like this is going to happen now where it's going to lead you toward not making [laughter] an unreproducible notebook but that requires you to recognize like oh these are things that I was doing these it's kind of weird like these library enforced it's trying to break your practices almost like stopping you from smoking or something like that [laughter] you know like it's going to just keep giving you these little like kind of push you in this direction of better practices which I think is great but yeah it's going to be hard for a certain subset of people who used to like I I just want to use it like a notebook >> I that's why I think I don't think Maro is a replacement of jur notebook I think if you want like jur notebook is still good for prototyping and if you like you just like want to throw away afterwards >> right >> and it's like because a lot of time right like you create like a quick notebook for like some proof of concept and you don't really ever touch it again and that's fine. But if you think about using notebook for production >> Yes. >> then then there need to be some restrictions on how you code. >> Yeah. Yeah. And it's definitely going to enforce them. >> Yeah. [laughter] >> Which I think is good. Yeah. It's different. If you were back in that position you had before and you were working with your ML engineer and you provided them with Mimo notebooks, would they you think they'd be happy with that as far as uh the reproducibility? >> Yes. I I still think that Marimo is a tool. >> Yeah. >> And it still need you like it's not like it will automatically make it you know reproducible if you use a marot tool. I think it's it's it helps you a lot but at the same time it still require like good practices like you know writing code that is clean and readable. >> Oh yeah definitely it's time to shine a spotlight on another Real Python video course and it touches on one of the tools we discussed this week. The course is titled Python Project Management with UV. It's based on a real Python tutorial by Leodonas Poso Ramos. And the video course is presented by Rodrigo Yra. And in the course, you'll learn how to use UV as a Python package and project manager, how it integrates multiple functions into one tool. You'll practice using UV for fast dependency installation. You also learn how UV automatically handles virtual environments, creating and managing them as needed to ensure clean, isolated project dependencies. It also can manage which version of Python your project's using. You also learn how to use it to initialize a project and how to continue the process using UV to build and publish Python packages to a package repository like Pippi. Like most the video courses Unreal Python, the course is broken into easily consumable sections and where needed you get code examples for the technique shown. All lessons have a transcript including closed captions. Check out the video course. You can find a link in the show notes or you can also find it using the search tool on realython.com. In the book, we've talked a little bit about providing other team members inside of a data science organization going from a scientist to like say an engineer or somebody else. And I think you kind of addressed this a little bit early on. You use the word shipping and to me that can mean a lot of different things. I guess what are the different ways is who is the consumer or user you're referring to there? Uh maybe in the case of your book like who are we shipping to? >> Okay. Can you give me the entire sentence? >> You mentioned like shipping an ML pipeline. >> Uh a huge ML pipeline. Okay. It's like when I think about the context of work, it's like data scientist try to give it to their teammate which is like data engineer or uh ML engineer. That's what I mean by shipping. >> Okay. >> So, it's not necessary that they will deploy the model themselves, but it's like hands it off to someone else. >> Okay. So, they can reproduce it and be able to use it. >> Yes. I guess it's twofold. I try to think of So, I was thinking about the CI pipeline. >> Okay. >> Continuous integration. Um so something that like a scenario is let's say there's already a machine learning model in production and you as a data scientist you have created a model with better performance then how do you put it into production? Let's say for the simplicity sake the for model in production is in the main branch right like all the code and all the versioning like let's say your your model leave somewhere else or your data leave somewhere else but all the versioning are in the main branch of the GitHub >> okay >> repository then if you set it up so that if it's been merged into the main branch then it will be into production Then how do you have up the checks necessary before it is merged into the manage range >> okay >> so the CI pipeline is uh one of the solution right so you first of all it's good practices that come with version control right yes >> you need to do it in like a different branch and then you create a pull request when you create a pull request then if you set up a CI pipeline which is continuous integration. Then it can run automatic test. So it can test to make sure I say it can run your unit test right making sure that they pass creating a report and also that will pull your data from other resources and and check to make sure that the performance of the new one is better. So you can like set up some test as a safeguard against merging your because a lot of time like you test it locally it's might not you know like it's it's not reliable right you want to have like a automatic test that like safeguard before it merge you into production I guess that's could be considered as shipping an ML pipeline. >> Yeah. Yeah makes sense. So that's a big part of the book then is you get into I feel that isn't a practice as much at the data scientist levels of using GitHub. This is just anecdotal data from my own use around people >> but and also I worked at a a couple different institutions, banks and other places where we had our own like git system you know for checking stuff in. And of course, unless we built like a CI/CD kind of pipeline inside of our own organization, we didn't have the advantages of something like GitHub as a tool. Is that something that you're explaining or kind teaching practices within the book? I teach git specifically but not really necessary go into like you know because the platform it can be like GitHub or any other tracking version control system but g is what uh I'm focusing on >> okay >> also I I try to go more into like use cases meaning that um let's say what do you do when you know you start a new repository right or like a new um project. What do you do when let's say you mess it up? You uh you are doing something you like experiment with something and then you mess it up like how do you how do you go back right? How do you revert it? So the good practices here is first of all you need to commit a lot like just every mile strong you make I feel like a lot of time they just they make like a big commit you know like they using version control but like [laughter] every time they commit it's like a big commit or they don't do pull request often enough and then they when it's merge it's like a bunch of merge conflicts to to merge especially when their their co-workers also work on you know the same kind of the same file but like their own variations. So it's good for them to commit small and commit often. Yeah. >> So that when incidents happen they can revert. So the book talk about those cases and other common cases when it come to >> like what do you do when incidents happen in a data science project basically with G. >> That's cool. Yeah, I think that's really vital stuff. It makes me think of video games a little bit in the sense that if you play like a a role playinging game, the there are people that will not save often and that are truly frustrated [laughter] when suddenly something bad happens and they have to go back, you know, two hours, three hours or whatever. >> Yeah. >> As opposed to somebody who's like saving often and can go back and just fix something small. But it's kind of getting in the practice of it. So I guess that's seems to be a repeating theme of like building building practices and getting used to doing these types of things. >> I think that takes me to the idea of testing which is again also a thing that's hard for I think data scientists sometimes to wrap their heads around. Mhm. >> I feel like that concept is a little foreign and maybe they don't know where to start and I don't know if you have suggestions on where people especially in say a notebook type of environment where a data scientist might start with testing. >> Yeah, definitely. So I think first you move uh some of the reusable code to a Python script, right? >> Yeah. Okay. >> Yeah. So it's also come with writing Python functions too. You want to if you want to write make have a easier time writing test. You want to write Python functions that are small doing one thing. >> Okay. like have one single responsibility and also ideally don't have global variables because okay >> that means like externally dependencies because that means that you need to incorporate that into your unit test >> and when it come to unit test I do think that you can skip like you don't really need to do unit test for all the functions right unit test should be to test the function functionality of your function to ensure that it works with a variety of cases. >> Okay. >> So I want to give an example like a lot of time data scientists like why should I test right? Because for example in a notebook I run A and I expect B and then I saw B like that's good enough like why testing >> right [laughter] test complete I'm done. >> Yeah. Yeah. Yeah. >> So there's twofold to that. First is you know to to rely on your like judgment is eronous like it's not reliable. Uh because let's say you expect a 00001 and you saw 00011 and like if you just scan in it you're like oh that looks good but it's not right. Right. So the fact that you actually writing it down if I put this if I put a in I would see b and the fact that you write down it make it like more reliable. That's one thing. Another thing is if you just test with one case, how do you make to the thing is it's not just prototyping, right? You want this code to go into production. So you want it to work with a variety of cases, >> right? >> And also another thing is you want to reuse this function for different things. So you want to ensure it work with a variety of cases. So uh you can unit test and with unit test you can for example you was wondering like so if I put A in I get B but if I put C do I have D or so like it allows you to test different hypothesis in if you just do all those in notebook it's very messy and but like with unit test is you know you you have a function here and you have unit test here and you run with a variety of cases So it's very clean at the same time unit test can serve very well as a documentation for your function because yes uh sometime people look at your function and they're like what is this doing [laughter] right because you you see I mean of course you should write readable function but even if you write readable function it's still hard for people like if it's a complicated function it's still hard to people to understand it. So if they see how it is done let's say uh with unit test a lot of time you like provide an example like a variety of example so you you put some like for example a multiply function so you see the input is one two three and the output is multiply input uh comma three then and you see it's like three 6 9 so you understand you don't need to like read through the function you can just read the unit test, >> right? >> To understand what the function does and >> you can look at the ins, you can look at the outs, you can see the results. Yeah. >> Yeah. I I like the writing it down thing that that that's interesting because it's like a thing that so often is taught to learn anything. You know, like one of the common things you have is just like, okay, I want to learn this concept requires this extra step of you pausing, taking and using this different part of your brain. And I feel like testing is is that it's like shifting programming into this other gear >> potentially making you think about how you would explain this. Think about well what are the test cases? What are the different types of ways I personally might reuse this code because we're trying to make it reusable and trying to make it more functional. So yeah, I like that that idea of like okay, it just it pauses you for a minute to to think about how all this stuff works. Often I feel like people create well definitely with AI people create tests that are very very nonsensical in the sense that they they're just like not only a b they'll make it just be boolean. It's like true, okay, we're good. Then we're tested and it's like no, like that's not really testing, you know, thoroughly of like what the function's doing. And >> yeah, >> and I agree with the documentation part. So >> yeah, I heard a episode somewhere on like founder of TD uh test drive development and he yeah >> uh he also work with AI now, right? But what he does is he write the test himself and then like ask it to write functions to test again the unit test. But like the fact that you write the functions yourself to like you know specify the behaviors right to make sure that it's yeah >> it's match your behaviors that you expected. >> Yeah. Yeah. Yeah. It's it's great to to see. It's hard for people to get over the hump though like of like having [clears throat] to do the step of it. It's, you know, almost like the first process of learning again of having to like write things down. >> Yeah, I I Yeah, that is true. I do think that you don't need to start like do everything, right? Just do for the most important thing and and then just just do that first and expand it as necessary. It does [clears throat] >> and I think to be able to people wouldn't do things that they don't think necessary, you know. So be able to understand the benefit of it too is like for example people just tell you hey you should do unit test you're like why why should I do it you just get lazy you know but if if you know the benefit of it you know what it's for then maybe you can focus on creating the test that actually matter >> yeah yeah I feel like the test-driven de development thing is is hard for an experimentter person who's trying out these kinds of things and and I don't know if somebody necessarily needs to learn that entire paradigm to start and in fact you talk about unit test almost exclusively so far are you using other tools or are you using things like uh pi test or other tools >> yeah I use pi test I really like pi test because >> it's very easy to use compared to unit test is and I find that the code the test also more readable as well with pi test >> yeah yeah you've mentioned a little bit about this sort of modularity of notebooks potentially, you know, keeping some of the functionality reusable. Are there other areas where you think that you should consider, you know, separating parts of your code? >> Yeah. So, I think values is another thing. So, I'll I'll give a scenario. Let's say you want to experiment and and I'm think about like maybe like a machine learning experiments, right? >> Okay. or like just data processing functions you want to try with different parameters and if you are working with a Python script this will apply to notebook too but the ability to so let's say you want to change something for the sake of experiments you would need to like dive into the code to like see where it is at and if you repeat that value multiple places you need to change it in multiple places and first of all that is sample zooming that's uh discourage people from experiments another thing is you can make error because you know let's say you change here but you forgot to change there then there will be some errors >> so what if you can just like extract those put in a different files and those files will only contain and contain the information about the parameters like which values you can use um which is like a YAML file right and you can just go in and you can um change >> yeah yeah >> then that's to me that's like very powerful you can separate it that will make it so much easier for people to change your values and also it keep your code less messages more readable and and also it's make your code more documented too. So instead of saying I don't know uh some processing function you multiply by 120 like people read your code and say what what is he or she or whoever like why 120 here right? >> Yeah. >> But if with the configuration you can say config dog processing dog some readable name then like the person can like understand what it is about. So it's also like more readable as well, >> right? I think it's interesting because it you see that pattern across so many areas of computing like the fact that you have a settings file for you know almost every program you can think of and it's stored in a similar way. Uh, and that might be, you know, preferences about like how I'm left-handed, so I I, you know, run things this way or whatever. To be able to migrate that into your code to see the advantage of how different users might use this or you might put on a different hat that today we're going to use this code to do this other thing. by having a list of variables and not I I don't have a computer science background so I sometimes throw these terms out and I don't know if I always get them right but I hear this term magic number uh used which is like to hardcode a actual value into your code >> and then you again you see this 120 somewhere in your code you're like what is that and what does that equal and as opposed to having it set aside in the configuration file where it's it's named as a variable and then that variable's used across the rest of it and you're importing all that stuff in. >> And yeah, I think that's great. That's a I think a really good practice. >> Yeah. And also with unit testing as well, like you don't need to depend on that variable, you know, like you can switch because like if it's not hardcoded, it's not in the code, then you can like text with different values as well. >> Yeah. Yeah. Totally. like it it kind of keeps it opens up so many possibilities without necessarily breaking your code. One of the things that you mentioned also that I think is interesting that we didn't harp on too much but when you do like a git commit a and you uh some you know people often will run the idea of a diff you know like see the differences between things and this ability to not change your code so much. MIMO is yet another win there in the sense that it is a Python file. When you change a Jupyter notebook, I don't know if you've seen this yourself, but the diffs are pretty crazy because it's all JSON and everything sort of shifts around and it's not very readable to see what has been changed. Uh just kind of going back to that idea. >> Yeah, even with the tool for like notebook diff which is much better, it's still not as good. >> Oh, okay. I didn't know about that. It's still not as good as um because if you think about it have the output and the sale it's like so many thing for you to compare like if the output is different it say different but it shouldn't be the case is what you compare is the code you are versioning control the code not the output you know. >> Yes. [laughter] Yeah. Yeah. Not the outputting and formatting or whatever. >> Yeah. >> Yeah. That's great. I like to ask everybody who comes on the show some questions and the first one is what are you excited about in the world of Python? >> I am excited about I don't know I I feel like there's a lot of modern tools coming out and I like to see more of like UV. I like what to see of more of Rough. >> Yeah. Um I like to see more marim mo and uh I I I think there's a lot of movement towards modernize the tools that we are using and [clears throat] and the fact that they are also being open source. >> Yeah. My my thing to add on top of that is that almost everything you mentioned there I'm trying to think of the name of the company that does rough and UV astral they're all astral stuff is all is all rust based and so it's very fast. >> Oh yeah. Rust is very nice. Yeah. >> Yeah. What's something that you want to learn next? This doesn't have to be about programming. Could be, you know, any creative pursuit you're interested in. >> I think I I like to understand more of like applications of LM in data science. I I have been writing more pieces about that. But I kind of want to like dive deeper into it. >> Okay. >> In the future. >> Are there particular tools that uh interest you right now? land chan land index uh land graph is >> okay >> what I'm I have been writing about I guess it's like a lot of things to to dig into >> yeah yeah >> but yeah I'm pretty excited about how easy it is to use them for different applications >> yeah cool well that leads me to the next question will people watch your experiments [laughter] and uh follow along with it what's the best way to follow your work >> they can just go to codeca AI I also another thing that uh I like to do is to send out newsletter uh free time per week and each newsletter it have like a lot like different code snippets >> okay >> and that's my ways of teaching something I try to extract um because again I assume my audience are very busy so I want them to be able to consume as much as possible in the minimal amount of time so they can go to my website, they can see my blogs, but they can also get um code snippets that are like featuring different functionalities of different tools in one co- snippet that can be consumed in two minutes. >> Nice. That sounds like a great way to I don't know, there's so much stuff that comes in your email today. It's nice to have something that's a bit of a gift there. [laughter] >> Yeah. Cool. Are you doing anything on the social media front? >> Yes. Um, I have uh LinkedIn and uh X slash Twitter. I have around I think 115,000 followers on LinkedIn. So that's kind of like my effort. >> Yeah, >> I post it like basically every day except for the week weekend. So people can follow me on LinkedIn as well, which they will find the link at the end of my website. I don't want to like give people too much, but they can just go to codec.ai and they will you'll find everything there. All right. Well, Quinn, it's been really fantastic to talk to you. >> Yeah, thank you for having me. It's very fun. >> I want to [music] thank Quinn for coming on the show this week. And I want to thank you for listening to the Real Python podcast. [music] Make sure that you click that follow button in your podcast player. And if you see a subscribe button somewhere, remember that the Real Python podcast is free. If you like the show, please leave us a review. You can find show notes with links [music] to all the topics we spoke about inside your podcast player or at realpython.com/mpodcast. [music] And while you're there, you can leave us a question or a topic idea. I've been your host, Christopher Bailey, and look forward to talking to you