Ready to Publish Your Python Packages? | Real Python Podcast #83

Key Takeaways

welcome to the real python podcast this is episode 83. are you interested in sharing your python project with the broader world would you like to make it easily installable using pip how do you create python packages that share your code in a scalable and maintainable way this week on the show real python author and former guest dane hillard returns to talk about his new book publishing python packages dain shares his research into creating python packages we talk about tools techniques and potential pitfalls of publishing your own packages dain also discusses his experiences unraveling projects and determining dependencies we also covered dane's recent conference talk at pike atham titled keeping code safe and modern with semantic searches and he shares some security tools and practices from the talk this episode is brought to you by sea data software the easiest way to connect python with data sql access to more than 250 cloud applications and data sources alright let's get started [Music] the real python podcast is a weekly conversation about using python in the real world my name is christopher bailey your host each week we feature interviews with experts in the community and discussions about the topics articles and courses found at realpython.com after the podcast join us and learn real world python skills with a community of experts at realpython.com hey dane welcome back to the show hey thanks so it's not been quite a year but pretty close yeah time is kind of warped lately since i last talked to you you've been working on your new book but i thought we could maybe start by talking about a recent talk that you did you actually just did at pi gotham so what was the talk about yeah so the talk is about it's called keeping code safe and modern with semantic searches and the idea is that you have a lot of signals that you can actually draw out of your code to understand things like type safety and syntactic errors and common sort of constructs that are either dangerous or likely to produce bugs and if you think about kind of the the swiss cheese model to security right you can kind of apply the same thing to correctness of code so the more the more of these kind of patterns you can cover with different tooling the more likely you are to have safe operational code that you can that you can deliver and it specifically talks about this tool called sevengrip which is not a python specific tool but is very good at working with python it kind of integrates with pre-commit and ci tooling and all this stuff and you know it's it's sort of like a generalized case of a linter you can think of it as okay where you could use pylint or flake8 or something to to check all the things that are in those libraries if you have something specific to your organization or you know specific to your project that you know you need to look for as problematic or something to produce a signal of some kind about you can kind of build those uh in a custom way with tools like some grip so okay and that's is it sem grep yeah i i think i say it in the talk too i think it's short for semantic grep but i don't know that there's like proof of that in their documentation okay that's fine and then just to kind of backtrack just a touch you you mentioned this model of the swiss cheese approach and i'm guessing that is sort of the idea that one slice of cheese may have a set of holes in it but if you layer another piece of cheese maybe at a different angle or something like that that then the coast those holes are going to be in different places and and maybe block things is that the premise yeah that's exactly right so uh the goal is to get enough slices of cheese that there's no one path straight through all of the layers of security right right okay yeah that's interesting to me that the that i don't know i've had several people on talking about you know we're talking about packaging today but you know i've had a handful of people talking about kind of the the security and upkeep of packages and and you know the different types of things of that you kind of need to pay attention to so that's that sounds like a nice talk to kind of get people started you know down that road of like continuing to look at the security of of your open source stuff yeah and there's this you know there's this uh i don't know what to call it there's a renewed emphasis maybe on shifting things left right so the earlier in your development process you can get feedback about things the better yeah and as you think about these linters and things the more you can really leverage those ideally before you ever commit your code in the first place you know the more likely it is that you'll continue having nice operational code in the future have you used other security type tools that do that kind of checking they're not a sponsor currently but sneak recently sponsored the show and i've looked at them as like another one that you kind of does like this sort of cross-checking of your projects views stuff like that yeah i haven't used them specifically um but i've used one called sonar cube okay which again is kind of a cross language tool and then i know there's tools like bandit that that look for they do static static analysis at the command line yeah those talks typically go live i probably don't have a date right now but probably in a few weeks after this episode comes out people could probably find it yeah i would imagine so and i wonder if any of it is available on twitch too because that's where they were streaming them from okay because i know twitch you can kind of cut segments from streams but i haven't checked all right well if we can we'll toss in a link there if we can find it to share yeah you were on before uh and we were talking about the book practices of the python pro and then we kind of got into a very philosophical discussion and the book was kind of more geared that way of this idea of like okay well what is a pro what is you know what does that mean and that sort of um was sort of a bit of a you know not only a mind state but just kind of like a process state of like you know as you're moving along as a developer but your new book is kind of coming at a very different tack and i find that super interesting the new book is publishing python packages so maybe we could talk a little bit about when you started writing this book and maybe the impetus of why you wanted to to tackle this topic yeah the the driver for this came out of a desire and a need really to come up with a homogeneous way to do packaging within our organization okay we have probably almost two dozen python packages that we maintain and really some of those are much bigger than they should be so continue to split that up further but each was kind of developed organically and not all of them were following the same practices or had the same format for all of their configuration and you know that that gets very hard to maintain very quickly so ended up doing some research a couple of years ago now into some of the best practices at the time and had learned a lot about things found that i was rather behind in uh in what i knew about best practices for packaging and ultimately ended up building kind of a a reference architecture if you will for publishing packages within the organization and with with some of the newer standards and developments in the packaging community in python how would you share that within the team so we actually have a what we call a reference architecture documentation within our organization that we kind of share okay the best practices we've defined for a variety of things and then we built like a cookie cutter template for a python package that kind of incorporates all those things so you can kind of get a new package up and running rather quickly do you mind explaining briefly like what cookie cutter is i think some people might think of that as just as a term but it's actually a kind of a tool right yeah so cookie cutter is this really awesome tool that can you can you can take the heart of a of a project uh for the file structure and the file contents and template out the pieces that someone might want to change and then you can provide that usually as a github repository and people can actually create versions of that project with all those variables templated out it has a nice command line interface and all that for bootstrapping a project from a template so it kind of like uh prompts you yeah exactly like in a command line sense right so it kind of walks you through and then it builds i don't know i want to call them stems but these parts of all the files that are needed within the project um so you could have like a cookie cutter for say django or i saw one for i think i talked about it on the show a little while ago about setting up things for like a docker container and like being able to have it ready to go what was funny about the article was the guy was actually throwing out the majority of it which was really strange i'm like why didn't you just make a new cookie cutter like why are we like throwing away 80 of this it was really kind of odd but um yeah but yeah that's a neat project yeah definitely i would recommend it if you find yourself repeating yourself with some regularity yeah yeah totally that's interesting that a lot of the sort of source of this uh started you know that sort of started making you think about like okay i need to do this within the organization and then how soon did that move into you wanting to maybe turn it into a book i guess as i learned a lot more about how far i was behind i also took stock of kind of across github what it looked like other people were doing and i saw that a lot of people are still using things that i was using that seem seem like they're the less recommended way of doing things for example using setup.pi for dynamic well not for dynamic uh information but using setup.pi when you don't need dynamic information it's better to use static metadata now things like that so just seeing where the industry is and even where the some of the tools have adopted certain peps so far kind of shows there's still a lot of catching up to do and i know i was very behind even on what was possible or what was defined in some of those peps and standards okay just wanted to honestly like more start a discussion and and make that more at the forefront than anything else yeah so it sounds like a bit of a research project and sort of documenting your path in some ways that's a really good way of putting it i think because where the first book was very much kind of uh i don't know i'd call it almost a thought piece from myself like here are here are the things that i think are interesting to think about right this one is very much more uh involved in let's do some landscape analysis let's do some understanding let's read deeply several of these peps and yeah definitely feels like more research i i never got very good at research in uh in school so sort of learning some things for the first time yeah i was terrible at it i dropped out um not because of it but i dropped out of college and part of my frustration was with the the the whole i guess it was 102 or whatever it was english 102 and i was just like this is so much busy work and it was like frustrating to me because it was like i i'm like man there's you know this is a long time ago so i'm like i'm thinking to myself i could automate this i could do this and so forth there should be ways to like make this process like so much smoother but it just seemed like turning in my note cards and all this sort of stuff was like so yeah i don't know just wasn't for me so i dove deep into music and anyway never turn back yeah in some ways yeah i was going to say that i recently also got a lot better at note-taking partly partly driven by this book i think but also like when you get to when you when you start getting older like it's much harder to retain information even even now i'm not i'm not that old right but it's harder and harder to keep all the context about everything in your head all the time right so note taking becomes super valuable and so this kind of like learning to take notes learning about research learning about making arguments has all has all been very fresh on my mind yeah no i can imagine do you have you developed like tools or systems for taking notes and i know there's a a bunch of interesting sort of platforms that i i see some of the other authors uh for real python use i'm my notes are very scattered and i gather lots of them i'm using a tool called drafts to do all this you know basically two years of podcast stuff now and it's nice because it's very portable across like all my digital devices but do you have some some tools that you like to use yeah so i have tried a couple of them uh in general what i enjoy as a model is this kind of bi-directional linking note-taking uh graphic graph based notes um and i think it tends to help me represent my thoughts a little more clearly than hierarchical note-taking systems and it it supports a bit of kind of the scattered model if that you know works for you too so i use obsidian these days okay and yeah i've heard that you know i wouldn't say it's significantly better or significantly worse than than other offerings out there but what i like is that it's marked down and you can keep your mark down with you it doesn't have to live in the cloud those kind of aspects are helpful [Music] see data software connect integrate and automate your data from python or any other application or tool at sea data we simplify connectivity between all of the applications and data sources that power business making it easier to unlock the value of data our sql-based connectors streamline data access making it easy to access real-time data from on-premise or cloud databases sas apis nosql and big data check out cdata.com to learn more [Music] i guess maybe we could kind of roll back to the core concepts here of talking about packaging and david and i definitely had lots of episode talking about packaging and changes in the landscape packaging but maybe i don't know if i've had anybody help kind of define the the kind of the concept of like okay well you know technically what are we talking about with you know what is a package in this case as far as the book title publishing publishing python packages is concerned um that's speaking to the distribution package definition okay meaning something that you publish out to some place where others can access and install it from it's a you know bit of an overloaded term in python because you have import packages as well right but so in the in the general sense a distribution package or a software package if you want to even go that general is kind of this idea of coupling coupling code whether it's source code or compiled code of some kind along with metadata about that code and maybe some supporting data files and things like that so this applies across several systems you know linux package managers and npm ruby gems and all that stuff kind of has the same high level concept i guess about coupling software with metadata so that you can search for it and install it and sort of build on each other yeah okay in the case of python the the one of the major package managers that you know people are using i guess that term can be confusing sometimes people might think of that as you know it's not a person who's managing the packages but this idea that the packages you know that we've talked about um pipi the python package index but the package manager is more of a a tool for you know people can sort of use to access the the package index is that correct yeah that's a good way to think about it and i don't like you like you're alluding to it kind of depends who you ask but package managers can also do the dependency resolution task that kind of okay fits into that space so right that was a big conversation i had last year with the people from from pip talking about their new uh ways of you know how it's going to resolve that actually we're rewriting the resolver right and updating it and so that was like important like make sure you update pip yeah i know it always asks you potentially depending on your your um distribution of python right um but yeah cool so then maybe we could i think the idea of like the the term publishing is kind of interesting too like how is that unique like you know in this case like what makes it a published package yeah so i think that starts to get at this idea that you've created some code that's useful and you may even put enough configuration around it to be able to do like an editable install from your file system let's say okay but if you want to be able to share that code in a in a useful way with someone else ideally in a way that doesn't involve them having to clone your github repo and fiddle with it to work on their system and so on publishing a python package means actually creating a built distribution maybe that's platform specific if you have if it's not a pure python package and putting that somewhere like i said that other people can access it and that could mean the python package index it could mean a private index that your organization maintains right you know it could mean anaconda yeah there's a there's a variety of these kind of locations that it could be used we talked about that idea of like the internal repositories and there was that weird kind of thing where people were spoofing names that was kind of messing up people's um you know the repositories that were inside of like organizations which i thought was interesting by creating names um on the greater uh larger ones that you know things could go and look for right yeah okay so i guess maybe we could talk about you mentioned there that some of the things that you're going to need to think about you know if you're going beyond just trying to share the code like like you said on github the the the problem with that potentially is you would have to write some pretty detailed instructions that explain you know what platform this is for and um you know what what this code works for but in the case of setting it up to publish you are going to need to go through a lot of these kind of very detailed steps of setting it up so that the python software that you're sharing in this case can be built and you know run on a huge variety of people's machines i mean of course you can limit it also like i recently talked to will mcgougan about his project textual and currently he is it's only working on kind of unix-based stuff and so he doesn't he's got to figure out how to get it working on on windows stuff too because there'll be specific things because of the the way it's interacting with these terminals but to me that i think that's really interesting that this sort of idea of like okay in some ways you're trying to remove your computer setup from the equation i don't know if i'm explaining that right is that part of the deal yeah i kind of follow what you mean by that which is that here and now at least uh most development we do takes place on our local machine right you have things like github whatever whatever that product is called that's that's doing development in the cloud and stuff now too but largely we we tend to do development on our machines and that represents or may represent some very small subset in some cases of of your target audience like my taking myself as an example i don't own a windows machine so you know i think we have my partner has a windows machine but if she didn't have that i would have no access to windows in any manner just because that's the technology i've happened to buy into so if i ever want to build something that targets windows you know i'm kind of at a loss if i'm thinking about what's possible on my own machine so then you start to to think about what's what's out there that you can build in something like github actions or jenkins or or whatnot that has maybe a docker image that's available to build for a variety of different platforms and there are there are these tools out there for for python there's like ci build wheel okay and the the mini linux wheel allows you to kind of build for a wide variety of linux platforms as well so yeah you kind of want to think about your computer as the the means but not necessarily the end okay because you don't want to end up in the in the situation where you say like works on my machine right especially if yeah you know if you're sort of you know quote unquote publishing it it'd be like sending out a magazine that you know nobody can read right um unless they had like a special like reader or had to come to your house to read it you know right so so in this case it's interesting to think about like i feel like this is a you know kind of a different audience right like that you've decided to write this book for um compared to the python pro book like who would you say you know specifically this this book is for yeah where i think the first book was really sort of a broad broad audience sort of maybe maybe skewed beginner book this one i think is certainly a more niche subject and and probably more toward the intermediate uh part of the spectrum okay and i think if you're someone who's finding themselves wanting to share their code finding that the way their organization currently shares code that isn't packaging and you want to learn more about how to package in a way that means you can install things that your organization has developed yeah just the same way you would install something you know like requests or numpy or any of the really popular libraries uh you know that's that's kind of where this is meant to to bring you up to speed yeah so so instead of like having to find the readme file that bob or jane wrote explaining how to install this particular set of tools they could simply you know use something like pip and point it at a you know internal repository in and be ready to go i think that's really cool were there other experiences that you had maybe outside of your work environment that made you want to write this book yeah i mean we well you say outside the work environment we did open source a package as well that's kind of a builder for rest clients built on top of requests and kind of is a declarative model for defining api endpoints and things okay so we've we've got a micro service oriented architecture and found ourselves kind of repeating a lot of the same here's the way to fetch the host and here's the way to get to the path and here's a way to do service discovery so we kind of kind of abstracted that all into a package that we thought others might find useful so we open sourced that i maintained a couple of other open source packages none of which were you know particularly popular i guess it's not like i'm uh managing something like textual at the moment but just seeing the kinds of things that can go wrong and the kinds of yeah even the kinds of code review that can happen i think there's something to this general idea of maintainership right that uh is often considered a significant burden and rightfully so i think a lot of the discussion is about attitude and abuse and conversation and feedback which i think this won't solve right but yeah potentially if if there's any part of the burden that you can unload into an automated process like i i would hope that people can can take advantage of that so you know if you're if you're finding your team is code reviewing and saying you know the formatting of these lines isn't as readable as it could be or you're using single quotes instead of double quotes uh don't don't make that a people problem try and try and standardize on that and make a machine tell you that yeah it's like that yeah those those nice code formatters kind of can take care of those those discussions without it really ever having to be elevated yeah and so the book goes through some of that too and talks about you know how to automate a lot of your core workflow to keep that out of your way and let you focus on whatever it is your package is actually trying to do okay i thought about going down this line of questioning because i haven't done this process i've seen you know some some of the resources that you know real python has and then obviously you know reading through your book and and covering this topic over the last year and a half i've thought about it for my own purposes like okay well you know do i have something that i want to do this do this with there's a author christopher trudeau who i work with a lot and he's created this sort of tool that he uses for his video courses that kind of types out a lot of the stuff so he's not having to do live typing you know instead of having like correct mistakes as he goes and stuff and i found it kind of a neat tool and so then as i review his projects i i can simply just install that particular package and work with it and so i think of that as like kind of this sort of nice sort of small package is that a good place to start is to to start with just like a like a personal project you know in i'm not sure uh you know i didn't dive all the way through the book and it's um currently a manning early access yeah and so it's not you know all up there to check out entirely but would you say that that's a good plan start with like a small personal project yeah that's a really good question and i would say from one perspective anything that helps you learn is is good enough right okay so if you're if you're a newer to packaging and you need to understand the ropes starting with something like that is is perfectly fine and there's even you know i don't know how many people know this but pi pi has its own test pi pi also so you can do a lot of like okay i i don't wanna i don't want them to get mad at me too much but you can you can do a bit of like junk and fiddling publishing there without worry that it will end up in in the real pi pi in a way that will actually get used out in the wild too much okay and i think that's a good way to learn the ropes understand the kinds of errors you might encounter during publishing gaps in your metadata things like that you know you could also spin up a i think they have docker containers for like a locally hosted pipi type mirror thing okay that you could sort of go through the steps to publish too yeah okay so you know from that perspective i think it's really good to kind of get your hands dirty and and really understand things so that they aren't magic yeah and then the other side of it too is you know you mentioned earlier that sort of package name squatting was a security issue yeah i also think it's kind of a hygiene issue in some sense like i always think back to when early days of any social media platform right where you're like dying to get the username that's your name or your handle you always use sure and then you find someone registered it three years ago and has zero posts and no profile picture you're like why did you need that i need it yeah it's kind of the same thing on a package index so when you find something already has your name that you wanted it's it's used up and you can't really maybe there are exceptional cases to this but you can't really just switch out from underneath people what that package actually represents yeah and pi pi also doesn't have name spacing at the moment so it's all just one flat global namespace that was interesting that i think that that's something that i want to say it was dustin ingram they're working on it right yeah he was talking about it right how um a couple different packaging things are starting to do that which i think is great yeah it might it might have only been on his wish list i don't remember for sure okay all right but it's certainly something that's being thought about right yeah i thought like gosh i have to go back and look but yeah that i i you know i think that would be really handy for you know a variety of organizations to to to think about that not so much for you know individuals or sing singular projects but you know if a bunch of stuff kind of goes together then that might be useful and also kind of limit some of the name space pollution of you know all these things that are out there right um i mean npm is one javascript is sort of notorious for having a bazillion packages uh because there's no batteries included in the language and all that right and they've actually been pretty successful with the name spacing model so lots of major projects that started with sort of one package have since split out into sort of the core package and all these peripherals that people might need and they've put those all under a namespace so it's you're always clear that this is the official thing uh because it's in that namespace but you don't have to pollute the flat global namespace with all those different things so nice yeah all right well i hope that kind of comes into play more kind of diving into the book about how your approach is different in this case there's a lot more you know code and and sort of taking people through the steps which i you know i think is great especially in this like the whole point of it is to like you said to just do this thing you know get your hands dirty and actually try it out yeah and i you know the the risk that you run there is to make it very recipe-like okay which in an article or a blog post or something like that where you're really just trying to hit home on here's here's how to go from zero to one on this thing is a perfectly good way to do it when you're when you're kind of going at a at the depth of a book it can be it can get very formulaic i guess to say here's the sample write that code run that code here's the next sample write that code run that code so i really wanna i find myself i guess being careful about making sure that the teaching is actually happening underneath all that right so yeah the code samples should really kind of drive home the concepts a lot of these concepts are really are really difficult to understand right so yeah i definitely see that there's lots of references inside of it and in explanations of these sort of core concepts um because there is you know there's a lot of there's a lot of background to cover you know like like i said i've tried to just touch it multiple times from the podcast sense but i feel like a book is really kind of the place to go for this because you have so many different sort of tools that can be used in this process and you're diving deeper into those as you go you really cover a lot about automating but also testing you know which is crucial to this whole process um which makes sense to me what are other things that you you thought about in this process of you know you know turning it into a book that sort of expands it from being like an article or whatever are there other things that you thought about that you felt like okay this definitely needs to be included yeah i would say at a high level my my thought process there was kind of like a lot of articles focus on you know here's how to do testing with talks or here's how to do pi test here's how to you know there there's sort of a theme to to it and it shows you one way to do one thing what what a book needs to do in in many cases and and what i'm trying to do here is like there's this there's this long thread through the thing which is that you need to get from having some code to having a package of code right and so the book actually tries to get to that very quickly it's not published code it's not um pretty code uh by any means but it's like you've now got a packaged piece of code by the way if you want to publish this thing and maintain it and not get totally burnt out there's all these other things you're going to end up doing anyway so why don't we talk about how to maintain all of that in a reasonable scalable way so that includes testing like you mentioned it includes type checking it includes automating the actual publishing once you get there yeah it includes at the end i hope to kind of talk about like okay now you've published this thing and you've got an open source package you might want other people to contribute to it so yeah how do you make sure you have a inviting community built around that so it's really i i have this interest in making sure that most things i put out in the world are thought about holistically and it doesn't always mean that i am going to go into the depth that any particular area deserves but i more so appreciate the end to end view and the big picture of like what is it we're really trying to do here yeah i feel like you know we we talked about that you know even though you feel like this is uh has a lot more step-by-step kind of stuff in it there's still a lot of that philosophical thoughts that are in this like one of the areas that i thought was really interesting is you talk about you know thinking about the code itself like you mentioned stuff about your work and how certain portions of the code and way things that are grouped or organized it may make sense that that these should be separated out and so you have a lot of concepts that you're you're kind of covering to to make sure that that you know again instead of it just being like a straight-up how-to guide you're adding a lot of the why um or the the thought about it in that sense yeah i think that's i think that's right and again the risk is sounding like i'm meandering all over the place right so sure hoping i haven't uh made myself just sort of a seem like i'm i'm just wandering around blindly too much but i think it's important to i don't know there's there's two types of development for me really one is i know exactly the outcome i'm looking for and i'm going to go i know the line i want to change and i know the test i want to change and i'm just going to go change those test paths okay great done then there's also the i want to change this code i know the last time i did it was kind of hard the tests have coverage but i'm not sure if it's the right coverage i don't feel great about making like doing any big lifts in this area at the moment but i i think i'll need to and by the way there's all this other stuff in the way uh like once i make the change i need the test to pass i need the code to be formatted properly i need the package to publish successfully you know so it can get kind of if not overwhelming at least tedious and kind of want it to kind of want to give people a sense of what kinds of processes are possible for them that help them do more uh actual development yeah that makes sense yeah i mean it's i don't know i don't want to say it's easy but like it's fairly formulaic in the sense that it truly is a formula that you're you're setting up to discuss you know okay this is how you could configure continuous integration so that it it's going to you continually publish this package and and run it through these processes and so forth but to take a step back and say okay but you know have you thought about like how the code's organized also like before you you decide to to do this which i thought was kind of interesting the idea like you're talking about cohesion and encapsulation and i thought that was interesting like is that something that at your work that you're like having to think about okay like how how could i divide up the this code so that it makes sense and then i guess you use this term of decomposition maybe we could talk about that a little bit like what does it mean to to have higher cohesion yeah so it's a really good question the challenge i found myself with in some of our code i mentioned a lot of these had developed organically we were actually in a place where we were installing some of these packages from the file system rather than publishing them and then installing them okay and what ended up happening because there's no true division there you know all the code on the file system can access all the code on the file system if it really wants to and what ended up happening is we had this kind of circular dependency or at least tangled dependency situation so bad that i had to write like a script to spit out a graphviz visualization of what was actually happening right to to see what we could actually extract and publish so okay you know this this concept of dependency hell right that you have diamond dependencies and this package depends on these two and they both depend on different versions of this other package gets even harder when you're doing it all from the local file system and then find yourself wanting to extract some code so this whole idea of kind of modularization and yeah directed graphs so that things only depend on each other in one direction it is really helpful for ensuring that code it doesn't have the wrong abstractions sometimes it doesn't have the wrong sort of architecture and organization because otherwise you find yourself depending on things that were never meant to be depended on uh you sort of the blast radius of changes are hard to understand sometimes too yeah so you mentioned using you know a tool to sort of graph out what was happening inside of that did that help to i mean obviously it's going to point to those sort of circular or diamond shapes or other kinds of shapes that that are you know kind of creating these funky redundant dependency kind of stuff at that point then i'm trying like how do you even start to tackle those situations i mean i guess it starts to give you some some of the instructions in that process but were there steps that you take right after that to say okay we need to you know tackle this first yeah so it was actually really fun in a certain way once we got this visualization because you could finally point at things and say oh that one if we if we could just publish that one okay you know it would it would make all these other ones so much easier or all these other ones would suddenly be able to be published so by visualizing it you actually get this you know if you see a node and all the arrows coming to it are flowing in the same direction it kind of tells you like i can i can publish this one now because it's not dependent on anything else in this graph things depend on it but it doesn't depend on anything else and that's when you can that's when you know that it's publishable because otherwise you have this chicken and egg problem right if two packages depend on each other or have a cycle between them in the graph you can't ever publish them because they'll they depend on something that's not published yeah okay right so you know which which was published first yeah exactly okay probably probably there's a way you could manage to do it if you if you really tried but yeah so uh it helped us get a sense for which things were already publishable that were hard to see just by kind of staring at the code and then like i said once you pick certain ones off new ones become publishable we're still actually trying to get a last few of those and that's because those actually don't have very good encapsulation and cohesion so they're these kind of mega packages that do a bunch of different stuff for a bunch of different people and as a result it's a big ball mud and we just have to find the right pieces to chip off of it and extract so that again all of the dependencies in that graph kind of flow one way okay so the i understand the the flowing of it and then cohesion is that kind of the concept of that that they they stick together properly is that yeah yeah so like like things together um birds birds of a feather if you will right so right you know have all looped sides of velcro you gotta have some hooks too to be able to get them to attach and and work together okay cool so i kind of wanted to talk a little bit about some of the other steps that are involved in this process that that may go beyond even what's in an organization though i'm sure that this would be crucial there but you know how much of this process kind of can be involved in trying to documenting you know like what's in the package and is that something that you had to spend a lot of time thinking about too like how do we document our packages and and create the create the required tools for that because there are certain i would guess there are certain sort of requirements for documentation to to publish to these indexes is that correct you would think so okay there's a minimal amount needed i would say uh your package needs i'm not i'm not going to remember all the core pieces that are necessary off the top of my head but like you kind of just need a name and maybe a description and okay that might that might be kind of everything so you you can get away with a lot but that being said i am a big proponent of documentation and in particular documentation that lives as close to the code that's related as possible so tooling like sphinx is something that i'm very big on because it it sort of takes your doc strings directly from all your classes and methods and functions and modules and pulls that into a documentation system and the book the book covers that so you're kind of able to write it once inside the code and publish sort of you know external documentation or i've seen like you know web docs and other things and a lot of that's coming out of a tool like that that's exactly right and it you know it it does the automated part with the code but you can also write prose alongside it and have it all kind of fit together nicely so the book the book goes into that as well yeah in one of the later chapters and was that ever a struggle at your work for creating documentation like that oh yeah okay you know it's i think it's the stereotypical programmer attitude that like oh the code documents itself right when you when you get a team of a big enough size depending what your code architecture and your team architecture is like and how you all operate you might find yourself going into some other team's code at some point and needing to change it and wondering what the heck this thing or that thing does right and having that documentation is one heck of a benefit when when you need to uh know more about why somebody added this method and all the reasons behind the logic that it has and we we ended up kind of i don't know i've started i've started really trying to leverage existing systems right so the same way that sphinx can do interoperability with other sphinx doc sites so like django may link to a page on the requests documentation and that it knows it knows something about the structure of that documentation based on the fact that they're both sphynx documentation sites okay and so we actually ended up kind of doing the same thing so each package we publish has its own isolated sphinx documentation and then we have a intersphinx thing kind of connecting them all that's nice it saves time in having to restructure or relink those kinds of things yeah and you start to get this kind of rich ecosystem around it right which is is what's out there in open source but not as not as prevalent within an organization [Music] this week i want to shine a spotlight on another real python video course it's about an important step in the development and sharing of your code the course is titled documenting python code a complete guide the course is based on a real python article by james mertz and in the course instructor andrew steven takes you through the reasons that documenting your code is so important the differences between commenting and documenting best practices for doc strings and he covers additional tools references and documentation projects i think it's a worthy investment of your time to learn best practices for documenting your code and to learn about tools like sphinx which we discussed during this week's episode and like all the video courses on real python the course is broken into easily consumable sections plus you get additional resources and code examples for the techniques shown all of our course lessons have a transcript including closed captions check out the video course you can find a link in the show notes or you can find it using the enhanced search tool unrealpython.com [Music] what were some of the things that you were most excited about in sharing you know inside the book honestly i i like sharing things that i myself was learning along the way right so i mentioned there's a lot of research and um yeah something i haven't worked a ton with is non-python extensions so understanding kind of the workflow for building package out of python code or a c extension and sort of visualizing the build workflow behind all that was really interesting to me and kind of getting a really solid handle around python build back ends and build front ends which are the the newer uh standard around how packaging actually happens was was also really fun so you know hoping that is something not a lot of people know about and would find useful in the book yeah because it's kind of going through a bit of a change right now right i mean in the sense that there are older tools that are somewhat being deprecated for this process and and some newer tools that are being preferred is that am i correct in thinking that yeah that's that's a pretty accurate characterization i think it's yeah it's it's in flux uh to a degree not to say that the standards are in flux right so the the pep that defines this build system was actually written i'm looking six years ago almost to the day okay so that's a that's a while ago um but there are like the the tooling that people are maintaining is only just catching up to this in some cases so it's it's interesting to see what that adoption gap has been and and in some cases i'm sure for good reason uh there's a lot to a lot of shift and there's a lot of people that these tools are supporting and it would be a detriment to the community to to get it wrong or to to do it too fast and kind of give everybody time [Laughter] yeah and in some cases this stuff is all still sort of put on the developer uh because there's not a great way to hide it inside the management tools so not only are you you're not really just changing it under the hood in some cases you really have to think about the developer experience directly so uh it's not an easy problem that's for sure yeah yeah do you have like additional resources that you'd want to share good additional things that people can kind of look at yeah so there's like the python packaging authority who kind of manages the python package index and and no well they they manage the python package index and you know they do provide a very good guide it's it's rather abridged i guess um you know it doesn't yeah probably for smart reasons uh it doesn't cover in depth all the different ways you could go about doing things and what is there i think is really good and i guess it's a it's a bit of a maybe a cop-out to the question but one way that i learn a lot of stuff is by doing sort of a global search on github okay a really common thing that i search for is you know if i'm if i'm looking to understand more about how i should use this field or what the syntax is or anything like that i'll typically add the name of the uh the field and then do like filename colon setup.cfg and that will just show me every file called setup.cfg on github that has that field in it and then i can just kind of look through those results and see like okay these five people are using it this way these 10 seem to be using it this way and then everyone else is using it this way so then you can kind of start to understand the nuances of those use cases and that's what i mean by like landscape analysis that i mentioned earlier so okay really gives you a sense of like real world in the wilds type of of usage so yeah i think of like um brett cannon had this post about pipe project tommel but i could see using that technique to to look at tamil files to kind of get an idea of like what's going on with them absolutely but yeah that kind of uh just to see how people are using it that's been a common theme for people you know again researching code and kind of learning it but not thought of it of like these other files that are not necessarily pure python files these are files that give instructions to all these other tools right yeah and in some cases provide metadata and and things so that's where your you know again a package is like software with metadata right so right yeah it's uh it's an ecosystem another another thing that i that occurs to me is we were talking about pi gotham simon wilson had a great really like speedy talk where he he basically went through the process of starting from zero and publishing a package to pipi okay in like 10 minutes so definitely definitely check that out too cool and he has an example repo that he he links to there that you can go uh look at as well so nice so i have these weekly questions i wanted to start with the first one is what are you excited about in the world of python yeah i think it's it's got to be what what will mcgougan is working on right now you know rich and textual both are really cool we're using invoke for a couple of things right now and they're fairly simple tools but i could imagine if we ever wanted to kind of enrich that experience uh and do it in a in a python way that those tools would be the first place i'd look um yeah i'm excited to just try to find a couple projects i might use in it yeah so yeah all right so what do you want to learn next again it doesn't have to be python specific in this case yeah i have been spending a lot of time i mentioned modularization yeah i've been spending a lot of time on that concept generally speaking so my sort of full-time job is