Serializing Data With Python & Underscore Naming Conventions | Real Python Podcast #187

Key Takeaways

welcome to the real python podcast this is episode 187 do you need to transfer an extensive data collection for a science project what's the best way to send executable code over the wire for distributed processing what are the different ways to serialize data in Python Christopher Trudeau is back on the show this week bringing another batch of pic coders weekly articles and projects Christopher shares a tutorial by real python author BTO jinsky titled ser ize your data with python this comprehensive guide moves Beyond XML and Json to explore multiple data formats and their potential use cases it's a deep dive into the topic and provides a thorough resource for future reference we also discuss a real Python tutorial about naming conventions in Python that use single and double underscores the piece covers differentiating between public and non-public names and apis writing safe classes for subclassing purposes and avoiding name clashes with keywords we also share several other articles and projects from the python Community including a couple of release announcements and news items a discussion about never being taught how to construct quality software building a small repple in Python using the key parameter in Python functions and methods a framework for restful apis using flask and SQL Alchemy and a rust based HTML sanitizer for your python projects all right let's get started [Music] the real python podcast is a weekly conversation about using python in the real world my name is Christopher Bailey your host each week we feature interviews with experts in the community and discussions about about the topics articles and courses found at real python. after the podcast join us and learn real world python skills with the community of experts at real python. comom hey Christopher happy New Year welcome back yes happy arbitrary calendar demarcation to you as well yeah well we have a collection of stuff that kind of happened over this sort of break that we took we got a few news items and then some kind of fun topics there's a bit of a theme going on with topics for me at least best practices and things that happen inside of python that I think are kind of interesting we again have a discussion which I'm excited to kind of dig into so I guess we should start with the news you want to get going with that you say you have a theme like we did it on purpose sure so uh over the over the holidays a big commit was made to The cpython Interpreter and that is the jit compiler if you're new to this idea jit stands for just in time and it's a process that dynamically writes machine code to optimize your software on the Fly other python interpreters have jits but this is new to the Cory python version and we're going to be linking to both the commit itself and a set of slides that talk about what's in it the commits kind of a fun read yeah Brent butcher who's the committer he got a little creative and wrote a poem in the vein of the night before Christmas so uh even if you're not interested in the code the the commits worth reading if you're digging into the slides they're posted on GitHub which does this weird thing with PDFs there's this little more pages button buried at the bottom every few slides that loads the next chunk I missed it the first time and I was kind of confused as to why the whole thing was rather short so I'm sure if you all go back in there and see that you'll immediately laugh at me and go well but the buttons right there what's wrong with you it's in front of your face so you know have a good laugh and then the other chunk of news for any Django knots out there yeah that's what they call themselves I I don't but bug fixes dropped for 4.2.9 and 5.0.1 if you're playing with 5.0 which was recently released in December you definitely want this one it's got a about a half dozen regression fixes in it so you want to go get on top of that yeah that's a fun read from brt Booker yeah that was very Christmassy themed so definitely check it out all right so we'll dig into topics here my first one is from our friend leodas poor Ramos who's having a nice little break right now so I'm hoping he's enjoying as his extended holiday break and this tutorial is about naming conventions in Python specifically ones that involve the underscore character for a beginner I think this is really Illuminating like you look at other people's code and you go boy that's an interesting way to name things why add this sort of arbitrary at the time as you look at it underscore character in different places there's a lot of con inventions that were built up and you can learn a lot by just digging through this article I think definitely for a beginner it's like I said Illuminating but for someone coming from another language it's going to help explain some of these Concepts and the ways that python does it python is a very Dynamic language we talk about that a lot here if you're coming from something I'll call it more rigid like Java there's different ways of handling these sort of Concepts and so forth um so there's five of them that are discussed five different ways that the underscore is discussed in this article the first that he covers in a lot of detail is the single leading underscore so it'd be like underscore and then the variable name and it indicates again this is the the core idea it has to do with sort of objectoriented programming and apis these sort of application programming interfaces not so much of a web API but how other program rers are going to use your code and interact and you know interface with your code and how they should use it so in the case of the single leading underscore it indicates that it's meant for internal use only so it should be only used within the the framework itself um as opposed to outside use sort of this idea of we'll dig into it more of public versus non-public methods and attributes there's another one that's at the tail of something um in naming things the single trailing underscore is used to avoid naming conflicts with python keywords or built-in names in case you have names that only make sense to match up with what a keyword is um I think there's lots of other ways around this but this is something you may see out there the third one was about the double leading underscore this one is sometimes called name mangling and it triggers this feature in this context of python classes and if you're having things that inherit and it can help with sort of auton naming things for you by creating it this way and he digs into that there's uh the most common one that we've talked about in the show which is the double leading and trailing underscore sometimes called the dunder methods with you know again two underscores before the name and two underscores after and that would indicate that it's a special attribute and or method what's interesting is right I think this week L on us had another article that came out that's talking about Python's sort of magic methods or Dunder methods special methods different names for the same thing and leveraging them so I'll include a link to that he doesn't go into a lot of detail about Dunder methods in this article this tutorial but he has an entire separate one that just came out here at the top of January and then the last is uh sort of interesting uses for a single underscore by itself and it indicates a temporary or throwaway variable and there's an interesting way that that works so to kind of dig into the heart of this thing he spends the most of time talking about this public and non-public methods and attributes and how this works a public attribute or method you can use them in your own code or the client's code whereas something that's private or in this case non-public for uh the use as we're describing it here you would use them only from with inside the defining class and its sub classes so he takes you through the thought process there provides copious code examples kind of going through it creating public and non-public names avoiding pitfalls and how knowing this will help you and I'm not sure how often you will use this I guess it depends on how you've kind of built your code he gives three major reasons for doing this the first is preventing incorrect usage of your code to be able to mark internal parts of your code with non-public names allows you to guide users through the correct usage of your code like what should be considered use for non-public or public use it improves the readability of your code if you're doing this thing of creating non-public stuff it it definitely identifies them easily and then it avoids name clashes uh using non-public names can help you avoid collisions in packages and modules and classes he describes how Wild Card Imports work with that and how non-public names using you know the import Star thing python enforces a a behavior about how things would not be imported by a wild card import that have that and then it goes into these other uses a little bit more name mangling um again another convention that's using the two leading underscores in an attribute or method names and when you name an attribute or method with using this python will automatically rename it by prefixing its name with the class name and it'll have like a single underscore after it so this is this is something will make a lot more sense sort of visually as you kind of go through the article the goal is to prevent name clashes in in inheritance and again to Define non-public members you always want to use the single leading underscore the trailing underscore in python names this is a kind of an odd use case in my opinion it's like something if you wanted to use you really wanted to use the variable name list so like list equals square brackets 1 two 3 well now you've if you've assigned that to the name list you've Rewritten over the list callable function that is built into python that you can create lists with and if you were to you know try to create a list using list parentheses and have it turn something into a list it it basically would say this is not no longer a callable function and so it's kind of an interesting thing I don't I would definitely avoid renaming keywords if you can but this is a way that you can do it by just adding an underscore after it it's an interesting again convention and then he digs into the dunder name special methods how they're used but again this other article digs way deeper into that the last one again was this idea of the underscore just by itself I haven't used this a ton but this is a common practice and one of the interesting things that always kind of surprised me is that if you're working in a repple session it's a special variable that will contain the result of whatever you just evaluated so like if you typed 3 + 12 15 will show in there you can actually type the underscore hit return and it will return that as this sort of variable sort of as a special return value so it's sort of assigned as the last thing you evaluate which is interesting I've seen it commonly used in loops and sort of other constructs this is a convention that people do I would prefer naming something explicitly but again it's common that you may see some somebody use underscore in this as it's going through and iterating through like a for Loop or something I use it a fair amount actually it's typically when you've got a tupple that you're trying to unpack and you're not interested in one part of the tupple okay just an easy way to see it rather than naming it because then you're producing a variable that you're never going to use it's kind of a signal to the developer that that third thing in the Tuple I I want the first and second thing you know like say we've got oh okay I've got first middle and last name in a tuple and I don't want the middle name I'm going to say first comma underscore comma last and that tells the programmer I'm not going to use the middle name it's potentially gon to be just tossed as it goes okay yeah essentially yeah okay so that that kind of a then is an indication a Convention of like temporary exactly probably using it okay and that's why it tends to show up in Loops a lot because you're doing that on the iterator out of your you know what you're looping on yeah a lot of these conventions are described in pep 8 if you haven't dug deep into pep eight and read through it definitely as a beginner that is useful because those are the unique things that are happening inside of python as far as how you should name things the way things could be used and he provides links and quotes to those sections of that and I was going to ask you Chris like not only the underscore one do you use some of these other ones is it are they common practices for you some I do some I don't a lot of it depends on the code a couple libraries that I've contributed to people are like hardcore about it that like if this isn't if this isn't a variable that I expect people to use then I'm going to put the underscore on front of it and I've even seen people get to the point of where they're like using the underscore internally and then the property smells a little like Java to me at that point um so like right right right I I will use it when I tend to use it a lot if I'm using a property where the property has to have a side effect effect so there'll be the internal version of it which has the underscore and then the actual property method that causes the side effect to happen so if you're getting into properties and descriptors that's that's a place where I'll use it okay deep into oop stuff essentially yeah back in the days when I was a Java programmer I was Hardcore about like this should be private this should be protected and I'm trying to save the next developer from themselves and honestly as I picked up python I kind of gave a lot of it up I'm sort of like you know what if you want access to that variable use it why why am I trying to protect you like who am I trying to protect and from what and particularly considering 90% of the time I'm the one maintaining my code right like I've become a lot less um Vigilant about it now to me it's I tend to use it for something that is like this really is supposed to be hidden for a reason okay but i' I've seen a wide spectrum of it in in code other people's code that I've helped contribute to what about the after variable name I almost never bother yeah I would I would come up with like maybe a two-word name or something yeah you know I'm not um I I try not to reuse soft keywords I try not to rename functions I find it just confusing yeah uh and most of the time you know the example you used where like list underscore well it's going to be a list of something there's there's going to be a better name for it so you know that tends to be my but I I'm picky about my variable names so yeah yeah I'm super picky about it you know even in teaching it I want to have something that is something Beyond The Foo and bar and things like that so yep there's a there's a funny uh well I think it's funny because it's like kind of over-the toop comment at the end of this it's the only comment that's there but and he's like great article and why I never want to go near Python and it's like okay you know great article you explain it well but craziness of python that's like I don't don't know if it's craziness but it is a dynamic language and so these were conventions that kind of came up uh you know in naming things and having to kind of plan stuff as opposed to having dedicated private and public and so forth yeah if special symbols bother you stay away from Pearl that's all I have to say there you go all right well thanks again leest this was a great tutorial and if you're learning python it's one of these sort of like additional reads that can help with the Namy conventions especially if you're doing the thing that we often suggest which is reading other people's code and you look at the code and kind of scratch your head and it's like those are lots of interesting underscores and weird places this uh might help you with that so so what's your first one my first article of 2024 is from Max Bernstein and uh his site is Bernstein bear.com which is funny if you're familiar with that particular Mandela effect um the article is uh building a small rep in Python and like all good titles that pretty much sums it up so if you've ever played with a command line tool that was reppel like well python allows you to override the behavior of the rippel making it your own an example this is inside of Dango there's a management command called shell which when you run it you just get a python reppel but it's loaded all of the Django configuration bits inside already so I've never looked at the code for it but I suspect it's using the same kind of techniques that Max is talking about in this post there's a module in Python called code that has a class called interactive console and you can instantiate that and then call its interact method and that runs the repple so essentially you can write a script that invokes the interactive session with an import in two lines of code and you're running the repple inside of something you control the interactive console object allows you to configure the reel's title Banner as well as an exit message so those are two easy things to make it look like it's your own thing and then the heart of it is a method called run source which gets called each time somebody presses enter in the repple and this is how you can start writing your own stuff adding commands or whatever overriding run Source has some consequences though and it can Muck with your access to the inline editing features the article goes on to show you how to use the readline module to put this functionality back in and as an example project he shows you how to build tab completion for anything in Python's M dictionary which includes things like the built-in functions those things you probably shouldn't be renaming that we just talked about yeah Max appears to have been building something on his own while he was writing this post because near the end he goes wait there's this thing called the CMD module The Command Module which isn't as low level and maybe I should have been using that instead it also allows you to customize the Ripple so the last chunk of the article talks a little bit about why you might use CMD versus using code and the interactive console and when you might use the lower level code and how so if you need to write an interative acve program building it on top of the reppel can actually save you an awful lot of time it gives you access to Features you'd otherwise have to roll yourself and you can like write python inside of it as you go along so if you're thinking about doing this the this article is not a bad place to start out and if you want to start trying to muck around here I wonder about some of these what we often call an alternative repple things like B python or PT python I wonder which methodologies they're using to to create those I yeah I suspect a lot of it depends on what you're doing some of them I think are probably not built on top of the repple because they're like complete alternative mechanisms yeah sure and then others be python seems to just sort of add a few of its own commands and a little bit of extra editing and it's not that far off from the kind of things Max is talking about in the article so it would surprise me if they didn't build it on top of it yeah the Jango one I'm you know all it really does is load the Django stuff in the background and make it available to you so I'm 90% sure all they've done is gone import import import call Interactive and there you go right so that makes sense great my next one is from Steven Greta who's written for real python several times and we've featured a few of his tutorials on here and he has his own blog site newsletter project I don't know what they call it uh his whole thing is known as the python coding stack and that's where this is coming from and hey it's another repeated thing that's used multiple times in Python in this case it's the word key his sort of jokey title for it is the key to the key parameter in Python the word key yeah lots of different uses probably think of key slash values in in dictionaries um we talked about key words just a moment ago several times God there's a few other ones but in this particular case what step is talking about is the how there's some built-in functions that allow you to change the behavior of how this function's going to work by adding key equals the two examples that he kind of digs into have to do with sorting uh he talks about sorted and Dot sort which have kind of similar functionality depending on what you're sorting he goes through the default behavior of how if you're sorting a list and it would sort it numerically or if you're working with text it would or strings it would be doing that alphabetically there's already a feature built into sort for saying I want it to be reversed but in this case let's say I want to sort by completely different rules and you can use this key and key equals inside of there as a parameter and you basically throw a function into it so you say key equals maybe you want to use something like uh the you know length of your strings and so you could use or maybe they were u a list of list or something like that so if you put Len in there you don't call it so you don't put parentheses after it you put key equals Len at that point it would then sort say these strings or like I said a list of lists or something like that by the length of the elements in there or characters so you can use any kind of built-in functions if you want something like Min or Max as a way to sort of choose how you want the Sorting to go you can also Define your own functions and create them as rules for sorting so in that case you would outside of where you're calling this sorting you would Define your own function as you've named that you would then put the name of the function after key and I've used this a handful of times um it's a very useful thing in data science when you're working with lots of different parameters to kind of clean up things or sort things or do other stuff so he gives a handful of other examples probably the more common way I've seen it used is not to define the function outside of it and then put key equals but put to put key equals and then have a Lambda right there so you're doing it in line so he goes through that shows you the process of setting this up gives a couple other examples and then he shows a few other places that you may see key that are other than sorting iter tools has a group by function that allows for a key uh so you could Group by the size of things or you know maybe you want to sort or group by some other kind of parameters there's a few other places where this key parameter appears inside of the pandas Library such as like sort index or sort values and again grouping so uh something to look out for a way to customize these functions to give you or methods to give you a lot more flexib ability just by adding this and what's nice is again if you are doing it as a Lambda that is one way to do it but if you are doing this often it's something that you could create your own little library of simple functions that you could then reuse across again if you're doing cleaning sorting preparing things and something like a data science setup um which was really common for me it's a nice quick read to go through lots of little examples he has a couple like kind of uh interesting ones where he's like try to guess what I'm trying to do here which is kind of fun to kind of look through so um thanks Stephen I'm intrigued to see how his his new thing goes he's he has some numbers in there of like how he's been growing this since nine months ago his python coding stack so good luck with that Stephen he's got some interesting stuff in there um particularly he's got a bit of a knack for analogy so yeah there's a couple better articles on um sort of computer sciency things where he has some very entrylevel explanations as to why they are and how they work you know he I think he ended up writing a article on cues where he was like showing teddy bears in boxes and things like that it just sort of made it clearer to rather than getting into the details of something that's sort of a high high level better understanding kind of stuff he's got some interesting writing on the on the site yeah [Music] definitely this week I want to shine a spotlight on another real python video course it continues one of the themes this week about best practices for naming variables functions and how to write highquality readable python code this course is titled writing beautiful pythonic code with pep 8 it's based on a real Python tutorial by Jasmine finer and in the course real python instructor Joe tusco takes you through the various python naming conventions building a beautiful python code layer layout which includes line lengths line breaking blank lines and indentation formatting comments whether they're block inline or documentation strings using linters for better code and additional tips and tricks to help ensure your code follows pep 8 I think it's a good investment of your time to learn the best practices from the official style guide for python code pep 8 like most video courses on real python the course is broken into easily consumable sections and all our courses have a transcript including Clos captions check out the video course you can find a link in the show notes or you can find it using the Search tool on real [Music] python. so what's your next one I've got a real python article by Guru baz zinski probably heard us talk about articles from him before he's kind of outdone himself that this one's even deeper than usual this is on serialization and it's called serialize your data with python serialization is the process of turning some data in memory into another format typically so it could be put onto disk or onto a network and then of course you want to be able to deserialize it back into memory later uh there are a ton of ways of doing this in Python and depending on what kind of data you're serializing why you're doing it and you know a whole bunch of choices along the way and this article covers all of them I'm pretty sure it covers all of them yeah it's uh so it starts out by explaining serialization at a high level and some of the complexities that you can run into so for example are you going to serialize into a binary format or text if you're using text what kind of text Unicode good old asy some sort of encoding that can survive being inside a URL your choice affects what you can do and who you can share the serialized data with processing binary data tends to be faster than text and also tends to take up less space but then it isn't generally as portable there's a chance the code deserializing your data isn't in the same language as your software so the portability thing might be an issue once you've made that highlevel format decision another thing to consider is whether your serializer supports a schema a schema is a formal description of the expected structure of the document schemas tend to enforce rules about what you are serializing for example XML on its own is schema list uh you can write a well-formed x document using whatever tags you like but if you want a web browser to be able to display it it better follow the schema for HTML yeah that might be a bad example as browsers are rather forgiving about scheme of violations but you get the general idea I've been talking about this article for a few minutes now and have skipped a whole lot and I still haven't got past section one so uh like I said very deep dive yeah section two talks about serializing python objects uh you can use the pickle module for example which is a python specific format or you can translate your object into something like Json there are pros and cons of both pickle is more powerful with fewer restrictions on what can be serialized but it's python specific and has some security risks so essentially you're going to need python to read it Json on the other hand has a few limitations you can't serialize all python objects with it but it's more or less ubiquitous across the internet making it kind of the king of data interactivity at this point H so much so that there's like five different versions of it as well and one of which includes schemas if you need them the article goes on to talk about serializing executable code serializing for HTTP message protocols hierarchical and tabular data and then some details on using schema based libraries pretty much everything you want to know about serialization is in here and possibly some stuff you didn't know you wanted to know so yeah as often with bartage very very deep article here and lots of good stuff to sink your teeth into yeah it's a a great resource which is a common thing we say for bartage and stuff it's one of those that you would come back to as you like oh yeah this is a nice reference it's nice that we do that here at real python that that's a common thing for us to say like oh I have a question about this and you may only be hitting that little portion of the article or the tutorial or if you want to dig dig deep into it this is definitely one of those things all right well we have a discussion this week which is kind of fun we haven't done one in a little while and this is one that you found about building quality software which you've talked about a little bit over the last several months we didn't really talk about it from this angle which I think is kind of good and we both have very different perspectives on it so I think it'll be fun to dig into yeah you want to introduce it sure it's based on an article by a gentleman named Florian bman uh and it's titled you are never taught how to build quality software and found it through a Associated hack news discussion of course the article talks about the fact that most schools focus on the mechanics of software development like algorithms yeah with a varying degree of coverage on topics like quality this is one of the things that immediately got nitpicked on Hacker News because you know anytime you say this doesn't get taught someone goes but I was taught it so right or I teach it was one of the or I teach it yes so I suspect some of it is an age thing I I I'm guessing it might get covered in schools a little more now than say when you and I went through through and a lot of it of course comes down to the there's a wide variety of school programs out there right and boot camps and such that focus on different things as well right so you know your choice and what you did wasn't necessarily what everyone else does the article goes on to talk about the fact that most companies focus on time and money and neither of those things tend to lead to quality products yeah there's an old adage which is fast cheap right pick any two and that tends to be a problem for projects every not just software the second half of the article has a subtitle which is how to get out of the hamster wheel and covers some ideas for addressing a lack of quality well in development uh he highlights knowing how to talk to management speaking their language so that they understand quality issues are also cost issues talks about the fact that everything is a balance you can't spend so much time on quality that the product never ships yeah connected to that the fact that you need some measures so you can know whether or not you're doing a good job or not and to move on as we kind of joke about it all the time and then the internet so there's there's there's a lot of lot of stuff including some uh interesting little branches here in The Hacker News conversations you and I have sort of different backgrounds in this stuff so maybe you want to chime in where you think and then we can compare and contrast a little bit Yeah well I definitely agree with the idea that this definitely wasn't taught when I was taking some of my you know engineering courses this was like really really early on again this was a long time ago and then when I got back into programming it wasn't something that was necessarily shared as far as like best practices and stuff like that and QA wasn't even really something that we dug too much into it was always like the the jobs that I had were always um almost like individual facing I was always working with either a small team of people and I was you know assisting them and building things for them or uh you know creating tools for them and then when I struck out on my own and kind of started my own I guess you call it like a software business where I was building tools for small businesses I kind of developed my own methodologies for doing this and it involved constantly iterating and I I guess it would be similar to way people would think of design Sprints and things like that where I would like meet with the client and like okay give me an example of what you're doing now hand me the literally the documents and things that you're creating and then I was creating input tools and other things that were mirroring things that they were doing but I'm like okay but we can make a software tool that combines all this together and simplifies it and does a lot of the work for you and so I would always be the boss never could answer the questions so I would always go to the individuals and ask them very specifically to get you know what they're doing and and they were my testers so it was kind of a very different working process and so I don't have that large team you know integrating things or building on top of other people's software often it's it was very much kind of solo development so I don't have as much experience I've talked a lot about it on the show in fact I had danne Hillard on he was back in episode 49 we we called it the challenges of developing into a python professional and the idea as you leave school you have all these skills and you kind of understand how to program but what do you need to know to work in a professional environment and actually do the work and they talk about a lot in the sort of description you know back and forth in The Hacker News thing where yeah but that's that's software design or whatever versus computer science and that kind of weird thing that how many classes were you ever learning truly about the design of software and creating quality software and so forth and almost sort of like adjacent to that project management those are definitely some of the things that I I did go back to school for a little while and did have a program where we went into project management and I did learn a lot of I think good skills in that just kind of even thinking about how to work with a team and sort of set goals and work with these the typical types of charts and stuff as you kind of work along I'm mean intrigued by things that were coming up in the discussions besides the noise that's in there and I had a question for you which is this sort of idea of I hear a lot of people talk about test driven development and then I also hear the idea about building tests along as you go as you sort of complete things and I feel like that's not necessarily test driven development at that point like there's a kind of a very different way of thinking of it where like you have to write the test and then the software is written around it which I think is interesting and then he had a third way of describing it that he feels like was the common experience that he was having in his article of we get all the way to 90% done and oh my God we need tests now we need to do QA and and see if all this stuff works and and that seems to be maybe the in his opinion a very ineffective way of doing it because it just ends up maybe wasting a lot of time so I don't know if you want to dig into that a little bit about the idea of I feel like testing is is a big part of you know building quality software today but I'm I'm intrigued by this idea and we don't necessarily need to get into the mantras of one or the other there's a there's a hardcore component to tdd which are sort of the people who came up with it that you know very much are the no I will write the test first and I know I'm done because it's green and it's past okay the challenge with that is you have to do a certain amount of upfront design because you have to know what the function is going to what the parameters to the function are going to look like in order to be able to write the test first yeah so you end up having to either on paper or in your head think about exactly what this thing is going to do a lot of the software I find when I write it I tend to be very iterative at it right it's it's closer to writing uh English I write something and then I go back and I'm like oh no that's not right and and it's small tiny little liter a even on a function right so it it it I very selom would know exactly what the test would be before I've written the function because writing it is part of expressing that at least for me personally right and so that that's why I've tended away from the strict version of tdd but part of what he's talking about in the article we talk about this sometimes in in the agile world and I don't want to get into the whole waterfall versus agile too much but there's there's a a phrase that gets used which is something called the illusion of progress and the idea is if you are leaving your tests until too late your estimation ability and your knowledge of whether or not you're done is incorrect because I'm estimating oh how much is it going to take me to write this code rather than how much is it going to take me to write this code and the tests and QA it and if I say this is going to take me two days and then I hand that off to some project manager then that project manager is going to you know that they always go oh well double the number or whatever some magic right Gant chart filled in yeah you're not actually grabbing an estimate about the work involved here and you're kicking the can down to the the road to say the QA team or whatever so to avoid that what you want to try and do is small cycles and in those small Cycles you want the testing included as part of that and to me spiritually I feel that still manages the idea behind tdd which is don't release anything without testing it don't really release anything with without having automated tests so I tend to code a little bit and then I write some tests and I code a little bit and I write some tests and okay and I keep an eye on my coverage and I make sure that my coverage numbers are high and I feel personally like that gets me as close as I need to without having to be so rigorous about The Upfront but there are people who disagree with me on this and definitely want to come at it from you know a more of a you know think about it first and if that works for you great it's not uh I honestly can't say it's better or worse I think it's part of it's a style thing it doesn't fit as nice ly with how I personally code so it isn't it doesn't tend to be what I do and and with everything whenever you have a label like this the labels end up meaning different things to different people and then you can have these weird how many angels did take the dance on a pin kind of arguments about you know MIT tdd or not well I don't I don't care what your label is this is the approach I'm taking is it working for us or not on our team right yeah there's a couple good references in there I just want to shout them out like a handful of people mentioned the philosophy of software design which I think we've mentioned that book a couple times and I'll include a link to it I feel like some of University programs they're Geared for different things they're not necessarily yeah so gearing you for the job you know so the the school I went to had both the computer science and a computer engineering department okay when I was there I was the first group of students in the computer engineering group that had a software specialty they invented the software specialty as an option when I was there that then turned into a software engineering program so this this University currently has computer science software engineering and computer engineering as three different programs and so like even within the same school let alone from school to school you see this huge difference in what you're emphasizing where you're spending time yeah and as an example both Us in the computer engineering side and the folks on the computer science side we both took compiler courses the computer engineering compiler course we were given a partially implemented compiler and then we had to add features to it the computer science people spent a large amount of time on the math and theory behind the compilers I still don't fully grasp what lr1 look ahead stuff is I've never needed it so I've never dug back into it and I'm not saying there's not value in that but I have built and mucked with compilers and understood them without needing that right so there's just different emphasis in that there's only so many hours in the week there's only so many years in the degree and you know where where you spend your time is going to change that right yeah and this is why you sort of see in the The Hacker News conversation about oh well we covered some of this stuff or we didn't cover some of this stuff right and and like there was a a quote in there which I saw which was I took one of these classes in my Master's programs this year this year and they were totally obsessed withl and it's it kind of amuses me that they're still like that's still there right and depending on and and honestly even within the same program some of it'll be what is the Prof interested in so you may get a bit of uh you know what the Prof is doing research in and and where that is right yeah generally I found my favorite profs were the ones who had worked and were like had come from the field and back to do their phds because they tended to have an awful lot more practical perspective on things and that's kind of how I learn so the they tended to be able to teach me a little more about you know the realities of this and what it means to write the software and you know what quality looks like when I was teaching myself SQL there were a a variety of resources I used I was using uh Linda and a few other kinds of like Resources with like video courses and so forth and I really enjoyed that part of where people were talking about the design of databases and so forth because I felt like again I was working with a lot of existing built things and kind of Building Solutions around them or reworking with them and so forth so I would see okay I kind of see why you structured it like this you know and and so forth but I can kind of also started to see like you know why you want to pay attention to things like you know okay I want to design this very specifically so that I can you know extend this and and grow it out that's something we got into deep when I was talking with danne about uh extensibility you know this idea of like okay like are you building this yourself into a corner you know um and and so forth and and those are things that it may not be in your education you know and it's something that hopefully you can kind of continue to keep learning and and keep finding resources to kind of keep growing on it I think that's something that there's only so much they can kind of give you in in a school program or even in a you know set of tutorials and so forth and it's it's definitely uh it's harder to talk about because it's really you know it's kind of like you get into like sort of case studies and things like that you know well and and you know although we apply the word engineering honestly the field isn't there yet yeah like if you look at other kinds of engineering where there's a repeatability to it uh you know everyone sort of talks about the whole you know Bridges thing right he's like well you know if we we could build software the way we build Bridges well no because that's not how we build software that's not what software is for right there's a degree of rigor that can be brought to the practice but you are building something unique and original every single time and when you look at projects in the building world that are unique and individual every one of them is tied to the oh my God that project went horribly right you look up the Sydney Opera house right and and like they weren't even sure while they were doing it whether or not they could make concrete do that and as a result it was like four times over budget and you know it took three times as long as it was supposed to and every single software project is that right so it's always there's there's a mix of both the art and the science and the only way to learn how to be a better artist is practice and you know and working with other artists right so the raw materials changed so much and there's that too yeah um but this is you know there there's a quote in in the discussion which is like internships are important and I think I fully believe that right like things like a co-op program or getting practice at doing these things can make all the all the difference in the world the other thing you know I I loved the quote if you believe you can ship bug free code it's time to switch careers um yeah there's another good one about well then the manager's expectation to have a you know AI integrated thing Shi by next month you know or something like that well and you know quality is a cost right and so there's always needs to be that conversation about what do we want to spend on this and how much time do we want it right so when word crashes it's annoying when your flight Guidance Software crashes well something crashes right so like obviously you want to put a different degree of quality control into that right yeah and then the other aspect of that too which gets into it and they talk about a bit of this in the discussion is the idea of the cost of patching right so a lot of people like well you know Hardware doesn't seem to be anywhere near as faulty as software well that's because Hardware tends to be harder to patch and because it's harder to patch you can't release and then release a fix tomorrow right yeah and and in fact you've seen this over our lifetime in video games right like it was very rare to find a bug in a video game in like an Atari console because you couldn't patch it and now they're like burned onto a ROM I I've I've got the title screen done I'm going to release it um and and be and because you can just download it on the Fly and because everybody's got internet access that becomes easier which means you don't have to spend as much time and money on the QA thing or you can do it a little bit afterwards yeah and there's you know debate as to whether or not that's right but it allows a flexibility and it changes sort of people's perspective on it yeah these smaller teams with all the Early Access stuff is really fascinating I you know I'm I'm probably involved or whatever you want to call it with uh several different game sort of projects that still label themselves I kind of think of it some ways as like version numbers like you know 0 Z you know something but they 0. z0000 Z something yes yeah yeah it's like well we're just going to keep adding features as we go and bug you know you know troubleshooting and so forth but you've already paid your money theoretically so like it's always kind it's it's an intriguing development and but I understand it I mean the the idea of shipping a AAA massive game like just sounds crazy to me you know introduce and drop a trailer or something and it's like coming in two years and you're like it's I you know Grand Theft Auto I want it tomorrow and yeah probably show up 2030 that's fine yes exactly but yeah I know it was a good article and uh interesting conversation if you can get past some of the the noise as usual um on Hacker News I guess I could go first here for projects I went through a handful of projects this week and I couldn't get a lot of them running I wish people would really specify like on GitHub Hub like what they were building it on you know what OS what what what version of the Python all that kind of stuff give me some details because like sometimes I just cannot stand these things up and it's very frustrating yeah I I ran into this uh um Michael Kennedy and I were doing a thing on talk python a few months back on a bunch of different HTM X libraries that were out there and some of them weren't like even just libraries they were like sample projects and stuff like that and and I remember running into one of them it was like has to be python 3. and you must use poetry and you must be standing on your head and if you look at it Crossways it won't work and I'm like uh or I'll just move on to the next one so exactly yeah and you know a lot of the more elaborate the project is you know the the more it's like that and so it's kind of interesting you know what what they're leaning up on um but I found this one that I like it's called flask mck uh with a dash in between the two sort of the subtitle is restful apis using flask and SQL Alchemy it's a project by Danny teasing I'll just read you what it says on GitHub flas muu you don't have to worry about the crud which is great flm is a batteries included declarative framework for automatically generating restful apis with crud the create read update delete end points in a flask SQL Alchemy application stack and as little as nine lines of code the thing I really liked about the project that I'll sh