Exploring Python T-Strings: Formatting Limitations & Introducing T-Strings

Key Takeaways

Welcome to exploring Python T-strings. My name is Christopher and I will be your guide. This course is about T-strings, a feature added in Python 314. They have an fstring-l like syntax, but instead of returning a processed string, they return a template object. The template object contains all the processed parts of the string, breaking it down into the sequence of strings and value parts known as fields. The fields themselves get represented by an interpolation object. These contain the name of the variable being interpreted in the field, its results, and more. In addition to learning about the different parts of a T-string and how they work, you'll also see several examples of how you can write your own T-string handlers and why you might want to do that. This course was tested with the beta 3 version of Python 314. T-strings were added in Python 314. So, you must use at least this version for the code to work. As I just mentioned, T-strings got added in Python 314. Although they've been talked about for quite some time. Back when Fstrings were first proposed, there was a companion idea called an Iring that got punted until later. Eventually, these evolved into T-strings. An F string gets evaluated by the interpreter directly and returns a processed string. Like with an F string, a T-string is denoted by a prefix, but in this case, it's a T instead of an F. The inside part of a T-string is the same as an F string. The result, though, is different. Instead of being turned into a processed string, a T-string gets turned into a template object. Frings do everything for you, but sometimes you might want to be able to interfere with some of the processing steps. A T-string gives you that kind of control. The two most common uses for T-strings are escaping user input before putting it into a result, like with SQL or HTML, where the user could mess up your output, and when you want to interpret a template in multiple ways, like when you want to log both in human readable and machine readable formats like JSON. Next up, a quick review of the different string formatting options in Python and their limitations. In the previous lesson, I gave an overview of the course. In this lesson, I'll be reviewing the various options for string formatting in Python and pointing out their limitations. PEP 20 is the Zen of Python. It's a poem thing that describes the philosophy of good Python code and Python itself. One of the lines is there should be one and preferably only one obvious way to do it. Even Python, the language itself, sometimes struggles to follow the zen of Python. There are currently three ways of formatting strings in your favorite language and this is due to the evolution of the language over time. The original mechanism is C style formatting where a string uses a percent placeholder to indicate where something could be replaced. So percent s for a string and percent f for a float. You then operate on this kind of string with another percent sign handing it a tpple containing a value to insert against each of those markers. Python 2.6 six added the string format method which instead of using percent as a placeholder uses curly brackets. You call format on the string itself kind of like you can do with upper, lower and other string methods. And the arguments you provide to format are what get used to help populate the string. And more recently frings got added in Python 36. This kind of string uses an F as a prefix before the string to indicate its type. like format. It also uses brace brackets, but instead of being placeholders, it takes variable names directly. Anything in scope can be used in the brace brackets themselves. Let's dip into the ripple to see these in practice. First, I need some variables to play with. And let's start out with the C style formatter here. The percent s and percent f in the string are the placeholders to be replaced. The s and f indicate the kind of value conversion to take place corresponding to string and floating point. After the format string, you use the percent sign to invoke the replacement process, handing it a tpple. The tpple needs to have the same number of things in it as the placeholders in the string. The end result is a string with the placeholders replaced. You really shouldn't use floats for money. They're problematic when it comes to rounding. But ignoring that, what if I want 1099 to only show two decimal places? Well, you can add information to the percent f place placeholder to specify the format of the result. Putting the 2 between the percent and f says to format the float to two decimal places, giving us 1099. a little more reasonably without those trailing zeros. One of the drawbacks of the C style method is if your string uses a value more than once, you have to repeat the corresponding variable in the tubble. Python 26 added the string format method to address this and in an attempt to make string formatting more readable. Instead of using a percent placeholder, the format method uses brace brackets. The number in the brackets indicates the position of the argument to use for replacement. I haven't done it here, but I could put brace zero several times in the string, and each instance gets replaced by the zero argument, our item variable. Similar to the C style, you can add format information. You can also use named values instead of a count for the arguments. To use a format specifier here, you add a colon inside the braces with the kind of formatting coming after it. In this case, I've used 0f to round 1099 to no decimal places. Quite honestly, I've never liked the format method. Part of that's because I was super used to C style, having used it in several other programming languages. Part of it was because it required more typing and the situation of argument reuse isn't all that common. And last, it also isn't as performant as the C style. That last reason is rationalization. It's true, but if performance were my primary concern, I wouldn't have been writing Python in the first place. Python 36 saw the addition of a third way to format strings, the fstring. This one I like, and I switched to it quickly in my own code. You indicate an F string by prefixing the string with the letter F, hence its name. Inside, like with format, you also use brace brackets. But unlike the format method, that's all you need. The Python grammar refers to the items in braces as fields. Any value that is in scope can be used in the field and gets replaced. A field can even handle function calls. fields also support the same format specifier as the previous method. So once again, I can do0 to round my price. There are still a few things you can't do easily with any of these three methods. In the next lesson, I'll talk about their limitations as a prequel to why T-strings got created. In the previous lesson, I reviewed the different ways of formatting strings in Python. In this lesson, I'll quickly run you through two limitations of the format mechanisms so you'll better understand why T-strings got created. An fstring gets processed in line by the interpreter. That means that any values used inside of it have to be in scope and usable. And in the case of using an fstring as an argument to a function call, the fstring gets processed and turned into a string before being passed into the function. Sometimes this is the opposite of the behavior you want. For example, doing the string processing might be expensive and you might want to delay its occurrence for performance reasons. And sometimes you want to be able to see what is going on inside of the string before it gets processed. This second case is particularly important when dealing with user input. Users can be unhelpful and sometimes malicious and they might give your program something it can't handle. If you put user input inside of an fstring without cleaning it first, you could run into trouble. Let's start by drilling down on when an gets processed known as eager evaluation. I'm creating a little logging function that prints some content to the screen, but only if it's turned on. If enabled is true, then print the message. Quick little test. There you go. To play with an F string, I'll create a variable. And when I log it, it gets printed out. The problem I've been talking about is this function call. Note that the fstring is getting processed before the function gets called. That means the string is being interpolated and 42 is getting embedded into it before it's getting passed into my log. This is the eager part. It evaluates it as soon as it can. In this situation with enabled false, that's a waste of time. It would be ideal if it didn't work this way as you're paying the cost of string construction even though you aren't using it. To get around this, it would be nice to move the string processing into the log function and only calling it if enabled is true to save on performance. In fact, that's why Python's logger still uses C style formatting. To demonstrate, let me set the logger up. Importing. I want to print out the results. So I need to configure the logger with standard out which is in the CIS module. The basic config call is probably the quickest way to set up a logger. Here I'm using standard out as an output stream so the contents will get printed to the screen and I'm setting the logging level to error. Any log messages below the error level don't get printed. This is a pence ear version of my enabled equals true from above. Now I'll instantiate the logger and finally I'll use it. The login calls error in this case take one or more arguments with the first argument being a string message and the rest being used as part of a format call. Note that although this is a C style string, it isn't using the percent operator to populate the template immediately. That's because the logger itself is making that call for you inside of the function call. The advantage is it only does this if the log level is sufficiently high. The output's a little muddled as the default logger shows the log level, which is the all caps error, the source of the log, which is root, and then finally the populated message. The debug method is a lower level than error. This is like enabled equals false in my earlier example. Since the string interpolation happens inside the logger and the first thing the logger does is check the error level. The string interpolation never gets called. You just can't do this with an string. Now, the example here is quite simple, but imagine if you're making loads and loads of login calls a second, or if the things being logged are large, complex objects that are expensive to stringify. If you use an string, you pay that cost, even if you aren't going to log anything. Spoiler alert, T-strings don't quite solve this problem. They kind of walk up next to it and smile at it, but they don't quite get all the way there. More on that in a later lesson. Another potential problem with fstrings being eagerly evaluated is when you need to do some processing beforehand. One example of this situation is what's known as a SQL injection attack. On the screen here, I've got a select statement that returns any row in the movie table of my database where the title is Batman. If I want to accept user input for the title, a naive way of doing that would be to construct this same thing with an string. Makes sense, right? The result of this string being interpolated when title is Batman is the same as the select statement above. We're good. Right. Right. Not right. What if user the big meanie gives you this as the title? Semicolon is used in SQL to separate statements. When this title gets interpolated, you have a select where title is empty and a drop table statement. Generally speaking, you don't want your users to be able to mangle your SQL in a way that is damaging to your database. It saddens me that SQL injection is still a thing. It's quite preventable if you know what you're doing. Just so I'm clear, in case the giant red X here was too subtle, never, never, ever do it this way. Most databases have a way to parameterize your calls. The syntax varies a bit, but the idea is kind of like the C style formatting. You put a placeholder in the string. In this case, it's a question mark. And then pass parameters to your SQL execution, letting the library create the corresponding SQL statement before running it. If mini user puts in drop tables, that will get treated as a string being looked up rather than as part of the SQL statement. The good news is, at least if you're writing Python, it's harder and harder to make this mistake accidentally. The reason this code is on a slide rather than me demonstrating it is both SQLite and SQL Alchemy libraries now have checks for this. If you attempt to use two different SQL statements in the same call, it errors out. That said, this problem exists anywhere where user input gets used. They can muck with your results. SQL, HTML, reg X's, all sorts of places. You have to clean your data first. But it's kind of easy to forget the cleaning data step. So, Python introduced a new kind of string formatting that allows a developer to introspect on what's being processed. Next up, T-strings. In the previous lesson, I showed you some limitations of fstrings. In this lesson, I'll introduce you to one possible answer to these limitations. T-strings. A T-string is like an fstring, but instead of becoming strings, they become a template object. This feature got added in Python 314. As its name implies, you specify a T-string with a T prefix instead of an F prefix, but the inside of the string itself uses the same syntax as fstrings. For safety reasons, you can't directly convert a template object into a string. Well, you can, but it doesn't interpolate its contents. It just shows you the same kind of object debug info you'd get in the ripple. The advantage of a T-string is what gets put inside the resulting template object. Python processes a T-string like an Fstring, but populates the template object with the parts, allowing you to do something with the result before rendering it. Let's head to the ripple to see how they work. Let me create some variables again. That's a bit much to take in all at once, isn't it? The output here is the contents of a template object. It has two key components. A tpple of the string parts of our T-string and a tpple of interpolations. An interpolation is an object in of itself containing information about the fields in the T-string. In this case, I get two interpolations, one for item and the other for price. Let's do this again, but break it down into pieces this time. I'm going to store it away first. Now let's look at the insides. The strings tpple in the template object has three things in it. The string before the field containing item, the string between the two fields and the string after the price field which in this case is empty. Now let's look at the interpolation objects. The first interpolation is for our item field. The interpolation object has four arguments. The value of the field, which in this case is the string shirt. The contents of the field, which is item, the variable I'm using, and then two more bits that I'll talk about in a future lesson. This is our second interpolation. Note that the value is of the resulting type. So for the price field, it's a float rather than a string. When Python processes a T-string, it returns a template object and that object has attributes. That means you can access those attributes directly on the T-string itself. This is the same T-string as above, but I'm directly accessing the strings tuple. The template object also has a convenience attribute. The values attribute is a tpple containing each of the values from the interpolation objects. If all you need are the results, you can use this to access them instead of going through the embedded interpolations tpple. And of course, you can also access the interpolations attribute directly as well. Before wrapping up, I just want to dig into one little thing just a bit deeper. Remember how with the original template there were three things in the strings tuple? Even though the price field was at the end of the string, this is the same but twice. When the t-string contains nothing but a field, you still get two empty strings. One to indicate that there's nothing before the field and a second to indicate that there's nothing after the field. The empty strings act as delimiters. A similar thing happens when you put two fields side by side. You get a blank string between them. In the next lesson, I'll investigate interpolation objects a little more.