Advanced Playwright Debugging and Test Resilience | DEM499

Key Takeaways

Thanks Katie. Hi everyone. How are you all doing today? You doing good? So my name is Debbie O'Brien. My name is Max Schmidt. Hello everyone. And we're here to talk a little bit about um debugging your tests. I know y'all hate testing and you all like have to debug, but we're going to show you how easy it is to debug your tests. And then I'm going to have a little cool thing at the end where I'm going to use MCP to actually show you test generation. So keep that in mind. The last couple of minutes are going to be uh on the edge, living on the edge. So, Max, debugging? Do you like debugging? Do you all like debugging? You do? Oh, wow. Really? Max, do you like debugging? I I love debugging. Spending hours debugging my local test, debugging them remotely. It's spending hours. That's the problem. We spend hours debugging. Okay. You know, Max is going to show you a couple of tricks to take debugging to new levels so you don't have to spend hours. All right, Max. Off you go. Show them how to debug the rubber duck. Oh. Oh, let me let let me maybe jump straight into the UI mode and take a look. So, maybe you all know UI mode. UI mode is playright's uh one of playright's features to run tests locally and uh it works on any IDE. It's structured like a IDE. So, on the left side you see all the tests and then on the right side you see uh how the test was looking like during the execution. So in this case I have this test. It's called a pagionation spec. And I was working on this feature but this test started failing. So let's take a look at this test and try to figure out why it's failing. I hate when tests fail. It just like a I just want it to be green all the time. But that's not how it works all the time, right? They be like in this case there are like 20 things we try to expect but there are only three. And I absolutely hate reading through the error messages when it comes to debugging. This is really helpful, but sometimes it's like, "Oh my god, I've got to try and figure all this out. Isn't there a better way?" Exactly, there is. But let's maybe spend a few seconds at UI mode. So on the right side, this is the normal way of debugging, right? You could now take a look at the DOM on the right side. How did the website look like? There are snapshots. There are console logs. But as you said, it's much easier. So let's just do something which we just added recently which uses AI to solve the tests and to fix them. So let's just copy prompt here and open our co-pilot here. And this is just like uh using any AI model. In this case, we use GBT4.1. Okay, just before you press play. Oh my god, it's too fast. Too fast. Um what did you just paste in there? What I pasted here is we call the playright prompt. It has all the interesting and relevant information in it. First it starts like with some instructions about playride then error details and then we go on and on how the page look like what we call the page snapshot. So this is essentially like the accessibility snapshot of the page and then later on we add uh the test source and the git diff. So the git diff is really interesting when you have like normally the tests in the same repository as your code. So you were working on your app but then some test started failing and then AI is trying to figure out uh which change caused it. And in this case if we scroll down you will see why the test failed and it said uh the test expected 20 movie list items after navigating to page two. We knew that before but however due to a local change in this file uh when we are on page two there is it's intentionally sliced to only three for debugging. So this is much easier to read, right? Much better than the error messages. This is like plain English. I can read this. I can get this. And it's giving us a suggestion of how we can fix it essentially. Yes. So let's try to fix it. So in this case, we need to go to this file, the movie list index.js. So let's go back to VS Code. I have it here open and remove it. Go back to UI mode. And ideally, if we rerun it again, it's all passing. So the alternative to this, right, would be simple. We could go and check all the local files. In this example, it might be really easy because there is only one file changed. But imagine you have 20 files changed, 30 files changed, and there's more complexity to it. So now it's passing. I think the key here is that the copy prompt button is giving the context to the AI. So it has everything it needs to make a better decision on how to debug your tests rather than guessing or trying to figure it out. It's got the context and that's all just built in for you. Yes. The combination out of git divs and page snapshot make this possible basically. But this was your iMode. Not everyone uses UI mode to run the test. For example, Debbie, I think she loves VS Code, right? Yeah, I'm I live in VS Code. So if I'm in VS Code and I'm just debugging, what's what's my solution there? In VS Code, we have something similar. So let's go into VS Code. And there I have this test. It's called search specs. And when I run this test, it's failing here. And uh let me rerun it again. and then see why it's failing and maybe we can figure out uh why it's failing. So in this case we as code we have the playwright extension available. It's essentially the same like your mode. So you see all the tests on your left side. You see the test thresholds and so on and you see errors. In this case it said test timeout exceeded. I could read now over this test but test timeout. I'm going to add a set timeout or something like that to fix it and then it's probably going to fail again later and it's that's not the solution here. Let's do the same. We have here this sparkling icon. We can just press it and what this does it uses the same context as before the git div the test source the accessibility tree of the page and ideally it figures out what the problem is. So here it said for example the test is failing because it's trying to click on avatar link. it doesn't exist but uh we are trying to search for twisters and it even like is integrated well into VS code. We only need to press accept here and we can rerun the test and then it's passing. So there's no context switching between like copying the prompt into your favorite large language model. You can only uh you can just do it in VS Code basically. In this case uh it's failed here but I think that's uh because of this uh slow Wi-Fi connection. So what do you think? Do you think this makes debugging better? Yeah. Okay. I'm excited for you all to give it a try and just go through your tests and debug it. Now, we have a good few minutes left. So, I'm going to move on. Now, actually, before we move on, this is also available. Yes. So, this is not only available in your IM mode, in VS Code for local debugging. This is also available remotely. So, usually you write your tests locally, right? You rerun, you run them inside GitHub actions. And then in GitHub actions you have your HTML reporter your trace viewer and there we have the same available there. You can copy the prompt paste it into your f favorite large language model end. So when it fails on CI you have that same context that you can just copy that prompt put it into AI and then basically turn around to your developer and say hey look this is failed on CI and this is how to fix it. Go fix it or you can even fix it yourself. That's which is pretty cool. So yeah definitely test that out. Test it out literally. Okay, we're going to move on to something really cool. This is a bit um we're gonna I'm going to try and do a live demo of how to generate a test. Now, when we talk about test generation, there are many ways. We can just go into C-pilot and we can say generate a test and copilot if it has good access to your code base. You can probably do a pretty good job. It's sometimes okay. But what about when you don't have access to the code? What if you're a test um engineer? You don't have the source code and you got to test the website. You can use our tools such as codegen and you can manually go in, click through everything and it will generate the test code for you. But what if AI had superpowers and could just do all this for you? And that's what we're going to show you with the Playright MCP. And we've had a lot of talk about MCPS today. So you should all know what it is. It's the model context protocol. What the Playright one does is it gives you access to the browser. Right? Playright is browser automation. And now we're basically giving Copilot access to the browser. It can now use the browser like a user would. And that's the cool thing. Now, uh, in order to make this work, you need to install the playright MCP. I've got it running locally. You could have it globally set up. I'm just got a local one here. And you can see I've got this running. What's really important is prompting. Letting know. Letting the um, AI know what you want is super important. And being very clear and concise. I'm basically telling it you're a playright test generator. I'm saying like you're given a scenario and um I can pull this across a little bit so you can see it a bit more. Uh I I want you to generate a test. Do not generate the test based on the scenario. I want you to run the steps one by one using the tools provided by the playright MCP and only after all steps are completed emit a playright typescript test and then save it and then run the test. That's what I want it to do. So um what I'm going to do is I have just a little couple of instructions. I'm just going to copy these here and I'm going to paste this in. Um, I always put my prompt in here into the co-pilot chat so has access to it. That's a reusable prompt and you can share that across your code bases and reuse it again and again. I'm using agent mode as you can see. I'm just using this model here, but you can use whatever model works better for you. Now, I'm going to press this my what I want it to do. I want it to go and generate a playright test of following scenario. I want to navigate to this URL. I want to search for the Garfield movie and verify the movie is in the list. Now, this is AI. Okay, this is a live demo. Let's see if it works. Let's see if it's going to do what it's meant to do. I'm going to press play. My hands are up and I'm going to hope for the best. Demo gods, come on. Come on. You can do this. So, it's saying first, let's navigate to the website. It's using the Playright MCP. It's navigated like my hands are up. I'm not doing anything. It's opened up a browser window. Okay. It's now having a problem because it's saying it can't type in the text. It sees the issue. So, now it's gone and run a page snapshot. It's taken a page snapshot of the accessible elements of the page. So, it knows what it can do. And the actual search icon has to be clicked before it can be typed into. And now it's seen that. So, it's gone ahead and clicked it. And it's typed in. It's put in Garfield. My hands are still up. Max, your hands up. Yeah, good. And uh it's uh basically found that movie. This is perfect. Right now, it's got all the information it needs. It's done all the movements. It's done everything on the page. So, it knows now how to write a better test. So, it's basically creating the test file. I'll just click here and show you. That's the test file that it's just created. And now it's going to go and run the test to make sure it passed. And there's the browser popping up. That's the playright running the test for us. Opening up a browser because we have in VS Code the show browser um clicked there. And I can see my Garfield movie has been selected and the test has been created for me. Now I'll just close that there and I can press keep. And I could literally continue further on this, right? I could install the GitHub MCP and then say create a pull request with what you've just created with that test and actually just push it to GitHub without even doing anything else. So this is the power of MCP. I'll run that test so you can see that it passes. And um it's pretty impressive and pretty cool what it can do. This is just a small short scenario, but you have to imagine in your use case what it can do for you. Very important as prompting. So, so playright MCP is like just the building block, right? There's playright MCP which allows you to access all the browser and the context and like the accessibility tree. Then there's we code which has access to the files. It can write files. It can run tests. There's GitHub MCP which could pull all the instructions and the feature requests from your issue list and uh VS code combines all of this and with just a single button click it creates such a test case. It's not hallucinating because it has all the context available. So it's very cool. What do you think? Do you think it's cool? Yeah. Excellent. What time have we got left? Three minutes. Three minutes. Excellent. So basically um I want to just point out what's very important when you're using this and when you're generating your tests is prompting. You need to tell the MCP, you need to tell the co-pilot or whoever you're using. Um you need to tell it it clearly how you want it to test. If not, it will just use the code base because that's what it's used to doing. So you've got to be very explicit in what you want to do. If you we're all going to become prompt engineers. That's our new job. So if you learn how to prompt good, you can basically then put that into the context and say this is the prompt, but it's a reusable prompt in VS Code like I showed you just here, which means inside your github.prompts folder, that's my generate test prompt that I'm going to reuse every time I want to create a test. So I only have to create it once. And you could put in whatever you want. Maybe you're using C#, not TypeScript. That's fine. You can just tell it that. You tell it what you want. And then you just go ahead. You could even actually use the the GitHub MCP to put all this in a in a GitHub issue and then just say read from the G GitHub issue, create the test and just go ahead and do it. And that's the power. It's cool. So any custom style guides, reusable functions, uh reusable fixtures you could put into the context and then it would pick them up. Exactly. So it's very very exciting. This is like early days and it's so cool what it can do. It's not just about writing tests. Playright MCP is about browser automation. So, you can use it for other things as well as just writing tests. You can use it just to navigate the browser. I actually used it to buy a table and chairs for my kids. And I just went along and said, I want to find the cheapest version of a table and chair and find me the shipping details. It filled in all the information for me that I didn't have to bother doing so it could find me the shipping price and then tell me the exact cost of it. So, it's really exciting what it can do. Um, do test it out. I really want to thank you for watching and I really hope you do. Um, test out the MCP. uh debug your tests and just have fun testing. If you do have any questions, we will be in the expert areas throughout the day. We also have a lab on playright. Do come along and join us and don't be afraid to come and chat to us. Thank you very much and enjoy the rest of build. [Music]