Playwright Can Do This?

Hello, TestJS Summit. It's time for another Playwright session. Playwright can do this.

And before we get started, let me quickly introduce myself. I'm Stefan from Berlin, Germany, even though I'm in Greece right now. I do front-end development, JavaScript development, for quite a little bit. And I work for a company that is called checkly, and we do API and end-to-end monitoring in the cloud. So if you want to maybe control and run Playwright on a schedule, in the cloud, to make sure that all your products are up and running at all times, you can have a look at checkly. And I'm sure you will find that interesting.

And let me tell you that I started testing over 10 years ago, and these were the technologies, the technology choices that I started playing around with. So my first UI and end-to-end test were written in Selenium. And then at some point, Phantom JS showed up, this new fancy JavaScript-driven framework or library to control headless WebKit. And then later on, there was Casper JS. So we had a thing with ghosts at the time, which was a testing framework on top of Phantom JS. And let me tell you that from the beginning, from the first possibilities to control a browser to test that the stuff that I actually put online works, I was a big fan of this approach because the idea of end-to-end testing was always appealing to me. Because you can test your websites and applications really from end-to-end, starting at the browser and maybe ending at the database and testing and covering all the things in between. Let it be APIs, web servers, or whatever you're building.

When I wrote my first end-to-end test, though, it was a terrible experience. I spent, for example, sprint over sprint over sprint with my colleagues and my team to come up with a good end-to-end test coverage, only to create a test suite that was slow, we didn't enjoy writing tests, and was flaky. It was the absolute worst-case scenario to end up with a slow test suite so that you have to wait 30, 40, 50 minutes to get a green light to maybe deploy a typo fix, but then also get false positives results so that you re-run your end-to-end tests to make sure that well, maybe the tests just were not right, and then they paced later on, you probably have been there. This is just the worst-case scenario that when you cannot rely on your test suite, and it is also slow. So we were there, we invested a lot of time, and at some point we decided that it's not worth it, and we went for, hey, let's run the tests on-demand, right? And maybe you have been there, too, but this is the moment when you pretty much give up on end-to-end testing, because what you want to do is you want to have your tests running all the time, on commits, on deploys, you want them to be on your radar at all times. And when you run them on-demand, well, that is probably not often enough, so that you will end up with an outdated test suite very, very quickly, because you forgot about it, and it's kind of negating all the effort that you put into place. And this is exactly what happened to me.

Luckily, things got a lot better. Puppeteer was the first step for official browser automation, backed by Google for Chrome, and then Cypress a few years ago appeared with solid developer experience, and you might have guessed it, for a long time now, I'm playing around with Playwright, and I think it's just a very, very decent solution for end-to-end testing.

So, let me show you what Playwright is about. In case you haven't seen Debbie's talk, so first of all, Debbie from the Playwright team talked about Playwright just a few moments ago. Playwright is a very inclusive testing solution. So, it includes all major browsers, covering Chromium, WebKit, and Gecko Firefox. It runs on Mac, Windows, and Linux, and you can write tests primarily in JavaScript and TypeScript. If .NET, Python, and Java are your kind of thing, you can do that, too. So, from an inclusivity standpoint, Playwright is top-notch. Additionally, if VS Code is your editor of choice, the VS Code Playwright extension plays very nicely to create Playwright tests, debug them, generate them, run them. You can do all this end-to-end functionality and testing functionality right from your editor. If this is your jam, you can use VS Code, so that is very nice.

And it took me a while to realize that Playwright is actually more than a browser control. Playwright Test, these days, is a full-fledged testing framework that comes with the usual describe functionality, before each, after each hooks, fixtures, test configuration. All these goodies are included in Playwright itself. So when you want to really go all-in on end-to-end testing, Playwright is a valuable solution here. It's easy to paralyze. So remember my story of running tests and waiting for 30, 40 minutes? With Playwright, you can paralyze all your tests with a single CLI argument or put it in your configuration to speed up what happens and what is going on. And by far, my favorite is that Playwright is built for quick execution with features such as auto-waiting and web-first assertions, because UI testing is usually that, well, you wait for something, you do something with it and then you wait for the next thing to appear to do something with it then. And usually this means that I had to put arbitrary delays here and there or manually wait for elements to appear or disappear. Playwright has all of this baked in and this gets me very excited. Debbie briefly touched on this, but I want to show you how this works in this session too. So, shall we have a look? Let's have a look at Playwright and run some tests. So, we go to VS Code and what you see here is on the right side my terminal. And on the left side, we have test.js spec.js file, which is a standard test file. And what we will test is this lovely website here, which is on localhost 8080. And it's the test.js summit site and it's pretty much the same side, but now it includes this huge celebrate button, because I think more websites should have confetti in them. Other than that, when we look at the project configuration of the project setup here, you will find a standard playwright config here and other than that, there's a test folder including test.js spec.js and this little facebomb.js is just my notes here in case you're wondering that. We're not using the VSCode extension here, so to run this project and run the tests that are included in it, what you can do is npx playwright test and you see already that there are 28 discovered tests using five workers and we see already parallelization going on here, but we're going to have a look at what happens here right now. We see that there are seven past tests and 21 skipped and when we look at the test file, we see that there's one official or enable test and the other three are skipped, but why is everything multiplied by seven? So when we go to the playwright config, you will see that right now there are various and multiple browsers configured to be run when we run playwright tests.

So there's Chromium, Firefox, WebCorp, Mobile Chrome, Mobile Safari, Microsoft Edge, Google Chrome. So this shows that you can run all of these browsers with a single command and to show you that this actually works, what we can do is we can pass the headed argument. And now we just see a bunch of windows popping up, emulating a specific browser and testing that all things are going well and are working properly. And I think this is already pretty, pretty cool to see all these windows popping up. But for now, I think we can just disable most of the browsers so that we don't have to wait for them going forward.

So what else do we have? Debbie showed in her talk that you can use the Vscode extension to generate tests. You can do the same thing from the command line pretty much too. So what I did here is I ran npx playwright codegen with a URL. And now what happened here is that the Playwright Inspector opened up. And now whatever we do in this little window here, let's click this year, let's do this and do this. You see that it's recording all the actions. I could now go here, copy and paste this, add some assertions and get a quick and a headstart using codegen if you want to do this. But let's have a look at the tests and start messing around with some tests.

So what we have here is first of all, a test that checks for the current page to have the correct title. And you see that there is a before each block that already goes to localhost. It checks the title, it locates the h1 element and then it does a string assertion that the headline has the correct test. So why not break this assertion here? Let's have a look at what happens.

So we can run this test and this will now fail because while tests, it will not have three Ss here and we'll retry a few times here. But what we now see is that we have a failed test. So how could we debug this? Even on the command line, playwright also comes with a debug mode. So what you see here is this browser session and again the player inspector that allows us to step through all the instructions of the actual tests here and to debug and see what is going on. And I think that's just a nice handy addition if you don't want to use the VS code extension.

So moving forward, what I want to show you is that first of all, playwright has the normal or the common screenshot functionality baked in. So when you're running a headless browser, very often you're wondering, Hey, what's the actual state of the website? So you can do the same thing. You can run a whitepage screenshot to take a screenshot of the current page and we now just generated a new PNG here, which includes the current status of this website. What is cool though, is that playwright has screenshot component, virtual regression testing baked in too. So what we also can do is what we have here is we have a locator for a headline. So let's do wait, expect. Now we can take the headline and we can say to have screenshot and let's give it the screenshot headline PNG. And when we, what happens here is that we are tying a, an assertion to a locator. We're going to screenshot this particular element that is matching this locator. And then for following runs, we're comparing the screenshots. So this is visual regression testing with one line of code. So when I now run this initially, it will fail because there is no snapshot, snapshot generated yet. But when we have a look at the result of this first failure, there is now a new directory that is testjess.spec.js.snapshots. And you see now here, this new snapshot that we just created. Going forward, when we now run the test again, it will always pass because Playwright will take a screenshot. We'll see if it's matching what we expect and moves on. So if we now adjust and mess around a little bit with this screenshot here, which is the baseline for comparison, and we run the test again, we will fail because we just messed with the comparison of the visual regression. And when we look into the test results folder here, now you see that this was the image that we expected with a lot of scratchy lines on it. This was the image that was generated in this current test run. And here we have the visual regression showing us, hey, something changed in your component that you have in your Playwright tests. And I'm still amazed that with a single line of tests instructions, you can have component-based visual regression testing.

To overwrite what we just had, what we can do is we can do update snapshots. And now we're back to the proper screenshot, which we will have here without scratch lines so that all our tests are passing. So this is good.

But let's move on and let's test that this very important confetti functionality is working. So what you see here is already that I have the button, that is the party confetti button. And I am locating all the canvas elements on this page. So when we look at this and we reload the page, you will see that there's actually quite a little bit of a delay for the button to appear. So what happens is that, first of all, we have to wait for this button to show up, then we have to click it, then we have to wait for the canvas to render all the confetti. And after the confetti is done, also, the canvas will be automatically removed. And this includes a lot of waiting.

If we would like to test this, but the nice thing about this is that playwright has all the waiting functionality built in. So what we can do is we can leverage the web-first assertions that come with Expect and we can say that, initially, we expect that there is no canvas element. Then we want to click the button and then we expect that there is one canvas element and that there is no canvas element anymore. And you see here that all of these instructions are asynchronous. We have to wait them. This looks like sequential code, even though there's a lot of waiting included here. So this button will only show up after a couple of seconds. Then the canvas element will immediately show up, but it will only disappear after a couple of seconds. And in PlayWrite, all of this is baked into the instruction itself. And I think this is just a wonderful way to streamline end-to-end testing code without putting waitforce and artificial delays everywhere. And you see here that we now have two past tests and we can break this by doing this, but I think that's just a beautiful and nice way. So we can move on.

And I want to show you something that I think is very, very cool too. So let's just fix the test again because it was broken now. So what you always can do in your test cases is you can always mess around with the network layer or with internals of the loading of a particular web page. So what you see here is that this test case get the page object in this test run. And you see here that we can listen for all requests and responses going in and out. So if we, for example, want to check the quality of a website and evaluate that there are no 404s or no 500s in API calls, what we can do is we can do a very quick status check. If there's a status response that is greater equal 400, we want to collect the URL and push it into an array that is not there yet. And let's create the array very quickly, const not found and do this.

And we have to push this, here we go. And now we have to reload the page because it was already loaded. And then we add a quick assertion, expect not found length to be zero. And this is all it takes to make sure that there are no 404 requests on your page.

So we now run this test case again. Let's see what happens here. Our confetti test is slowing things down now a little bit, but it looks like we don't have any 404 resources or 500 in our little local house websites.

You can also do further stuff. You can start blocking requests. For example, if you want to speed up your end-to-end tests by blocking fonts or blocking tracking or blocking third party requests, because they are not necessarily needed in your end-to-end test suite at this moment in time, you can do that too with all the internals that PlayWrite provides you with the page on object.

And then lastly, I want to show you how you can leverage native browser APIs to make sure, for example, that your website is fast enough. So what you see here is that I use the page evaluate function, which is the way to implement your own little data aggregators inside of this particular website. So PlayWrite already provides a lot of functionality. But if there is something that you're missing and you want to hook into the context of the particular website, you can always use page evaluate and give it a function that runs in the context of this particular web test case in this website.

So what you see here is that I'm using page evaluate, and I'm instructing a new performance observer to figure out the metric largest contentful paint, which is then passed back to the testing scope. And what we now can do is we can do expect and I'm casting LCP to be a number because right now it's a string to be less than, I don't know, let's take 800 milliseconds here. Let's run this. Let's see if it's fast enough on my local machine here. Um, this looks cache. Let's see. Here we go. Cool. And these were just a few things that I wanted to show you how we can get started with playwright, running it on the command line. If the VS code extension is not your thing. So let's finish this up.

So really playwright comes with so much functionality, includes auto weighting, reference assertion screenshots, the test runner is nice. Retrying, tracing, VS code extension, inspector debugger coach, an image snapshots. There's really a bunch of functionality in there and the team is releasing a lot. So watch out. There's a lot of things, a good things coming.

But we at Checkli re-run playwright every few minutes for our customers, and I just want to share some common best practices. So first of all, what we've seen is that short and idempotent scripts usually work better for the testing and the monitoring use case. So instead of having and testing your entire user account flow, what you usually want to do is you want to split that in different tests to make that easily to parallelize. So what we often recommend all our customers is that they should have a create a login and update and a delete all for their account properties or resources so that all these kinds of things can run in parallel. And this then also means that if one fails and you really know what failed, what's going on, and you get a nice little message, Hey, this particular test case failed, and it's also very important that these kind of scripts are idempotent. What does that mean? I think it's a very fancy word. But this basically means that your test cases clean up after themselves so that you can run them repeatedly, which is very important for the monitoring use case. So whenever you create something, your test case should clean up after itself so that you're not cluttering a database, and these kind of test cases should run at all times.