Testing Mail Service With Playwright

Hi, everyone, my name is Kat Kniotek, I work as a quality engineer at Zoopla, and thank you for joining my talk today. I will be talking about testing mail service with Playwright. I will show you how easy it is to add e-mail verification or reset password user journey to your end-to-end tests. And honestly, I really like how clever you can go here with Playwright. So let's start.

Depending on the application you are working on, you may or you may not communicate with the user by the e-mail. But if you do, you would send them newsletters, order confirmations. If you work on the e-commerce website, reset password link if the action is requested by the user, e-mail verification request and any sort of receipts or notification they may ask for. And you usually use the third party providers to handle this, may transportation for you. And those would be Amazon SES, MailChimp, Postmark and SendGrid, just to name a few. And it's a really good idea because we can rely on them to make sure that e-mail is actually delivered. But the problem statement of this talk is what gets delivered to the user. Can they click the link and reset their password? Does verification e-mail actually take them to the correct domain? If body content of the e-mail actually matches our design system, and of course, if by mistake, we don't send any sensitive data or any random data like this test as a subject of the e-mail. So today, I will show you how all those checks can be performed with help of playwright. So let's start.

So to describe the e-mail communication in a production application, you will have application communicating with third party service, and third party service will be responsible for delivering e-mails to the user or user groups. In a test environment for this demo, instead of sending requests to the third party service, we'll be sending e-mail to fake SMTP service and then from there, we will be able to access data from playwrite tests. OK, I just introduced a new thing, a fake SMTP server, what it is. You can think about it as a container that will be capturing and storing all e-mails sent by application in a test environment. And for this purpose, you can choose one of many tools available. I listed just a few here. Some of them are paid and some of them are open source. It's up to you which one you will use. For this demo, I'm using Mailhug. All of them usually come with really nice graphical user interface that you can access in your browser that looks exactly like the mailbox. You can browse e-mails, you can delete e-mails, store attachments or forward e-mails. It's great for manual verification of what gets sent to the user. That sort of fake SMTP server will also come with great documentation about their API. So with Swagger. Swagger would list all available end points on the service with methods that are available and how the payload should look like and what you expect to see as a response of the API call to the server.

Here's the screenshot of the example call to the Mailhook API. So as a search point I'm querying here. Based on their documentation, I provided query params to get all emails that has been sent to the test user at example.com and on the right-hand side, you can see how the response looks like. So good, because status is 200. And then you can see how many emails have been sent to this user, so the service endpoint returns three.

Brilliant, because I was just testing a little. And all of them are listed as email objects in the items array, so on line five, and each individual email object would have ID, from, to, content, and body. And if you look closer, below line 34, well, from line 34, you can see that body is actually HTML body, HTML code. So again, here we can do some assertions as well. We can check if the certain strings are present, if the certain elements are present, let's say button or link, but also verify if styling, right, styling has been applied, right? Just to mention, I'm using Thunder Client here. So, tools similar to the Postman or Insomnia, if you're familiar with them.

A fake SMTP server comes with some nice features allowing manual testing, but we'll use the same SMTP server in our play write tests, so let's have a look how to translate what I just showed with, let's say, API testing to play write tests. Let's start with pseudo coding, what do we want to do? So, first, our test will need to have precondition and that depends on your application. That could be the action that triggers email verification to be sent. Let's say, user creates the account or using completes the order and order confirmation gets sent and so on. Because I'm not connecting to any real application here, it was just a script that triggers sending email.

Okay, so what do we want to focus in the test? We want to do exactly what I did manually with connecting to the SMTP server. So, I want to access SMTP server, I want to get most recent email that has been sent to the user. So, in a response there was items array, so I want to get the first element of it, so with index zero. And I want to get an HTML body part, because that's where I want to do assertions on. And the great magic feature of Playwright, you can actually use this HTML body to render a new page and do the same assertions as you would do on a regular page in a browser. So, you can click verify button link, you can check if certain elements are present and you can also verify if by clicking the verify button, you will open a new tab and you can do assertion if the new tab which is also page, will have expected URL. Okay, so, that's a lot to do. So, where to start? I will start with accessing SMTP server from my end-to-end tests. And for this purpose, I will be using fixtures. What fixtures are? So, probably in your application, you have Playwright Config. Well, in your end-to-end tests, you have Playwright Config and you specify base URL value there. And that would be fronted of your application client code, the base URL of your old tests. But you can specify also fixtures, so like isolated contexts that will be used in a certain test. So, here I'm creating the fixture called email API and that one will have separate base URL value, extra HTTP headers and so on.

So, that fixtures allows you to add maybe different endpoint tests from the same repository test also backend or other services like here example, Mailhook. So, as you can see, I provide the base URL value which is running on localhost Mailhook server. I pass authorization as an extra HTTP headers and I also specify to ignore HTTP errors as true because I'm trying to connect to localhost. And this API context is defined on line 11, will be used on all calls made to email API.

Okay, and how it looks like actually when you use it. So, on line 14, here I see I can see that yeah on the left-hand side you probably don't see this 14, it's 14, line 14, I do exactly the same what I did in the Tandr client whether it was manually testing endpoint. So, on the API context that I just defined, I do use the get method to query the search endpoint for the query params the same as in the Tandr client example. So, I query all the emails that has been sent to the test user and then I return the response.

And here, this may look a little bit uh complex, what is happening here? Here on line 26, I get the first email, so items index one, content and body to get the HTML body of the most recent email that has been sent to the user. What is happening on line 26 might look a little bit confusing, so when I was showing in a tender client how the body looks like, the HTML body of the email, maybe you noticed that they were appended equals 3D strings in some places. This is because the type of encoding used for email transportation called quoted printable. So here on line 28, I just tidy up the email body from this encoding. Looks complex, but I rely on helpers libraries to do it for me. And then at the end, I just return the HTML body.

So now this HTML body can be used to render the page. So here in a email page object, I define my custom go to method. That takes as a parameter, actually the HTML code from the email. And on line 14, I instruct playwright. Every time I would be going to forward slash email URL, I want this route request to be fulfilled with the email body. So then when on line 16 actually I go to this URL, what will be rendered in the browser will be actually email body. And then I will be able to interact with it as I would do with any page. So I will be clicking verify button on line 22 and wait until new tab will be open and will return the new tab.

Putting it all together on line 15, basically that's how the test will look like, so obviously first precondition. But then as I said, I specify the separate fixture so that's how I imported it to the particular tests, email API on line 15. So, then I use the email API to get email body on line 18 for the test user, the test user I use for manual verification as well. Once I have this HTML body online, I think it's line 12, I can use this HTML body to render the email page, click the button on this email page and then assert that actually URL of this newly opened tab matches test.js summit. Okay, that may work, may not, but if I will run the test actually in a debug mode, I can see browser opening and Playwriting Inspector opening beside it, so I can step from the test and see that it actually works. So what is happening here? That's the actual first page that opens, because the API call happens in seconds, just before the browser opens. So here we are in the step that I go to forward slash email, and the email is actually rendered. And here, player attempts to click element button with ID Button 1.

And once that one is successful, the new tab is open. And as you can see, the URL is exactly the one that we expected to appear here. So when I've seen it for the first time, I was like, whoa, that's amazing. You can actually get HTML from the API call and use this HTML to render the page. Yeah, it's amazing. And going from here, you can think how many other assertions you could do on the actually rendered page.

So you can do visual regression tests. So do the screenshot comparison to make sure that nothing has changed in the version of the email that you are sending now comparing to the baseline image. Here you may say, okay, I sent to the user as a maybe title or, I don't know, a header of the email. I rendered their username or their first name or initials. And that will be different with every test run, because you dynamically generate test data. So here with Playwright API, you can exclude this area from the comparison. So let's say mask the H1 locator. And that will help you with the testing the email if the data is dynamically generated.

You can also, as I mentioned, once we have this email rendered as a page, you can do all sorts of assertions like as you would do on a regular page. So let's say that check the verify button actually has the correct attribute or if the state is changing when you hover over and so on, so on. So I hope that I showed how the playwright can be used for testing email body and what actually gets sent to the user. So I checked that email got delivered to the user. I checked functionality of the button so you could see that action click has been successful, that the navigation happened to the correct URL. With Playwright you are also able to do visual check on the email, on the content body and obviously all other assertion on DOM elements as you would do.

OK, that's end of the talk. I hope that you found it useful and the links that I just posted here will help you to maybe try and build a similar solution and apply it to your tests. As I mentioned, I was using Mailhook so I provided a link to Mailhook Swagger. Mailhook docker image with OAuth so you can just run it on locally. More explanation about equal 3D, so the quoted printable encoding. A few words about Tandr client, so my go-to tool for manual API tests and link to contact with me if you would have any questions. So, yeah, I just look forward to hear your questions now. If you want to access the presentation, you can just scan the QR code and that will take you to the slides.