Don’t Make These Testing Mistakes

Hey, this is Gleb Bakhmutov from Cypress.io. Thank you for inviting me. I want to talk about common mistakes people make when writing Cypress tests. And first, I want to remind that we are still in climate crisis. Despite COVID slowdown, we have to act and act now. You can change your life or better join an organization that fights together. You can join multiple organizations. I recommend both here.

In this presentation, I'll cover common mistakes in Cypress tests. And then I'll talk about something we're trying to do to minimize the number of mistakes by improving our documentation. I'll finish with a discussion of our github repository. You can find the slides online and if you have questions, please find me on Twitter.

Let's start with Cypress mistakes. So when people start writing Cypress tests, we forget that Cypress tests run in the browser. I know it's a simple fact, but it's easy to write something like this, require file system module and then try to read the file. Well, this will never work because the Cypress test executes in the browser. You would not be able to execute this code in your application browser code, right? So instead of trying to access the file system and operating system directly from your spec, you want to go through the Cypress command that we provide to access the file system, node code and your operating system. You can read file, write file, execute any program or execute node code using task.

The task is the most powerful command that's one that executes the node code that you write. For example, a very common use case is try to connect to a database. For example, if you want to reset the database before the start of a test. If you write your plugin file like this, it runs in node, you can reuse part of your application code to connect to the database and then, for example, truncate the table on a task. We have very good examples of truncating the database and reseeding it with data in our Cypress real-world application. You can see that we are executing the task DB seed before each test.

One little detail that I want to point out, if you look at the spec file names, well, here's an example of API test and here's an example of a user interface test. A very common mistake is not picking the right type of a test. If your test is hard to write, hard to maintain, has a lot of boilerplate, well, maybe you picked a wrong type of a test and are struggling against magrane. If you're testing a user flow for web application, it's an end-to-end test. If you're testing an individual piece of code like a function or a class, you probably want to write a unit test. If you're trying to test a server and how it responds to a REST or a GraphQL request, you want to write an API test.

If you want to see how a page looks and if it looks the same, or maybe some of the CSS styling has changed, you don't want to use individual assertions. You want to do a visual test that compares the page or an element pixel by pixel. If you want to test an individual framework component from React or Vue or Angular Component, you want to write a component test. If you want to see how your page behaves in a different resolution, you want to run an end-to-end test with a different viewport. If you want to test accessibility, you want to write an accessibility test using a plugin.

Finally, if you want to run Node code — not the browser code, but Node code — and test it. Well, you cannot do it with Cyprus today but stay tuned because we are working on the Cyprus Node test runner.

We often see end-to-end tests that are way too short. On the other hand, sometimes, we see tests that are way too long. It's kind of a contradiction. You don't want to have too much money or not enough money. But here is an example of a test with too short. I think these tests were written by someone who's used to unit tests where every test just has one assertion. In this case, we're getting an input element and asserting every attribute in a separate test. This is way too short. It's unproductive. We recommend putting all assertions related to that element into a single test.

On the other hand, this test is probably way too long. This test goes through multiple fields on multiple pages, filling them up one by one. I believe this test takes way too long to be effective. Imagine if you're working on a feature and you have to make a change in your code and then wait for 30 seconds, a minute, five minutes, just to go through all the steps again and again. You're not very productive.

If you split the long tests into smaller ones, you will actually detect the problem faster because when the test fails, you'll know exactly which feature has failed. If you split longer spec files into shorter spec files, then your CI will be able to run those spec files in parallel, speeding up the overall test run. And to be honest, the longer the test is, the higher is the chance that the browser will run out of memory and crash, so you'll have fewer CI crashes if you use shorter specs.

Just remember, when you're writing a Cypress test, all commands are asynchronous and also they are all queued to be executed one after another. A very common mistake is to use a local variable, like username, and then execute visit and find the element. But remember, those commands did not run. They were just queued up. So if you're trying to access that variable immediately, well guess what? It will still be undefined at this moment, and so the else branch will always run. And your if branch never runs. So the right thing is to move all the code that gets the element into the then callback that executes after we visit the page and find the element. And then we can click on the right element depending on the element's text.

Another common example when writing end-to-end tests is not having enough assertions. This is an example of a navigation test. It goes to the home page, finds the link to the about page, clicks on it, confirms we're going to the about page, then clicks on the link to go to the user page. So this is the flow we want to test. And the test seems reasonable. It actually finishes. We are finishing on the user's page. But if you look closely, there are a few red flags. For example, when we click on the about link, cypress command log claims that we clicked on an invisible element. We also seem to have found the user's text before we navigated to the user's page. How is it possible? Well, in this case, we can use the site post command and go through the test one by one or use the time travelling debugger and go over the commands to see what happened and how did the page look at that particular moment. For example, if we go into the get command that finds the text about on a page, well we used that command to check that we are on the about page but look what happened. We accidentally found the text about on the main home page and not on the about page.

We accidentally found the text about on the main home page and not on the about page. So we actually passed this test accidentally and for the wrong reason. What we want to do, especially every time we do navigation, is to add assertions. For example, if we click on the about page link, well our application will do something in response. So before we do another command, we want to assert that that action has finished but we actually navigate to the about page and then we want to make sure that the text is there. If we look at the result and we hover over the contains command, well now we can see that clearly we are on the about page and we find the text that we expect to find.

In general, just like a zebra, try to alternate commands and assertions to make sure that the test runner doesn't run away from the application and in fact does what you expect it to do. When you're adding assertions, you can add positive assertions. For example, trying to find elements and make sure there are two elements found and each has a class completed is a positive assertion. But assertions that start with NOT keywords like an element should not have class completed or the loading element should not be visible, these are called negative assertions.

Be careful with negative assertions. It's easy to make a mistake. Here's just one example. Imagine my application is loading data. While it's loading the data, it displays a loading indicator. You write a test that visits a page and then says the loading indicator should not be visible. Well, the test passes, everything is happy, right? Well, there are a couple of red flags. Look at the negative assertion. The loading indicator is not visible. But it seems to be happening before the Ajax request happens. How's it possible? Well, let's investigate. In my particular application, I can delay loading the data by passing a query parameter. So how does it look now? Well, the test finishes and then we see the loading of the data. So something is wrong. Our negative assertion passes before the actual event happens. And that's actually correct. Our application doesn't show the loading indicator by default. So our assertion passes immediately, even before we load the data. The solution that I usually recommend is to always have a positive assertion that checks if the action starts before using a negative assertion. In this case, the positive assertion is the loading element should be visible and then it should be invisible, and now it passes. This test is not the best test actually because it's flaky because the loading indicator is so fast. What we really want to do is to make sure that we slow down the request using the cyber intercept command, and then it's clearly visible, and then it goes away.

The biggest mistake I think in all of that is that people skip reading the documentation. If you're not reading documentation, you're making a mistake. We know that documentation makes or breaks projects. We can see it.

Our people tweet or our users tweet. For example, my organization was evaluating automation framework. I was pushing for Playwright, but Cypress won due to perceived more available documentation and community support. We won for the right reason, I think. Every time I read our support forum and chat, I feel pain because I see so many questions, and every time a user has a question. To my opinion, it's a failure of documentation, it probably will be a support issue, or maybe a lost customer for Cypress dashboard.

We really want our users to find all the questions by themselves. Unfortunately, writing good documentation is hard because often, it's contradictory. Some users need questions answered in different order. For example, a first-time visitor might ask, show me Hello World, while a repeat user might ask, show me how to do x first. And these demands are contradictory. We optimised our documentation structure for beginners. We put Hello World first because we think there are 10 beginners for every advanced user. Every advanced user used to be a beginner. And then we add all our documentation to our structure. And we scrape all our documentation and every answer can be found by search. We even scrape not just our documentation, but our blog posts and our examples so you can find all the various pieces of information. I even look at the searches with no results. And then I take those searches and I write docs to answer those questions. You don't even have to use the browser to search our documentation. We released an NPM module called SysSearch that allows you to search our docs right from your terminal so you never leave your favorite environment.

If you want to know more about the way we structure our documentation and solve this problem, we have a couple of presentations. My personal goal is to have ten times more Cypress examples and recipes than we have right now across all our documentation properties, docs, examples, blog posts, and even videos. The big problem there is to keep examples up to date and correct. There is nothing worse than finding an example, copying it, and then finding out it doesn't work. Here is an example how we solve it. Imagine there is a question in a chat. How do I toggle a checkbox? I see that question, and I start writing a markdown document where I describe the question and provide HTML for the mini web application that has it. And then I write my test code in JavaScript blog. Well, we have a markdown preprocessor for Cypress that allows us to use markdown files as specs.

So we test those examples. Then we push and publish that markdown. It becomes a static documentation page. And then I answer the original question with a link. Of course, those documentation pages have been scraped and they're available for anyone to find the answer later. So we do not answer support questions directly, especially for private support, because it doesn't scale, because we'll have to do it again and again. And so instead, we update the documentation, create a new example or write a blog post or a recipe, and then answer with a link, as you can see in my example. I believe that the support team should really automate their own jobs away, but by writing more docs and more documentation, I mean examples, until all the users can find the answers themselves.

So I want to finish with something important. Many people comment on our GitHub issues, and they're angry and frustrated because there are no solutions to their issues. Well, we have a lot of open issues in our repository. Our number of issues every month is tremendous. So for example, in January, we had 300 issues in our repository. We closed 200, but 78 were open.

A single 24-hour period shows a lot of activity. Even worse is number of mentions and comments and updates that happens every day. This is my mailbox and it shows just emails from Cypress.io Cypress repository over maybe about a 30-hour period. There is a lot of activity in our test runner. As you can see, I read most of it.

If you have a problem, I suggest we first search our documentation. As I've shown, we put a lot of thought into the docs and searches. If you don't find an answer, please search our GitHub issues. Sometimes the GitHub text search is bad. So what you can do instead is look at the label that starts with topic. We have 60 something topics. Maybe there is a topic that matches your problem, like visibility, cross-origin, local storage, or network. Click on that label and look through the issues. Open and close. Maybe there is an issue already describing your problem.

If you don't find an issue that describes your problem, no big deal. Open a new one, we love it. So the most likely scenario for us to fix your problem or provide a walk around or a solution, is for us to be able to reproduce it. If we can run the code and see the problem ourselves, the chances are high that we'll be able to fix it. Now you can use one of our starter repositories, like Cypress Test Tiny or Cypress example recipes as a basis for your reproduction. Or you can prepare your own repository that we can install and run. Whatever you do, the most important thing is please, fill the issue template. Sometimes we get the issues where we just see a screenshot or a piece of custom code that says it doesn't work. It doesn't work in what version of Cypress, what do you expect to see? Do you see any errors in the console? We have no idea.

So please fill the issue template. One shortcut you can take is to really limit your example to what's really needed to recreate it. So if you don't have a plugin file or a support file, just set them to false, so we know that Those things are not important. And limit yourself to just reproducible example like that.

I want to leave you with the last piece of advice to summarize. In order for you to write good test without mistakes, please read the documentation. Thank you. I'm Gleb Bakhmutov. Find me online. And also find the slides here. Thank you very much. Take care.

Hello, everyone. Hey. So what do you think? Were the answers correct for the poll? The answers were absolutely correct. I myself picked the code coverage because I like seeing that green code coverage badge. I thought more people will pick all of above. So this means there's still room to grow for Cypress and make it more fun to write tests. Yeah. That coverage is actually it's not the goal, but it is a nice way of seeing how good you're doing. My current job, I was hired to help improve testing and I went from 12% to 80% that was my job. Or it was like 17% to 70%.. I don't remember the exact percentages, but that's like, yes, I did a good job. So I hope people also get that satisfaction when their coverage percentage goes up. But people are here to ask questions. So let's go to the first question of the audience. I have a question from Marcus, what would be the advantage of using Cypress Node test runner over existing test runners, like in example Jest? So we were thinking, right? What is your dev experience while writing tests in Node? And I'm not talking about unit tests like the smallest piece. What if you wanted to test your express server? Start it up and then execute requests against it and see what it does. And if there is a crash, what's your debugging experience when the test fails? Right? So we were thinking of doing the same thing that we've done for browser tests. Right? That means have a container where your node process runs, where you can absorb everything.

That container can spy on every network request. It can spy on all file system access calls, on other process calls. Right? And will give you a graphical user interface. So you'll see all your node tests, just like you see end-to-end and all the commands in a test. And then you'll be able to trace back how your code behaves for every step of a test. All right? So this debugging experience, the GUI, the test results, the ability to actually go back to the failing test and not rerun it again. Right? Like you do now. But actually being able to understand while fail step by step, that will be different. So that's the added... It's basically to fill up our experience, if I can sum it up like that. Yes. Yeah. All right. Thank you.

So next question is from Lias. Does the use of data tests or test selectors is good? Absolutely. We think so. We recommend putting special data attributes on your elements, what you want to select. For example, a button for clicking or text for assertion. So we have best practices guide in our documentation, but explains our logic. We think dedicated selectors are easy to understand and kind of give you a hint, Hey, use me for your testing. Don't remove me. All right. Don't accidentally change me like you can do with classes. A lot of people like using Cypress testing library that selects by role, by tax, by form input label. We absolutely encourage that as well. If you find you write better test using Cypress testing library, use those selectors as well. Just make sure it don't use some weird tan or deep X-path selector. Right. So that's the only thing.

Next question from Artem. Wouldn't network based await be better for ensuring the navigation is finished? Right. So when you write Cypress test and you visit the page or you click a button, it navigates to another page. How do you know that it has finished? Right. How does your user know that it's finished? Well, the user probably either sees the URL change. Right. That's one way. Another way, the user sees the new page. The user doesn't care if there is a network call that happens. No, the user wants to see the update page. So we'd advise you to observe the page, just like the user, and know that you finished the page like that. You can observe network calls if you, for example, spying on the data, if you're confirming the deep behaviour of your application. But don't make it your primary observation, I would say.

But don't make it your primary observation, I would say. So you kind of tie into the ideology of, well, you've mentioned it, testing library that tests as much as possible like the real user. Yes. Yes. Yeah.

Number three from Artem. Oh no, that's the one we just had. Sorry. A question from nof3412. What would be a good pattern to test a bigger user journey, like a shopping experience without checkout? Go for the whole story, just write the test that loads the page, picks an item, puts it on the cart, picks another item, puts in the cart, checks, you know, clicks on the cart, asserts number of items is correct. You know the payment information, shipping information and then click submit. Now when you click submit, you don't want that call to actually go to some third party payment gateway, right? You probably want in your test to intercept that call using our scientist subcommand and confirm that what's going to the third party is what you actually entered and expect, right? So you write the full test, but then you start testing all the edge cases, right? What happens if shipping is incorrect, right? Then you might kind of bypass and split that long test into parts, you know, filling, you know, picking items would be one test, but maybe, you know, entering shipping information, right? So start with a simple user journey all the way to the end, maybe stop outgoing call, confirm it goes out correctly and that will be a test.

Okay, okay. That's the last question we could do. We have no more time. But if you ask the question in the chat, and you need that sweet, sweet answer, GLEB will be in his speaker room available. So go down here to the timetable, click on the speaker room, and GLEB will be there shortly. GLEB, I want to thank you a lot for joining us today and hope to see you again soon. Absolutely. Thanks for having me. Happy testing everyone. Take care.