It's a (Testing) Trap! - Common Testing Pitfalls and How to Solve Them

Hello, everyone, and welcome to my session here at TestJS Summit this year. I'm so glad you're spending your time with me to learn about all the things I myself screwed up. Yeah, I know, that sounds harsh. But honestly, I hope to learn from my past mistakes together with you all. So thank you so much for being here and let's get started, right?

Let me start in another way. I'm Ramona and welcome to the premiere episode of Moe presents FarmEarth Test. Side note Moe is a little nickname for me. But let's stop being silly and let's get serious instead. I'm working as a software developer at Shopware, which is a company providing an open source e-commerce platform. But I have a couple of years' experience in quality assurance as well. So I know both views of the product or of an application, that of a tester and that of a developer, and I strive to make the experience of both better. Let's call it developer and tester experience alike. And this is basically the reason why I started to think about this session, this talk.

To get to the point, let's think about the following situation or it's more thought. There are a couple of movies I loved watching in my childhood. And even as an adult I re-watched them numerous times. And one franchise of those is Star Wars. And there is one quote in particular from the 1983 Star Wars film episode 6, which is Return of the Jedi, which sticked into my mind. And it's this one. It's a trap! Well, every terrible mimicking skills of me aside. It's really a memorable quote, said by Admiral Ackbar, who is the leader of the Mon Calamari rebels. And it was said during the Battle of Ender, where the Alliance mobilized its forces in a concerted effort to destroy the Death Star. And Ackbar encounters an unexpected ambush there, which leads him to exclaim this quote. So yeah, it's a nice quote, right? You think? You will wonder what it has to do with testing. And yes, I think it has to do something with testing. It's a nice allegory when it comes to testing. Testing has so many perks and advantages, and I don't think I need to convince you about that here in this session. But all those value, all those wonderful things when it comes to testing can be outshadowed though by pain points caused by various reasons. Many of them can be considered traps, and they may feel like an ambush. Things you did in best intent, of course, but turning out to be painful in the long run or even earlier.

Well, when it comes to my beginnings, my tester or developer career, I think about three pain points, especially when it comes to, here, pain. The first being slow tests, which can happen by various reasons. And you can imagine, if you want to finish a feature or a task or a ticket, and you want to merge your pull request so badly, but you need to wait for the pipelines to run, that's awful.

Second, and maybe more important, I think about tests which are painful to maintain. For example, a test I don't understand when looking at it months or even years later. Well, even fellow team members asking me what I thought I wanted to achieve with this test, so well, I think bad things happen, right? But it's not the worst pain point I've felt when it comes to working with tests. It's about this one, tests who give you no value, no clear result at all. You may call them hyzm-fails or hyzm-tests, like the famous hyzm bug, which only occurs if you look away, don't debug results. And there's the end boss, flaky tests, which are non-determinant, tests who fail to deliver the same correct value in between builds. Imagine a test passing the first time and failing the second, and passing the third it says you cannot trust, right?

But, well, it doesn't have to be this way. Let me show you my biggest failures when it comes to tests, and how to correct them. How to, yeah, what to keep in mind when it comes to writing good tests and to avoiding these pitfall traps. Well, due to the time limit, I cannot mention all of them, but let's speak about others, like some I encountered or your experience later on in the Q&A, or even later.

The first and most important thing inside of my mind is the following situation. Please look at it first. Maybe you find yourself familiar in it. Well, think about your brain doing a task, coding. And your brain is full with the main production code. And you have no headspace left for any additional complexity. Every task which comes on top shouldn't eat headspace then, or be exhausting to you. Otherwise, it will be painful to you. So eating more headspace is against the intent of testing, and even cause team to abandon testing at all in the worst case. So, avoiding tests to eat headspace is the most important thing in testing in my opinion. And I'm not alone with that. Joni Gilbert describes it as the golden rule. And it is the following one. It's basically keep it stupid simple. Tests should be designed plain simple in every case. And it doesn't matter if we are talking about unit or end-to-end testing. Our goal should be the following.

One should look at your test and get its intent instantly. Without thinking about it, basically. So yes, test is not totally like production code. Not completely. And yeah, I know that you should treat your test code with the same care as you do with production code. But some best practices applied to production code aren't suitable for tests or are in conflict with tests. For example, when I'm thinking about duplication or the dry principle. So please try to use less to no abstractions inside of your tests, to keep them comprehensible. So that also means be cautious when it comes to commands or page objects in NGINX for example. I like to describe it as a flat or minimal test design. So testing only as much as needed belongs to that as well. In this case, you will keep your tests comprehensible and delightful to work with. And this is a principle which can be applied to basically all the following examples.

So I'm speaking about examples right? Let's take a look at them. Look at this little code snippet. Especially when it comes to the test title. It's a just snippet. But it's still applicable to end-to-end testing as well. When you look at this test, do you know what it wants to accomplish? For example, if you look at your test log, if this test is failing, do you know what it is about or what could be wrong inside of it? Well, it should throw an error. But which error? When should it be thrown? Anything? Remember our golden rule. You should know instantly what your test is doing. So we need to improve it, right? Let's take a look. This one is the three part rule by Woi-Ashiwab, which could come in handy in this case. It will help you with keeping clear what your test wants to cover and accomplish. It's well known from unit testing, but maybe it's not that bad to keep it in mind when it comes to end to end or integration testing as well. The three part rule basically says you should have a test containing three parts. One, what is being tested? In this case, it's the property. Two, under what circumstances and scenario we test, which is the usage of the deprecated property in our example. And three, what is the expected result of the test? In this case, the error of thrown. So we have this little title here, which is not that little anymore, but it's more comprehensible. And if you think about longer test titles, long cat is long, I'm sorry, test name is long. Long cat likes long test titles because you can see the result of the test in first sight. And you don't need to read through all logs or the source code to get what's wrong inside of this test.

Alright, next trap. Look at this test. Do you see and get it in first sight? If yes, I got to say props to you, but if you don't, that's totally normal and no problem at all, because this test structure is a mess. You see, declarations, assertions and actions are really not needed without any structure at all. So how can we change this test to get it easier to understand? Well, I like to use the triple-A pattern, which is short for arrange, act, assert. So again, this is relevant for unit testing, so I'll use a just example here, but, yes, it's not that good for end-to-end tests, which tend to be a bit longer, but it's still worth mentioning here. The triple-A pattern says, you should structure your tests in three parts. First, the arrange part, which is dealing with the setup to bring the system to the scenario the test aims to simulate. Think about variables duping or mocking, for example. The second one is about running your test. Basically, the act part. Running your unit under test, do your steps, actions, etc. to get to the result state. And third, the assert part, which is quite self-explanatory. So you do the assertions and checks of your test scenario inside of it, which makes the triple-A pattern another way of designing your tests in a simple and unique way. And you can basically get the test to send in first sight, which is really, really important.

Alright, next one. Another slide, another trap. And we got BB-8 from Star Wars as a guest. And he found a name which might be familiar to us. But not to him, not to BB-8. It's FUBAR. Well, we as developer know that FUBAR is often used as a placeholder name. But if you see it inside of a test, do you directly know what it stands for? Again, this way your test may be more difficult to understand at first sight. So we should avoid that. But what should we do instead? Let's take a look. In this case, I brought you a little cypress test. So it's an end-to-end test in this case. But this advice is not limited to that. You can use it in integrational unit testing as well.

This test should check if a product can be created and read. So in it, I want to use names and placeholders connected to a real actual project. So when it comes to the t-shirt, I use t-shirt Akbar because it's a t-shirt with Akbar. I don't know. And when it comes to the manufacturer, I want to use Base Company as a name. So that you can see that I am dealing with a product, for example.

Well, if you don't want to invent all the names by yourself, you can use Faker to generate data or even import it from a production state. So maybe you see I want to stick to the golden rule we mentioned even when it comes to naming. New jobs, same tests.

Did you notice these selectors here? They are CSS selector and you you might say now, OK, but, well, I can have unique CSS selector they work right? Why are they problematic? Well, because they are prone to change. If you, for example, refactor the application and change classes, the test may fail even if you didn't introduce a bug in the test. Failing without a bug is a false negative, giving no reliable report on the application as you just changed the implementation and it don't report an error, right? So this trap refers mainly to again tests in this particular case. But in other circumstances it might apply to unit testing as well as soon as you use selectors. So look at selectors you must, Yoda would say.

So you shouldn't test implementation details at all. Instead, you should use other things. For example, in place of a CSS selector, try to test for something a user would call attention to. For example, a little string you see inside of your application, or a heading, or the words inside of a button. Or even better, choose selectors which are less prone to change, for example, data attributes. If you refactor your application, it might be the case that you will change some CSS classes or styling, but changing data attributes, especially if you name them like data test or data CY, isn't that common, so they are less prone to change.

Last but not least, there is a topic I cannot stress enough, so I need to introduce it as a trap as well inside of here. So it's about these fixed waiting time things. Like waiting for half a second every time. And even worse, it's in a command here, so it will be executed way too often. And in best case, it will slow down the test way too much. And it's not necessary at all, because our application can be faster. But in the worst case, we will wait too less time on causing flakiness. So fortunately there's plenty of things to do to avoid fixed waiting time traps. Let's take a look. All the ways center around waiting dynamically, and I prefer more deterministic methods that most platforms provide.

I brought you my two favorites to use. The first being waiting for changes in the user interface of your application, for example animations to stop, a loading spinner to disappear or anything to be loaded. Another possibility would be the waiting for API requests. Cypress provides you neat features for doing that. In this way, your test will stay stable and reliable while managing time efficiently, like this bad boy here.

Let's go back to Admiral Akbar. The Battle of Endor turned out to be a success, of course, with teamwork and a couple of countermeasures included. Refer that to testing. It might be a lot to do, especially when it comes to legacy code, and it might need a change in mind for test design or lots of refactoring, but it's worth it in the end and you will feel the rewards.

So the most important things, really, I want you to remember, is referring to the golden rule we talked about earlier. All examples pay in on that. All pain points emerge from ignoring it. It's about keeping your tests easy to read, to maintain, and to keep them easy to understand. It's one sentence I want you to remember from this talk. A test should be a friendly assistant, no hindrance to you. Tests should feel like a routine, not like solving a complex mathematical formula or something. So, let's give our best to achieve that. See, R2D2 here is catching bugs with ease here, and I want you to feel that as well. So, that dealing with tests feels clean and like fun, and not exhausting, right? So, yeah... What else can I say now then? Thank you. Thank you for your time and for listening to me. And if you got questions or want to discuss any other traps which might occur to you or other traps I experienced, please join the Q&A session or talk to me on this conference or afterwards on Twitter or basically wherever you found me. So, let's keep that and see you next time. Goodbye!

How you feel about the results, Almona? Welcome! Hello, everyone! So, I'm feeling really relieved because most people answered the way I did. I experienced all of that, as you might have noticed, and it's wonderful to not feel so alone or feel like an idiot because I screwed up so many times and I hope we can make the best of it and to learn together and discuss together and yeah, let's see! I like the lessons you mentioned today, definitely learning from experience and for all of the situation that we are in today.

Let's move to the question, we already have some in Discord, Sev was saying hi and asking if you have an opinion on dry, don't repeat yourself versus dump descriptive and meaningful phrases when it comes to end-to-end tests. In other words, if you use a code snippet in several tests, would you just copy and paste it to keep it simple or create a custom command for it? I love this question, because it's a struggle I myself feel a lot, because as a developer, of course, I like to decide in favour of the dry principle, to don't repeat myself too much. But as I said, as a tester and also as a developer, having comprehensible tests is the most important thing. Sometimes I would say don't give too much weight on dry principle and more on understand your tests. I think if I use a code snippet in several tests and it's not a big code snippet, I would use a custom command, but I would try my best to name it in a really descriptive way and to use some documentation, for example, definitions for autocompletion and stuff, which is considered a duplicate, but I think it's really important in this case, so that all people seeing this custom command do know in first sight what it is all about and what it does. So you can have it comprehensible and have some kind of dry inside of your tests. So it's I think the best thing in the middle, but I know it's really, really difficult to decide in favor from one or the other. Indeed, it's hard to be in between, but sometimes there's the good place to use it.

Liaz was wondering how to force frontend dev to use selectors. I think you mean to use selectors in favor of strings or whatever, so when it comes to selectors, it's really important to distinguish which selector you use when it comes to CSS selectors, I would rather prefer using strings or to wait for any text the user would see in your application, because CSS selectors are really prone to change. And this could be a thing to give you some failure tests, even if nothing's broken inside of your application. But if you're using other selectors, like for example, data reviews, which I myself a huge fan of. You can make it that they know they won't need to update their tests that much, because when it comes to data attributes, and especially if you name them like data CY for Cypress attributes, or any other descriptive phrase, a developer who's working on your code does know that it's meant for testing and won't change it. So your tests will fail for the right reasons and not just for an outdated class or anything like that, which is less work for the developer, right? I think that might be a wonderful argument to convince them. Definitely, we can gain them on our side with that, indeed.

Martine, oh my god, it's alive, was asking how would you compare arrange, act, assert, sorry, arrange, act, assert to given, when, then, and use which of them in which situation? Encountered any traps with this? Well, I'm so glad you ask this, Martine, because that was the thing I needed to leave out of my track because of the time frame. I wrote an article about that, that some tests in my daily doing I'm working at a huge application in the e-commerce sector, which tends to be quite conflict. And even in unit tests, it needs sometimes a bit more context in the test as you should have when it comes to unit testing. So you shouldn't call them unit testing to begin with, but when it comes to integration tests in Jest or something, you sometimes need to test the DOM, which sometimes require you to check on the before and after states of your component, for example. And in this case, arrange act assert is no good thing to use, because it's basically really difficult to press your tests inside of the structure in this case. Even when, then it's more suited in this case when it comes to integration tests or if you need some sessions on the state before and after. So you can consider it like that. Arrange my tests, I think it's what I'm giving. So all the thing you need to arrange is the given part. Act in my tests. I think that's when something happens. So that's the middle part. And assert the results.

So that's the middle part. And assert the results. So in this case, you can do the assertions on the DOM in the given part and it's still okay. And it's still a valid structure. And it won't violate AAA in this case anymore. Indeed, Maria, Maria K85 was wondering, it is considered a bad practice to have repeated assert sequences in a single case. Example being, filling one form, submitting it, expect the data to be submitting, click edit and expect the form to be active again. I think it depends on the purpose you use it. If you are writing a quite short unit test or short integration test, I would consider, as Martin said, you can then do another structure, but when it comes to end-to-end testing, you cannot avoid using those assert sequences. So, in this case, I think we can bend the rule and say it's okay, as long as it's easy for you to read your test. So, maybe use some comments to name some sections so that the other developers don't have a hard time to understand your test. Indeed. Yeah, it would be like that.