AI Can Generate Tests for You, Now What?

Hi everyone, and yes, we are living in a very exciting period, right? Exciting and also worrisome period, I know. But come to think about it, we become more and more productive. We become so much, so much better. We can do things that we could have done within weeks that we wanted to do before 2023. And now AI can help with almost everything. It even can help it generate tests without us asking.

But now that comes to another question. What will it be? How are we going to scale it? How are we going to make sure that AI can go beyond the whole concept of generating tests and make sure our product is still in high quality and AI can still help us at the maximum that it can? Well, let's take a look.

Before that, my name is Maya. I'm a staff software engineer in Salesforce. And I'm passionate about AI accessibility and frontend. I am hosting a podcast called Build with Maya where I share my insights, experiments, and experience working with AI and my journey in high tech, you know, working with viewables, soft skills, and so on. So feel free to subscribe if you want to learn more about AI and the high tech career path. Good enough. So now back to our talk.

Yes, AI is making generating tests become so much easier, right? Right now, if you go to Qlarko, you ask it to implement a website, implement features, it can go instantly to add in, throw in some freebie context. For example, it can help you generation end-to-end tests, unit tests, and you'll no longer be blocked by testing at the end of the implementation where you have, you just want to load your PR, but you forgot about unit tests and now you need to dedicate some time for that. It makes this become much, much more streamless, much easier. But that also means that it becomes cheap, so cheap to generate tests. Yes, you can actually forget about that, but that means that the cheaper the test generation is, the cheaper your code can be. The quality of your code can also be, and which means that the bad tests now are very cheap to come up.

Let's take a look why I want to say that, okay? So this is the test. This is a very normal test, integration test, where we actually test the provider, generate image capability, and we validate the result. This test is a valid test because it does validate the real provider integration. It validates the actual response, and when you run it with the right configuration, it passes you locally, right? But does it mean that it works? Does it mean that it really guards your code base? Well, look at the dependency. So this kind of test depends on so many unforeseen things, such as network stability, latency, rate limit, token, budget, and module behavior, because different image generation modules behave a little bit different from the other.

It can generate something less, something more. Maybe it doesn't generate at all, or you're talking about budget, right? So if you place this test in a CI-CD for PR, if something happens, something causes it to fail, now here's a couple of questions. How do we know the failure is caused because something is wrong in our implementing logic? Or because something else happened in one of the dependencies we discussed? Well, this is exactly what I call the trap in generating tests, because the test generated is invalid, but the place the AI put it is absolutely in the wrong place. And because in the wrong place, at this point, it's no longer valid, because what it runs, you cannot control whether this fails because of us, or it fails because of something you don't know.

And that comes to another thing. More tests doesn't mean that we have better confidence here. You can have 100 tests in your code base. You can have 100% coverage. Every single line is covered. Doesn't mean that your code is safe, because confidence doesn't come by code coverage. It comes by knowing which risks are we trying to guard it from, knowing how to verify it and when the verification should run. And by verifications, I mean when we need to check whether we should put the validation for the provider for image real integration in the CRCD, or we should put it in the small test.

So that comes to a thing. That's why we're here. And that means when we do the AI generation test, it suddenly doesn't give us more confidence. It gives us something that hidden qualities doubt that we are not aware of yet. OK, so let's take a look at an example. So this is a new feature for my blog post, sorry for my website, where I have a page called blog list, where I display the list of all my articles. And I wanted to add a search. OK, so what it does is just show you a few thousand list of the blogs, available blogs by tag, by description, by title. Sorry. And it also has an empty state, have no result state, and so on and so forth. And I focus on making sure that the search has to be fast. The result has to be real time and no sterile result. And it has to come with accessibility. And of course, because my page will grow and my website will grow, it has to be also scalable so that it won't be hard to maintain it in the future.

Sounds very simple and straightforward, right? So let's take a look in the validation, all the potential tests that we need for this feature. So we have a bunch of things like the most important one. We need to make sure it works by filtering, post, by title or tag. So this is the core functionality. And then we have the clear when the query is empty, and keyboard navigation, accessibility, which is excluded, and so on and so forth. OK, now come to another interesting section. If you're working with testing, you know about the pyramid of unit tests, integration tests, and end-to-end tests, right? The classic pyramid. We also have another thing called Agile testing queries, where we divide our tested variables into four quarters of queries, the Q1, Q2, Q3, and Q4.

This allows us to know whether a weak test is important for each section and whether the test should be automated or manual. OK, so in this case, we have a bunch of tests, assuming that Clarko or Gemini or Codex have already generated the test for you, ready to use. Now, how are you going to know which of these tests belong to which quarters defined here? Because each quarter will matter for whether you put in CI-CD or you put in manual smog tests or manual human interaction tests.

OK, let's take a look. So, fill out the post by title tag, go into unit tests, because that's the core implementation where you actually validate the technical logic. Q2 is the acceptance test. We can put it like clear when the query is empty. Q3 is when we test all the unexpected behavior or user behavior and user interactions. So, we have no result, keyboard navigation, search responsive, and screen reader accessibility. It should belong to Q4 because it belongs to tools. We are using external tools like screen reader in order to validate our product, our feature. And so on and so forth.

With this mapping, now we have a clearer picture of how our test fit into each category, each quarters, and how we can validate our feature much better and more systematic. Right. But now come the questions. How are we going to fit in AI in these testing quarters? Because you actually have to manually sit there and validate each test. So, if AI generated for you 200 tests, how are we going to scale it? How are we going to make sure that it may fit into each and whether it's important enough or we should fit in this quadrant? Well, that's the thing. So, now we're looking at a new testing module workflow. The first thing is, let's say, in Q1 when it's automatic, everything's almost automatic. So, what AI can help? Well, this is the most straightforward. This is the things that you saw AI always generated for every peer. They can absolutely help with unit tests, component tests, anything that is very much really logic, technical detail, technical limitation, coding related.

So, it can be edge cases, helping marking, set up the unit test framework, so on and so forth. And human, we, as the developer, human developer, we will direct it to understand the technical behavior in order for it to come up with the test itself.

What about Q2? Q2 is kind of hybrid between automated and manual because it also has some part of technical verification. It accepts for business acceptance tests also. So, for this, AI can absolutely help us to give some example of what is considered accepted so we can understand more about our feature, what we should write, what we should expect, or what is the criteria in order for our feature to be considered accepted.

For Q3, usually it's manual. But I can see that nowadays the gap or the boundary between the manual and Q3 and automated is a bit blurry. Anyway, for Q3, it's an exploratory test where we do all this testing, the usability, user interactions. So, for this, AI can definitely help with ideas or questions about the usability, about user interactions for us as the human to provide the judgment and to actually connect with the reality and decide what kind of scenario we want to test for manually.

AI doesn't know everything for it. Maybe P0 or P1 or P3. But we may know that maybe the P3 actually P1 in some case. For Q3, usually it's manual. But I can see that nowadays the gap or the boundary between the manual and Q3 and automated is a bit blurry. Anyway, for Q3, it's an exploratory test where we do all this testing, the usability, user interactions. So, for this, AI can definitely help with ideas or questions about the usability, about user interactions for us as the human to provide the judgment and to actually connect with the reality and decide what kind of scenario we want to test for manually.

And lastly, for Q4, where we interact with build tests with tools, well, AI can definitely help with us in performance, accessibility, benchmark, compliance, security ideas. We hold all the definition of what is the threshold, what is the CRCD, where the CRCD come in, and where the risk. We own the whole risk. We own the threshold. We own the CRCD gate. And of course, AI can also help us to set up the pipeline for accessibility, the benchmark for performance, and so on. So this is how we should work with AI in the testing work runs in the current time.

And that comes to another thing, that the traditional workflow where we work with AI in a way that AI receives the feature docs, AI implements it, AI writes tests, and our job is to sit there and review the code and ask it to integrate with it to make it become better and approve the test. Sorry, approve the PR and move on. Now, for that, we change. So instead of just four steps like this in the AI write test, we have a different workflow, a slightly more scalable workflow where we do receive feature docs. AI receives feature docs.

AI receive feature docs. We own the requirement and context. The feature docs, by the way, is the single sort of true what we call now product requirement description. This is where you receive it from the product managers in order to know how we're going to build a feature, right? And like anything, we need to have a clear, concise, and full of information. In order for AI to be able to suggest the scenario and which quadrants it actually belong to and analyze the info and use, we can also give it a strategy template that contain all the information that is needed in order for it to process the feature doc and map it into the correct format that we want. And when our job is, we can integrate with AI or we can also work on ourselves to review the strategy and decide what priorities for each of the tests that were generated by AI and decide what are the priorities to move on from there, how and when to implement it, and what to implement.

And this step two and step three are very close to each other. And step four, the last step, only then AI will start to implement. AI will help us to prepare the pipeline, to help us to generate the automated tests when it should be automated, and help us to validate our task and move on and safeguard our code. Then this is the new workflow for working with testing that should be instead of just generating text.

With that, let's take a look on how I'm implementing with my blog post search. So in this one, I gave it the feature request document where I have the functional requirement. Like you see here, I have even accessibility of non-functional requirement with the key requirement acceptance criteria and so on and so forth. I built a skill called test plan generate, where I give it the template of how the template should look, what is accepted, what is the definition of it, P1, P2, and P3, something like that. And in Clarko, I asked to generate the test plan for this feature based on the files, the feature docs. And here we go. We have the test plan, which everything divided into a quadrun. And then we have also each of the quadrun, we have some scenario. The test belongs to the scenario. And what is the priority of the test? And what's the key test we should prioritize first according to what we understand? And that's it. That's all we need in order to start. From this moment on, we have a system for our agile testing workflow, where we can actually control what it should be implemented for, what test is matter for us more, and what test that AI should focus and we should review. And we will not go out of control where we will still PR with 100 tests or a lot of code line for test that it make us very hard to concentrate. And now comes the next question. How are we going to scale from this individual skill to this team workflow? So at the team workflow, the first and foremost, we need to create a standard. Everything else, coding, styling, working style, everything have a standard at the team level. So why would I suggest it to do a strategy template the same way that we do in our team? A strategy template for everyone to share with everyone on the team where people can actually take it and plug into the workflow and modify it according to their preference and work with their own AI to execute based on this setting strategy templates. And then we can improve from there. And of course, for each of the red ball, we also have to have a testing rule. We have a definition of done.

We have everything. It's like a standard guideline where we consider what is accepted, what is the standard alignment, what is consistent consistencies in order for us to work at a team with AI as the team member. And for anyone who review have to have a review checklist, a CI CD gates where we safeguard our code base as usual, even now more before, because we have AI code and we don't want it to go wrong out of control.

And lastly, like just think about like we work in with other human. We're working with AI when now we have another team member called AI, whatever you name it. And then we have to do the metrics and learning loops after we do all of this to know whether this checklist is not good enough, whether the strategy template can be improved. And then we can see the metrics. And from there we can improve our learning, our process, our workflow in much better and scale more. And this is the sixth step to make sure that we have a good testing model, workflow at the team level.

And with that, remember, better confidence comes from strategy and not just from adding more tests. 100% code coverage doesn't mean your code is 100% safe. And how do we make sure that? We have to use health to write scenario, not test scenario first, knowing you have to understand yourself what features are you building, what failure it can be, what failure you want to catch first in order to make sure that it will not reach the customer before you roll, before, when you roll out the feature. And lastly, not every test that generated belong to a CICD. Let me see, a test that can be valid, but it will not belong to a CICD, it will make it into a non-valid floppy test and it make our effort go waste and it make our workflow not working and expected and we're not productive anymore.