AI Can Generate Tests for You, Now What?

Hi, everyone, and yes, we are living in a very exciting period, right? Exciting and also worrisome period, I know, but come to think about it, we become more and more productive. We become so much, so much better. We can build things that we could have done within weeks that we wanted to do before 2023. And now AI can help us with almost everything and even can help us generate tests without us asking.

Well, how are we going to scale it? How are we going to make sure that AI can go beyond the whole concept of generating tests and make sure our product is still in high quality and AI can still help us at the maximum that it can? Well, let's take a look. Before that, my name is Maya. I'm a Staff Software Engineer in Salesforce and I'm passionate about AI accessibility and front-end. I am hosting a podcast called Build with Maya where I share my insights, experiment and experience working with AI and my journey in high-tech, you know, working with people, soft skills, and so on. So feel free to subscribe if you want to learn more about AI and the high-tech career path. Good enough.

So now back to our topic. Well, yes, AI in making generating tests becomes so much easier, right? Right now, if you go to Cloud Code, you ask it to implement a website, implement features, it can go instantly to add in, throw in some freebie context. For example, it can help you generate end-to-end tests, unit tests, and you no longer be blocked by testing at the end of the implementation where you just want to mark your PR but you forgot about unit tests and now you need to dedicate some time for that. It makes this become much, much more streamless, much easier. But that also means that it becomes cheap, so cheap to generate tests.

But that also means that it becomes cheap, so cheap to generate tests. Yes, you can actually forget about that, but that means that the cheaper the test generation is, the cheaper your code can be. The quality of your code can also be, which means that the bad tests now are very cheap to come up. Let's take a look why I want to say that, okay? So this is the test. This is a very normal test, integration test, where we actually test the provider, generate image capability, and we validate the result.

This test is a valid test because it does validate the real provider integration. It validates the actual response, and when you run it with the right configuration, it passes you locally, right? But does it mean that it works? Does it mean that it really guards your codebase? Well, look at the dependency. So this kind of test depends on so many unforeseen things such as network stability, latency, rate limit, token, budget, and module behavior because different image generation modules behave a little bit different from the others.

It can generate something less, something more, maybe it doesn't generate at all, or you token out budget, right? So if you place this test in a CI CD for PR, if something happens, something causes it to fail, now here's a couple of questions. How do we know the failure is caused because something's wrong in our implement logic? Or because something else happened in one of the dependencies we discussed? Well, this is exactly why I call it the trap in generating tests because the test generated is invalid, but the place the AI put it is absolutely in the wrong place.

And because in the wrong place, at this point it's no longer valid because once it runs, you cannot control whether this fails because of us or it fails because of something you don't know. And that comes to another thing. More tests doesn't mean that we have better confidence here. You can have 100 tests in your codebase, you can have 100% coverage, every single line is covered, it doesn't mean that your code is safe because confidence doesn't come by code coverage. It comes by knowing which risk are we trying to guard it from, knowing how to verify it and when the verification should run.

And by verification I mean when we need to check whether we should put the validation for the provider for image real integration in the CI CD or we should put it in the smog test. So, that comes to a thing. That's why we're here. And that means when we do the AI generation test, it doesn't give us more confidence. It gives us something hidden in confidence, hidden qualities, doubt that we are not aware of yet. Okay, so let's take a look at an example. So, this is a new feature for my blog post, sorry for my website where I have a page called blog list where I displayed a list of all my articles and I wanted to add a search. So, what it does is just show you, fill out a list of the blogs, available blogs by tag, by description, by title. Sorry. And it also has empty states, it has no results state and so on and so forth.

And I focus on making sure that the search has to be fast, the results have to be real-time and no stale results and it has to come with accessibility. And of course, because my page will grow and my website will grow, it has to be also scalable so that it won't be hard to maintain it in the future. Okay, sounds very simple and straightforward, right? So, let's take a look in the validation or the potential test that we need for this feature. So, we have a bunch of things like the most important one and we need to make sure it works by filtering post by title or tag. So, this is the core functionality. And then we have the clear when the query is empty and keyboard navigation, accessibility, which is excluded and so on and so forth.

Okay, now come to another interesting section. If you're working with testing, you know about the pyramid of unit tests, integration tests and end-to-end tests, right? The classic pyramid. We also have another thing called Agile testing quarens where we divide our test area into four quarters of quarens. The Q1, Q2, Q3 and Q4. This allows us to know whether a weak test is important for weak section and whether the test should be automate or manual.

Okay, so in this case, we have a bunch of tests assuming that Clark or Gemini or Codex have already generated the tests for you ready to use. Now, how are you going to know which of these tests belong to which quarens defined here? Because each quaren will matter for whether you put in CI-CD or you put in manual mock test or manual human interaction test. Okay, let's take a look. So, filter post by title tag, we go into unit test because that's the core implementation where you actually validate the technical logic. Q2 is acceptance test. We can put it like clear when query's empty.

Q3 is when we test all the unexpected behavior or user behavior and user interactions. So, we have like no result, keyboard navigation, search responsive and screen reader accessibility. It should belong to Q4 because it belongs to tools. We are using external tools like screen reader in order to validate our product, our feature, and so on and so forth. With this mapping, we have the – now we have clearer pictures of how our test fit into each category, each quarens, and how we can validate our feature much better and more systematic. Right.

But now come the questions. How are we going to fit in AI in these testing quarens? Because you actually have to sit there and validate each test. So, if AI generated for you 200 tests, how are we going to scale it? Right. How are we going to make sure that it may fit into each – and whether it's important enough or it should fit in this quarens? Well, that's the thing. So, now we're looking at a new testing module workflow. The first thing is, let's say in Q1, is when it's automatic, everything is automatic. So, what AI can help? Well, this is the most straightforward. This is the things that you saw AI always generated for every peer. They can absolutely help with unit tests, component tests, anything that's very much really logical, technical detail, technical limitation, coding-related. So, it can be edge cases, helping marking, set up the unit test framework, so on and so forth. And human, we, as the developer, human developer, we will direct it to understand the technical behavior in order for it to come up with the test itself. What about Q2? Q2 is kind of hybrid between automated and manual because it also has some part of technical verification. It's acceptance for business acceptance test also. So, for this, AI can absolutely help us to give some example of what it considers acceptance so we can understand more about our feature, what we should write, or what we should expect it, or what is the criteria in order for our feature to be considered accepted. And for human, we own the business intent because we know what we want to do best. AI is not the one who wants to have this feature. We are the one who wants to have this feature. We know what intent we want to use the feature for, and we know which priority matters to us. AI doesn't know everything for it. It's maybe P0, or P1, or P3, but we may know that maybe the P3 actually P1 in some cases. For Q3, usually it's manual, but I can see that nowadays the gap or the boundary between the manual and Q3 and automated is a bit blurry. Anyway, for Q3, it's exploratory test where we do all this testing the usability, user interactions. So, for this, AI can definitely help with ideas or questions about the usability, about user interactions for us, that the human to provide the judgment and to actually connect with the reality and decide what kind of scenario we want to test for manually. And lastly, for Q4, where we interact with build tests with tools, well, AI can definitely help with us in performance, accessibility, benchmark, compliance, security ideas, and we hold on the definition of what is the threshold, what is the CRCD, where the CRCD come in, and where the risk. We own the whole risk. We own the threshold. We own the CRCD gate, and of course, AI can also help us to set up the pipeline for accessibility, the benchmark for performance, and so on. So, this is how we should work with AI in the testing work runs in the current time. And that comes to another thing, that the traditional workflow where we work with AI in a way that AI receive the feature docs, AI implement it, AI write tests, and our job is to sit there and review the code and ask it to integrate with it to make it become better and approve the test. Sorry, approve the PR and move on.

Okay. Now, for that, we change. So, instead of just four steps like this in the AI write test, we have a different workflow, a slightly more scalable workflow, where we do receive feature docs, AI receive feature docs. We own the requirement and content. The feature docs, by the way, is the single sort of tool, what we call now product requirement description. This is where you receive it from the product managers in order to know how we want to build a feature, right? And like anything, we need to have a clear, concise, and full of information. In order for AI to be able to suggest the scenario in which quadrants it actually belongs to and analyze its info and use, we can also give it a strategy template that contains all the information that it needed in order for it to process the feature doc and map it into the correct format that we want. And what our job is, we can integrate with AI or we can also work on ourselves to review the strategy and decide what priorities for each of the tests were generated by AI and decide what the priorities to move on from there, how and when to implement it and what to implement. And these steps two and step three are very close to each other. And step four, only then AI will start to implement. AI will help us to prepare the pipeline, to help us to generate the automated test, when it should be automated, and help us to validate our thought and move on and safeguard our code. Then this is the new workflow for working with testing, and that should be instead of just generating text.

So with that, let's take a look on how I'm implemented with my blog post search. Okay. So in this one, I give it the feature request document where I have the functional requirement. Like you see here, I have even accessibility of non-functional requirement with the key requirement acceptance criteria and so on and so forth. I built a skill called test plan generate, where I give it the template of how the test plan should look, what is accepted, what is the definition of it, P1, P2 and P3, something like that. And in Clojure code, I asked it to generate the test plan for this feature based on the files, the feature docs. And here we go. We have the test plan, which everything divided into a quad run. And then we have also each of the quad run, we have some scenario. The test belongs to the scenario, and what is the priority of the test, and what's the key test we should prioritize first, according to what it understands. And that's it. That's all we need in order to start. From this moment on, we have a system for our agile testing workflow where we can actually control what it should be implemented for, what test is matter for us more, and what test that should focus and we should review. We will not go out of control where we will still appear with a hundred tests or a lot of code lines for tests that make it very hard to concentrate.

And now comes the next question. How are we going to scale from this individual skill to this team workflow? So at the team workflow, the first and foremost, we need to create a standard like everything else. Coding, styling, working style, everything have a standard at the team level. So what I would suggest is to do a strategy template, the same way that we do in our team, a strategy template for everyone to share with everyone on the team where people can actually take it and plug into their workflow and modify it according to their preference and work with their own AI to execute based on this strategy template, and then we can improve from there.

And, of course, for each of the Red Bull, we also have to have a testing rule. We have a definition of done. We have everything. It's like a standard guideline where we consider what is accepted, what is the standard alignment, what is consistency in order for us to work as a team with AI as the team member. And for anyone who review have to have a review checklist, a CI CD gates where we safeguard our code base as usual, even now more before because we have AI code and we don't want it to go wrong out of control. And lastly, like, just think about like we working with other human. We're working with AI. Now we have another team member called AI, whatever you name it. And then we have to do the metrics and learning loops after we do all of this to know whether these checkers is not good enough, whether the strategy template can be improved. And then we can see the metrics and from there we can improve our learning, our process, our workflow much better and scale more. And this is the sixth step to make sure that we have a good testing model workflow at the team level. And with that, remember better confidence comes from strategy and not just from adding more test. 100% code coverage doesn't mean your code is 100% safe.

How do we make sure that? We have to use health to write scenario, not test scenario first. Knowing you have to understand yourself, what features are you building, what failure it can be, what failure you want to catch first in order to make sure that it will not reach the customer before you reach the customer when you roll out the feature. And lastly, not every test that is generated belongs to a CI-CD. Let me see. A test that can be valid but it will not belong to a CI-CD, it will make it into a non-valid flockey test and it make our effort go waste and it make our workflow not working as expected and we're not productive anymore.

With that, thank you for joining my talk. I hope this session brings you some insights and feel free to ask me any questions on my social media, connect to me in any case. We love to hear your feedback and to hear what you thought. Thank you.