Go Find What We May Have Missed!

Hi, my name is Maaret Pyhäjärvi, and for the last year and a half out of my 25-year career, I've worked at Vaisala as a principal test engineer. A nice way for me to frame what I do for my work is this idea that I'm quality control for the testing that is happening in our teams. I go and assess the results that we're able to provide by staying within the team as one of the team members for a while.

A lot of times I frame my assignment as go find at least some of the things others have missed. I repeat this from team to team, usually spending six to even twelve months within one team, with the idea of leaving things better after I'm gone and helping the teams grow in the way that they do testing. I have done this over my career quite many times for various different products and teams, one of them being this particular example here on the slide where I asked a developer of an API-based testing tool for permission to test their application and use it as training material in some of the conference talks that I've done. Then they gave a podcast interview saying I basically destroyed it in like an hour and a half. This is a common experience that developers tell of me with a smile usually, hopefully at least, on their face.

And it is usually related also to the fact that I have by that time also had the conversations on, you know, I didn't destroy the application. The only thing I might have destroyed is the illusion that was never the reality. So you might be very proud of your application. You might be doing already a good job. And there still might be things that you're missing. And your customers might not be telling you.

So in all of this work that I've done, I've kind of summed it up as a recipe for better teams. How do we go about finding the things we're missing? And we start with two kinds of testing. There's the testing that is kind of framed as an artifact creation, whether it creates automation or checklists for repeatable tests for later. And then we have the other kind, testing as a performance, kind of like improvisational theater, where you look at the application. Application sort of speaks to you. It's like your external imagination. And it makes you more creative. And whatever you learn, you can then turn into the artifact creation part of the testing. You need both of these sides. They give you very, very different things where the artifact creation style gives you specification, feedback, regression, and my absolute favorite, granularity, knowing based on the results that you're getting, what was the change that broke things, and from the logs, what is now broken, without having to spend multiple hours or even days analyzing your results before getting to the actual fixes. These are things that you can get from the artifact style of testing.

On the performance style, it gives you a little bit more vague things in many ways, but also, it gives you kind of like a guidance. You know, the direction, are we going to a better direction? Is this good? Is there still more feedback, more conversations to be had? Is there something where we need to build our understanding and improve the models? And my, again, absolute favorite, serendipity, lucky accident, meaning that sometimes, some of the problems that we need to find are two interesting combinations of all kinds of things we didn't think of, that we just need to give it time. So, there's a saying, a quote by Arnold Palmer, a famous golfer, that it's not just that he's lucky, it's just that he has been practicing. So, kind of like that's the general idea with this style of testing. So, framed from the sides, we need something in the middle for the better teams.

And the thing we need in the middle is, of course, different kinds of tests. Whether it comes from the point of view of creating artifacts, or whether it comes from the point of view of performing testing and thinking what kind of things we might still be missing, we probably will test against different levels of interfaces available in the system, and try making a balanced set of all the different shapes of tests, be they small, medium, large, unit service UI, or unit integration system, or end-to-end, whichever words you end up wanting to use. You probably also will not have just these different kinds of tests where you're basically then just kind of growing the scope of it. You also probably would like to have in those better teams some kind of ways of faking, mock, stub, spy, fakes, whatever you want to call it. Ways of faking either the service responses, the data, or any of the components that you want to leave out of the testing scenario so that you can have a focus feedback. But also, you want to test with the real integrations, again, because of serendipity you are most likely going to see something different there, and that is what your customer will end up using anyway, not the mocks that you have created. You'll probably have a foundation of functionality, but also the three other key things. It needs to be responding for the customer's requests fast enough. It needs to be easy enough to figure so that you know what to do with the application. And the disfavored users should have mechanisms keeping them away from your system so that whatever the business purpose is that the system serves, the information also is safe from other people causing you harm. So this is kind of the frame that I think that we need for the beta testers.

And I wanted to give you a small example of what typically applying something like this looks like on an application. I took a small example application which was created basically for using this or showing the idea that you can have a front-end and you can have a back-end and you can mock the back-end responses. So there's a very simple React front-end, very simple React app and the possibility of changing whether you are working against the actual or the mock server is already in the user interface.

The app is clearly created for testing purposes in the sense that whenever I kind of take it out from my notes there and I start running it the first few minutes goes into updating whatever dependencies there are. And after the dependencies have been updated I usually have to think of it for a moment on the fact whether you know I have any test cases that I could run. I actually do not have any, so I'm very much relying on the manual performance, attended performance type of testing with this application.

So I might start with reading about the application, figuring out that it's actually using an open weather map API for the current weather, and thus summarizing things for that particular API. And with showing things in this outline, even here, I can notice it does things around city names, city IDs, coordinates, zip codes. It's supposedly returning me different kinds of responses. There's something of formats, units of measurements. And just alone, the outline already creates a vast amount of ideas of what I could do. So I would probably want to start with a new application, trying to test it with something that I could see a basic positive scenario, something that is fairly easy for me to verify. So just checking the location where I am at, and seeing the results of that right about now, would give me the results that I can verify just by opening my window. And yes, very surely, it kind of shows exactly what I was expecting the case to be for Helsinki, Finland.

Similarly, I can imagine other locations, Sydney being one of the locations that is kind of furthest I have ever traveled from my own home. So it feels like a perfect place to go and test. But comparing to my own location, I'm not actually getting much of a difference here, broken clouds, few clouds, not relevantly different. So I was hoping to see sunshine and a whole different icon here in the user interface. But no luck with that one. But again, generating more ideas, I realized that I could and I have before written Sydney with a typo. And I realized that I'm kind of surprised that even when I typo things, trying out Sydney and many other different similar small but usual typos, it seems that the API actually gives me the right answer even though I am not very precise on my request. And this makes me, of course, a little curious, but I kind of just make a note of it, mental note of it.

Remembering the API description, I remember that there was an idea that I can also query, at least from the API, I can query based on a postal code or like the location code. And I first want to just try out my own location, this very neighborhood that I am in in Finland. And to my surprise, it locates me not to Helsinki, Finland, but to Vantaa, Finland. And that's considered a different city, so not quite okay. And also I was supposed to be entering cities. This doesn't seem like a city to me. So getting some kind of ideas on what I can do with the application. Similarly, thinking of, you know, postal codes that we all know, 90210 from the fairly famous series back in the days. I'm expecting to see Beverly Hills and I'm again disappointed, noticing that maybe there is a reason why this should only allow me to enter a city so that I don't always get disappointed when I enter the postal code of an area. So I wasn't definitely expecting to end up in Mexico. I go through, having looked at the application interface, I go through a little bit of the specification and I find that there's, you know, nice things about ideas of different kinds of weathers.

Like this could be a thunderstorm, there could be a drizzle, there could be even a hurricane going on. And since the application is very much current weather, me trying to guess right now where in the world there might be a hurricane, that's not going to work very well. And similarly, me right now being here in Finland in the middle of the night with clouds, I can't really verify the daytime, nighttime icons and whether those work correctly that I could find in the specifications with the data that I have at my hands.

So what I usually then do, you know, trying to do proper testing is take control over the data. There's many ways of doing it. And mocking of course is one of the ways that we could consider. And changing into the mock server, well starting the mock server at first, I can run some of the test cases that I have prepared that set up stubs for different kind of settings. And I have here a stub with weather at Utsjoki. Utsjoki being a location in Finland from where I saved a response on summertime at a time when there is no sunset and no sunrise, because that's how certain locations in northern Finland are in the middle of summer. So I set up that Utsjoki and I set up Utsjoki data so that well, I only go and change the specifics of what is the weather like. So the times and all that stuff, I leave as it is. But I can edit a thunderstorm in. And yes, definitely. Now, I can verify that yes, if this is the response we would be getting, I could see a new kind of an icon, I could see the texts. And only parts that I was expecting to change are changing. And obviously, the first time I did this, it wasn't quite so straightforward figuring out what data belonged together. But it took me a little bit of investigation. I also know by now that while thunderstorm icon works quite nicely, there's other icons available that the application hasn't yet implemented, so they won't work quite as perfectly. Similarly, I noticed with the application that when the sun doesn't go down at all in the summer, I would definitely expect that the sunrise and sunset times are the same. But it also kind of drills me into realizing that there's a typo, it's not sunshine, it should be sunrise, and it's no more easy to pass it by, and that I am pretty sure that this application that I've picked up online is by someone on the IST time zone, meaning somewhere in the Asia time zones, rather than a neighbor of mine here in Finland. So it gives me ideas of things that have been relevant for the creator, but also will limit what the application is capable of doing right now.

Investigating a little bit more, kind of, you know, being curious about the temperatures, very quickly going into code, I figure out that there's a very interesting thing going on here with the application turning temperatures to celsius. They come in as kelvin and I very quickly realize that this is due to the fact that when setting up this stub and the URL there, there is a tiny typo in that URL which ends up not giving me the celsius for this application right from the API ending up with this extra need of implementing the transformation separately. If I would see this with time handling it would be a much bigger concern, because time handling and all the exceptions related to time handling it's notoriously awful to create your own, so please use whatever you have and, well, temperature's likely the same. So with this example what I want to show you is that for the better teams and for pretty much any teams aspiring to be better. We start with our thing. In this case my thing, our thing, my team's thing, is the little application created in React, and it is benefiting from integrating into somebody else's publicly available API. It is running on top of some libraries, it's running on top of an operating system, and all of those are part of the things and integrations we need to be testing. And it is somehow also running on the side of other things that shouldn't have an impact on it, like browser extensions for example, that still might, there's many different things that might need to coexist. There is some kind of expectations from the users and the stakeholders.

Usually they're somehow reasonable and they're also part of the things that we need to consider and looking at all these things together that's how we approach testing. If you learn from the testing you have missed that you thought it should work differently, you had an expectation that it should work differently, but the empirical evidence shows, trying it out the program disagrees with you, the program is going to win, and there will be a different program before you can get the program to agree with you, so we go about making those changes based on the empirical evidence. We try to avoid the speculation build up, we need to address how reliable, how trustworthy our testing is and all in all we approach the applications, the systems, the integrated systems, the users needs and the users expectations, we approach them with curiosity and wanting to understand how things work.

Automation will play a part, but automation is driven by someone attending to that automation. So whenever automation fails it's kind of like an invitation for us to go and explore further and we are exploring to create the automation in the first place. When we have good questions about things that need to be addressed with the application like for example does it run on the long-term, automation might be very useful for us in understanding that. But it all starts from understanding our own architecture, our own responsibilities and caring for our users not on just the things we personally wrote but everything we intentionally and unintentionally integrated with.

Have a great time testing I hope you enjoy that. Welcome and hi thank you so much for your talk it was a really really great one. What do you think about the results we got gathered here? I was seeing these numbers and I was laughing here all by myself because I'm kind of a strong believer that there's no individual who's ever responsible for these kind of things. It's a collaboration and there's these layers where we're trying to catch things and the idea that almost 20 percent 19 percent would find one individual but it's not always the most usual suspect which is the testers. It's an interesting thing to see in the polls. In a year if someone does this it will be all in the majority category. It was a bit surprising to see how the numbers change on the side who's going to be blamed for as the answers came in and was like, okay, okay. Of course, as you mentioned most of the times I think people will turn back for the tester. But from 80 I think we have a good start and we can aim for 100 perfect. Maybe someone just playing with the buttons. Looking at it don't do that.

Great. I think we should get some questions then. Henry is asking, what is in your opinion the best tools for API testing and API testing automation? I work with a lot of different teams with a lot of different technologies. Right now, I'm of course personally, since I'm in a team with lots of Python, I'm convinced that Requests and Python is the best tool. Half a year ago I was working with a JavaScript and Java team and I was absolutely convinced it was SuperTest because that's what we were using back then. It's really about what the team is, but I'm much more inclined to code oriented tools, like something that will go nicely with the code instead of the kind of GUI oriented tools. So I believe that that's the general favorite that I go for. That's great. I really believe that, again, there is nothing to rule everything else, like only one solution, and it changes on the context that you are working on at the same time. When and how often do you perform exploratory testing? Post release? I know this is a topic very dear to you. Yeah, it is very dear to me. So I don't perform anything other than exploratory testing, I answer.

But automation for me is part of how I document my exploratory testing. And automation is my way of being able to explore some of the things that a regular human without use of automation couldn't. So for me, all of that is exploratory testing.

Maybe more interesting thing is kind of the split between feature testing and release testing, which is the two concepts that we use kind of on the system level. And about 99% of our work happens on the side of the features, we do very basic checks anymore at the release time. But this is something that we are still trying to move towards no manual work on the release time. Whereas, the thinking work needs to happen while we build the features.

Indeed, indeed. Mark was adding that we do exploratory testing mostly close to major release. We invite all people in the company, no matter if developers or salespeople to join it. But the challenge is always how to teach them to be a successful, of course, exploratory tester in almost no time. What's your approach in test tours as they try this till now? I haven't brought people together to do exploratory sessions like this for a few years now. When I bring people together, I bring a little smaller group, usually about a group of 10 at a time. And we do ensemble testing, meaning we share a single computer. And I get to facilitate so that everyone gets to learn how to do that exploratory testing and use everyone's skills on that. So having done ensemble testing and letting them run and roam free, they're likely to know a little bit more about how to do proper exploratory testing. You could give people ideas of tours or ask them to go and find whatever bugs they're worried about, investigate areas of their expertise, any of those are actually great start-up points. But it depends so much on what information do you consider valuable and giving that guidance to your team of explorers.

Oh, that's a good one. Indeed, we do something similar when we create pros and cons of this is is not what we ask for when we do some test parties too. And Martin said that you mentioned things your team co-exists with are also very important. How would you go about identifying the relevant things to co-exist with? By looking around kind of I have a mental checklist of looking at things that we depend on. Whatever is in the operating system, whatever is in the network, whatever is in the platform, whatever is in the neighboring teams. So kind of like being able to label all of those and figure out who those are, asking for architecture pictures and just pointing at a box on that architecture picture. That probably helps us a lot. Another maybe simpler way is to ask for an end-to-end test environment and just make sure that everything goes through that so then you don't have to understand all the dependencies, but you have to see that they run together in an integrated environment.

Yeah, I think that that's important. Thanks so much, Merit, again, both for the talk and your answer to these questions. We are very thankful to have you around. It was great.