Go Find What We May Have Missed!

Rate this content
Bookmark

Coming into software with an exploratory testing mindset is like coming to a multi-layer canvas with lots of information and an open ended task: find what we may have missed! This is the assignment for us all in software teams in our quest for quality.


Framing the search of how our system falls short of expectations is easier when we are able to see software from its user’s perspective. However, useful tests aren’t a collection of end-to-end tests we automate, but great tests to leave behind will decompose the testing problem differently. In this talk, we learn about using architecture as a filter in decomposing tests and look at an example of taking control over the API responses to test a React frontend.


Users don’t know or care if the problem is in the frontend and services your team provided if it fails to meet their expectations but you care. Granularity of feedback matters. Recognizing the same problems in incomplete scope - half-done features or only in frontend or APIs - is a skillset the software industry needs to be building.

This talk has been presented at TestJS Summit 2021, check out the latest edition of this JavaScript Conference.

FAQ

Maaret Pyhäjärvi is a principal test engineer at Vaisala with a 25-year career in software testing.

At Vaisala, Maaret Pyhäjärvi works as a principal test engineer, focusing on quality control for testing within various teams.

Maaret Pyhäjärvi usually spends six to twelve months with each team at Vaisala, aiming to improve their testing processes.

Maaret mentions two kinds of testing: artifact creation (automation or checklists for repeatable tests) and performance testing (exploratory, improvisational testing).

Maaret believes both are important because artifact creation provides specification, feedback, regression, and granularity, while performance testing offers guidance and serendipity.

Maaret approaches API testing by using code-oriented tools like Requests in Python or SuperTest in JavaScript, depending on the team’s technology stack.

Maaret uses exploratory testing continuously, integrating automation as part of documenting her exploratory process.

Ensemble testing is a method where a group shares a single computer to collaboratively test and learn, facilitated by Maaret to ensure effective exploratory testing.

Maaret identifies relevant dependencies by reviewing architecture diagrams, understanding the operating environment, and ensuring everything runs in an end-to-end test environment.

Maaret handles test data by mocking server responses, setting up stubs, and editing data specifics to create various test scenarios.

Maaret Pyhäjärvi
Maaret Pyhäjärvi
27 min
18 Nov, 2021

Comments

Sign in or register to post your comment.

Video Summary and Transcription

Maaret Pyhäjärvi, a principal test engineer at Vaisala, emphasizes the importance of balancing different types of testing to build better teams. Testing the application with different locations reveals potential issues with its behavior. The speaker highlights the significance of testing integrations and dependencies, including libraries and operating systems. They prefer code-oriented tools like Requests and Python for API testing. Exploratory testing is the only type of testing they perform, and they encourage others to participate in it as well.

1. Introduction to Testing at Vaisala

Short description:

Hi, my name is Maaret Pyhäjärvi, and for the last year and a half out of my 25-year career, I've worked at Vaisala as a principal test engineer. I go and assess the results that we're able to provide by staying within the team as one of the team members for a while. I repeat this from team to team, usually spending six to even twelve months within one team, with the idea of leaving things better after I'm gone and helping the teams grow in the way that they do testing.

Hi, my name is Maaret Pyhäjärvi, and for the last year and a half out of my 25-year career, I've worked at Vaisala as a principal test engineer. A nice way for me to frame what I do for my work is this idea that I'm quality control for the testing that is happening in our teams. I go and assess the results that we're able to provide by staying within the team as one of the team members for a while.

A lot of times I frame my assignment as go find at least some of the things others have missed. I repeat this from team to team, usually spending six to even twelve months within one team, with the idea of leaving things better after I'm gone and helping the teams grow in the way that they do testing. I have done this over my career quite many times for various different products and teams, one of them being this particular example here on the slide where I asked a developer of an API-based testing tool for permission to test their application and use it as training material in some of the conference talks that I've done. Then they gave a podcast interview saying I basically destroyed it in like an hour and a half. This is a common experience that developers tell of me with a smile usually, hopefully at least, on their face.

And it is usually related also to the fact that I have by that time also had the conversations on, you know, I didn't destroy the application. The only thing I might have destroyed is the illusion that was never the reality. So you might be very proud of your application. You might be doing already a good job. And there still might be things that you're missing. And your customers might not be telling you.

2. Testing: Artifact Creation and Performance

Short description:

In testing, there are two kinds: artifact creation and performance. Artifact creation provides specification, feedback, regression, and granularity. Performance testing offers guidance, serendipity, and helps discover unexpected problems. To build better teams, a balance of different test types is needed, including faking components and testing with real integrations. The system should be responsive, easy to use, and secure. An example application demonstrates the use of front-end and back-end testing with mock responses.

So in all of this work that I've done, I've kind of summed it up as a recipe for better teams. How do we go about finding the things we're missing? And we start with two kinds of testing. There's the testing that is kind of framed as an artifact creation, whether it creates automation or checklists for repeatable tests for later. And then we have the other kind, testing as a performance, kind of like improvisational theater, where you look at the application. Application sort of speaks to you. It's like your external imagination. And it makes you more creative. And whatever you learn, you can then turn into the artifact creation part of the testing. You need both of these sides. They give you very, very different things where the artifact creation style gives you specification, feedback, regression, and my absolute favorite, granularity, knowing based on the results that you're getting, what was the change that broke things, and from the logs, what is now broken, without having to spend multiple hours or even days analyzing your results before getting to the actual fixes. These are things that you can get from the artifact style of testing.

On the performance style, it gives you a little bit more vague things in many ways, but also, it gives you kind of like a guidance. You know, the direction, are we going to a better direction? Is this good? Is there still more feedback, more conversations to be had? Is there something where we need to build our understanding and improve the models? And my, again, absolute favorite, serendipity, lucky accident, meaning that sometimes, some of the problems that we need to find are two interesting combinations of all kinds of things we didn't think of, that we just need to give it time. So, there's a saying, a quote by Arnold Palmer, a famous golfer, that it's not just that he's lucky, it's just that he has been practicing. So, kind of like that's the general idea with this style of testing. So, framed from the sides, we need something in the middle for the better teams.

And the thing we need in the middle is, of course, different kinds of tests. Whether it comes from the point of view of creating artifacts, or whether it comes from the point of view of performing testing and thinking what kind of things we might still be missing, we probably will test against different levels of interfaces available in the system, and try making a balanced set of all the different shapes of tests, be they small, medium, large, unit service UI, or unit integration system, or end-to-end, whichever words you end up wanting to use. You probably also will not have just these different kinds of tests where you're basically then just kind of growing the scope of it. You also probably would like to have in those better teams some kind of ways of faking, mock, stub, spy, fakes, whatever you want to call it. Ways of faking either the service responses, the data, or any of the components that you want to leave out of the testing scenario so that you can have a focus feedback. But also, you want to test with the real integrations, again, because of serendipity you are most likely going to see something different there, and that is what your customer will end up using anyway, not the mocks that you have created. You'll probably have a foundation of functionality, but also the three other key things. It needs to be responding for the customer's requests fast enough. It needs to be easy enough to figure so that you know what to do with the application. And the disfavored users should have mechanisms keeping them away from your system so that whatever the business purpose is that the system serves, the information also is safe from other people causing you harm. So this is kind of the frame that I think that we need for the beta testers.

And I wanted to give you a small example of what typically applying something like this looks like on an application. I took a small example application which was created basically for using this or showing the idea that you can have a front-end and you can have a back-end and you can mock the back-end responses. So there's a very simple React front-end, very simple React app and the possibility of changing whether you are working against the actual or the mock server is already in the user interface.

QnA

Check out more articles and videos

We constantly think of articles and videos that might spark Git people interest / skill us up or help building a stellar career

Network Requests with Cypress
TestJS Summit 2021TestJS Summit 2021
33 min
Network Requests with Cypress
Top Content
Cecilia Martinez, a technical account manager at Cypress, discusses network requests in Cypress and demonstrates commands like cydot request and SCI.INTERCEPT. She also explains dynamic matching and aliasing, network stubbing, and the pros and cons of using real server responses versus stubbing. The talk covers logging request responses, testing front-end and backend API, handling list length and DOM traversal, lazy loading, and provides resources for beginners to learn Cypress.
Testing Pyramid Makes Little Sense, What We Can Use Instead
TestJS Summit 2021TestJS Summit 2021
38 min
Testing Pyramid Makes Little Sense, What We Can Use Instead
Top Content
Featured Video
Gleb Bahmutov
Roman Sandler
2 authors
The testing pyramid - the canonical shape of tests that defined what types of tests we need to write to make sure the app works - is ... obsolete. In this presentation, Roman Sandler and Gleb Bahmutov argue what the testing shape works better for today's web applications.
Full-Circle Testing With Cypress
TestJS Summit 2022TestJS Summit 2022
27 min
Full-Circle Testing With Cypress
Top Content
Cypress is a powerful tool for end-to-end testing and API testing. It provides instant feedback on test errors and allows tests to be run inside the browser. Cypress enables testing at both the application and network layers, making it easier to reach different edge cases. With features like AppActions and component testing, Cypress allows for comprehensive testing of individual components and the entire application. Join the workshops to learn more about full circle testing with Cypress.
Test Effective Development
TestJS Summit 2021TestJS Summit 2021
31 min
Test Effective Development
Top Content
This Talk introduces Test Effective Development, a new approach to testing that aims to make companies more cost-effective. The speaker shares their personal journey of improving code quality and reducing bugs through smarter testing strategies. They discuss the importance of finding a balance between testing confidence and efficiency and introduce the concepts of isolated and integrated testing. The speaker also suggests different testing strategies based on the size of the application and emphasizes the need to choose cost-effective testing approaches based on the specific project requirements.
Playwright Test Runner
TestJS Summit 2021TestJS Summit 2021
25 min
Playwright Test Runner
Top Content
The Playwright Test Runner is a cross-browser web testing framework that allows you to write tests using just a few lines of code. It supports features like parallel test execution, device emulation, and different reporters for customized output. Code-Gen is a new feature that generates code to interact with web pages. Playwright Tracing provides a powerful tool for debugging and analyzing test actions, with the ability to explore trace files using TraceViewer. Overall, Playwright Test offers installation, test authoring, debugging, and post-mortem debugging capabilities.
Get rid of your API schemas with tRPC
React Day Berlin 2022React Day Berlin 2022
29 min
Get rid of your API schemas with tRPC
Today's Talk introduces TRPC, a library that eliminates the need for code generation and provides type safety and better collaboration between front-end and back-end. TRPC is demonstrated in a Next JS application integrated with Prisma, allowing for easy implementation and interaction with the database. The library allows for seamless usage in the client, with automatic procedure renaming and the ability to call methods without generating types. TRPC's client-server interaction is based on HTTP requests and allows for easy debugging and tracing. The library also provides runtime type check and validation using Zod.

Workshops on related topic

Designing Effective Tests With React Testing Library
React Summit 2023React Summit 2023
151 min
Designing Effective Tests With React Testing Library
Top Content
Featured Workshop
Josh Justice
Josh Justice
React Testing Library is a great framework for React component tests because there are a lot of questions it answers for you, so you don’t need to worry about those questions. But that doesn’t mean testing is easy. There are still a lot of questions you have to figure out for yourself: How many component tests should you write vs end-to-end tests or lower-level unit tests? How can you test a certain line of code that is tricky to test? And what in the world are you supposed to do about that persistent act() warning?
In this three-hour workshop we’ll introduce React Testing Library along with a mental model for how to think about designing your component tests. This mental model will help you see how to test each bit of logic, whether or not to mock dependencies, and will help improve the design of your components. You’ll walk away with the tools, techniques, and principles you need to implement low-cost, high-value component tests.
Table of contents- The different kinds of React application tests, and where component tests fit in- A mental model for thinking about the inputs and outputs of the components you test- Options for selecting DOM elements to verify and interact with them- The value of mocks and why they shouldn’t be avoided- The challenges with asynchrony in RTL tests and how to handle them
Prerequisites- Familiarity with building applications with React- Basic experience writing automated tests with Jest or another unit testing framework- You do not need any experience with React Testing Library- Machine setup: Node LTS, Yarn
How to Start With Cypress
TestJS Summit 2022TestJS Summit 2022
146 min
How to Start With Cypress
Featured WorkshopFree
Filip Hric
Filip Hric
The web has evolved. Finally, testing has also. Cypress is a modern testing tool that answers the testing needs of modern web applications. It has been gaining a lot of traction in the last couple of years, gaining worldwide popularity. If you have been waiting to learn Cypress, wait no more! Filip Hric will guide you through the first steps on how to start using Cypress and set up a project on your own. The good news is, learning Cypress is incredibly easy. You'll write your first test in no time, and then you'll discover how to write a full end-to-end test for a modern web application. You'll learn the core concepts like retry-ability. Discover how to work and interact with your application and learn how to combine API and UI tests. Throughout this whole workshop, we will write code and do practical exercises. You will leave with a hands-on experience that you can translate to your own project.
Detox 101: How to write stable end-to-end tests for your React Native application
React Summit 2022React Summit 2022
117 min
Detox 101: How to write stable end-to-end tests for your React Native application
Top Content
WorkshopFree
Yevheniia Hlovatska
Yevheniia Hlovatska
Compared to unit testing, end-to-end testing aims to interact with your application just like a real user. And as we all know it can be pretty challenging. Especially when we talk about Mobile applications.
Tests rely on many conditions and are considered to be slow and flaky. On the other hand - end-to-end tests can give the greatest confidence that your app is working. And if done right - can become an amazing tool for boosting developer velocity.
Detox is a gray-box end-to-end testing framework for mobile apps. Developed by Wix to solve the problem of slowness and flakiness and used by React Native itself as its E2E testing tool.
Join me on this workshop to learn how to make your mobile end-to-end tests with Detox rock.
Prerequisites- iOS/Android: MacOS Catalina or newer- Android only: Linux- Install before the workshop
API Testing with Postman Workshop
TestJS Summit 2023TestJS Summit 2023
48 min
API Testing with Postman Workshop
Top Content
WorkshopFree
Pooja Mistry
Pooja Mistry
In the ever-evolving landscape of software development, ensuring the reliability and functionality of APIs has become paramount. "API Testing with Postman" is a comprehensive workshop designed to equip participants with the knowledge and skills needed to excel in API testing using Postman, a powerful tool widely adopted by professionals in the field. This workshop delves into the fundamentals of API testing, progresses to advanced testing techniques, and explores automation, performance testing, and multi-protocol support, providing attendees with a holistic understanding of API testing with Postman.
1. Welcome to Postman- Explaining the Postman User Interface (UI)2. Workspace and Collections Collaboration- Understanding Workspaces and their role in collaboration- Exploring the concept of Collections for organizing and executing API requests3. Introduction to API Testing- Covering the basics of API testing and its significance4. Variable Management- Managing environment, global, and collection variables- Utilizing scripting snippets for dynamic data5. Building Testing Workflows- Creating effective testing workflows for comprehensive testing- Utilizing the Collection Runner for test execution- Introduction to Postbot for automated testing6. Advanced Testing- Contract Testing for ensuring API contracts- Using Mock Servers for effective testing- Maximizing productivity with Collection/Workspace templates- Integration Testing and Regression Testing strategies7. Automation with Postman- Leveraging the Postman CLI for automation- Scheduled Runs for regular testing- Integrating Postman into CI/CD pipelines8. Performance Testing- Demonstrating performance testing capabilities (showing the desktop client)- Synchronizing tests with VS Code for streamlined development9. Exploring Advanced Features - Working with Multiple Protocols: GraphQL, gRPC, and more
Join us for this workshop to unlock the full potential of Postman for API testing, streamline your testing processes, and enhance the quality and reliability of your software. Whether you're a beginner or an experienced tester, this workshop will equip you with the skills needed to excel in API testing with Postman.
Monitoring 101 for React Developers
React Summit US 2023React Summit US 2023
107 min
Monitoring 101 for React Developers
Top Content
WorkshopFree
Lazar Nikolov
Sarah Guthals
2 authors
If finding errors in your frontend project is like searching for a needle in a code haystack, then Sentry error monitoring can be your metal detector. Learn the basics of error monitoring with Sentry. Whether you are running a React, Angular, Vue, or just “vanilla” JavaScript, see how Sentry can help you find the who, what, when and where behind errors in your frontend project. 
Workshop level: Intermediate
Testing Web Applications Using Cypress
TestJS Summit - January, 2021TestJS Summit - January, 2021
173 min
Testing Web Applications Using Cypress
WorkshopFree
Gleb Bahmutov
Gleb Bahmutov
This workshop will teach you the basics of writing useful end-to-end tests using Cypress Test Runner.
We will cover writing tests, covering every application feature, structuring tests, intercepting network requests, and setting up the backend data.
Anyone who knows JavaScript programming language and has NPM installed would be able to follow along.