Panel Discussion: Future of Web Testing

Hey, hey. Hey, hey, I'm really happy to be here. I'm thrilled. I'm super excited to kick off this panel about the future of web testing. And I'm so excited because of two reasons. One, a lot of things, great things happening now in this field. Just to give you an example, I checked today in GitHub and there are already 17,000 repos dealing only with testing in JavaScript. 4,000 of them were written in the last year. So, so many things and innovation happening now in testing. The second reason is because I probably have the ideal panel to discuss all of these trends. Really some of the most influential, the people who are building this innovative tool are here with me.

So, how about we start with a quick intro. Let's start with you, Nancy. Would you like to introduce yourself? Sure. I'm really excited to be here. My name is Nancy and I am a solutions architect working at a company called Rango.io. We're a tech consulting agency. So, worked with a lot of clients in the past, enterprise and startups, often times with their testing strategy. Cool. Great to have you, Nancy.

Kent, Zidane, I'm not sure you even need to introduce Kent, but we will, Kent, please tell us about yourself. Yeah, sure. Thanks. Super happy to be here. I love the conferences put off by Git Nation. They do a great job. So yeah, I created testingjavascript.com, where I can teach you everything that I know about testing. It's awesome. So take a look at that. And actually, there's a discount coupon just for TestJS Summit.

Hey, I'm really happy to be here. I'm thrilled. I'm super excited to kick off this panel about the future of web testing and I'm so excited because of two reasons. One, a lot of things great things happening now in this field. Just to give you an example, I checked today in GitHub and there are already 17,000 repos dealing only with testing in JavaScript. 4,000 of them were written in the last year. So many things and innovation happening now in testing. The second reason is because I probably have the ideal panel to discuss all of these trends, some of the most influential people are building this innovative tool here with me. So how about we start with a quick intro? Let's start with you, Nancy, would you like to introduce yourself? Sure, I'm really excited to be here. My name is Nancy and I am a solutions architect working at a company called Rango.io. I also work for a tech consulting agency, so you know, worked with a lot of clients in the past, enterprise and startups, oftentimes with their testing strategy. Cool, great to have you, Nancy. Kent, Siddharth, I'm not sure we even need to introduce Kent, but we will. Kent, please tell us about yourself. Yeah, sure. Thanks. Super happy to be here. I love the conferences put off by Git Nation. They do a great job. So yeah, I created testing-javascript.com, where I can teach you everything that I know about testing. It's awesome. So take a look at that. And actually, there's a discount coupon just for TestJS Summit. So I'll share that in the discord. And I also created TestingLibrary, which is kind of a funny name for a library. But it helps you test DOM-related stuff, so anything that has to do with the DOM, anywhere you can find the DOM. And actually, there are other implementations for React Native and stuff like that, too. But it just gives you utilities that you need to test UI. That's the main idea behind that. So that's what I do.

So I'll share that in the Discord. And I also created Testing Library, which is kind of a funny name for a library, but it helps you test DOM-related stuff, so anything that has to do with the DOM, anywhere you can find the DOM. And actually, there are other implementations for React Native and stuff like that, too. But it just gives you utilities that you need to test UI. That's the main idea behind that.

So that's what I do. Cool. And we're about to discuss, of course, Testing Library very soon.

Oren, good morning, San Francisco. Hi, everyone. Pleasure to be here. Super excited to be here with everyone. My background is mostly building products for developers the last 22 years. I'm guessing the standard you know is the visual validations. I was one of the founding members, weeks an early developer, and now the founder of the last six years, founder of Testing. We focus on AI helping test automation.

That's great. And we're also going to discuss testing. Jason, hi. Everyone, I'm happy to be here. I'm Jason Palmer. I work at Spotify for the last almost 10 years as an engineer and now a PM. I'm also a core contributor to Jest. I created Jest JUnit. If you're running Jest in a CI environment, there's a pretty good chance you're using that. Amazing. Great. So we're about to discuss now a bunch of topic, a bunch of hot trends in testing. And each of the panelists is going to share his thoughts and experience about each of these hot trends. So how about we start with the CI world? And there is a new giant player, GitHub, now playing in the CI world in the past two years.

And funnily enough, I just checked today, and each of the big vendors have approximately a hundred extensions or actions, some call them orbs. GitHub, in two years, already have 7,000 GitHub Actions. So it's growing crazily. And my question to you, let's start with you, Ken, is this just another big giant eating the market, or are they doing things differently? Do they provide new capabilities, opportunities for us developers?

You know, years ago I did a survey, not like a survey, but I took a survey of all of the CI services that I thought would be good for the company that I was working with at the time. And I tried a ton of them, and I ended up with SnapCI, which is no longer a thing, which is kind of sad. There's GoCD now as their thing, and that's what I used for my company. But then for open source I was all in on TravisCI and it was, it served me really, really well for years. But I don't want to get into this too much, but Travis kind of changed the way that they do things in a way that made it very difficult for open source. And so I've been totally moving everything over to GitHub actions. I've been kind of thinking about it anyway. And my goodness, it's such a nice experience. So there's a reason that I think so many people are going over to GitHub for managing their GitHub actions. There can be some concern with the monopolistic approach, I guess, but I don't know. I'm just trying to ship stuff. And GitHub Actions makes it really easy to do that. So I like it.

Can you share one example of any delightful experience, something that just helped you specifically to ship? Yeah. Well, you mentioned the 7,000 different GitHub custom actions that you can use. I didn't have to build any custom or, yeah, custom action of any kind. It was just putting together all these different ones to make things come together in a really nice way. I wouldn't say that it was super straightforward. There are definitely some nuances and intricacies and things like that. But once you have it set up, it's actually pretty understandable what you have. And I think that, yeah, just the sheer amount of actions that you have at your disposal is super, super helpful.

Yeah. And Nancy, I'm curious to hear your thoughts on this matter. Yeah, I think it's actually super cool. I really like the approach GitHub went with, instead of just focusing on being another CI tool and just focusing on their infrastructure, they really took that open source approach, really encouraging and fostered the community to contribute, and focusing on that marketplace aspect rather than it being an afterthought. I think that's where it's really powerful. And that's probably why we've seen so much growth there so quickly.

And really now GitHub is not limited by how many developers are putting on to really build out their CI or really understanding the user's need because it comes organically, right? They're allowing the community to kind of dictate where this goes. And I think that's super exciting to see. I think it's hard to predict where it will go in the future, especially, you know, a lot of the clients I worked with in the past. Like moving, switching over to a whole new infrastructure is just a huge undertaking and oftentimes not the top priority and just not worth the effort right off the bat.

Yeah. Interesting. Both of you mentioned that the marketplace is the thing, like up front, they build it for the marketplace and this is the core and not some extension like it was in previous CI vendors. Yeah, yeah, sensible. And Oren and Jason, feel free to jump in whenever and everyone. We are a spontaneous here and so it's not- One thing I really like about GitHub Actions, I guess, is that it's kind of one of the CI providers that's taking the like Docker based approach, which is really nice. And it's kind of a new wave that I've seen, I guess, in the last few years. It definitely makes it easier for folks to be able to like reproduce CI steps locally and kind of craft a decent CI environment locally and make sure that it works. And then you kind of, it's a lot less toil basically than you might have with other CI environments.

Yeah, yeah. So bottom line, I think anyone should evaluate the GitHub, at least evaluate it as your next CI. And now moving to the next topic, which is testing library. Well, probably everyone here heard about react testing library. I learned recently that it has sisters, which is X testing library for Cypress, for Puppeteer, for almost any UI framework. And if you're not sure how popular it is, recently the state of JS big survey was published. And people chose a testing library above Jest and Mocha, as their most popular testing framework. Of course, it's not the same thing. But it's, I mean, it was quoted as the most trendy testing library of the past year. And surprisingly, we have the creator Ken. Question for you. So, I'm, I know Cypress. I can do things for this with Cypress. I know to build stuff. And I'm going to start now a new project. Why should I use, for example, Cypress testing library and not Cypress? Essies. Yeah, well, I mean, I think you should use Cypress for sure.

The testing library is a collection of utilities that you use within testing frameworks. It's not a replacement for Jest or Mocha or Cypress, but rather a utility that you tack onto those things. With Cypress, there are built-in commands that you can use instead of what you get from Cypress testing library. However, adding Cypress testing library gives you more confidence, which is the ultimate goal. It offers a nice way to query and interact with the output DOM, making it easier to read and write tests. By using testing library, you can increase the level of confidence in your tests.

As a developer, the main motivation for me to use testing library across different UI frameworks is to reuse my knowledge of the syntax I'm already familiar with and to follow best practices. Testing library encourages testing like the user and focusing on the behavior rather than the internals. You can use testing library wherever you can find a DOM, which is a great advantage.

In summary, the testing library is not a replacement for testing frameworks but a utility that enhances them. It provides a way to query and interact with the DOM, increasing confidence in tests. Using testing library promotes best practices and encourages testing from the user's perspective. It can be used across different UI frameworks and is a valuable tool for developers.

Whenever I commit code, it should get shipped. And for me to be able to do that, I need to have confidence. And this is what testing library offers is a really nice way to not only query, in Cypress, it's pretty much just for ... Elsewhere, it gives you not only queryability, but also interactivity with that output DOM. And the reason I say you don't need that for Cypress is because Cypress has great interactivity functionality already. So it just adds the ability to query your document really, really well. And not only does it make it easier to read and write those tests, but it also, you end up with more confidence because of the types of things that you're writing in your tests. And so that's the reason. It just helps you increase the level of confidence. That's the most important thing.

So just to make it even clearer. So the main motivation for me as a developer to use testing library across different UI framework is mostly to reuse my knowledge that the syntax that I'm already familiar with. Or it kind of pushes me towards best practices. Being like hey, test like the user, don't focus on the internals.

What exactly? Yeah, exactly that. So it's, the fact that you can use testing library wherever you can find a DOM is really awesome thing. But that's not why you use it. The reason that you use it is because it helps separate your tests from implementation details, which is why you get so much confidence out of it. And originally when I wrote testing library, it was actually React testing library specific to React. And then I realized that there was just a very thin integration layer between React and the rest of the DOM. And so I extracted all of that. And then we made Angular testing library and Vue testing library and Cypress testing library. There's a test cafe testing, like anywhere you find a DOM, you can use testing library for it. And that actually is a huge benefit. So you can have your unit and integration tests that are maybe running in Jest or wherever, and they're all hitting the DOM. And then you can also have your end-to-end test. And they look very similar from the way that you're querying and interacting with the DOM, which I think is a huge benefit too.

Sounds like a great opportunity. And I mean, can you recommend resources, places to learn more about testing library? Yeah. TestingJavascript.com is a pretty good place to learn it. And I did share in the discord a short URL, maybe I'll share it again, because people are pretty active in the discord, but yeah, for a 25% discount to testingJavascript.com. But of course, that's not the only place to learn. Like you said, it's very, very popular. There are tons and tons of courses that you can pay for, and also videos that you can watch on YouTube. I've given many talks. I've got lots of free content on my blog. And then the docs are really good. So you can go to testing-library.com and learn more. I personally really like this approach of test from the user perspective, almost as production, and I think that this mindset also inspired many now of the testing strategy of the organization level, testing strategy are being disrupted. So let's play a small role game.

Let's say that you are the chief testing officer or the CTO and I'm your developer, and I'm working traditionally. Well, I'm writing unit tests, a lot of unit tests. Then after three weeks, when I have the version working with 100% test coverage, hoo-hoo, then me and the QA person starts writing some integration tests and then some end-to-end testing, then manual testing my version and put in production.

Now we have a meeting and we do our planning for 2021. What exactly, and Oren, let's start with you, what exactly should I change in this approach for the next year? So when you start planning, I think first of all you need to understand what you have and how do you want to go. We all know the pyramid, right, and we all say this is the amount of tests that we have, unit tested, unit tests is not enough, and deciding on over there, everyone tries to reach the hundred percent test coverage. Code coverage is really hard, it's really hard to get to 100% code coverage and sometimes it's not worth it.

So you have those different layers and you need to decide how to start. And sometimes, by the way, if you don't have anything, sometimes it's even nicer to go actually against the tradition and actually start from the top. End-to-end testing and the integration tests give you high coverage very, very fast. And they give you a lot of confidence, as you said, close to production to understand, or you can even test in production to understand what you got. The other thing that you need to make sure is that you don't neglect those other layers and you need to make sure that you keep investing in them because they give you the feedback faster. So the end-to-end are super critical, they give you high confidence, and high coverage in seconds, where they usually, the challenge over there is what Kent said, and I totally agree, is the trust.

You need to trust your tests. If you have tests that are running and you don't trust them, I don't want them. I don't, I prefer 10 tests, they give you 20% coverage and I trust it, then 100% that it's flaky, and when the CI would go, I would say, oh, it failed again, I don't know why. And then what happens is people take tests out, and they say, I don't trust it, let's take it out because I want to release, or they're starting this thing called rerun, you know what, let's run it again, and again, and again, and again, let's rerun it, until everything passes. And that, of course, it covers a lot of bugs that you don't catch because of those flakiness, you might have a real bug, people think that bugs are only deterministic bugs, that you say, hey, I caught this, and this will always reproduce, the most challenging bugs are the one that don't reproduce all the time, when you have that one out of 10, it's gonna happen, and you can ignore that. And I keep telling people, if you had someone, a robbery, you can't say, the robbery is not gonna go up in the same place again, you need to make sure, how do I find the person that did this, how do I find it, and that means how they had all the information, and that brings us to the root cause analysis, understanding, you need to invest in the fact that you'll know when something does break, what happened, do I get, is it, we started with the screenshot, then, you know, the dom, I wanna see the dom, the network request, the console logs, I want to have all the information I need, so that when something does happen, I know I can pinpoint and say, okay, this is it, and of course, the more you go up the dom, it's harder to say, to pinpoint, and as you go more to the, more even toward the unit tests, you can say, oh, here's the function that, the method that went wrong, this is the state. So here gives us, up to here, we see a very, we have a lot of coverage, slow response, that means it's harder running on every commit, and going down, making sure that we have the amount of tests in each, in each part, to know exactly as fast as possible what went wrong.

Yeah, yeah, that makes sense, the wider your tests are, then you need more to invest in setups that prevent flakiness and, yeah, makes tons of sense. Jason, many of your work is related to flakiness, so I'm curious about your overall thought, but also specifically to this, flakiness in integration tests.

Yeah, flakiness is a big problem, but it's also kind of a fascinating field as well, something I'm sort of weirdly interested in. I can sort of say at Spotify, we had kind of a large flakiness problem. That's kind of how I got introduced into working and testing to begin with. The library that I wrote, JestJUnit, the entire reason I wrote it was because I was working on the Spotify desktop client at the time, and we were trying to tackle test flakiness, trying to reduce it significantly. And so the first part in doing that is just basically knowing what tests have run, which one failed, how fast are they, et cetera, et cetera, putting them in a database and sort of trending these things over time. This is kind of the fundamental starting point for just understanding what is flaky and then how are you going to address this. So it's a fascinating field. There's probably tons I could talk about, way more than this conference.

But I guess kind of getting to the original question, something I feel that's important to say is that far too many organizations that I've seen, even now, but particularly in the past, don't really recognize the fact that testing and sort of quality in general actually speeds up teams. So there's this concept in agile called sustainable pace, which I don't know, I just don't see a lot of teams actually following this or really understanding this. A lot of teams are really moving very fast, as quickly as they can. And this often means writing no tests or writing fewer tests. But if you can move just a tad bit slower, and focus on quality and keep that quality high, this actually pays dividends over years and years and years. So this is something that I think is important, not just to developers, but companies as a whole.

I think that one of the big things we actually see is actually that developers do own quality. You mentioned, hey, there's a QA person that's going to say, hey, I think it's becoming more and more that developers own the quality. And they need to say, hey, I feel okay with releasing on a Friday, because that means that you feel secure enough, you're saying, hey, I trust this test, it's not someone else's job to make sure everything works. That's my job. That's my responsibility. Sure. Sometimes it's a good idea to release on Fridays. I would like to add that I feel like quality is not, you know, moving it to having developers be responsible for that is not enough. It's I feel like it's an organizational thing that everyone needs to be aligned on. And I think that's where your quality strategy really becomes impactful, is when it actually helps your whole organization understand what quality means. And that's what enables you to actually move forward and be able to have those confidence in those tasks. Um, which is why I think test coverage isn't always reliable to do that, because it's often just express as number of tasks or a percentage. And it doesn't really at all speak to what they actually do, especially for those people who are non technical, or really remove from the day to day code. Like what are they? What does this test case actually mean in terms of, you know, how does this safeguard my business, and it doesn't say any of that. So a lot of the things that I think is actually important to consider, besides test coverage, or what type of test you're writing is actually be able to identify, you know, features and flows are actually really high impact areas, or high touch areas for your users, whether that's like a dashboard, or a homepage or checkout flow, and really concentrating your testing efforts there, focusing on a few high impact BTE tests. That's what's actually going to give you confidence, you know, understanding your users, are they on Chrome, are they on, you know, desktop versus mobile is going to help narrow down what, you know, what browsers you're actually going to be testing on? Does it make sense to run all your automation tests through every single browser? So a lot of those things, I think, is more important to think about than just the number of tests.

And then, again, that part about communicating about your quality strategy and having everyone within the team and company understand that, I think it's often overlooked, but it's super critical. Yeah. Yeah. And there is a repeating concept here. Start from the user, focus on the user. Okay. We have five minutes left. So, I think that it's absolutely now a significant trend to move from Pyramid to something that is more Daimon or Trophy or Honeycomb.

Start with integration test or end-to-end. By the way, some of the earlier TDD books actually also recommend this. Start with wider tests and then do your unit and TDD stuff. But now as a developer, we all know praise integration test. When should I write now unit test? And also I keep hearing about the new thing, component test. Kenneth, I'll direct first the question to you. What exactly is component test and what is the new role of unit tests?

You know, it all comes down to confidence and how much confidence you can get out of things. If you unit test, let's say you get 100% unit test coverage, you can still have so many integration problems. And so you get a lot of problems there. So let's say, all right, never mind, we won't do 100%. Let's do like a mix here. And then as you start doing more and more integration tests, you realize that you don't need as many unit tests, because you're already covering that code from your integration. And so the, where I, when you keep on down this path, where you end up finding most of your unit tests are in those maybe like pure functions that are doing complicated logic. And those typically they run really fast and they're pretty straightforward to write and maintain. And that's where I find myself doing most of my unit tests. I actually don't find myself doing a whole lot of module mocking in those types of tests either. So I'm pretty much doing component or integration tests. The component is like React component or Angular component, whatever. Integration would be like on the node level. You're integrating a bunch of your modules and stuff. And so yeah, I find myself doing mostly that sort of testing. And then I fill in the gaps for like these hardcore computational or like these algorithmic things that are difficult to cover with the broad brush of a integration test. Yeah. So, there's still a room for a unit test and this should be stated clearly.

A question to back stage, we're almost out of time but we have a lot of fascinating topics still uncovered. So, if we can get five minutes more, please let me know. I'm moving to the next topic which is a promise for years that it's going to affect the testing world and maybe this is the year where it's happening. I'm referring to AI-based testing. Oren, you are the chief, the testing. The testing, I know, is a lot about AI.

Question, I mean, can you explain really in layman terms what does it mean to me, really in simple words, features. How can it augment or help testing? We're trying, I think what everyone is trying to get to the same thing, to trust your test. You can look at it at the authoring the test to have as much coverage as possible. I think Nancy said it in an amazing way. Wait a second. How do you define that? How do you see that? I hope that one day we'll get to talk about not just code coverage but also user coverage. Not just the shift left. Shift right. What do we know about production? What do people interact with? What do we not? First of all, AI can help with the authoring. Can we generate, can we look in production and generate automatically 1,000 end-to-end tests in two days just by looking at what's going on in production? Can we filter out of that? What are the tests we want to keep as an end-to-end test? And generating those integration component or unit tests, can we remove the flakiness? Right now things that are basic in every platform was, hey, was one CSS selector. There's one way to find an element and click on it. A machine can look at thousands of them. Can we do that automatically and reduce the flakiness if we understand that a machine can find an element better than us? Yeah. Like identify selectors from previous records that lead more to flakiness. Exactly. And improve that over time. Yeah, makes tons of sense. I would love to discuss it for at least 10 minutes more, but we are just almost out of time and I want to cover the last very important topic.

Story, Jason and Nancy only one minute to cover this. There is a big bloom of tools that related to network interception, network mocking, a lot of new capabilities. You specifically, Nancy, wrote one of these tools. Can you first explain to us why it first record the network? I wish I have more time to talk about this because I think it is super exciting. I think the biggest thing that the network mocking really solves is it takes the most difficult part out of being able to write integration tests. I wish I can elaborate what I mean by integration test because I think it is one of those testing that is super misunderstood and no one really aligns on what it does but I think it really solves that flaky issue that Jason was talking about, you can really focus on things that you can actually visually see, things that are as close to possible as what the users can interact with. I feel like network mocking really allows you to be able to do that without making it super slow and flaky and cause all of those issues. The plugin specifically that I worked on actually tags on what is out of the box already from Cypress, a lot of you who do use Cypress knows it comes with a lot of mocking capabilities and it kind of just tags on top of that to really solve for I think the biggest issue with all of this which is how do we actually manage our mocks, right? Mocking is obviously the number one issue but number two once you're able to mock how do you manage hundreds of, you know, files of mocking that needs to change often and you know that's where you know the plugin Cypress AutoRecord that I wrote really helps to address that but again I love to continue this discussion I know we're out of time. Yeah happy to talk with anyone on discord. Yeah I took a look at it and it looked amazing like re-verify that your integration and practice are a excuse things they are. We've got one last minute grace jason I'll really be glad to hear your take on the network interception monitoring. Yeah sure, just really quickly I mean I think it's an important part of testing you have end-to-end tests and the most typical reason why they're flaky or why they're slow is going to be network requests so if you can kind of take those out of the equation especially if you have something like VCR where you can automatically generate stable mocks, you turn these end-to-end tests into something that is far less flaky and far more performance and basically is testing the same thing using hopefully the same system so there's a lot of benefits to it I would encourage you to look at it especially Nancy's plugin there is a pretty great example. Yeah, thank you Jason and unfortunately we're out of time so really many thanks to all of you for this great insight should I try to summarize it in one sentence is production user, production user leave the internals to the end start from the chaos of production the network from the user from the UX, this is where most of the tooling now innovate and this is where testing should start from and then if you have something complex inside then well we have the traditional, traditional unit and other tools many thanks for all of you for being with me today.

Cool. And we're about to discuss, of course, Testing Library very soon. Oren, good morning, San Francisco. Hi, everyone. Pleasure to be here. Super excited to be here with everyone. My background is mostly building products for developers in the last 22 years. I'm guessing the standard you know is the applicative, the visual validations. I was one of the founding members, Wix and early developer and now the founder of the last six years, founder of Testing. We focus on AI, helping test automation. That's great. And we're also about to discuss, we're also going to discuss testing. Jason, hi. Hey everyone. I'm happy to be here. I'm Jason Palmer. I work at Spotify for the last almost 10 years as an engineer and now a PM. I'm also a core contributor to Jest and I created Jest JUnit. So if you're running Jest in a CI environment, there's a pretty good chance you're using that. Amazing. Great, so we're about to discuss now a bunch of topic, a bunch of hot trends in testing and each of the panelists is going to share his thoughts and experience about each of these hot trends. So how about we start with the CI world? And there is a new giant player, GitHub, now playing in the CI world in the past two years. And funnily enough, I just checked today and each of the big vendors who operate there for years have approximately 100 extensions or actions, some call these orbs. GitHub in two years already have 7,000 GitHub actions, so it's growing crazily. And my question to you, let's start with you Kent. Is this just another big giant eating the market or are they doing things differently? Do they provide new capabilities, opportunities for us, developers? You know, years ago I did a survey, not like a survey, but I took a survey of all the CI services that I thought would be good for the company that I was working with at the time. I tried a ton of them and I ended up with SnapCI which is no longer a thing, which is kind of sad. There's GoCD now as their thing. And that's what I used it for my company. But then for open source I was all in on TravisCI and it served me really, really well for years.

I don't want to get into this too much, but Travis kind of changed the way that they do things in a way that made it very difficult for open source. So I've been totally moving everything over to GitHub Actions. I've been kind of thinking about it anyway. My goodness, it's such a nice experience. So there's a reason that I think so many people are going over to GitHub for managing their GitHub Actions. There can be some concern with the monopolistic approach I guess, but I don't know. I'm just trying to ship stuff and GitHub Actions makes it really easy to do that. So that's, I like it.

Yeah. Can you share one example of any delightful experience? Something that just helped you specifically to ship? Yeah. Well, you mentioned the 7,000 different GitHub Custom Actions that you can use. I didn't have to build any custom or yeah, custom action of any kind. It was just like putting together all these different ones to make things come together in a really nice way. It was just very, I wouldn't say that it was super straightforward. There are definitely some nuances and intricacies and things like that. But once you have it set up, it's actually pretty understandable what you have. And I think that, yeah, just the sheer amount of actions that you can, you have at your disposal is super, super helpful.

Yeah. Yep. And Nancy, I'm curious to hear your thoughts on this matter. Yeah, I think it's actually super cool. I really like the approach kind of GitHub went with. Instead of just focusing on being another CI tool and just focusing on their infrastructure, they really took like that open source approach, really encouraging and fostered the community to contribute and focusing on that marketplace aspect rather than it being sort of an afterthought. And I think that's where it's really powerful. And that's probably why we seen so much growth there. Quickly, and really now sort of GitHub is not, you know, limited by, you know, how many developers are putting on to really build out their CI or really understanding the users need because it comes organically, right? They're allowing the community to kind of dictate where this goes. And I think that's super exciting to see. I think it's hard to predict where it will go in the future. Especially, you know, a lot of the clients I've worked with in the past, like moving, switching over to a whole new infrastructure is just a huge undertaking, and oftentimes not the top priority and just not worth the effort right off the bat. Yeah, interesting.

Both of you mentioned that the marketplace is the thing, like upfront, they build it for the marketplace, and this is the core and not some extension like it was in previous CI vendors. Yeah, yeah, sensible. And Oren and Jesse, feel free to jump in whenever and everyone, we are spontaneous here.

So one thing I really like about GitHub Actions, I guess, is that it's kind of one of the CI providers that's taking the, like, Docker-based approach, which is really nice. It's kind of a new wave that I've seen, I guess, in the last few years. It definitely makes it easier for folks to be able to reproduce CI steps locally, and craft a decent CI environment locally, making sure that it works. It's a lot less toil than you might have with other CI environments.

Well, probably everyone here heard about React Testing Library. I learned recently that it has Sisters, which is X testing library for Cypress, for Puppeteer, for almost any UI framework. People chose testing library above Jest and Mocha as their most popular testing framework in the State of JS big survey. It was called the most trendy testing library of the past year. We have the creator here. Ken, question for you: Why should I use Cypress testing library and not Cypress for a new project?

Of course, it's not the same thing. But it's, I mean, it was called as the most trendy testing library of past year. And surprisingly, we have the creator here. Ken, question for you. So I know Cypress. I can do things for this with Cypress. I know to build stuff. And I'm going to start now a new project. Why should I use, for example, Cypress testing library and not Cypress? Essies. Yeah.

Well, I mean, I think you should use Cypress for sure. So testing library is a collection of utilities that you use within testing frameworks. So it's not a replacement for Jest or Mocha or Cypress. It's more of a utility that you tack on to those things. Um, and so, like, but that said, with Cypress in particular, there are built-in commands that you can use instead of what you get from Cypress testing library.

And the reason that I can suggest adding in Cypress testing library instead of using like cy dot get, for example, is because it gives you more confidence. And that's all that we're really after. I know that some people, some testers really like to use testing to kind of enhance their design process and everything so they can build out their code. And that's that's great, if that's what you want to do, you know, TDD is awesome, whatever.

And that's that's great, if that's what you want to do, you know, TDD is awesome, whatever. But for me, I'm mostly and when we talk about running tests in CI, the types of tests that I want to run in CI are the types that can tell me can I ship this or not. I want everybody says don't ship on Fridays. I want to be able to ship on Fridays. I want to ship whenever, whenever I commit code, it should get shipped. And for me to be able to do that, I need to have confidence.

And this is what testing library offers is a really nice way to not only query in Cypress, it's pretty much just for querying, but elsewhere. It gives you not only query ability, but also interactivity with the, that output DOM. And the reason I say you don't need that for Cypress is because Cypress has great interactivity functionality already. So just adds the ability to query your document really, really well. And not only it doesn't make it easier to read and write those tests, but it also, you end up with more confidence because of the types of things that you're writing in your tests.

So, yeah, that's the reason. It just helps you increase the level of confidence. That's the most important thing. So just to make it even clearer. So the main motivation for me as a developer is to use testing library across different UI frameworks. It's mostly to reuse my knowledge that the syntax that I'm already familiar with or it kind of pushes me towards best practices, things like, hey, test like the user, don't focus on the internals. What exactly? Yeah, exactly that.

So it's the fact that you can use testing library wherever you can find a DOM is a really awesome thing, but that's not why you use it. The reason that you use it is because it helps separate your tests from implementation details, which is why you get so much confidence out of it. And originally, when I wrote testing library, it was actually React testing libraries specific to React, and then I realized that there was just a very thin integration layer between React and the rest of the DOM and so I extracted all of that.

And then we made Angular testing library and Vue testing library and Cypress testing library. There's a Test Cafe, testing like anywhere you find a DOM, you can use testing library for it. And that's actually is a huge benefit. So you can have your unit and integration tests that are maybe running in Jest or wherever, and they're all hitting the DOM. And then you can also have your end-to-end test. And they look very similar from the way that you're querying and interacting with the DOM, which I think is a huge benefit too.

Sounds like a great opportunity and where can, I mean, can you recommend resources, places to learn more about testing library? Yeah, testingjavascript.com is a pretty good place to learn it. I did share in the Discord a short URL. Maybe I'll share it again because people are pretty active in the Discord, but yeah, for a 25% discount to testingjavascript.com. But of course, it's not the only place to learn. Like you said, it's very, very popular. There are tons and tons of courses that you can pay for and also videos that you can watch on YouTube. I've given many talks. I've got lots of free content on my blog. And then the docs are really good. So you can go to testing-library.com and learn more.

Yeah, I personally really like this approach of tests from the user perspective almost as production, and I think that this mindset also inspires many now of the testing strategy or the organization level testing strategy are being disrupted.

So let's play a small role game. Let's say that you are the chief testing officer or the CTO, and I'm your developer, and I'm working traditionally. Well, I'm writing unit tests, a lot of unit tests. Then after three weeks, when I have the version working with 100% test coverage, then me and the QA person starts writing some integration tests, and then some end-to-end testing. Then you manual test my version and put in production.

Now, we have a meeting, and we do our planning for 2021. What exactly, and Oren, let's start with you. What exactly should I change in this approach for the next year? When you start planning, I think, first of all, you need to understand what you have and how do you want to do it. We all know the pyramid, right? And we all say this is the amount of tests that we have unit tested.

Unit tests are not enough. Even deciding on over there, everyone tries to reach 100% test coverage. Code coverage is really hard. It's really hard to get to 100% code coverage, and sometimes it's not worth it. So you have those different layers, and you need to decide how to start. And sometimes, by the way, if you don't have anything, sometimes it's even nicer to go actually against the tradition and actually start from the top. End-to-end testing and the integration tests give you high coverage very, very fast.

And they give you a lot of confidence, as you said, close to production to understand, or you can even test in production to understand what you got. The other thing that you need to make sure is that you don't neglect those other layers, and you need to make sure that you keep investing in them, because they give you the feedback faster. So the end-to-end are super-critical. They give you high confidence and high coverage in seconds, where they usually... The challenge over there is what Ken said, and I totally agree, is the trust. You need to trust your tests. If you have tests that are running and you don't trust them, I don't want them. I prefer 10 tests that give you 20% coverage and I trust it than 100% that it's flaky. And when the CI would go, I was like, Oh, I fail again. I don't know why.

People take tests out and they say, I don't trust it. Let's take it out because I want to release. Or they're starting this thing called rerun. You know what, let's run it again and again, and again, and again, let's rerun until everything passes. And that's, of course, it covers a lot of bugs that you don't catch because of those flakiness. You might have a real bug. People think that bugs are only deterministic bugs that you say, hey, I caught this and this is over as they produce. The most challenging bugs are the ones that don't reproduce all the time. When you have that ones out of 10, it's going to happen.

How do I find the person that did this? How do I find it, and that means how they had all the information and that brings us to the root cause analysis, understanding you need to invest in the fact that you know when something does break, what happened? We started with the screenshot, then you know the DOM, I want to see the DOM, the network request, the console logs, I want to have all the information I need, so that's when something does happen, I know I can pinpoint say, okay, this is it, and of course, the more you go up the DOM, it's harder to say, to pinpoint, and as you go more to the, more even toward the unit test, you can say, oh, here's the function that the method that went wrong, this is the state, so here it gives us, up to here, we see, a very, we have a lot of coverage, slow response, that means it's hard to run it on every commit, and going down, making sure that we have the amount of test in each part, to know exactly as fast as possible what went wrong. Yep, that makes sense, the wider your test are, then you need more to invest in setups that prevent flakiness and yeah, it makes tons of sense.

You need to make sure, how do I find the person that did this? How do I find it, and that means how they had all the information and that brings us to the root cause analysis, understanding you need to invest in the fact that you know when something does break, what happened? We started with the screenshot, then you know the DOM, I want to see the DOM, the network request, the console logs, I want to have all the information I need, so that's when something does happen, I know I can pinpoint say, okay, this is it, and of course, the more you go up the DOM, it's harder to say, to pinpoint, and as you go more to the, more even toward the unit test, you can say, oh, here's the function that the method that went wrong, this is the state, so here it gives us, up to here, we see, a very, we have a lot of coverage, slow response, that means it's hard to run it on every commit, and going down, making sure that we have the amount of test in each part, to know exactly as fast as possible what went wrong. Yep, that makes sense, the wider your test are, then you need more to invest in setups that prevent flakiness and yeah, it makes tons of sense.

Jason, many of your work is related to flakiness, I'm curious about your overall thought, but also specifically to this flakiness in integration tests. Yeah, flakiness is a big problem, but it's also kind of a fascinating field as well, something I'm sort of weirdly interested in. I can sort of say at Spotify, we had kind of a large flakiness problem, that's kind of how I got introduced into working and testing to begin with. The library that I wrote, jSJUnit, the entire reason I wrote it was because I was working on the Spotify desktop client at the time, and we were trying to tackle test flakiness, trying to reduce it significantly. And so the first part in that is just basically knowing what tests have run, which ones failed, how fast are they, et cetera, et cetera, putting them in a database and sort of trending these things over time. This is kind of the fundamental starting point for just understanding what is flakiness and what is flaky and then how are you going to address this. So it's a fascinating field. There's probably tons I could talk about, way more than this conference, but I guess kind of getting to the original question, something I feel that's important to say is that far too many organizations that I've seen even now, but particularly in the past, don't really recognize the fact that testing and sort of quality in general actually speeds up teams. So there's this concept in agile called sustainable pace, which I don't know, I just don't see a lot of teams actually following this or really understanding this. A lot of teams are really moving very fast, as quickly as they can, and this often means writing no tests or writing fewer tests. But if you can move just a tad bit slower and focus on quality and keep that quality high, this actually, you know, pays dividends over years and years and years. So this is something that I think is important, not just to developers, but companies as a whole.

I think that one of the big things we actually see is actually that developers do own quality. Yoni, you mentioned, hey, there's a QA person that's going to say, hey, what's up. I think it's becoming more and more that developers own the quality. And they need to say, hey, I feel okay with releasing on a Friday. Because that means that you feel secure enough that you're saying, hey, I trust this test. It's not someone else's job to make sure everything works. That's my job. That's my responsibility. Sure. Sometimes it's a good idea to release on Fridays. Yeah.

I would like to add that I feel like quality is not, you know, moving it to, having developers be responsible for that, is not enough. I feel like it's an organizational thing that everyone needs to be aligned on. And I think that's where, like, your quality strategy really becomes impactful, is when it actually helps your whole organization understand what quality means. And that's what enables you to actually move forward and be able to have those confidence in those tasks. Which is why I think test coverage isn't always reliable to do that, because it's often just expressed as number of tasks or a percentage.

So a lot of the things that I think is actually important to consider besides test coverage, or what type of test you're writing is actually be able to identify features and flows that are actually really high impact areas or high touch areas for your users, whether that's like a dashboard or a homepage or a checkout flow, and really concentrating your testing efforts there, focusing on a few high-impact ETE tasks. That's what's actually going to give you confidence, understanding your users. Are they on Chrome? Are they on desktop versus mobile? It's going to help narrow down what browsers you're actually going to be testing on. Does it make sense to run all your automation tests through every single browser? A lot of those things, I think, are more important to think about than just the number of tasks. Then again, that part about communicating about your quality strategy and having everyone within the team and company understand that, I think it's often overlooked, but it's super critical.

Yeah. There is a repeating concept here. Start from the user. Focus on the user. Okay. We have five minutes left. So, I think that it's absolutely now a significant trend to move from pyramid to something that is more diamond or trophy or honeycomb. Start with integration test or end to end. By the way, some of the earlier TDD books actually also recommend this. Start with wider test and then do your unit and TDD stuff. But, now, as a developer, we all know praise integration test. When should I write now unit test? And also, I keep hearing about new component test. Kenneth, I'll direct the question to you, what is the new role of unit test?

You know, it all comes down to confidence and how much confidence you can get out of things. And if you unit test, let's say you get 100% unit test coverage, you can still have so many integration problems. And so, you get a lot of problems there. So, let's say, all right, nevermind. We won't do 100%. Let's do like a mix here. And then as you start doing more and more integration tests, you realize that you don't need as many unit tests because you're already covering that code from your integration. And so, when you keep on down this path, where you end up finding most of your unit tests are in those maybe like pure functions that are doing complicated logic, and those typically they run really fast, and they're pretty straightforward to write and maintain.

And that's where I find myself doing most of my unit tests. I actually don't find myself doing a whole lot of module mocking in those types of tests either. So, I'm pretty much doing component or integration tests. The component is like React component or Angular component, whatever. Integration would be like on the node level you're integrating a bunch of your modules and stuff. And so, yeah, I find myself doing mostly that sort of testing. And then I fill in the gaps for like these hardcore computational or like these algorithmic things that are difficult to cover with the broad brush of an integration test. Yeah.

And then I fill in the gaps for like these hardcore computational or like these algorithmic things that are difficult to cover with the broad brush of an integration test. Yeah. So, there's still room for unit test, and this should be stated clearly.

Question to backstage. We're almost out of time, but we have a lot of fascinating topics still uncovered. So, if we can get five minutes more, please, let me know. I'm moving to the next topic, which is a promise for years that it's going to affect the testing world. And maybe this is the year where it's happening.

I'm referring to AI-based testing. Oren, you are the chief, the guru at testing, the testing, a lot about AI. Question. I mean, can you explain really in layman terms, what does it mean to me, really in simple words features, how can it augment or help testing? We're trying, I think what everyone is trying to get to the same thing, to trust your tests.

So you can look at it at the level of like offering the test to have as much as coverage as possible. And I think Nancy said it in an amazing way that, wait a second, how do you define that? How do you see that? I hope that one day we'll get to talk about not just code coverage but also user coverage, not just the shift left shift, right? What do we know about production? What do people interact with? What do we not? So first of all, AI can help with the authoring. Can we generate, can we look in production and generate automatically a thousand end-to-end tests in two days just by looking at what's going on in production? Can we filter out of that? What are the tests that we do want to keep as an end-to-end test. And of course, generating those integration components or even unit tests. Can we remove the flakiness? Right now, things like they're basic in every platform was, hey, it was one CSS selector. There's one way to find an element and click on it. A machine can look at thousands of them. So can we do that automatically and reduce the flakiness if we understand that a machine can find an element better than us? Yeah, like identify selectors from previous records that lead more to flakiness. Exactly. And it moves that over time. Yeah, makes tons of sense. I would love to discuss it for at least 10 minutes more. But we are just almost out of time. And I want to cover the last very important topic.

Story, Jason, only one minute to cover this. There is a big bloom of tools that relate to network interception, network mocking, a lot of new capabilities. You specifically, Nancy, wrote one of these tools. Can you first explain to us why it first recorded the network? So I wish I had more time to talk about this because I think it's super exciting. But I think the biggest thing that the network mocking really solves is, you know, it takes the most difficult part out of being able to write integration tests. And I wish I can go into elaborate what I mean by integration tests, because I think it's one of those testing that that's super misunderstood and no one really aligns on what it does. But, you know, I think it really solves that flaky issue that Jason was talking about. You can focus on things that you can actually visually see, things that's as close to possible as what the users can interact with. And I feel like network mocking really allows you to be able to kind of do that without making it super slow and, you know, flaky and cause all those issues. The plugin specifically that I worked on actually tags on what is out of the box already from Cypress. A lot of you who do use Cypress know it comes with a lot of mocking capabilities and it kind of just tags on top of that to really solve for, I think, the biggest issue with all this, which is how do we actually manage our mocks, right? Mocking is obviously number one issue, but number two, once you're able to mock, how do you manage hundreds of, you know, files of mocking that needs to change often? I feel like that's where, you know, the plugin Cypress Autorecord that I wrote really helps to address that. But again, I love to continue this discussion. I know we're out of time. Happy to talk with anyone on Discord. Yeah. I took a look at it and it looked amazing.

Verify that your integration and practice are a few things they are. We got one last minute, Grace. Jason, I'll really be glad to hear your take on network interception monitoring.

Yeah, sure. Just really quickly. I mean, I think it's an important part of testing. You have end-to-end tests, and the most typical reason why they're flaky or why they're slow is going to be network requests. So if you can kind of take those out of the equation, especially if you have something like VCR where you can automatically generate stable mocks, you turn these end-to-end tests into something that is far less flaky and far more performance, and basically is testing the same thing using hopefully the same system. So there's a lot of benefits to it. I would encourage you to look at it, especially Nancy's plugin there is a pretty great example.

Yeah. Thank you, Jason. And unfortunately, we're out of time. So really many thanks to all of you for this great insight. Should I try to summarize it in one sentence is production user, production user. Leave the internals to the end. Start from the chaos of production, the network, from the user, from the UX. This is where most of the tooling now innovate. And this is where testing should start from. And then if you have something complex inside that, well, we have the traditional tools. Many thanks for all of you for being with me today. Thank you. Thank you. Thanks, everyone.