How I Test a Million UI States with Every Merge — Visual Testing with Storybook

What's up, everybody? How are you doing today? OK. It's OK. It's OK. I need more coffee. Man, my flight over here, I'm working on a couple hours of sleep, so I apologize if I things are a little bit rough today. But I'm super-excited to be here, super-excited to be hanging out with you. I'm a little bit worried about the timeline because like 20 minutes, I mean, like, I can talk about the subtle differences between number two pencils for 20 minutes. So I don't know how much of this we'll be able to thoroughly cover, but I'm excited to get through as much of it as we possibly can.

Now, my goal today is just to convince you that just snapshots are not enough for what we're doing today and that we need more robust tools for testing our UI. So we're gonna go through that. My name is Chan, Michael Chan, Chantastic, whatever you feel comfortable with. This is how you can find me online. Earlier this year, I gave a talk called Taming the UI Multiverse, which was a little bit more of like a feely, squishy version of this talk. We're gonna take a little excerpt of that and then jump into a little bit more code. But, if you want more stuff that's just talking about the feeling of UI development, how hard it is, and some of the challenges that we feel but don't really talk about a lot of times, this is gonna be a good follow-up talk for you.

So jumping right into what we're talking about today, who here is familiar with the testing trophy in React? Okay, only a handful. Okay, okay, hands are slowly starting to go up. So I think probably a good half of you. The idea behind the testing trophy is kind of a shape of how many tests we should write in any regard, and the biggest of those is in that integration slice of the pie. This is kind of a distribution of the types of tools that we use for the different types of testing. But integration is supposedly the biggest piece of that. Now I propose that there's this big piece of integration tests that we haven't actually integrated into our development workflows, which is visual testing. So that's what we're going to talk about today. And the tools that I've been using for this are Storybook with Chromatic. Those are the tools we're going to talk about. Now both of these tools kind of derive from this mindset of being component-driven. And so what's the idea of component-driven? How does it work with visual testing? Well, the idea behind it is really boils down to, how much value can we extract from one component? We have this really cool, isolated nugget of user interface. How can we, I guess to borrow a violent metaphor, how many birds can we kill with the stone of a component? Sorry, if you love birds. I love birds. I actually had a podcast for a little bit where we just reviewed bird songs.

A little different than a React podcast. One of the things that we haven't, components have really become every part of our life. But something we haven't explored a lot is how we integrate components with browsers in isolation. Up to this point, visual testing has often been bringing the full stack of our application into the browser, testing it. And that can be really hard to test, getting the environments right, getting all the personas right. Components allow you to just kind of like, inject a state and go really fast.

So, I want to describe the problem a little bit, and I call this the UI multiverse. And it's a description of the multidimensional challenge that we have when we're building UIs. And yes, like all of us, I was watching a lot of Marvel at the time that I was thinking through this. I also call this the 10-ish dimensions of web UI, or 35,000 perfect states. So, every UI starts with the thing that everyone thinks that they're building, the very clear vision of what we're going to make. But we all know as modern web developers that any view is potentially three views, an error state, a loading state, and then the successful loaded state. And then it goes even further, because like, we have like, different variants of those We could have a spinner, we could have a skeleton, the errors could be a 404, which is distinctly different from a 500. And then for our successful views, like, these multiply like rabbits, right? Because maybe we have 6 breakpoints or so that we actually think about and care about. But like, we could be supporting any number of breakpoints in reality. This gets a little bit more complicated multiplied by the number of browser engines that you have to support. Unfortunately, these are getting better year over year, but there are still subtle differences. Now, I would argue that the subtle differences aren't necessarily important to account for as long as you have a consistent experience inside of those browser engines. And then we have usabilities, right? So, are you using, like, touch, are you using sight, are you using your ears? In America, the CDC suggests that at least 4.6% of Americans have blindness or low vision. Now, it's really interesting because I've spent a huge amount of time testing and reworking views in Internet Explorer to save our app for, like, one and a half, 2% of users, but we dedicate so little time to the 4.6 plus percent of users who aren't able to use our sites with their eyes. And we also have, like, device capabilities. I'm not going to talk about that a lot, but it is part of the, like, dimensionality of UI development. And then these views get multiplied by all the complexity in our apps. So it could just be, like, authorization, that's one metric, but then you have, like, we all know how quickly props and state and the authorization of an application can really, like, balloon out of proportion. And if you have the privilege of having an app that's so popular that you need to localize it across different locales, well, now you have at least two versions that you have to worry about. So just take all of those views and multiply them by two. And if you've ever had to do this, you'll realize how poorly your apps are probably suited to just make that switch, because anytime you use a margin right for anything, like, all of that just is something you've got to fix now. And then docs. Docs are always a huge problem. We create these component libraries and we expect people to use them diligently, but we need to put in the work to make good documentation.

So these are the real problems that we have to face. Now let's talk about my personal problems. I've been working for the last, like, 12 years in, like, design systems, component libraries types of work. And so this is what the challenges that I was facing over the last handful of years looked like when I came to these solutions.

So for every view, we would have six view ports, three browser engines, three device capabilities and four user abilities. So that's 216 perfect states per view. So at scale, let's say it's two authorization types, like, user types. It was actually, like, four or five. But I want to keep these numbers as low as possible. We weren't doing right to left. So we just had the one. So I'm going to cut that. And we had two UI frameworks. So we needed our stuff to work in React and on Rails. Wait, React and, yeah, I got that right the first time. So that's 864 perfect states per view.

Okay. Let's take this a little bit further. So now we're going to talk about style. We have color schemes. Color schemes are very important these days. People want their dark mode. We had two contrast preferences so you could have it be a low contrast or more contrast. And we also supported ten discrete applications. So we had this kind of, like, white label type of design system that could be themed for individual applications. If you multiply those out, you get this rough number of 34,560 ideal states per view, which is kind of nuts. And I think that we look at this and we're kind of, like, we're accustomed to not thinking about this. Because if we did think about it too much, we would just kind of not do the work. And I think that that's the approach that we've taken to testing a lot of this stuff. easy to do all kinds of testing.

Except visual testing across all of these various dimensions. Now, just to show you that I'm not way out on left field. This is a Figma file. Someone wrote a blog post documenting all the variants in Figma. And they landed on just in these two color modes, 1,134 variants. This isn't multiple contrast support. We're not talking about viewports or logic or authorization. So just buttons alone. So it adds up really fast.

Now, again, we're not going to have a ton of time today. So I want to leave two links with you. We're going to cover as much as we can. And then I'm going to leave you with these. If you go to Chan.dev 1M, that's going to be the GitHub repo that we're working from in the exact state that I have right now. And see, is this counting up or down? Oh, my gosh, down. So then YouTube.com slash at storybook.js. I've been covering a lot of these topics in depth. So anything we cover today will also be available there. And you can see me doing some really debasing poses as well. So enjoy.

Okay. Let's jump into demos. Okay. So we're going to look at Storybook real quick. I'm sure many of you have seen Storybook in its very, like, basic state. This is the storybook that gets generated for you when you run NPX Storybook in it. I've added a couple libraries just so I don't have to NPM install today, but this is really what you're looking at. Now, at its base, it's a component catalog. And like many people, when I first saw Storybook, I was like, I don't need that. I know how to put a component on a page.".

Like I'm going to be fine. I would just spin up Gatsby, dump out all my components on there, or whatever. However, if you go a little bit deeper, you see there's actually a lot of really cool things that we start extracting from components. Remember, our goal is to get as many things out of a component as we can.

So if we jump over to this docs page, we can get code examples, which is pretty sweet. Copy and pasteable. But then we also have this really cool thing where if you are adding type annotations using or TypeScript, well, we can actually just pull all of that out for you and give you a really nice interface documentation for this component. But we can take that a little bit further as well. You shouldn't have to be a developer to be able to see how all these things can... How all of these interfaces interact with each other.

So since we know all of the type interfaces, we create a playground for you by default. So I can say I actually don't want this to be a primary button. I can make this text really long if I want. Change the color. We have all kinds of stuff in here that you can use. So really cool. So I'm gonna go back to Canvas. That's available for all of these components if I click on this add-ons pane right here.

Now another cool thing about stories is that I can just kind of inject certain things into this. So if my components are well designed, they're pure function of props, well, then I can just kind of throw in the data that I want. And so that'll render with the data that I want. I can pass in objects, all that kind of stuff. Now another cool thing is that components aren't just a function of their props all the time. They're also a function of the interactions that can be done to them. So we also have answers for that, as well. Let's look at this page component that we have. It's a composition of all of the buttons and the header that we have right here. And we're going to click on this logged in state. So we have logged out. You'll see there's the log in and set up button, and we go into logged in and we see this logged in state.

How did we get that? We actually have an interaction that we can run using testing library on top of these components, which is really cool. That means that we can derive stories from the components themselves, but also the interactions that we expect people to take on them.

Now, let's jump into code a little bit. I have never before done live coding in front of a crew, so I've tried to do as much as I can to kind of make it go well, but bear with me. So we have our storybook running. I have a bunch of examples if you decide to look at this repo later. I have a bunch of to-do's over here that you can jump to various parts.

I want to show you just the ergonomics of a story first. We import the component that we want to document. We just export default a little bit of metadata, the title where we want that story to render, the component under test to see why that's important. And then we can pass in kind of default arguments if we want. Think of these as props in react land. It's a disambiguated term for other libraries.

Okay. So, what do we have? These are what the test looks like. We could write out JSX like this. But we have the object format where we just say, hey, we want to do a primary test, we're going to assume we're testing this component, we're going to throw these args on there. We can run it without args. We can pass in whatever args we want. And we actually compose args. So, down here we can say, hey, I want to use all the arguments from the large and the primary button to compose this new story. So, we'll just spread those out and then we have a new story. That's what stories look like. Super easy so far.

Now, let's go to the next thing. Right here. So, one of the values of storybook is that we can capture these states that are kind of unideal. So, up to this point, we have this like Jane Doe story over here, right? And this is like a very, like, ideal name. But if you've been doing web development for a really long time, you know that these views are kind of sent with these ideal states, not the extreme cases. So, let's do an extreme case right here.

We're going to do John Jacob Jingleheimer Schmidt. So, with a long name, and I'm going to show you one more thing. We can do some really cool stuff. So, I'm going to do another test right here, or another UI state.

We'll create a story using the long name, using the arch from the long name. But we're also going to set a viewport. So, one cool thing about this object syntax that we have is it allows us to do parameters of all kinds. So, I don't have to kind of imperatively put this story into a certain state. I can just say, hey, the viewport that I want is going to be this mobile 1 viewport. So, did I save that? No, got to save it. I'm going to give this new story, and then here we go. So, as I jump to that, I can see, like, well, this isn't exactly ideal. And then we can kind of make some decisions based on that. But we're just effectively like parameterizing all of the things that we care about in this specific story.

How much time have we got? Two and a half minutes. Sweet. Okay. We're going to go a little bit long. So, we've, you know, yeah. Okay. So, cool. So, we had that. So, these stories are really cool. Now let's jump into play functions real quick. So, we saw those really cool interactions. I want to jump over here into play functions. Now, how do we start doing those interactions? Well, this is what that test looks like. So, we have, again, we're going to go through this. We have a page. So, we're importing that whole page.

That page is the subject of all of these stories, and we're just actually here. Let me show you one more thing. I have done, just a little bit, this is all just, like, regular code. You can write your React in here. I have a page, and this is just kind of passing in those functions, and I've done a very, very small little mock where I have a tiny component that I insert that, you know, catches these actions and then kind of returns a resulting state. So I just didn't want to skip over that. Really easy to just kind of, like, make the comp- put the components in the states that you want.

Now, from there, we just use testing library to actually, like, interact with this story. So we take our canvas, we grab that login button. So we just use our coreys to grab that. And then we just say use event, click on that button. Super easy. Now, we'd actually compose these stories as well, so we saw a composition of, like, all of the arguments and parameters. But we're going to also test these, compose these stories. So I can say, hey, everything from the logged in play function, I want to run that, but then I also want to do this other thing. So I'm going to write a test for log in and then log out. Okay, super cool. We have, let's see. Where is it? Log in and then log in and then log out. Now this looks on the surface to be the same as the logged out, but you can see that we've run both of our interactions over it to get into that state. Now we also run jest assertions in here, so I'm going to just import expect storybook jest, and then write some assertions here to make sure that it is in fact getting into those states. Did I save? Yep, I did. So we click the login button, we assert that Jane Doe is there and then we click the logout button and then assert that the login button is there again. Pretty neat. Pretty neat. Pretty neat. Okay, let's see. And then, check this out. This is super cool. So I installed the storybook test runner library, and we can just run yarn test storybook.

And that's going to run tests ingest using headless chromium for all of our and just running all these tests.

Now I want to show you something really cool. Now this goes back to our like dimensions of UI. We really haven't had a really good way to like test accessibility.

I want to show you something really cool that we're doing with the test runner. So I'm going to jump in here and say I'm gonna make a couple changes to the test runner. Now, I'm not a super good coder, so I literally everything you're seeing here, I just kind of copy and pasted. So this is on our blog. We augment or we give some configuration to the test runner to say that we want to run the axe play write library during that phase. So I'm going to run, let's see, yarn test storybook again. It's going to run all of our tests. Oh, shoot. What did I do? I didn't save. There we go. Again, not a good coder.

Okay. We're going to run that again and we're going to get a few errors. I already know what these errors are, but you can see that it ran axe play write on all of our stories and now we have the ability to say, like, okay, so we're not getting contrast on a couple of things and then make some fixes. So these are all CSS. Super easy. I'm going to make some changes real quick. That color was a little low. We're going to change some contrast for all of these. Run our tests again. Oh, I got a reload. I got a reload because I changed CSS. There we go. Run these again. Oh, what did I do? I probably didn't save a file again. Remove a comment.

Oh, thank you. Yeah. Mob programming. Now we're getting into it. Okay. Cool. Now all those tests pass. Super-awesome. Oh, yeah. Let's do it. This is really cool. Because this protects the user experience of something that if you're a visually oriented user of the web, it protects the experience of something that you don't always like get to see or like kind of is not always in your mind. Because it's this thing that's kind of like invisible to you. But it's the way that people interact with the web a lot of the time.

So that was super-easy. Like, just wildly easy. And now we have the benefit of all that stuff. So that is... Okay. So that's all of those things. I want to show you one last thing. Check this out. We can even do this... And I said in the... I had that slide up in the beginning. Burn all snapshots with fire. Well, this is probably the one, I think, acceptable use case of snapshots, which is to generate... Check this out. I'm generating snapshots for the... What is it? The accessibility tree.

Now we can know for certain if something that we changed actually resulted in a regression in the accessibility tree. So super, super, super, super, super cool.

Okay. Now, for this part, there's a lot of NPM installing. And this will probably be the last thing that we cover today. And so I wanted to fast forward through that. But I wanted to show you how you can get visual regression testing in just less than two minutes. So I'm going to run through this real quick.

We've got a video. So, first of all, we take this library. This is the one that we're looking at today. We're going to put it on GitHub. We go to Chromatic.com, sign in with the account. There we go. Add a project. Find our project on GitHub. Super easy so far. We're going to add our Chromatic library to our Git repository. Quick and easy. We're going to run this script, npx Chromatic. With our project token. It's going to take longer than this to build. But then we see our build online. There it is. So we can go to our project and see all of the things that it did for us. So it shows all of these snapshots, and then also it shows these stories in the post-interacted state. As well as DOM snapshots, as well. So we can know when something just in the DOM changed. We get a free hosted storybook right out of the box. And we can test across multiple browsers.

So that's that thing that I was talking about, we want to keep things consistent across browsers. Since we have a little more time, I'm going to show you how to install GitHub Action. Again, all copy and paste here. I ain't trying to write any code. So we copy and paste that. We're gonna throw that into our GitHub Action. Paste. Create a secret for our GitHub Action. Settings. Secrets. Action. Again copy paste, all the way. Easy, easy. We add that secret, and then commit and send it. Go over to GitHub Actions, and then we watch these things go. The first one failed, that's my fault. But this one's good. And we go and we can see our build. So now every time we push something, it's going to run all these tests for us. We have an army of robots just testing across all of our browsers, all of our viewports, everything. Super easy.

I think I'm pretty sure that I'm well over. So we're just going to leave it at that. I want you to go check out this Readabook, because there are some really cool additional steps in the, what is it? I think we have, what is it? 14 through 17. Some really cool stuff talking about integrating with Chromatic, testing your themes, all that kind of stuff. So anyway, thank you so much for your time today. I hope that you learned something new about Storybook and Chromatic, and start visually testing your apps. Thanks for having me.

Thank you. Thank you, Michael. Yeah! Why don't you step into the interview chamber. Let's do it. Am I on this side? Yeah, you indeed did go a little bit over, which is a shame, because I've been looking forward to interviewing you. How much over did I go? I'm sorry. About 7 minutes. We only have three minutes left. Oh, dang it! That's all right. Here. But you are now going to be safe from... From all the questions, I know. From all the questions, yes. Sorry. Well we have the Live Q&A, right? Here, this is for you. Thank you so much, Michael. Please go with your third hand. Do you want me to... Oh, shoot! Oh, no, I'm good. Okay. Yeah, go ahead. No, I'm good. Thanks.

All right. Let's get into the questions. I'm not sure... Do we see the questions? Yes! I think the most upvoted question is what are the most common mistakes we should avoid when using Storybook? I know this is a big question but what is the one big thing that people tend to screw up? Oh man. That's really... That's a hard question. I don't know if I have a really good answer for that. I think... Like something that I think is really important is like finding ways to not write a story for every exact state. So one of the examples that we didn't get to was I'm testing themes. And so it's really easy to say like, oh I want, you know, I'm gonna have this story and I want to present it in this theme and that theme and this contrast mode. But those kind of balloon out of proportion, right? And so now you have like seven stories to just see the various themes. One of the examples that we didn't get to is setting all of that as a global parameter. You can make new controls or add-ons that you put in the top bar really easily. And now you can just change those parameters via the top bar. And then in your testing environment, in Chromatic, you can say hey, just use this snapshot, which is a grid of themes. So now you don't have to think about themes of a story. You can apply to any story. So I think finding strategies like that, where you can keep the stories nice and simple, but then use the robots to do all the work for you, is the dream. Yeah, I think that was kind of like the question there as well is how does Storybook help you check thousands of states and I think that's kind of answered. Yeah, exactly, because I don't want to write stories for every single viewport. I just want to know that when I push it, that nothing changed. So, yeah.

Cool. All right. Let's do one more question. This is the most uploaded one. With this testing library, just integration, should we now do our unit tests directly on Storybook? Is that the way we're going? Oh! Ah! Well, so the goal of the test runner is that you can integrate it in whichever way feels best to you. So if you have a team that's a lot more like engineering, and they're comfortable in that Jest environment, well, now you can bring some of those into the Jest environment and use that. However, maybe your line is a little bit different.

You want to do a lot more stuff on the visual side, put a lot of it on chromatic, and then just have some of these things where you're, again, like testing the accessibility tree for regressions, like that kind of stuff, run that in Jest. But I, as a person, tend to stay away from hard lines or recommendations and just say, hey, whatever supports the team that you have, and like their comfort, like you should go towards that.

Yeah, I think that's a good note to finish on. We have so many questions for you. I think this might be the most questions. Oh wow. Well, then I'm sorry I spent so much time. But there will be the live Q&A. So if you do want to ask any questions for Michael, you can do it on the next break in the speaker Q&A room. Unfortunately, now we need to thank you, Michael, for your time and bid you adieu. Thanks for having me. All right, cheers.