Tame the Component Multiverse

This is a talk about UI testing, a tooling problem that we critically underestimate and its impact on both our applications, our experience building them and I think the broader web in general.

Now, I want to talk about some tooling that I think is missing from our current testing tool set. So, we're going to talk about some tools, some strategies that I think that we should start employing as frontend developers and UI engineers, and hopefully answer the question that some Just Fan has been asking since the second I took this virtual stage, when will this idiot take down this slide? Not quite yet.

So, I love the web, and I love great software, and I've been really disappointed as we've started to see the web take a bigger and bigger piece out of software development in general. With the advent of React Native and Electron, the web has started to move more and more places in software as a whole, and it's started to bring a pretty janky experience to everything across the board. Now, selfishly, I want the web to be better, because I use the web all the time. I love the web, and I want to make sure that we have the testing strategies in place that ensure that we protect user experiences at every step of the way.

Now that we've got the intro out of the way, we can finally get rid of this slide. So, who TF am I? This is the point in the talk where I'm supposed to convince you that I have a lot of credibility, and so I'm just going to give you a general introduction to who I am, and you can decipher yourself if you find me. Trust whether you're not.

So, first of all, I'm Chan, Chantastic or Michael. I go by anything, whichever you prefer is fine with me. I used to host a show called React Podcasts until, unfortunately, I burned out. It was just a ton of work. If you've ever done a podcast, it's a lot of work, but I got to speak during my time on React Podcasts with some of the just most brilliant developers and people I've ever met in my entire life. So, if you're interested in that, and you'd never heard of the show before, there's a great catalog, and if you want to hear from some of your favorite developers, it's there for you.

Now, I did kind of, like, in my burnout phase, just fall in love with the game Destiny, too. So if you're a Destiny player, hit me up and we can do some raids or Crucible together or something like that. Now, it wasn't all fun and games. I did get to work with the React Core team on the React Working Group, which was a really awesome experience, just bringing React 18 to life in the community and making sure that everyone knew how to transition their apps and take advantage of some of the new features. I also started one of my favorite online spaces for creative, curious, and kind React developers. That's at discord.gg. We have a ton of people in there, and I invite you to join if you like hanging out in discords. Most relevant to this talk, I have about 12 years of experience inside the developer productivity design systems and front-end architecture space. That's really where I've cut my teeth in web development and where a lot of the experiences that I'll share today are going to come from. I'm now working at chromatic, and we'll talk about that service as we go through this talk.

Our goal there is to improve the UX of the web. If you've been in the React and JavaScript space for a while, you've probably heard of the testing trophy. Let me put that on screen right now. This is the testing trophy, as visualized by Kent C. Dodds. It's based on a tweet by Guillermo Rauch that says, write tests, not too much, mostly integration. Humorously enough, it's based on a summarization format that Michael Pollan used to summarize The Omnivore's Dilemma, which is a really good book, and a really good summarization format for a lot of things like testing. Anyway, that's where that comes from. This is the testing trophy. For this talk, I want to kick it on its side and talk about it as a continuum and place some of our testing and testing strategies atop it. When we look at this visualization, we have a handful of things.

We have Static, Unit, Integration, and End-to-End. These are all types of tests that we can write to support an application. In the realm of Static, TypeScript is dominating right now, so I'm just going to put TypeScript over there. End-to-End, we hear a lot about Cypress. They definitely have the mind share. In the middle, we have Jest, or similar libraries, spanning the unit test and integration space. I put Jest in there because I think it's mostly well-integrated with JSDOM and Testing library, which are really popular tools in the React space specifically. This is kind of where we're supposed to be writing most of our tests. There is a little bit of complication here. We have two categories and three tools in here. And anytime I see a distribution like this, where it's not particularly clear what's happening, I wonder if we don't have two kids in a trench coat, if there's not some kind of idea that is kind of a little bit fuzzy that we haven't quite pulled out yet. Now, if I were to take a guess, I would say that the concept that we haven't really mastered is visual testing. So that's what this is going to be about, visual testing. I'm talking about this slide and I'm going to be a lot, and I'm going to be calling this the UI testing continuum. So if I refer to that, this is kind of the continuum that I'm speaking of.

Let's talk about the UI multiverse. This is the problem that I see front end developers having to face in the visual integration testing space. Now I also kind of refer to this sometimes as the 10-ish dimensions of web UI, or if I'm feeling particularly cheeky, the 35,000 perfect states. Let's get into that.

So anytime we design a web view, we think about it in its ideal state. This one perfect view, maybe we sketched it out on a piece of paper or we mocked it up, but this one perfect ideal state for this view. Now we all know that life is not perfect and we'll have to augment that with both error and loading state. We had a third dimension, which is the complications of loading and error states, which we might have dedicated skeleton views for a specific page. We'd have different errors handled different ways because a 404 is significantly different from a 500, and so on and so forth. Now we have an additional dimension, even for our successful pages, which is breakpoints. So we don't just make one single view. Typically for the web, we have a handful of breakpoints for a responsive website. It's pretty standard these days. Now I typically run in the vicinity of six, but for the sake of this, I'm going to keep it small to four. Now that multiplies again when we start thinking about browsers and how our views actually interact with the browser. Right now we have typically three or four engines that most applications have to target, and so I'll put four there. Now we take that and we multiply that again times user abilities. So this could be keyboard navigation or mouse navigation or, you know, having the site read via a screen reader. We also have device capabilities. Now this is a little bit different than user capabilities because a device can have touch or keyboard or mouse or can be spoken to. And those are all ways that we can interact with the content. Now I don't have any solutions for this part of it today, so I'm going to not add any slides to our expanding universe of pages. So free pass on that one.

Now most applications these days, at least ones that kind of are services, are going to have some level of authorization as well. And I've just kind of duplicated this a handful of times to show different authorization types. But in addition to this, you can think about logic in your pages. Any page that has, you know, like a Boolean that changes the view, but what I haven't represented here is combinatorial logic where you have two props that kind of make a third or fourth or fifth possible state. So I know that you might be thinking like, oh, wow, you just added a bunch of pages. But this is actually like a pretty minimal number when you consider how complicated components get. And we'll talk about that a little bit more in a second.

Now if you support multiple languages, then you really are going to multiply this entire stack of perfect views times every language that you support. This is particularly important if you support right to left, where the page will actually flip effectively for the proper language support. So pretty big numbers we're talking about here.

Now, if you work in the space of design systems or component libraries, you know that there's also this dimension of organization, where you might have the whole world under your Or control. But your galaxy of components is actually under incredible pressure by the parts of your organization that adopted, both intentionally and accidentally or unintentionally. And how some of those teams or parts of your organization might even have the autonomy to choose the view layer that they use. Maybe they're not using React. I know in my case, we were using Rails and React. But some of you might have parts of your application in Backbone or Ember or Vue even. And building a library of components that's comprehensive really involves supporting all of those libraries. So I can't really assume what challenges your particular design system and component library are under. However, I can tell you a little bit about mine and do the maths on the problems that I've been solving over the last handful of years, and multiply that out.

So these are the table stakes. So my application for every single Vue, it would have six Vue ports, three browser engines, three device capabilities, and four user abilities. That brings us to 216 perfect states. Well, there's a little bit more to it. We would have two authorization types at a minimum. Typically it was more like four or five. We didn't support right-to-left, because we're a US-based company, but we did support two frameworks. That brings us to a total of 864 perfect states. But it doesn't end there, because we haven't even talked about style. So we would support two themes, two contrast modes, and all of those could be multiplied by the 10 applications, which would have a discrete theme for each application. Totally nutty. But it actually brings us to an incomprehensible 34,560 ideal states per component. Now this doesn't handle even all of our authorization possibilities, and definitely not any combinatorial logic from props. Absolutely mind-boggling. Now I want to talk about the logic part of this for a second, to just show you how much UI states can grow exponentially. This was a blog post that demonstrated a button that was being built and all of the possible variants for not even additional contrast modes, but just two themes, light and dark. And all of the ways that a button could be represented in this application. And they had 1134 variants.

Just for this one component. Now, I think that we have a really big problem. And a lot of times we don't allow ourselves to feel the weight of this challenge. And even right now you might be squirming because the first time that you've actually thought of this all the way through is right now. And a lot of times we get by by just not thinking about this. By just focusing on our one task at hand and kind of letting QA figure out how to report back the errors to us, or we just do Twitter driven development where we push it out knowing that there might be bad things and people will tell us where they are.

Now I think that one of the biggest challenges we face is in test creation. So I want to talk a little bit about the suitability of testing strategies across this continuum. So first, static is really great for interfaces. So languages are great for testing interfaces. Unit tests are great for testing implementation of a unit of code. End to end tests are really great for flows, sign up, CRUD, update billing, et cetera. And in this integration part, we have coordination. Now ideally, this is coordination of components or code. They are separated from the full stack, which is what end to end tests. Now integration is primarily for, I'm going to limit it to non-visual things. And then visual is going to be that integration point where our code actually gets represented inside of a browser.

Now let's talk about the suitability of these testing strategies by role. Now, I think in most cases, we're going to see engineering take a lot of the static unit and integration tests. It depends on your organization, obviously. And on the other side, QA is kind of largely dominating the end to end side of things. But in this visual side, we have a lot of just kind of meh and hopes that it looks good to us on our machine, that it will look good to everyone else, and that maybe those other two fields will kind of close in the gap a little bit for us. I think that if we want to have better test coverage in here, we need to have tools that work the way that our UI authoring does, and make them quite a bit more declarative.

Now, Jest on one side does have snapshot testing and Cypress does have some component testing, but they have challenges in the way that they feel when authoring for visual tests. So how do we produce tests in this visual integration gap? And I think we do it automatically. Now I want to talk about a few things. And I call this the declarative UI testing. This will be storybook, chromatic, and continuous integration. And I want to thank my coworker, Varun, for preparing these slides from a deck that he shares with our customers.

So storybook. When I think about storybook, storybook really sits in the gap between design and development. And we have this idea of component-driven development where we actually really take advantage of components and the fact that they're isolated from the rest of the component environment. So we take a look, we take inspiration from tools like Figma, which have these ideas of sticker sheets for components, and we bring that into the code side. And we make this environment that is available to anyone with access to it, however you want to share it, that really visually represents the components and combines that with documentation and the ability to interact with them. So these are really living components. And we bring a lot of tools with us that we can share now in this front end workspace environment. So we have viewports we can codify that with the viewports we support in our application. We can use tools like measure instead of having to open up an inspector and measure all of this stuff we can just have that and make sure all of our gaps are right and the sizing is right between elements. We can outline things for debugging UIs, all inside of this front end workshop environment. Additionally, we have accessibility tests built in. This is typically something you have to install either in your browser, or through some kind of command line tool. And we can bring that into our front end workshop environment and make sure that everyone has the exact same information about the accessibility audits and whether or not they're able to support the full spectrum of users in our applications. And for me, I know I do a lot of console.log.wats, or console.log.wats, and we can have a full breadth of ways to be able to test our components using controls, archetypes, etc. We have automatically generated documentation inside of Storybook that allows you to, just by writing a component, have documentation that shows the interface of that component and the props available to you. And we have controls, so once those interfaces are identified, we now give you a playground so anyone indifferent to their ability to actually code the component can tinker around with some of the props and make sure that what they're trying to, the button, in this case that they're trying to achieve is possible with the interface available. Now there's this quote, I had the chance to sit down with Ryan Bayhan at Shopify, and they focus a lot on the cross-functional power of their teams, the ability for teams to focus on the expertise that they have, but have empathy for those other disciplines as well. And he says, interactive storybook controls improve our team's cross-functional literacy and communication around react, and I just love that.

Now chromatic is the piece of this where we can actually get some automated testing out of this. Storybook is really great at solving the organizational side of things, but chromatic is where it becomes a testing tool. Now chromatic, when you integrate with your CI pipeline is able to do a visual test against previously accepted visual tests. So when you upload your first one, you kind of define that as a baseline, and then every pull request after that will be compared to make sure that no unexpected changes have been introduced. Not only is that available for static views, but we can also do that for interactive things as well. So think about a modal here, we can actually have a button that enacts a modal, and then we can run our tests against the fact that that was actually clicked and opened. Pretty cool. And we can do that across all of the browsers that we support as well, and in all of the viewports. And this is all integrated into your CI pipeline, so by sending a pull request, now you can run your army of robots on all of your visual tests and make sure that nothing changed. And so, Brad Frost, who is just a titan of the frontend space, put this really beautifully. He says, testing has historically been really rough for us because it's been like a picture in your mind's eye, a modal, and imagine clicking on a button that opens that modal. But by writing a story, you get the documentation for the component, tests, and a playground all for free. Now, I've been really excited to work with teams like Redwood as well to integrate even further storybook testing using Testing Library, MSW, and Prisma into the framework and getting some really cool stuff out of it with things like Redwood's generators, which would just generate all of these various states of success, login, you know, error, and loading states for you out of the box. Really cool.

So we talked about storybook, chromatic, visual testing, why should you care? Well, I believe that we are similarly cursed, and that's because we love the web. The web is awesome. We believe in everything, everywhere, all at once, now and forever, accessible to all. And for all its faults, no platform but the open web really makes that promise. And if I'm being super honest, there's something fun about creating these worlds, these universes that I am fully in control of. And when they push back, I have to learn new strategies for kind of re-containing them again. And keep this universe, this UI multiverse that we've created under control.