Beyond React Testing Library: Testing React Libraries (and library-like code)

Hi everybody. Today my talk is called Beyond Testing Library, Testing React Libraries and Library-like Code.

Just real quick about me. Lenz Liebertronik. I work at Apollo GraphQL where I'm a maintainer of Apollo Client. I'm also the author of RTKQuery, a co-maintainer of Redux Toolkit, and due to my ADHD I've written too many libraries that I kind of have to maintain now. You can find me on GitHub and pretty much everywhere as Fryneas except for Twitter, where I just go by Fry. The handle was free. It's four letters, so I took that one.

Generally we need to see how I came to getting to this talk and I've been writing a lot of tests for our code bases and generally I love using React Testing Library for that. And it's absolutely amazing. But there is a but in React Testing Library because React Testing Library tests for what I'd like to call eventual consistency. But on the other hand, when I write a library, I need to optimize for a hot code path because the library that I'm writing and that you are using should add the least impact on the performance of your application. If I do an additional render in my library, that means that a thousand applications out there also have that additional render. So I need to test for that and I have a bunch of special requirements that I need to test for, so let's get over these.

Our first special requirement is what I already said. I want to make sure that components that use my library don't re-render more than absolutely necessary. The second requirement is a big React concern in general and that's that you don't want tearing. You don't want a single hook to return an inconsistent result, but also if you use two different hooks in the same component that maybe look at the same data source, they also should look at the same data source at the same time. And the same also over multiple components. If you use the same hook or different hooks all over your application in different places, they have to at the same time always give you back the same data, which isn't always easy. And then this goes back to the first one, but a little bit more specialized. I want to render not only as few times as possible, but also as granularly as possible. Only the right component at the right time. So why do I say we need to go beyond testing library for this? Let's look at an example for a simple hook tested with React Testing Library. Here we have a usequery hook that returns an object that has a loading state and a data property on it. And really the best case scenario is what you see on the bottom there. First you have loading true and data undefined, and that directly goes into the final state, loading false and data hello world. And that would pass this test. This test tests synchronously for loading to be true and for data to be undefined. Then it waits for a while until loading is false. And then we test that data is hello world. But there are a lot more scenarios than just this best-case scenario that actually pass this test.

The first of those scenarios is something unrelated in that hook does something that's not output by the hook, but it costs us additional renders. And these renders would not turn up in this test. And I would not have any way to test that or to ensure that that doesn't sneak into the library after the fact.

There are also other things that can go wrong. For example, data and loading can change on different renders. So we have this intermediate render where loading is still true. And data already goes to hello world. And that's what I meant with tearing earlier, tearing inside of a hook. We want to avoid that at any point. But this test won't avoid it because we test for loading to be false first, and then we test data. But data is not tested in between this data to be undefined and loading to be false. So it could actually take any shape in between. And actually also until loading is false, loading could also take any shape. So yeah, this would pass this test.

And then there's even one more really weird specific use case. With the right timing, this can pass. It will be a flaky test, but it can pass. Let's look at why this can pass. Because it looks like if you look at the test, this shouldn't pass. But the problem is that waitFor() creates a promise. And even after this loading to be false assertion succeeds, that promise still has to be awaited. So there's a single tick in between. In which moment React could start rendering against something else. So you'd need to write a very specific test, and it would not always succeed. But this would add a flaky test to your codebase. And it's something that I want to avoid at all costs.

So all of what you've seen right now is okay in a normal app. You hopefully use TypeScript, which will prevent completely wrong situations in your code, and would also catch that loading puppy state. And your UI eventually gets into that target state, and the user will just see a short flicker or something. It's kind of okay.

But it's only okay in an app. It's not okay in a library. Because for one, we have this bug, and a thousand apps out there get that bug. And the other thing is, you don't add one library. You add a few dozen libraries. So if each library only brings one bug to your application, you haven't written a single line of code yourself, and you already have a dozen bugs in your app. So as a library, we need to test differently, and we need to test more accurately.

The same doesn't only go for us library authors, by the way. It also goes for that in-house library that you have at your company that's shared by ten teams, and one person maintains it. It also goes for the absolute hot-code path that you have in your app, if you only have one app. This is like the code that's called everywhere and needs to be optimized. So we tried a lot of different things to count renders and to do assertions while rendering, and some of them worked for a while. Some broke when we changed from React 16 to 17 and 18. Some others broke the moment we added suspense to the picture.

There was one solution in the end that really stuck and worked, and that was using the React Profiler component because this React Profiler component, if we zoom in, has this on-render callback, and every time a render finishes, this callback is called, and I can synchronously take a snapshot of all the values that I'm interested in. So I take that snapshot, I just push it onto an array, and I let my app run. I don't make any assertions, and after a while, I just look at the array of snapshots down here and make sure that there's that first render where we have true and undefined, and then the second render is false and hello world, and that afterwards, we wait for another hundred milliseconds and no third render comes in. The problem with this is it's really not intuitive to use. It checks a lot of the boxes that we had before, but you don't want to write this in every test. But there's that thing that I'm a library author and I know how to build a library, so that's essentially what I did or what we did.

So the next slide is a problem. The title of this talk is Beyond Testing Library, and when we released that thing last week, it became part of Testing Library. So with this talk, I'm introducing the Testing Library React render stream, which is a testing library that's based on this profiler that we just saw to test a hot code pass render by render, and I can't stress this enough. This is not what you usually should use to test your whole app. This is to test the important pieces, and most of the time, those important pieces are a library.

Let's look at that. We have the test from before, and we're going to rewrite it to get away from that horribly confusing profiler thingy. First, we're going to replace all that rendering logic from before, and we use this API called Create Render Stream, which gives us an object back that has a few properties on it. One of them is Replay Snapshot, and one of them is Render. First we create a hook component, in which we call our hook, and the result of that we immediately put into Replay Snapshot.

And then we call Render with that hook component. Render is the same render you are used to from React Testing Library. It just adds a few wrappers and does a few things more, but essentially calls the same thing. Then we rewrite those test cases down there. We take another thing out of the render stream, which is this Take Render function. By the way, this is a super neat trick. If you use these lone blocks, you can reuse the same variable name every time again, and you also make sure it doesn't exit the scope. So I can use the name snapshot for a variable here and the name snapshot for a variable here, and I don't have snapshot1, snapshot2, and snapshot25, because who wants to do that? Essentially, we take the first render, and we await take the first render, which means the render could already have happened, or it will happen in the future. And then we wait until it happened, and then we look at it. And we assert that loading is true and data is undefined. And then we take or wait for and take the second render, and we assert that loading is false and data is Hello, world! We still have that weird promise results set timeout length test thingy down there, and we can also write that nicer. We just say await expect take render, not to re-render. This is like a default timeout from, I think, 100 milliseconds, but you could also configure it if you wanted to wait longer or if you want to test to get faster and have some confidence in your code. Of course, we still have that hook component up there and where our React testing library has the render function and the render hook function, instead of create render stream, we just have render hook to snapshot stream, where you put your hook up there, and you just get a take snapshot method back. And then you can just look at all the snapshots that were created from the hook. Every time that hook causes a re-render, a new snapshot is taken and we can step through them piece by piece by piece.

So, let's look back at those requirements from earlier. Which of those are we already matching? We can ensure that the component doesn't re-render more than necessary. We did that. We can check for this consistency inside of one hook call. We can also do it in a component because like the example two slides before was a component, not a hook call per se. We could do more there. We still got that question, like one hook and two different components. So let's look at that case next. Let's look at that for a second, why it's important to do that. You can have one data source, but two different ways of triggering updates in your hooks. There's a useState to trigger an update on a component re-render. There's useSyncExternalStore. And up until React 18, those had different timing behavior. React 18 would batch them, but it would batch all useStates together, and it would batch all useSyncExternalStores together, but they would still end up in two different re-renders.

So if you have one hook that uses one of the APIs and another hook that uses the other API, you would still get like one render where things are kind of weird. So here I'm going to test two different hooks that actually had those two different implementations at some point in time in Apollo Client, useQuery and useFragment. And essentially I'm not taking like one snapshot with one hook, but I'm just putting two properties on my snapshot, like the last useQuery result that we saw and the last useFragment result that we saw. And I'm also not using replaySnapshot here, but I'm going to be using mergeSnapshot.

And if we see how we use that, we create two components that we render side by side. We could also do both in one component, but this is like a little bit more detached from each other, which makes a nicer test. And in one of those components, we merge the useQuery result into the snapshot, and in the other component we merge the useFragment result into the snapshot. We render both components side by side, and then we look at those snapshots, we do a takeRender, we take the snapshot out of the render, we make our assertion on the useQuery result, and we do the assertion on the useFragment result. We repeat it the second time, and now we know that definitely both hooks rendered at the same time with the same content, and in the end we check again that we don't re-render again.

Going back to the special requirements, now we can do this over multiple components, which is important. The last thing is that granular re-render. I want to know which component re-renders, if I re-render a whole tree. And for that, let's look at what triggered this special requirement, which was suspense. Because here we have a loading component that shows a loading state, we have an error component that shows an error state, we have our component itself which calls useSuspenseQuery and just gets data back, and then we have an app that puts everything together with a suspense boundary with a fallback with an error boundary with a fallback with a component. For the sake of the slides not exploding, we ignore the error state for now, and then we actually look at building a test case with this.

So we look at this three different ways now. The first is what you actually already know, which is taking a snapshot from data, so we say replaySnapshot, and we add that replaySnapshot result to a component on the side. And then in our test we can take renders, we look at the snapshots. The first render is hopefully undefined because that component never rendered, because all we see is a loading state. On the second render, everything should already be fetched, the data should be there, and our snapshot should be replaced.

The second way to look at this, because this gave us some information, but it didn't give us full confidence that everything is working, is to look at the actual DOM. By the way, could you pass me a water, please? Thank you. I'm sorry, it's a lot of talking. So now we want to look at the DOM. And we don't only take normal snapshots with that replaySnapshot thing, we also have a snapshot DOM option. And this is off by default, you can turn it on. It means that every time a render happens, a full snapshot of the DOM at that time is taken and a complete copy of your DOM is created. This is not for free. That means you need to opt into it. It takes up more memory, probably also slows down your test a little bit, so use it sparingly.

But in this case, we take the first render and we look at the DOM of the first render by using this withinDOM thing of that snapshot. And then we just do what we would do if you write a React testing library test. We just use the normal testing library DOM assertions. So withinDOM getByTextLoading gives us that DOM element with a text loading. We assert that that's in the document. Then we re-render. Then we assert that that same thing is not in the document anymore, but that Hello has been rendered because that's like the greeting that we got back from data. Those two things are already great, but we still can't assert that certain re-renders don't happen and stuff like that.

There's a third way. This one requires us to rewrite our components a little bit more. It's a useTrackRenders hook that we just inject into each of those components on the list. And then we can say something like expandRendersComponent to strict equal appLoadingComponent, which means that in our first render, the app component renders, the loading component renders, but the component component did not render. And then we have that second snapshot that we take. And here we look at the rendered components again. We see that the app, like the parent component, did not re-render, but the child component did.

So component rendered. And if you put all of these together and we add the error component back, this is like a full test. And this test gives me confidence to ship code that actually does what it should do and nothing more.

Going back, our last special requirement gets a checkbox. All of this sounds pretty cool, but I want to add a bunch of words of caution. One I don't need to add myself. We released this library last week. We threw it on Twitter. So let's just do Andrew Clark have the stage for a second. He says they test React themselves similarly, but also to only use this for libraries or library-like code, because there's a chance that React might just change when it renders. So if you do a React upgrade, it might break half of your tests while actually everything is kind of fine. You just have to know that this new React version does things differently. And maybe it shows you something. Maybe not. So if you do this in your normal app test, you might get panic over something that is just to be expected. Like leave that pain to the library authors. A few other words of caution that even for libraries, this is not always a good approach because you kind of need to write your own test-only components because you have to inject this Replay Snapshot thing into your component. You have to inject even like that could go in as a prop, but what could not go in as a prop is that use track renders thing.

But for the use track renders thing, there's actually like a little bit of hope on the horizon, which is something discovered from Ken C. Dots a few weeks ago, who found a way to inject the React dev tools into his userland code to use the logic that makes these frames flicker when you get a re-render to detect which components were re-rendering at which time. So I'm looking forward to using that at some point, but it's probably going to always be an experimental feature because there are no guarantees on that. One last warning, we all are used to write component tests with ACT everywhere without really knowing what ACT does. At least I didn't know, and it's not mentioned anywhere in the documentation except that you should use it. And it turns out that ACT batches renders together, which is great in a normal test, except if you want to test when renders happen. So the idea here is to not use ACT, which adds a few problems on top. Suspense boundaries are in React always shown for a minimum time of 500 milliseconds. So our tests suddenly were 80 seconds longer, but only in React 19 and React 18, it's fine. And we want the React team to make that configurable. There's a discussion that's been stalled for too long and nothing is happening there. So currently we are patching React to configure it ourselves, but hopefully down the line, this will not be necessary anymore.

And we only patched it for the test. So all good. The real world usage means that we use this in currently 544 tests for Apollo Client. We've been using this for about one year before we released it publicly, and it really fixed so many flaky tests in our code base. And at some point I just was annoyed with the flaky tests. So I did an infinite loop that always rerun our test suite until it didn't crash anymore. And I just test by test by test that was failing, because we had tests that were running fine 99 times, but failing the hundredth where timing with normal React testing library was just wrong.

All that said, you've been warned, but I hope you also kind of see promise in this, so go try it yourselves. It's out in 1.0.3 or something like that. So we already had a few typos to fix. Testing library slash React RenderStream. And with that, we get to the Q&A.

My first question is, I hear your talk, and I think this is amazing. Looks really good. I'm a bit more dumb than you are, but I just use ends like ends render. I forgot the exact thing. You could test for a method being called. Yeah, but not renders. And the thing is that normally React calls the method once per render, twice if you use strict mode, and between one and a million times if you use suspense. The moment that suspense enters the picture, it just doesn't work anymore to have that correlation between the render function executes, and there's actually a commit to the DOM, which is what we're more concerned with. And you can measure how many components are re-executed, but everything else could influence that. Upcoming React features like, there's an upcoming off-screen thing that they were talking about, that would probably affect a lot of things. Things re-executing but not necessarily re-rendering but maybe re-rendering. Like this gives me confidence, just counting up a counter already doesn't and will even less in the future.

Okay, so I can understand that. Thank you. Let's see what the audience has been asking. First question is from Ben Durant. Does this library rely on any spooky React internals that will get me fired? No. Nowadays you only have to warn your users that they cannot upgrade. So you have been warned. But yes, the internals of useTrackRender access React internals to find out which component you are calling it from. But if you don't want that, you can also just give it a string argument and that string argument is used instead. And that will not access any React internals anymore. You're still not using the normal React library as a dependency for this, but the React library, which I also maintain, which just suppresses all the warnings. If you access internals, because I think we shouldn't, but it's still fine and nice to do sometimes. We shouldn't, but it's still fine. That's one sentence. Yes. Fair enough. No, I can live with that. Yeah.

Like, I said on stage, it's fine, but I also warn people. Like, right? I did my thing. Yeah. It's fair warning. Yeah. No. Does anyone feel like it's good what he's saying there? Or are you scared? Anyone more confident now? All right.

Next question is from our valued audience member, Anonymous. How long does it take for the 544 tests to run on the Apollo client and when do they run? They are part of a much bigger test suit, which is I believe like around 2,500. In CI that takes about five minutes because we're still on test, which I honestly still like and don't care about those five minutes. I would sign up for five minutes. Don't worry. Oh, that's okay. Or you run them locally, which means it's like a few seconds. So it's perfectly fine. On every pull request, on every commit to any branch in Apollo client repositories. So quite often. Cool. Cool.

Next question is from Matthew F. I feel like this way of testing can often rely on implementation details. What's your take on that? Yes. Okay.

Yeah. Yeah. Cool.

Next question is from Matthew F. I feel like this way of testing can often rely on implementation details. What's your take on that? Yes. Okay. Yes. That's perfectly fine. If I write a library, I want to guarantee certain implementation details. That's the difference between testing a library and testing an application. In an application, I test for eventual consistency, I test that the UI gets where it should be. But if I write a library, I need to make much tighter guarantees. And that's why I say this is great for libraries. This is great for that super hot code path, like that hook that you call in every second component of your app. You want to test for an implementation detail there, just not anywhere else. Because that additional render adds up with every other additional render. And if you can have one place in your code base that you optimize, and in the end you eliminate 500 additional renders as a result, that's amazing and that's a win. But what I sometimes do is just add those 500 renders, and then eliminate them a week later and say to my boss, look what I did, I made it a lot faster. You can. Don't tell anyone, MadHead. My colleagues in the audience, don't tell anyone. If you write that test, you can also just take a few more renders in between and remove them over time. Yeah. Oh yeah, like gradually improved. Nice. Right.

Next question. I don't really do that. TK is asking, has this made writing tests more enjoyable? So much more.

Yeah. Honestly, it feels much better for the tests that we are doing and need to do, because it gives us exactly the confidence we need that we couldn't get before. It was literally impossible, or with a profiler thing, you don't want to write that twice in your whole life. Cool. Then maybe one, maybe two more.

Anonymous asks another question. If it's too complex to write a test specific component, how do I avoid shipping used track renderers in my component, as it's only relevant in tests? That's why you probably don't want to directly render your component, but you write a test-specific component around it. You can't get around that. I don't see a way around that. Yeah, so I will have a counter component and then a counter test renderer component that just adds the used track renderers. Well, realistically, if you write an app, or if you write a test, better, you probably write a little bit of JSX thing anyways, and that's where you would do that. I see it a little bit more difficult if you have internals of a component. You probably can't test those with this. You would go for DOM snapshots and just look at it from the outside, how the DOM changes, and you still get the information on how many renders you have, but you don't know which part of your implementation detail rendered when, and then that's fine. You probably can't do everything with this, but it's a great toolbox to do a few things.

All right. I guess that's all the time we have. A little Dutch lesson, because you said something that's funny to me. DOM is the Dutch word for dumb. So for me, I always read DOM snapshot. Super fun. That's all the time we have. So, Lens, thanks a lot. Thanks for having me.