Designing for Failure: The Senior React Dev's Production Toolkit

That's what I want to hear! I bring a rubber ducky to every single talk of mine because it not only helps wake people up a little bit. As you saw, I'm from Switzerland, so it's cheaper to afford than therapy over there, and it's easier to debug with, too, for the OGs that want to debug stuff with instead of running to our AI assistants. I loved the last talk because there was a hint about we need to go back a little bit and reaffirm the fundamentals, and so we're going to talk just about that today, about how do we design for failure and go a little bit to some fundamental engineering concepts to be able to do so, and what better way to look at how to design for failure than exploring some real-world failures together.

So a little bit more about myself. I'm a staff front-end engineer at Small PDF. Has anyone here used Small PDF before? Okay, I'm going to ask another question. Anyone paid for Small PDF before? All the hands have gone down. I work on the monetisation team for Small PDF, so when the hands go down, it means I'm not doing something correctly at my day job. So I hope next time I come to React Summit we can have a couple more hands up. Funny other thing about working at a PDF company is that, after talks, a lot of people come to ask me PDF-specific questions. In almost two years of working at Small PDF, I've not touched the PDF part of the product. So I reserve your questions around PDFs for some other experts for that.

But around that, I've worked in connected TV industries. I love web performance engineering leadership. I'm also a RayCast ambassador, a big fan of RayCast. Anyone here a RayCast user? Yeah, I'm seeing some more hands up. Fantastic. Co-founded Zurich JS, and then also a Stripe community builder. Now I like to take the titles of my talks and sort of reframe them into a question. So for designing for failure, I like the question to be a little bit more, to be answered around how do we build resilient UIs in React. Now, even though we're talking a lot about React-related stuff or even some Next.js stuff, a lot of these principles are stack-agnostic. So whether you're a Vue developer, Angular developer, you can take these patterns and apply them to multiple different stacks. You're not just stuck with something.

To understand resilience a little bit more, let's look at the definition of resilience. And what may be the first thing that you do is you go on Google, it's like, how do I implement software engineering patterns? And you're going to see some complicated explanations of that. And I like to go back to just English vocabulary and how do we better understand the words that we use in engineering, because they come from non-engineering spaces, too. So what is the Google definition of resilience? The act of springing back or bouncing from something, you know? And the question is, why do we care about making our React applications resilient? Why do we want it to bounce back if it gets knocked down? At the end of the day, your applications, your web applications are integration layers. There's so many things being integrated before something makes it all the way from local host all the way into production. And each of these integrations essentially serves as a point of failure that exists in your application. You're dealing with all sorts of APIs. You're dealing with payments, authentication, feature flags, client caches, optimistic updates, and I can name a bunch more. And each of these have different ways in which you want to apply resilience to.

I want to speak a little bit also about outages, because when something isn't resilient or something fails, naturally there's an outage or something goes wrong. And the easiest ones to notice are total outages. Now, I know the room is a little green over here, but does anyone have the favorite color red? Anyone like the color red? Depends on the day, right? The easiest outages to notice are the total ones. This is a total outage. Something went wrong really, really fast. Payments just weren't working at all, and so you see red alerts everywhere. Things are going wrong. And to give you a little bit of context, small PDF has around about 30 million users per month, right? And so when you get a total outage and something goes wrong really, really fast, the red bars climb just as fast.

The hardest ones to notice are the subtle outages. Ones that are almost like death by a thousand cuts. The ones that bleed slowly over time. And those are more when there are partial issues that exist. When one singular API's down, but the rest that are fine. The ones that may be regional, where things are only, you know, in Switzerland, everything works, but the second I come to Amsterdam, everything's failing. Then there may be conditional issues where only retry mechanisms fail, but it only happens on Safari. Then you have race-driven issues, and let's say a user's trying to pay before the payment gateway is loaded, so they're clicking a PayPal button, but nothing's loaded to be able to start executing a payment. And I just wanted to take a snapshot of our error dashboard using track.js and small PDF. If you're looking at the numbers over there, in seven days, we have 732,000 errors. With around about 22 million page views, a quarter million users impacted, and guess what? We're 17 per cent down.

If you're looking at the numbers over there, in seven days, we have 732,000 errors. With around about 22 million page views, a quarter million users impacted, and guess what? We're 17 per cent down. That's a good week. What's so interesting at this scale is that I almost see this as a sea of errors, right? If you take a rock and you were to throw it into the sea and there are all these waves, you would never see the ripples, right? If you were to have a pond instead, a much smaller body of water, and you throw a rock, like an issue going on, you would see the ripples and be able to detect something going on.

So in such a vast amount of issues, how do you know what's the right thing to detect and react to, whether it's a paying user that's having an impact, or a free user, or it's a payment flow that may be more important than a PDF download? And so I want to show the first incident of one of those bleeding cases, where we had invalid postal code payment failures for Canada and Great Britain, and around about 4,400 users were being impacted. And what happened was we had a geo IP endpoint that was trying to infer a postal code for a user based on the IP address, and it wasn't getting it for the right, it wasn't passing validation essentially for Canada and GB. But those are not our highest ARR markets necessarily, or where we have the most users, so compared to all the other countries that we operate in, you're not really actually going to see a spike in errors significant enough to trigger an alert.

And so this ended up lasting for 35 days. However, that cost in over £30,000 of lost revenue essentially. Whereas if you had a total outage and things started spiking to the point that you noticed it within five minutes, I doubt that it would have been the same amount of monetary loss. Let's look at another one. 3D secure payments failing for nine months. Has anyone here ever experienced an outage longer than nine months? Okay, cool, we got a record, right? That's an extremely long amount of time for something to be going wrong to the point that you have leaking revenue from your system, but it's not considered a global outage. You're still able to download PDFs, compress them, merge them, so on and so forth. And so at scale, detection is much harder than prevention, and so that's why preventative measures like introducing resilience patterns are that much more important, because you cannot detect everything.

So what are typical resilience patterns that we could employ? The first one I want to introduce is a circuit breaker. And again, while we're introducing it here in an engineering concept, circuit breakers exist in nonengineering concepts, software engineering concepts. Anyone have fuses at home that they've ever played with? A fuse breaks, breaks the circuit, saves your house from catching fire or something like that, right? Similar case in software engineering. It prevents an application from repetitively executing an operation that's likely to fail. And so how does that operate? You have three states that exist within a circuit breaker.

A closed state, and normally, I don't know, it takes a minute for me to get this in my own head, because when I think closed, I usually think red. So in this state, closed is green, because the circuit is closed. And you have two systems operating, connecting, and moving. You have a half open state, and then you have an open state. So we already talked about the closed state. There's normal operation, requests are passing through, you're making API calls to upstream providers, everything's all good.

Then, after a failure, we go directly to an open state. Everything pops open, and a threshold has, for example, been reached. And once that threshold has been reached, you're blocking requests at that point because you already know something's not gonna work, so why bother keep on trying. And then what happens is after a timeout has been reached, let's say 30 seconds or two minutes, whatever it may be, you slowly start testing a connection again. And if it passes, you close the circuit again. And if it fails, you pop it back open. And the way I like to reference this is imagine you've really annoyed your best friend to the point that they don't want to talk to you again, and they need space and time.

That's exactly the circuit opening up. And then every now and then, once they've gone farther away and given them a little bit of space, you go check in with your friend and poke them, are you okay? Are you fine? Are we good? Can we be friends again? And you have that connection that reexists, right?

So treat your API sometimes as they can get a little bit annoyed by too much interaction. But how does this look implemented in a case like with Next.js? We can do this very easily with the API handlers that we have in there. So what we could do is take a very traditional get request, and what we do is create a product details breaker. So this is just a product details fetch request that we want to be able to do. We fire the product breaker, we await the data, and essentially, if everything's all good, we just pass the circuit status and the data along.

If anything has gone bad in the catch statement, we will return a fallback of some sort. And going a little bit more into that product details breaker, essentially over there we're defining within the fetch product function a couple of parameters around the circuit breaker functionality around the timeout, the error threshold percentage, and when that timeout resets. And this is using an open-source library called a possum, so it's pretty simple to integrate something like this. The next one I want to introduce is the retry pattern. Who's used a retry before? I think it's like the first one that anyone picks up, right?

One request fails. This is the reaction of the application. Launch all the retries now. And let's say your API has a global rate limit that you're contacting, and you have an outage like this where Recurly's API quote limit has been exhausted. And so what happened in this case is that we queried one endpoint for our payment service way too many times, but because it was a global limit and we started retrying too many times on failures, once we hit the limit, anything else like creating subscriptions also failed even though we were reaching the limit for a request that was unrelated to it. So you can have a really big impact on your platform if you retry too much, and not every error is or should be a retryable error, and we ought to refine when and how we execute these retries.

Beyond that, you have exponential back-off which is beyond the simple retries. You have three default retries like you have in the TAN stack query, retry one, two, three, they execute immediately. Otherwise, with an exponential back-off, you have the first request, then you wait maybe one second. Then you have the second where you wait two seconds, then three seconds, you're giving a little bit of breathing room. But this in and of itself can also be problematic because if you have multiple clients doing this at the same time, what can happen is that when an API comes back up, you can then have multiple clients hammered back down to the point where it goes down again, and this is where you introduce something called jitters.

Retry jitters are essentially beyond that exponential back-off time, you essentially add some randomness to each request to make sure multiple clients aren't triggering requests exactly at the same time. There's some randomness. If you're a bit of a backend engineer, you can think of this like salting a hash password to add that additional randomness, same thing we're doing with a request, and this avoids this thundering herd problem. The time-out pattern introduces another approach. In this case, a demo is shown with a financial dashboard. By simulating latency in an API response, a strategy to bail out from tasks taking too long is presented. Browsers have longer time-outs for such situations, and custom time-outs can be implemented to optimize user experience even in suboptimal conditions.

First, we're going to look at a financial dashboard over here, and what I'm going to do is I'm going to simulate some latency in an API response where we're making a request on the server to be able to fetch data, to be able to have it in the initial HTML and send it to the client. What happens in this case is because we're doing this work in, let's say, get server-side props in Next.js, everything is blocking. You're not sending anything to the client until the requests are resolved, and let's say those requests take a super long time, your fastest rendering time can only be as fast as your slowest request. And so at some point, you may want to implement a strategy where you bail out from something that's taking too long, and browsers have much longer time-outs for such things, maybe even up to 30 seconds, so you may want to have your own custom ones for this. So this is an example of how that may have been implemented for that financial dashboard. You have a get server-side props, we're doing a prefetch over here with some sort of request for these transactions, we're dehydrating that, we're passing in the props, and what we can add is a layer on top is just a function to create that prefetch where we maybe pass a time-out of some sort that uses a promise.race under the hood, and we pass a 500 millisecond time-out or something else, maybe even a two-second time-out, and then depending on if the time-out triggers first or the request results first, then we can bail from that, and then because there will be no cache ready on the client side when the hydration takes place, Tanstack query will match up the query keys and try to essentially make sure that we have that data being fetched on the client side, instead of trying to prepare it on the server side. In this case, we're creating a great user experience in the best case scenario, but in suboptimal conditions, we're still optimizing for user experience and bailing out from those conditions. Another way to go about this, if you've got a later version of Next.js or you're working with a much more modern stack, you can implement streaming which allows you to start that same request on the server and carry it through to the client, and everything can resolve in its own manner, and you're not blocking anything, essentially. So Next.js 16.3 is worth trying out for this kind of stuff with partial pre-rendering, cool techniques you can do to add resilience and make sure you're doing blocking work with suspense.

The next one I want to ask is anyone ever heard of a bulkhead pattern before? Hands up. Okay, a couple of people. I can guarantee you at least 50 per cent if not 90 per cent of the people here have used a bulkhead pattern before. So the bulkhead pattern isolates a part of a system into a separate pool, isolates one part of it that fails, or gets overloaded, or something happens that does not impact the rest of the system. Error boundaries, right? Anyone here heard of error boundaries? That's a lot more hands. So, bulkhead patterns, something that's more a broader software engineering term, again, something we use every single day, but it's really good to be able to associate these patterns, and so who here is just throwing a global error boundary on the application and they're happy with that? I see that very, very, very commonly. So we're not surprised when we see a scenario like this where we have multiple different pieces of UI and then, boom, application error, or I'm going to type something in here and then all of a sudden I type the wrong thing, I'm going to crash again, right? Why is one part of the UI impacting an unrelated part of the UI? And even further from that, if I open up Sentry, and I was just capturing Sentry, whatever the raw stuff that I was receiving from those errors being thrown, I see error, type error, error. Anyone know what's going on? I would ask the rubber ducky, but it's a little tired right now. Okay. So, I think that's also a little bit unacceptable to not be able to diagnose fast enough, so I really want you to think about trying to build quadrants when it comes to error boundaries, trying to isolate as close to where you think relevant functionality should be scoped, and so, looking at this quadrant error boundary, we're doing a couple things over here. The first thing is that we're actually going to be defining something called an error tag, and we'll see how that comes in handy in a second. Every time you define an error boundary, you pass an error tag and using the type system to ensure that it ends in error, but it doesn't matter if it just ends in error, you can essentially use the string templating to be able to put boundary error at the end, or you're taking advantage of the type system to enforce some level of tagging that you can propagate to your error reporting tool like Sentry to make it much easier to debug when a scoped piece of your application goes wrong. In that component, it then passed that to this report error function, which takes this error tag, and then, in that, we're essentially just creating a new error from that, passing the original one as a cause, and then we report that to Sentry. What does it look like when we start to isolate things a little bit closer to where they're going wrong? Not only do we now have unique UIs that exist just for where things are going wrong, like, if my nav bar is broken and for some reason errors out, why should my widgets also suffer as a result? You can then have user experiences that are adapted to the failures that are happening. But beyond that, because you're passing error tags, you now receive things like timer countdown error, input render error, you can isolate exactly where in the codebase the issue happened, and that means because you're using also the type system to guarantee that they're all unique identifiers, what you can do is you can then just copy and paste whatever the title of that error was into your IDE, search by it, and you know exactly where that error happened rather than having to run through a stack trace. Because a lot of us get super overwhelmed when we receive a lot of Slack messages and alerts from how many errors happen in Sentry, how many errors happen that we get via email. Sometimes we just mute and ignore those channels, and what's great is we'll receive an email like this with a button no worky error, and I know exactly what went wrong in my application, and I can triage effectively rather than getting overwhelmed. But all in all, how do we measure success with this? And that's where I want to talk a little bit about dora metrics. Anyone here heard of dora metrics before? A little bit about DevOps research and assessment metrics, and they're around four core things. Deployment frequency, how often you deploy code, lead time for changes, time from code commit to running a production, mean time to recovery, which is how quickly a service is restored after an incident, and change failure rate, the percentage of changes that cause incidents. Now why do I love these metrics so much? Because I think we've sometimes over-glorified unit testing coverage sometimes, or all these other metrics that are much easier to game, and I'm sure these dora metrics can be gamed too, but a lot of us focus a lot about how many unit tests we have, or how many our AI are generating, integration tests, end-to-end tests, what lint rules, how our perfect Cloud MD files are, or whatever else it is.

It doesn't matter. All these DORA metrics help contribute to better understanding end-user experience, and that's all that matters. Is the thing available? How quick can I fix something once I have a commit to be able to push to production? How often am I releasing something where things are going wrong? And that's all we should be thinking about. So, so long, and thanks for all the fish.

I just want to start by saying, amazing talk. I have a specific quote I am actually going to use. When you talked about how errors, it's like an ocean. It could be an ocean or a pond. And when you have a small project, not many users, every error is like a big ripple. You can see it. But you throw a pebble in the ocean, you don't see anything because you've got these waves, and then even waves, some are tsunamis, some are waves. And it's really interesting. And I think, as well, what you talked about, that example of a massive outage that you fix in five minutes may stress everyone and not cost the company money, whereas the small outage that has been going on for 30 days, that example, that was really, really brilliant. So I just want to pause and just say, I really appreciated the talk.

We do have a question that came in, which is about anti-pattern. So what's the most common anti-pattern you observe intermediate developers using when they're handling production errors in React? There are two core ones, and I mentioned both of them. The first is over retrying. So if your example are retrying on just any error, if response is not okay, just retry. That's not what you should be doing. Like, why are you retrying on a 400 response, for example? That may be that whatever you're sending is not valid, there's no point in retrying, something like that. So being able to drill down, what are the errors you're receiving from the APIs and segmenting them into retryable errors and not. And not just going for the default of three, but trying to essentially detect how many times you're querying an API and how far you are from reaching an API rate limit, that's also incredibly helpful, because just a default of three is a good rule of thumb, but it should not apply in all cases. Especially when a product is at a massive scale, you want to make sure that you're not hammering an API's limits, or at least you negotiate with whoever has the API limits, like if we negotiate with Recurly, maybe they're able to increase those limits depending on the use cases, so optimising when you retry.

The second one is error boundaries. You already saw a couple of hands go up. Error boundaries are always that massive global try-catch you put at the top of the application, and I've generally had a hard time understanding why we aren't able to isolate that a little bit closer to where the errors could be going on, but if we're able to do that and pass not just the error tags to error boundaries but any point of failure, error tags should be passed to fetch requests, anything you may be wrapping in a try-catch, those are just simple things you can move away from what you're currently doing as practices. I love that, and that kind of leads us nicely into the next question, which is how do we find, I know you gave us some patterns, but a lot of times people have slightly different systems, and even finding errors is going to be very dependent on the way their system is built, where failures are, what failures are important or not. What are some pattern recognition approaches, should we say, for people to figure out what patterns they should be implementing, maybe some new patterns, how can they discover them? That's a really good question.

I think also the most common answer to most questions in software development is it depends, right? It's my first one of the day. Right? I mean, there's two parts, right, there's discovering what are more patterns that exist than the ones that I just mentioned, and just being able to actually just Google software resilience patterns will already take you down a rabbit hole in and of itself. There's a lot to explore there. Then the other part is also opening yourself up to the English dictionary, which is how do you relate some of these patterns to real life, and you may already have been implementing patterns in real life that are resilience patterns that you may want to be able to shift and use within your software products. And then the other part is, okay, now I have a code base, and I am understanding what problems I have, which pattern do I apply? Do I apply the retry? Do I apply the exponential back off? And the best answer I can give is actually testing stuff out, right? If you're doing stuff without measuring success, and not just Dora metrics, but in that same sense of measuring, with retries, how far am I from hitting an API limit, maybe put a threshold where if I'm 90% used up in terms of my API rate limit, then trigger a warning and I can reevaluate my retry strategy, that's where you may consider, maybe, okay, I'm going to add some exponential back off or some other strategies there, but being able to measure the pattern you put in place is already a great way to start, and figuring out what iteration of it is the right one for you. Otherwise, I can't give a simple example of just go for circuit breakers here, go put error boundaries everywhere you can conceive, and now basically have an AI agent run and add everything automatically. That's not going to work out for everyone.

Nice. All right. So imagine someone asked the question, should I use an error boundary? You might have answered, it depends. And so let's talk about that it depends. So this next question is, how do you decide when to use an error boundary or to handle it in a different way? And what is your process? So it depends on how many, oh, yeah, great, it depends, right, how many points of failure exist within the component you're trying to isolate, right? Are you doing multiple fetch requests, multiple calculations, multiple things that could go wrong? Even if you have something going wrong in Sentry a little bit higher up, or you're receiving a lot of errors and that an entire page, essentially, is erroring out or sending you stuff, can you isolate in the stack trace if it's happening somewhere lower down in the tree and find a reasonable middle ground for where in that component tree you may want to slot an error boundary. And you can always move that lower or higher up in that tree, but the first part is figuring out where are you hitting that threshold of a lot of errors being populated and then where could you move that down and isolate a little bit more and where is it reasonable to talk with your UI designers on having an error boundary fallback specific UI to be able to have a better user experience because each error may give you a different reason to be able to put another UI there.

Some errors may want you to opt for contacting support, other ones may be just retry buttons that reinitiate the request or restart the process of streaming a component down, so every interaction is different. And I love that case of every interaction is different, because it's, like you said, user experience is the end goal. And it's also not just catching errors so you can fix them, it's so that a user knows we are aware something went wrong and here is the path to remediate it as well. So I really, really enjoyed that. I love this one because it's kind of interesting. Would you recommend having an error boundary on top of the top most parents to catch all the errors? I think you kind of taught this, should it be on each child level and where do you do it? And also when you create an error boundary, what detail do you get about where the error actually is? So should we have almost nested error boundaries? You should definitely be having nested error boundaries. You should not get rid of your global one in favor of more scoped error boundaries. You should keep the global one because you're assuming you will have put your essentially or try to catch statements or your error boundaries everywhere where there could be a conceivable error. And humans make mistakes at the end of the day. There's an engineering resilience and there's also human resilience that exists within the ways teams operate to be able to avoid failures in software products.

So it goes way beyond the scope of engineering, too. So making sure that you also have something to fall back to, which is the worst case scenario, the global boundary, and then tracking that through your error tracking system and then being able to choose where to isolate further afterwards. You end up saving yourself rather than assuming you put everything in the right place. All right, Faris. I think it's time for lunch. All right. Thank you so much. Really appreciate it. I know there's a few of us still here. Can we give Faris a massive round of applause? Thank you.