Caching, Payloads, and Other Dark Arts: Optimizing UX in Suboptimal Conditions

How do we optimize user experience in suboptimal conditions? So I got a fantastic introduction, so I'm not going to take too much time. My name is Farris. I've had experience in the connected TV industry, building applications such as Discovery Plus, Eurosport, GCN, worked across FinTech, fitness technology. I love working on open source stuff like Raycast. And I also organize a community called Zurich JS.

As you can see, sometimes I like hanging out with the React Summit Program Committee, as in the photos. So what I want to do to start off is I want to set the stage. And to set the stage, I want to show you this photo. Now I'm from Switzerland, but I'm also from Pakistan. And over there, the traffic isn't the easiest thing to deal with. People buy a bunch of different cars, predominantly Japanese cars. And over there, they're used in every way. They're not intended to be used. So a car will be bought and it's going to not be serviced. It's probably not even get the right fuel in it. And it's essentially an engineer's job to have built this thing to last in conditions that it was not actually built in.

And some of these cars actually go all the way to 620,000 miles. I converted that from kilometers for you just to make things work so you can understand it. But that kind of brings us to the conversation on like we build applications not in sort of the perfect factory at home with a great Wi-Fi connection, so on and so forth. Maybe even Ethernet connection. But our applications aren't actually used in the real world. We started them on localhost and localhost got stable network connection, great bandwidth. You're a single user using the application in an ideal environment. And the real world is very different from that. The real world, you have high latency, unreliable networks, mixed devices between mobile phones of not all the same quality and CPU powers.

So you have throttle CPUs. Then you can get into places where you have partial caches, stale data, flaky APIs. So there's a lot to deal with. And so the problem statement that I want to discover today is what challenges do developers face creating a seamless user experience due to complex data fetching specifically and management? But that question really is, how do we build optimal solutions in suboptimal conditions? And that does not just apply to the real world, but also in startup conditions. You don't have everything handed to you. And sometimes you have to work with just what you have around you.

Now, why do we bother wanting to actually build these experiences that are fast and seamless? Amazon actually did a study in 2006 where around about 100 millisecond increase actually reduced their sales by 1 percent. Back in the day, that was around about 100 million in revenue. And today that would mean over 3 billion. So at scale, this makes a really big difference. But just to get into a little bit more of like what are these golden windows that we want to render our pages in our applications or components? Zero to two seconds is quite ideal. But to dig a little bit more into it, I like to go over the response times of three important limits over here. And there's an excerpt from a chapter on usability engineering.

If you render something in point one seconds, the user feels like the system is reacting instantaneously. It's a really nice experience. If you have something load in one second, it tends to be around about the limit for users flow of thought. Anything longer than a second, if you're clicking through a bunch of buttons and you're going through different workflows, if it takes longer than a second to react, you break your train of thought. And this can be really detrimental to user experience. And if we go all the way to 10 seconds, that's really the limit for keeping the user's attention. And these days with all the TikToks and the six second videos here and there, I am pretty sure that 10 seconds doesn't even hold up. And we already lose our patience and are out about five or six.

In terms of numbers, the bounce rate increases around 32 percent when load time goes from one to three seconds, escalating to 123 percent at 10 seconds. At a React conference, we delve into two core technologies: React, a library, and Next, a framework, with principles applicable across frameworks. Tanstack Query, an asynchronous state management library, is highlighted for optimal state management. Discussing API types, distinctions between public APIs like Pokemon API and internal APIs used within cross-functional teams are emphasized.

Interfacing with servers involves API application programming interfaces, varying from GraphQL to REST APIs. Public APIs, like the Pokemon API, are considered black boxes with potential for breaking changes. In contrast, internal APIs are tailored for specific client needs, often collaboratively designed with cross-functional teams. The constraints of using public APIs differ significantly from those of internal APIs, where backend teams can customize endpoints to suit client requirements.

You can apply them to Nuxt and a lot of different things. So shout out to Daniel Rowe there. And then also Tanstack Query, which is a powerful asynchronous state management library. If you're not managing your asynchronous state with Tanstack Query, you should be doing that. There's nothing else you should be using. And if you need convincing, I'm sure Tanner's around to do plenty of that. So those are the two technologies.

What I want to touch on is also API types. When we're interfacing with servers, we're interfacing through API application programming interfaces. But not every API is the same. And I'm not just talking about GraphQL APIs versus REST APIs. I want to talk about, for example, public APIs like the Pokemon API or the Star Wars API. Public APIs can be seen as black boxes. You have no control over them. They're not built by you. You know, what can happen also is that they could be subject to breaking changes.

The constraints of using an API like that is very different to, for example, an internal API. So an internal API is an API generally dealt with very often if we are working in cross-functional teams with front and back end engineers. And those are internal APIs where maybe they're a little bit freestyle because the back end is designed specifically to serve your client. So if I want to be able to show a graph of some sort, I'll tell my back end team I need an endpoint structured like this. Let's work on the open API spec together. And it's designed in-house. There are no external consumers.

There are no external consumers. So maybe there's limited adherence to REST design principles. Maybe it's even lacking authentication. There are a bunch of different things when we're entering that world of internal APIs.

The final one I want to focus on are partner and product APIs. In this category, your API that you're working on is sold as a product or designed for third party consumers. So there are limiting factors here because you may have to abide by deprecation policies. And generally, when you're designing those APIs, you're going for widespread compatibility over internal optimization.

With all that stuff in mind, let's put our tech lead hat on and let's say we got a Jira ticket and we're going to solve a problem together. This is a problem I actually experienced at a FinTech, you know, two or three years ago. Our front end team needs to design a dashboard chart that displays the distribution of birthdays for the top 5000 users in our system. What's the data source that we have? The data source is available via an API which returns a list of users with all the data. The backend team is unable to provide a new endpoint or data warehouse that returns the required data format that the front end needs.

So key information is highlighted in yellow over there. What's the data source that we have? The data source is available via an API which returns a list of users with all the data. Check. Fantastic. We've got one thing down.

What's the limitation that we have along with this? The backend team is unable to provide a new endpoint or data warehouse that returns the required data format that the front end needs. And there's a limit of a thousand years you can return for each request.

So now if I put my hat on a little bit, I'm like, I need to have data for 5000 users, but I can only get a thousand at a time. If we design architecture, what does that look like? I need to create a user interface. If I need to get a thousand users per single request, I'm going to make five parallel requests. Two things can happen.

I have a loading state initially to show the user, distract them a little bit. Maybe they're rubber duckies floating all over the page. And if something goes wrong in that API request, I have an unhappy path. Maybe you persist the loading state, and maybe you have a custom retry mechanism that you have. You either retry all of them, or you retry the ones that failed. Not ideal. But the alternative is that you have a happy path.

You may receive the data back from the API. You map the data, set it into state. You render the UI. And now if you're not using any external library to manage any of this state, what could happen is that a state that's completely unrelated to the data you retrieve from the API could actually cause maybe a user effect to re-render. And then now you go through that pain where all the data fetching starts all over again, and you may land yourself in getting a 429 error. That's not great.

This is actually the code stuff using the Nuxt theme. Another shout out to Daniel. So let's look at the code snippet for how this may look. I still see a lot of codebases that use user effects at the end of the day to manage their data fetching. I mean, TanSeq create at the end of the day also uses user effect under the hood. That's not a bad thing, but it's very hard to get it right. So you may create a hook called useFetchData.

In there, you're getting the fake data from a faker API. You're passing a quantity. Maybe you want five. Tomorrow, the system extends, you want 10 requests to be made. You get it back, you're going to map the data. So you get the birthdays by month, and you're going to map it to the chart data, and you're going to set it in state. Fairly self-explanatory.

So let's do a demo and see how that user experience is like. So I load the application, five requests are made, and it's painfully slow. Let's take a moment of silence. See how long this takes. OK, there. We're there. OK, we got all the data. It took a wonderful 13 seconds to load. I'm sure you already started getting distracted and thinking about other things. Now, if you look at all the data, you see we have date of birth, ID, location, login stuff. For some reason, we probably have secrets in there.

Oh, there are passwords also in the API request. We probably don't need that. This is not great. Maybe it's a little bit of an extreme example. But let's say I want to go to dark mode, and now I'm loading everything all over again. So we've got a couple of problems in this initial design. Maybe it's a bit of an extreme one. But if you're great at math, you will have calculated that the total combined payload of those five requests was 1.58 megabytes, 80% of unused data. So let's look at the key factors of this initial solution. So you have custom retries in place. Let's say those need testing, maintenance, subject to bugs. Not great.

There's no caching by default. Your state. It's a little bit complex to manage globally if you're using context providers and set states. It's really hard to manage state and react. Then you've got to do data mapping effort on the client side. Maybe you'll swap in a little bit of use callbacks and use memos to save yourself a little bit some re-renders. Not great. You need to do some explicit memoization there. And the payload is not optimized for state, nor is it for caching.

Now, to go to the next part, I want to bring up something, 14-kilobyte rules. Anybody ever heard of the 14-kilobyte rule? Okay, one person, because he has attended my talk before. So that's a bit cheating. So in 14 kilobytes fits within a single initial congestion window for network requests, minimizing round trips. So this is a piece of information that we can do to actually improve this architecture. How does this work in practice? So let's say your client is going to go to the server and say, I want to see website.com. It's 14 kilobytes or under.

It's going to say, Hey, all good. I'm going to return that in one single request. I don't need to make a round trip to the server. Good. I've rendered everything. Now, if I have a 15 kilobyte website, it's going to go request it, get the first 14 kilobytes, say, Hey, I've acknowledged everything. Can I get the rest of it? It's going to return the final kilobyte, and then it's all good. So actually, the difference between 14 kilobytes and 15 kilobytes can sometimes be 600 milliseconds, whereas the difference between 15 and 16 kilobytes is pretty much negligible. Now, this is around about when the 14 kilobyte rule was a little bit more prominent, but now the wording's a little bit obsolete, but the principle still sticks around.

Most modern CDNs can actually have an initial congestion window up to 46 kilobytes. This is very variable. Don't try to focus everything around 14 kilobytes, but focus on this idea of initial congestion window. And to focus on that, we have to talk a little bit about the stall. The stall is that cost of that round trip we talked about. When we get the initial data, we've got to go off the server. Can you give me some more of that? And that round trip cost really depends per user. On LAN connection, it's one to five milliseconds. You're not going to notice it.

Typical broadband, 30 to 70 milliseconds. That's pretty OK. And on 4G, 5G networks, it might be a bit more noticeable on 80 to 200 milliseconds. So the general idea is you want to get your critical rendering payload in the initial congestion window. And that's where I want to introduce this back end for front end pattern. It's going to be your best friend forever for making this a little bit better. And this is going to be specifically the pattern called the transformation bridge or the aggregation proxy.

There are multiple ways to go about BFF pattern. The great thing about it is you can adapt it for platform requirements. So let's say you have multiple clients. You want to use a single proxy to serve all those clients. It can be really, really helpful if you have a mobile client, TV client, web client. You do decoupling of concerns. You separate the effort of being able to fetch the initial data, mapping it, sending it to the client. You can do a lot of focus for optimizing for performance. And what you can do is eventually create an initial structure that looks like this. To render my chart, I only need this data. I just need the months and the count for each month and maybe the total users.

So if we evolve this architecture, how does it look for us? Using Next.js, we're going to use an API handle that Next.js provides. The user interface starts with one API request, then progresses to making five parallel API requests at the handler level. This leads to two directions: happy path or unhappy path. In the unhappy path, a single failure code may be returned, initiating a retry journey.

On the happy path, all data is mapped on the server side, eliminating the need for front end memorization efforts. The data is then sent to the client, set in state, and rendered in the UI. However, improper state handling may lead to frequent triggering of use effects. Evolving the architecture involves using the useFetchData hook with a fetch function hitting the API handler, providing benefits of failure isolation and clear separations of concerns.

Data mapping is handled at the proxy level, ensuring only necessary payload values are optimized. Local storage limitations of three to four megabytes can be a concern for offline caching. Utilizing a proxy with a single endpoint can significantly improve UI rendering speed, as demonstrated in a second demo.

But you're still subject to that issue. If you have not handled your state correctly, you may be able to be triggering use effects more often than you actually want. So we take that hook that we built and we evolve it to this new architecture. We're actually just going to take that useFetchData hook and we're going to pass a fetch function that hits our API handler. And what that gives us is a couple of benefits of failure isolation to a singular client request. Your client doesn't have to manage multiple requests.

From a code-based management perspective, there's a lot of clear separations of concerns. You can have separate teams work on it. It doesn't just have to be an XJS proxy. You can also have ways to do this in Quick. You have ways to do this as a separate Node.js microservice or in any language. Data mapping is done at the proxy level. You don't have to worry about it at the front end level. You only get what you need. For the payload, it's optimized to the minimum viable values that you need.

And what's nice about that is that local storage, if you were to do some offline caching, actually has limits to three to four megabytes depending on the browser being used. So if you were to get that 1.58 megabytes and you were to actually make more requests than just that because you're not having a lot more than if you have a lot more than one chart, it could get problematic. You may run out of local storage at some point. So let's look at a second demo and see how just adding this proxy with a single birthdays endpoint actually makes it a lot faster to render the UI. We've done nothing else except change where the request is made and we're returning only the minimum viable data.

And we've shrunk that down to 300 bytes being returned in around about five to six seconds. So we think about network bandwidth with that original 1.58 megabyte payload under Wi-Fi conditions takes 340 milliseconds. That's pretty all right. 4G conditions, 700 milliseconds. 3G conditions, 6.5 seconds. So we're breaking the flow of thought here.

Now, if we move to just having a 300 byte payload, the biggest tax here generally is that round trip to the server and stuff like that. It's actually the ping and everything. So for a network bandwidth at 300 bytes payload, if we're looking at Wi-Fi, again, 90 megabytes per second takes 200 to 210 milliseconds. At 4G, 200 to 210 milliseconds. At 3G, 200 to 210 milliseconds.

When you only return what you need, you're exponentially saving on the time it takes for that network request to resolve. And your limiting factor isn't the size of the data. You have a lot of other limiting factors. Now, to get into a little bit of this, I'm going to talk about ingress versus egress, too, because there are multiple ways to go about this back-end for front-end pattern. Ingress is data coming into a system or network. Cloud providers like CloudFlare, AWS, they generally don't make you pay for ingress because they want data to get into your system.

It's more with egress when data goes out of a system or network and they're going to start charging you. So you don't necessarily have to worry about cost too much with a solution like this. And there are two ways to go about it. We talked about Node.js microservice, Next.js API handler. That's the difference between a function as a service or an infrastructure as a service.

Function as a service is your Next.js API handler, so on and so forth. Single stand-up Node.js service is your infrastructure as a service. And you can compare those a little bit because at your function as a service level, you can have actually a very high ingress bandwidth, so data coming into the network can be done at really high consistent speeds. So that's your AWS Lambda, your virtual edge functions.

The only real disadvantage you have here now with function as a service is that you have cold start times. So it takes around 100 milliseconds, maybe even two seconds to heat up for the first request and then subsequent requests are going to be fine. Infrastructure for service, sometimes a little bit more expensive. It stays hot the entire time. Bandwidth is a little bit higher. It's very hard to calculate that bandwidth. And that's your AWS EC2, your GCP compute, stuff like that. And it's always running. You don't have to care about cold starts and things like that.

You don't have to care about cold starts and things like that. There are multiple ways to go and navigate this. So I mentioned TANSTAG's query earlier. I'm not going to get too much into it, but there are a load of benefits just out of the box when you use TANSTAG query. So I highly, highly recommend it. And we're going to now plug it into our system to see if we can evolve it even further.

Now, we're still making that single request. We're having React query handle that. React query is handling the loading state here. We don't have to do any set states ourselves. If we have the unhappy path, things fail. React query handles retries. Not only can you handle standard retries there, you can also do exponential backoff. So you do maybe one request. Wait one second, two requests. Wait two seconds. The exponential backoff really helps give some breathing space. And then you can also add jitters in there to avoid thundering hurt problems, which I'm not going to have time to get into today.

Let's say you have the happy path in this scenario. You map the data on the server. You're going to send the data to the client and you're going to cache it. If you render the UI and then you have a state update that changes, that's unrelated to that chart, for example, what's going to happen is that TANSTAG query is going to go check for in-memory cache. If it's a hit, re-render the UI. If it's not a hit, it's a miss, then go and get another network request. But now you're saving yourself on re-renders here. So now if we change that hook that we had previously, it's going to be using this use query hook. We're passing query keys to have a reference to the data that's stored in the query cache. Managing stale time and we pass that same function to get it from our proxy level. So what's improved here now by delegating this work over to TANSTAG query? You have control of your loading state, errors, deriving response states. Caching that can be optimized with multiple different strategies.

You can also use local forge for offline caching with across IndexedDB, WebSQL. Then re-renders are limited over here, and you can manage data staleness a lot better, and you've got resilience out of the box, which is a big lifesaver when you're dealing with complicated situations.

We can go one step further, and I'm slowly running out of time. We can talk about this more, but I'd also like to bring up the topic of query criticality because not every API request carries the same importance. We generally make multiple API requests when we have either a single-page application, multi-page application, or when getting data from various sources.

If we take one more step with this architecture to a final place, we're going to actually switch things around and start our journey on the server. You have GetServerSideProps and Next.js to preload data on the server, populate the initial HTML, and send it to the client. Let's make that request on the server. We're going to use React Query's prefetch in there.

So a little bit of query criticality we can do to prioritize or handle requests differently. Don't treat every request the same. If we take one more step with this architecture to a final place, we're going to actually switch things around and start our journey on the server. You have GetServerSideProps and Next.js to preload data on the server and populate the initial HTML and send it over to the client. Let's make that request on the server. We're going to use React Query's prefetch in there.

What's going to happen is that five parallel requests are made under the hood. The server cache is set, dehydrated, and then the client is hydrated. React Query derives the state for you on initial re-render. By looking at the GetServerSideProps, we're creating a prefetch. Passing a timeout is significant on the server to handle delays effectively. It's crucial to have a timeout control to avoid blocking initial HTML rendering if the request takes too long.

Another point to consider is the optional query made from the same proxy, followed by dehydrating the state and passing it for React Query to pick up the hydration process. Prefetching helps reduce UI delay and balances server-side and client-side loads efficiently. This strategy allows for pre-populating HTML selectively and recovering on the client side in case of server failures for consistency.

If the request takes too long to resolve, the user won't see anything because we can't send any initial HTML. So why is the timeout important? The timeout's important because if I don't get my data resolved in 2.5 seconds, I want to bail on the request. I couldn't render this in an optimal way. I'm going to go and recover on the client side.

This is really important because you don't want to block anything being sent to the client. Then we have an optional query being made. We can dig into this a little bit later in the Q&A. And then you just fetch from that same proxy, dehydrate the state, you pass it, and then the hydration process React Query is going to pick it up.

Now, for example, we're doing prefetching to make this a little bit better. We're reducing UI delay. We can actually balance a little bit the amount of load of things being fetched on the server side versus on the client side. So you can actually choose what you want to pre-populate your HTML with versus what can come later. This is a data recovery strategy that we have in place.

So I have retries. So if things fail on the server, we can recover on the client and ensure consistency. And we're using query criticality here, which I can't get too much into in terms of managing which are the queries that are critical to rendering your interface or optional. And let's go for a final demo.

So I'm going to start on a website over here that's not the demo website. And what I'm going to do is I'm going to go to my demo three and you can see that things are loading, loading, loading and loading because it's got a five-second timeout on it. It took a long time for the server request to be fulfilled and then render my UI. Whereas I'm going to show demo four over here where I put a 500 millisecond timeout. And what happened is that it bailed the request, went into a loading on the client and tried to recover.

Now, we talked a little bit about real-world results and you may be asking, why did most of these UIs take so long to render? Because I did all these demos on the airplane on the way to New York. So all of these were done right here on an airplane in these network conditions. And so by just changing around where we're making the request, you can have an incredible difference in user experience. And we didn't do that much. We didn't do any crazy memoization or anything like that. We just shifted some things around and understood a little bit more about networking.

Now, the question is, Faris, instead of jumping through all these hoops, could we have not just had the backend team make the endpoint? Not really the point. The idea is sometimes in a startup space, you've got to be pragmatic, but you also have to sometimes be curious, be creative. And I'm thanking you very much for your time. First of all, I want to thank you for preaching the good word of Tanstack query, because the number of times I've come across people that try to implement it themselves or aren't using something like it.

So, yeah, if you're not using Tanstack query, definitely look into it. It's gonna make your life a lot easier in your react apps and fetching. So this first question from Anonymous, what's the difference between using a backend for frontend proxy to filter data versus using GraphQL? So with GraphQL, what you have is, well, it depends. It depends if you're back for frontend proxies made in REST or GraphQL, because GraphQL works a little bit differently with GraphQL. You're generally making a post request to then get the data and you're getting it from a single endpoint. So you actually can't even make use of it's a little bit harder to make use of the natural caching that you should get at the get HTTP layer.

But, you know, the difference really is over here that it depends. GraphQL is generally being structured as an infrastructure as service. You're having it as a single individual, either monolithic backend or fragmented across multiple microservices. What's generally easier here to set it up as a function as a service proxy layer is that if you're dealing with a full stack framework like Next.js, it's very easy to just copy and paste the code you had on the frontend, shift it to a handler. And probably in ten minutes you're done rather than having a GraphQL service handle this for you. So there are just a couple of different key differences there. It's more if you're already using Next.js it makes your life a lot easier. Definitely.

So we don't have any questions in the queue, but I'm curious. So we showed a lot of different scenarios. Right. So we have the scenario where you're trying to load the data on the server. If it resolves in time, render the page. If not, we could potentially break out, render on the client. And it does seem like more modern metaframeworks like Next.js is making this a little bit easier. But I guess, what's your advice for people out there that are trying to make their web pages more responsive and implement these patterns? So any practical advice? How can they take this home today with them? Understand your user base. Where are your users based? If your users are based, all my users are based in Switzerland, all my servers are based in Switzerland, for example. Maybe I don't need an entire CDN network distributed across the entire world. Understand what are the majority of the users using your platform. Like for small PDF, we have a lot of users that are based in India, based also in Brazil. So, for example, there's a limiting factor in terms of the devices that they're using. And so having analytics on geographically where things are placed can be very interesting because I've even heard some arguments in terms of like, oh, you should put the proxy closer to the user or closer to the server where it's originating from, getting the requests over there. The idea very much is how much analytics do you have? How much do you understand your user base? I've had I've been at startups, for example, to where they're working. I was consulting for a startup and they were building an incredible Datadog dashboard. And they're spending 300k a year on this incredible engine you're building it. And they had zero users. Pragmatism is at the key, if at all. Do not prematurely optimize if you don't need to. This over here was just to unblock a front end team and to move things forward. So it's very much wait to maybe hit the problem first or understand your limiting factors. Work within that. This solution's probably gone a little bit too far. It shouldn't have gone that far. Wonderful. And I think it's great advice. But that's all the time we have for questions. If you have more questions for Ferris, he's going to be over in the glass room. But thank you so much. Thank you very much. I'll be stuck in the glass box there.