Data Loaders - Elevating Data Fetching in Vue

Hello, I'm Eduardo, I'm a Vue.js core team member, I work as a freelance front-end developer and I'm the author of uRouter and Pina, among other libraries, and you can find me in person in Paris in some of the meetups or online as Pozva.

And today I want to talk to you about a topic that is very dear to me because it relates a lot to the router, which is data loaders. This is way more than just data fetching, actually, but I think data fetching is going to be the most common use case for this feature.

And currently, if you are interested or keen into learning all the technical details about this feature, you can already do so by reading the RFC. I even have a website up for it. But as you can see, this is pretty long. It's really going through every single technical detail that I could think of. There is really a lot, but it does cover more than documentation would even cover because it also covers some edge cases. Of course, this is a little bit boring to explain, so my goal today is to kind of make you skip the whole RFC by just showing you how to use it. Of course, I won't be able to go in all the details of the implementation of data loaders, but I will still show you what problem we are solving and how we are solving.

So the problem that we're solving is data fetching, which is hard. There are many ways of doing it, and I'm going to show you some ways of doing it. We have an initial data container that we're going to fill with the data that we want. We have some ref, and then we can call some API on mounted if we want a way that we're going to data and then we can display it. But then sometimes we also want to update based on the row, like an ID on the params. So then we move that to a watcher, we watch the params, and then we fetch again. Then what we might think about is, well, we still need to handle the loading state because initially the data is empty, so I might want to display some loading state. So we add that and then we have to toggle that loading state to true then to false. And then we think, oh, wait, but maybe there is an error, so I need to handle that as well. So then you have an error, which is again another ref, and you handle the error as well, and then you update the template.

Now, as you can imagine and probably experience, these get reposed very quickly. And because it's something so common in applications, you end up repeating these a lot. So you're usually abstracting to a composable, and you can even use already existing composables. So for example, you could use a view query from task stack, and so you have just four lines for what we're doing in 20 maybe, or 15 lines. And this is really nice because it also offers all the features like cache, at the cost of some, of course, heavier library, but it does move apart from the route layer. So it doesn't integrate into navigation anymore. And as a result, we have a more independent data fetching, which has its own benefits, but also some inconveniences. So data loaders are here to close the gap. And so the goals of the data loaders is to integrate with the navigation cycle, so it will block or not. You can choose whatever you want.

The navigation until all the data is ready, it can change or abort the navigation. So for example, in a navigation guard, you can return false to abort the navigation. So it will block or not. It can change or abort the navigation. So for example, in a navigation guard, you can return false to abort the navigation. Or it can change or abort the navigation. So for example, in a navigation guard, you can return false to abort the navigation. So you could also do that.

It deduplicates all requests and data access. So that means that all the requests are going to go in parallel and there will always be only one instance of one of the requests going. And then we have semantic sequential fetching on it. That means that by default, things are parallel, but if we want, we can make some sequential fetching. And also, we want to delay all the updates until all data loaders resolve.

So that means that in complex applications, we might have data that is being resolved from different IPA endpoints. And in our page, we might display that in different parts of the application. And so, some of the APIs might be faster than others, and some might resolve faster than others. And we don't want to display the new data until everything has completed, because otherwise, we're going to display an old page with the new data. And also, we want to roll back. So we want to avoid setting the new data if anything fails, because in the end, we're in a navigation. So we're staying in the old page. We want to update the data.

So to give you a glimpse of what this API looks like, we have two scripts. And I know you're going to think, why would I want to add another script? There is a reason for this. We cannot put everything in script setup, because the things you put inside script setup are inside of the setup function of the component. So they are not exposed outside the same way. They are not shared among components instances. And also, they happen when the component mounts. They are within the setup function, so you need to mount the component to access that. So the script, the regular script, gives us access to properties that are static and are not related to mounting the component. So that's where we're going to define the loader.

And as you can see, I'm using define loader, but I'm not importing that function anywhere. This is because this is just sort of code. There is not one define loader. You can have many define loaders, and the idea is there are many implementations of loaders that contain different features. For example, the basic one, which is fetch every time, and then we have another one, which is a bit more complex that has caching on top. And then you can implement other loaders that will integrate with GraphQL, Vue query, Vue Fire, anything. And so the idea of these loaders is that their API can be different, but what they return is always the same. And so in the end, we always have access to the data, to ease loading property and error, and one reload function that allows us to manually trigger a refetch.

Now, of course, the composable is responsible for if it should refetch or not, that's up to the composable, which is pretty nice, but this is the base API, and all implementations can add anything they want. They can add more things. Now it's time to show you some real code. So I have this application running. It's communicating with a public API, which is the Art Institute of Chicago. So it's just all of art. You can search. You can look up the artist and many other things. It's well documented. And what I'm going to show you is how to migrate this application to use the loaders.

So first step, what we need to do is we need to install the unplugging Vue router. Now, of course, here I already installed that. It's right here, but I'm still going to show you how to use it. So we need to add it to the VConfig. It's right here. We can configure it if we want. This will give us two things. It will give us file-based routing. So here you can see you have a page folder with two files, index and search, and here we have the router, and I'm not declaring any routes. It's just automatically detecting them. And also I'm using Vue router auto, which is tight. So all the routes that we see are going to be tight.

If I try to take the current location and check the name, I can see that it's also tight. So this is pretty nice. It gives me some extra features that I'm going to see.

Then on main, you have to install some of the data loader plug-in. So this is going to add a navigation guard that handles all the data fetching, all the data loading, sorry. And so right now what I have is just a regular old-fashioned unmounted fetch. So not only I have a pagination that doesn't work, as you can see here, we have a lot of pages, but none of them can be seen, only the first one. So I want to refactor these to use the data loaders. So how do we do that?

I'm going to start by adding another script. So I move up here, I add the script, same language. And I'm going to define a basic loader. So we're going to see later that we have different kind of loaders. And here we can pass the function that I saw before, but we can also pass the route. Now, this is a type route. So it allows us to say this loader is supposed to be running on these routes. And this is going to coerce the type of the location, the two. So we have all the params available. So in this case, none. So we don't use anything we cannot use.

So here I'm in the index, and I'm going to use sync to, although I'm not going to use to yet. And here I'm going to return what I have here, getHardwareList. And so I defined the loader, but what I need to do is to expose it. So I need to tell the router, hey, this loader is attached to this page. So here I'm in a component that is added to one of the routes. So this is what we call a page. So what I need to do is to export it, and I'm going to give it a name as a composable because it's going to give us a composable. I'm going to call this useHardwareList. And so I'm doing two things. I'm defining the useHardwareList, which is going to give me access to the composable here. And also by exporting it, I'm telling the router, hey, you should load this loader.

And so here, instead of having this, I'm going to use the RWorkList here. And now I get the data, and I can do RWorkList. Now I have all the properties. I have misloading, reload, and error, which I'm going to use later. But right now I'm going to stop here.

Okay, so let's take a look. And right now we have, if I reload the page, I can see, you can see that on top, I have this small loading bar that appears while we are navigating. And as you can see, it's taking some time because this API, I'm requesting on purpose too many things, so it goes a little bit slow. But it integrates into the navigation cycle. And so now we can change the pages. So for example, here you can see that we are navigating to a different page, and it's taking some time, but the content doesn't change. This is because we're not passing the page here. So this function allows me to give a page, which I'm going to do here. And I'm going to use the par, I have some helper here to parse a page on the query here. Now by the way, you can move these up or not, it doesn't really matter, Vue handles it anyway.

So by doing that now, I have access to the different pages. Now I think HMR is probably broken here, but I should be able to navigate through the different versions. Now as you can see, every time I click on one of the links, it reloads every time it does a new fetch.

Okay. So this is something we can improve later on. Now even though that we have the loading state appearing on the top, we can also handle it locally. So we can do ease loading here and error if we want, and we can add it, for example, here. So I'm going to add some ease loading. And here I'm going to have the error if any. So I won't be able to see the error because I'm not handling that right now. If we have an error, we're just going to cancel the navigation anyway. But we can see the loading appearing when we load one page or the other. And we can also see that the information is part of the route. So if we reload the page, you can see that we end up on the same results page.

All right. Now I have another one that is a little bit more interesting is the search. So the search is more interesting because it doesn't contain all the information. So when I do a search, what I'm going to retrieve is just an array of the title, but then I don't have the image of the art. So I only have a low quality image. So it's like a very, very small, very, very tiny image that looks like I think it's like five, 12 pixels or something like that. It's really small. And so the idea is we're going to do all the requests to get all the images, and then we're going to swap them. So we're going to just fade into the new images. So for example, here I can search things which are not going to work. They're going to add into the query, but they are not used. And they are only used if I reload the page. So let me show you the code. We have the exact same code as before. We have the unmounted, and we're just doing the search artwork. So we're going to move here and add another script. And same as before, use artwork. And same as before, we do use artwork, search results. And so we're going to define a basic loader, this time on search.

And here we're going to use another function that we have, which is search artworks. Now this one expects two arguments. The first one is going to be a query. So this is going to be a plain text, full text search. I think that's how we call it. And then we have the page and the limit. Now I'm going to set the limit to a bit more. So maybe 20. And then I need to pass these two parameters.

So as you can see, I have already connected the parameters to the route with some composables. And what I need is to retrieve them here and pass them. So I also have some helpers, query search, which is going to validate the parameters. I'm just using the same name here and here. Page equals parse query, page query. So this is going to be do.query.page, same as here.

Now, here's the interesting part. If I don't have a valid query, which in this case, it can happen if we pass an array. Or like if we manually have here, and q equals other. So this is going to be an array on the query, that's how we define arrays. So we want to change the navigation so we can abort it. And we can do that by returning or throwing both words through a navigation result.

And here we can pass the navigation result that we want to pass. So for example, false, which is the same as returning false in the navigation guard. And it's going to cancel, just abort the navigation. And by throwing, we're saying, okay, we stopped here. We don't need to wait for anything else. And also is going to make typing friends works work a little bit better.

All right, some reformatting here. And now I need to use it. So here, I'm going to remove this and this. And I'm going to use search results, data search results. I think that's how we call it. And I'm going to use error and is loading as before. And now, so let's go back to the homepage. Let's reload the page just in case. Okay, search. And so I have this search here. And we can try again, another search, see if it works. As you can see here, the navigation only happened once the data was ready, which means that the page is rendering only when we have all the data ready to appear, except that not exactly right, because we don't have all the images.

So now we're going to handle the images. So how do the images work here? We only have the ID of the artwork, which means that we need to do another fetch, another call to the API to get all the images. So I'm going to add another artwork images. And I'm going to use defined basic loader again, search, and again. There we go. I don't think I need the two.

Now here's the thing. I need inside of these loader, the result of these other loader. So what I can do is I can retrieve all the results here, and I can await them using the same composable. Now this usage is very different from the one we have below, because below we are not awaiting it, and we are extracting multiple references. Here we only get access to the plain data. We only get access to the content of that other value.

So it is a little bit more complicated than that behind the scenes, because these value might be different from what we have here, depending on the timing. Since we're still in the navigation and not every loader has been resolved, we don't have everything already connected. So this allows us to have consistent data loading.

Now what I need to do is to retrieve all the URLs, and I conveniently added already an API call for that. So I'm going to wait here, get artwork images URL, and we can have here a search, results.data.map, and I'm going to do artwork, and I think it's just the ID. So now what I'm going to do is put this in a map. I don't really need it to be a map. It could be many things. It's just going to be simple this way in this specific use case. So images equals new map, and I'm going to have the number, so that's the ID, and then the URL. And then I'm going to do a for, so I'm going to loop over all the URLs, and I'm going to set the map. So image.id, image.imageURL, and then I'm going to return the images. Oh yeah, they can be null. So not all artwork have an image, by the way. So that's why it doesn't work. So if I do this, it means that I'm waiting for all the images to be ready in order to appear. And right now, I also need to add them here, because I'm only displaying the thumbnail. So I'm going to uncomment this, which allows me to give access to the images, which is this map I just created. I haven't used the Composable yet. So I'm going to use it here, and I'm going to say images. Did I just name it images? Yes, I did. And now, if I reload the page, I can see that as soon as I enter the page, all the images are ready, and the transition happens immediately, because there is a CSS animation.

What if we don't want to wait for these whole set of images to be there before navigating? We can do that by setting the loader as lazy. So this is similar to the lazy that you know in Nuxt, if you have used Nuxt. We can also configure other options. So we have commit, key server. It depends. So for example, I don't want these to be waiting on the server, so I can say false. And that means that on the server, if we're doing NSR, this loader will just not be run at all. Now by being lazy, it means that we are running the loader as soon as we can, but we are not awaiting it.

So here, if I go to a different page, it takes some extra time, as you can see, to get all the images here. This is because we are still doing the other request. You can see it on the side here. We finish, and then we have the second one going, but the page is already displayed. Whereas if I have it on false, I'm going to reload the page. You can see that the first happened, the second one happens, and then we navigate. It's a little bit fast. One, two, three. But that's how it works. So in this case, I prefer to have it lazy.

There are many other ways of handling this, by the way. We don't need to do a loader specifically. But it allows me to show you that we can have sequential loading if we want on demand. It can be lazy. It can be avoided on server. It's highly configurable. All right. There is still one annoying thing here, in my opinion. It's that every time I click on next, I have to wait. I have to wait for the API to come and go. But I just did the call, so I could cache this. I could be faster. So I'm going to show you another loader.

This other loader is based on Pina Colada. So Pina Colada is still experimental, but it's the layer that allows me to add the caching to the data loaders. So the API is going to be very similar. It can be used by itself. So it's similar to view query in that aspect, but it has some core differences that allows me to integrate it better into the data loaders. So let's go into the index again. And we're going to change this into a defined colada loader.

So colada loader doesn't take a function. Instead, it takes an object. And this object is going to be very similar, but we need a query. And I need to add also a key. Let me just add this for the moment and fix the syntax. All right. So we need to add the query and also the key. The key is going to be used for the cache. And in this case, we're going to use a function because we always want to add any parameter that is used into the key.

So here is going to be the art words. And then we can add parameters. So in this case, it's going to be just the page. So I'm going to double the work here of parsing the page. It's not a big deal. It can be refactored, but it should be good enough. So I'm only changing the loader here. The rest of the API stays the same, but because we're using a colada loader, we have access to all the things, like a status. We have refresh and refresh and reload. So we have different functions, which we're going to use in other cache.

And now, just by changing that, if I go, so I'm going to reload the page again. If I go to a different page, and then go back, you can see it's very fast. And then after five seconds, because that's the default cache time, it's going to load again. So if I change that, for example, in this specific case, I don't think I need to fetch very often. So I can say, hey, the still time, oops, it's not here. It's here. The still time can be, so in milliseconds, 60. So that's one hour. It's a bit long, but it does allow me to have a cache that works for a long time. So I can go back, forward. Sometimes they're slow.

You can see here, it's instant. All right. So now let's do the same for the search, which is a little bit more complicated.

All right. We go back here. Glada loader, and then same as before. Query. So here. And the key, which is going to be very similar. We're going to have not words, but here we're going to have the queue, which is going to be the parse page query, no, parse query search on to the query.queue, and page, which is going to be the parse page query. And that's it.

So just by changing that, now I reload the page. Let's unsearch. I can see that on the network panel, I'm fetching not that often. So the IDs here is for the images and the other one is for the search. There is a lot more to the loaders that I don't have the time to show. For example, the Glada loader, because of its implementation, is going to only trigger if any of the used properties of the route change. So for example, if you have a pattern or another query or the hash that changes, it's not going to re-execute the loader, which is not the case for the basic one. And so they are all tested through a suite of tests that enables common specific behavior that allows me to reuse them, and so combine them however I want.

Since I don't have the time to show you more than that, I want to give you some links. So first is unplug and rerouter, the repository on GitHub, and the documentation, if you want to go further and start using this. Note that this is still experimental API. There are some bugs, but we're getting there. Pina Colada is saying it's still a work in progress, but promising and very interesting in terms of UX for the end-user when it comes to the data fetching. And last but not least, if your team or company is interested in implementing these, or getting ahead of others by knowing what is coming, or having their say on what's more important to implement, to fix, etc., etc., hit me up to work together as a freelance, and always looking for opportunities where I can balance my open-source work and just life. So I have a link. I will share the slides later on, if you can click on the link. That's all I wanted to show today. I hope you give this API a try, and please share the feedback you have, all the bugs, all the issues, all the ideas. We have a lot of users already, so I think it's nice. We have a lot of users already, but I don't think that many are using the data loader yet. Thanks, and enjoy the rest of the conference.