Beyond API Mocking

♪♪ Hello, everybody. Thank you for joining. I hope you are healthy and well, and would love to hear about API Mocking. My name is Artem, and I am a JavaScript engineer which has a huge love for open source. I've been doing open source for more than five years, during the course of which I was extremely lucky to author more than 20 packages that by this time have more than eight million downloads on NPM. And today I would love to talk with you about testing and specifically about API Mocking and the role it plays in it.

But before we begin, I'd love to ask you a question. Why do we write tests? Well, we certainly don't write them to gather code coverage. It can be useful at times, but unfortunately, it doesn't mean that our product works or works well. We are not writing tests to check each and every little piece of logic we wrote because that would be testing our code instead of testing the intention behind the code, which we shouldn't do. I think why we write tests is first of all, to gain confidence, to rest assured that the software we build is functional and our customers are enjoying and loving it. And to gain that confidence from testing, we should follow two principles. And the first one is to test as a user. And the user here doesn't mean customer necessarily. Of course, if you're having an e-commerce website and you want to double check the success scenarios in different pages, you will be performing user actions there. But let's say you're writing a test for a function or a class. Then the user for that class would be another developer. So you need to put their expectations against your class and write your test following by that expectations. I like this quote by Kenzie Dots that says, the more your test resembles the way your software is used, the more confidence they can give you. And I think it summarizes this principle pretty brilliantly. The other one principle that can gain us more confidence during tests is to establish clear boundaries. I think you're pretty familiar with this because this is the reason why we have different testing levels. So when we want to focus on a single unit of code, we write unit tests, then perhaps you want to check that pieces fit together nicely and we do some integration testing and we can wrap up with end-to-end tests to check the entire system. And I see mocking as a tool that distributes these boundaries. Let me give you an example. Let's say we have this orange square and we want to test it. Now, in a real application, this orange square probably does quite a few things and it may depend on other squares like this blue one. Now, because of this dependency, we can no longer focus our tests on the orange cube alone and we need to somehow account for the blue one. So this is where we can mock the blue one to substitute it with a seemingly compatible cube but it's going to be different. And because of this, we can control this dependency and connection between modules and make sure that our test of the orange one is focused.

And API mocking is a technique that allows us to substitute API communication, so HTTP requests, in the same manner. There are two main practices when it comes to implementing API mocking in your projects.

The first one is to use a mocking server. This is a pretty straightforward setup and it means that you have a standalone HTTP server that will substitute a production one. And while it's fairly easy to set up and it has some sort of abstraction syntax to write the routes and responses, I think it has a few disadvantages. So, primarily, no matter what are you going to use, write your own server or use a third party library, you're going to end up pulling dependencies and the whole process will feel like you're writing an actual server but you're not doing so. Then you need to ensure the operability of your server, so it starts and stops at the right time before or after your tests, and then there are no runtime exceptions that can crash the tests. And the worst thing of a mocked server is conditional URLs. And this is what I mean by them. This is an abstract example of code where we have some fetch call to an API backend.com. Now, we don't want to hit that production URL during testing, so we probably introduced some environmental variable that says that, hey, if you're running under tests, just hit this other URL at localhost because this is where we have the mocked server running. Now, the issue with this is that during the test run, the code will never hit this line, which it does in production. And let's say we made a mistake here. We mistyped a protocol or we missed a few slashes or dots. Now, this is going to result into a perfectly passing test on CI while perfectly crashed app for our customers. And the reason that happened is because we introduced a deviation. So the app that run during test is slightly different than the one that runs on customer machines. And the more you alter the system under test, the more you're testing a different system entirely.

I think the other approach how to do API mocking is to use a mock client. So this is a quick example of spying on window.fetch and saying that each call of fetch should resolve into this JSON which we specified. What this is going to do is that instead of calling the actual fetch function in the browser or during the test, it's going to always call this mock and return this fixed JSON.

Now probably we don't want all requests to return the same JSON. So we want to introduce some logic where responses depend on requests. And we end up writing this API client mock, which has this logic. And then we realized that we also need to account for some business behavior and also keep account for headers and other things to make this API client mock feel realistic. And in the end, we end up with a huge obstruction, which we wrote just for the sake of testing.

There are also a couple more issues with mocked clients. For example, because we're taking out the client and replacing it, we need to ensure that requests and responses make sense, that they're spec compliant, and they are similar or identical to what is going to happen in the actual app. The input to such mock client is also always considered valid. So unless we somehow validate the request and responses, the mock client is going to digest them and perform them, even if they have a typo, for example. Then clients are quite different, and the way response and requests are handled in Fetch may be different for XHR and Apollo. So we need to take account on that as well when writing it. And this creates a very tight coupling between our mocking and actual libraries we use, which makes it very hard to migrate and to see our software evolve.

The worst thing is that requests never happen with the mocking client. That's kind of what we wrote, right? So instead of calling actual Fetch or Apollo, we're calling the mocked counterpart. So the actual logic never executes. Let's keep that in mind and think about what could be a better approach.

So here we have an app. It can be an actual app running in the browser or just a part of an app in the integration test. The point is that this app makes a request. Now, we want this request to happen. We want the actual request client to be called and request to leave the app. Why? Because this is what happens in production. And if we stay close to this, we will get more confidence from our test. But we don't want for this request to hit an actual server. That way, if the server is down or malfunctioning, our test will fail. But the point of our client-side test is to test the client, right? Not the server. But we still want to receive a mocked response back so we can control different server scenarios per test and it will be reproduced reliably.

So, we always tend to choose either mock client or server. But there's actually a thing in between, so we could choose neither. And that thing is called service worker. Service worker is an API that is a kind of a web worker, a JavaScript module that lives next to your application in a browser and executes in a separate thread.

This is an example of a worker file. So, workers have certain life cycle events and one of them is a fetch event. Worker triggers this whenever your app makes any kind of request. And in this handler, I'm writing that I want to look up the cache response and if there is a cache response, just respond with it, so don't do actual requests. But if there's no cache, execute the fetch as usual.

Now, you can see how with this, there's a huge potential to implement caching like this on the client side, but this got me thinking, what if instead of cached responses, we could return mock responses? Let's try that out. So the same worker, but this time in the fetch event, we're responding with this hardcoded response body and status code 200. And now, whichever request our app makes, it's going to leave the app and it's going to hit the worker and in response, this static response will be returned.

This is pretty great, so let's just use Service Workers for mocking and the problem is solved. But if you try to do that, you will be faced with a number of challenges. First of all, Service Workers have limited execution context and this is mainly due to security considerations. You cannot access the window object or DOM and obviously your client side code and your functions, utilities and libraries. Worker can get out of date and each time you register a worker, this does not necessarily push the latest worker to the browser and you may end up with an absolute worker, which is not nice. When you hard load the page, the worker will stop and you would have to reload the page to see it running again. This creates a distorted developer experience. Workers also control unrelated clients and they do it because the worker establishes connection with the client based on its URL. So you may open a completely different project on the same URL, on localhost, and see an unrelated worker kicking in and trying to mock responses. And this is just confusing.

Despite those challenges, I like the idea. And a couple of years back, I wrote the library that's called MockServiceWorker or MSW for short. And it leverages the ability of ServiceWorker to intercept requests and does that gracefully. MSW is the closest thing to an actual server without having to create one. And this is thankfully to ServiceWorkers. Let me show you how the workflow with the library looks like before we dive into internals. So first things first, I'm going to tell the library which requests to capture. And I'm writing these things called request handlers, just functions that described request information and which response to produce.

Here I'm targeting a REST API request, which is a GET request to slash user endpoint. And I'm producing this mocked response, this JSON body with first name equals John. Once I'm done with the handlers and I described all the server behavior, I'm creating a worker by calling setup worker and providing my handlers. And then starting this worker, which is going to register and activate the worker in the browser.

I import this module in my app, and I can see that now when I make a request to slash user, it gets intercepted and I receive the mock response. But how exactly does it work? So whenever the app makes a request, it gets intercepted by the worker. The worker clones the request and sends it back to MSW because MSW doesn't execute on the worker Instead, it runs next to your client code. So you can use your favorite languages, libraries and utilities there. MSW tries to find the handler for this request. When it does, it returns the mock response to the worker and worker uses it to respond to the original request in the client.

Now with MSW and due to service workers, there is no more this ghost request, request that never happened. Instead, you can clearly see requests in the network tab because they actually happen, the same as in production. There is no more request clients tab in. We don't meddle with fetch or other libraries and because of that we're getting full specification compliance. Because request client is called and response is composed on the service worker site. If any of these parts is invalid, an exception will be thrown. And sorry, and there's also no more conditional URLs, which is, again, because of service worker and we can get identical application behavior the same as our app does in production.

So when would you use MSW? Well, first of all, you can use it for testing. Once you've written the request handlers, you can re-use them for integration tests and also for end-to-end tests with Cypress or Puppeteer. You can push this concept even further and use MSW for development. Let's say your back-end is not ready or it undergoes a change or you want to experiment with some third party API. You can just try it out with MSW first. The next one is debugging, and that's my favorite. Because whenever there's data-related issue, you can just create a handler for a specific request and try to match what response crashes your app. Once you do, you have a reliable way to reproduce the issue and thus to fix it. You also can use MSW for prototyping. When you're building your next awesome product, and the back-end may not be ready, but you can still write client-side code and you can use MSW to act like a back-end. So here we can try RESTful API. And maybe tomorrow we want to have a peek at how our application would feel with GraphQL. So we just use GraphQL request handler, and we don't have to commit to the entire ecosystem of GraphQL just to try it out.

There are much more features that MSW offers. For example, it's written in TypeScript, so you can use type definitions to annotate your requests and responses and make sure that you're safe. It requests client-agnostic, so you can use MSW with any request-issuing client, be it Native Fetch or React Query or Apollo. There is no library-specific logic there.

You can also use MSW with Node.js, so to run it in Jest or when you develop in Node.js servers to mock third-party communication. Companies like Google, Spotify, and Gatsby are already using MSW. And you are one command away from trying it yourself, so don't miss out. Make sure to check out the repo at msw.js.msw and also the docs at msw.js.io where you can read more about the library, its API, see some usage examples, and also check out the comparison with other API Mocking tools out there.

If you're a more hands-on learner, definitely look at that YouTube video to mock a basic HTTP response that goes from start to finish, configuring MSW and getting the mocks ready in under four minutes. And also don't miss the Epic React course by Kansi Dots, which is a huge collection of material on how to build awesome React apps. And it features MSW in its testing section.

Make sure to follow API Mocking on Twitter to stay in touch with releases and library updates, and you can also follow me just to say hi or ask a question. I'm always happy. I hope this talk was useful to you and now you can write more confident tests and perhaps MSW can contribute to that. Thank you.

Maybe in the meantime you would like to tell us something about what's going on with the MWS? No, sorry, MSW, that's the right order. I heard there are some nice things brewing? Yes, definitely. We've got a few people joining to the team and with more folks using it and contributing, we get a lot of requests. And we're now working on improved GraphQL integration so we can support subscriptions. And we're also pushing for WebSockets because, well, we use WebSockets, but there are not many solutions that support mocking that.

Yeah. And we're trying to also support more maybe approaches in development and testing. You have, for example, I know that folks prefer a data-driven approach so they can define the schema and then they can generate different entities and have the mocks based on that. So we're also working on the data modeling solution to help out.

That sounds awesome. So the community is really growing. Yeah, it's crazy. I'm ready for it. Okay, cool. I see there are some questions coming in. Stephen wants to know, could you integrate MSW and PECT? Do you have any info on that? I haven't tried that, but I don't think why you wouldn't be able to. So I guess it's a cool opportunity to try it out and let us know. Okay, cool. So, Stephen, if you try it out, please let us know what the answer is. Stephen wants to know, can you enumerate some differences between Cypress MOC feature and MSW? I guess that by Cypress MOC, they mean like Cypress Intercept, perhaps. I haven't heard about the MOC API. But in any case, I think that in Cypress, request interception is done through stubbing of fetch or XHR, so that would be the main difference. Because when you use service workers, you don't have to do that, like I mentioned in the talk. You actually allow your app to make the requests. And everything happens, all the request client code is called, and then the service worker intercepts it in the network level. So I suppose that would be the main difference. But you can use MSW with Cypress as well and have it running alongside your app in the Cypress and then have it handling the MOCing. Okay, cool. I think that also maybe answers the next question. Vatilis wants to know, would you use MSW in production as some kind of API gateway? Well, there are a few people who use MSW in production, but it's mainly for educational purposes.

I don't think that it's a good idea, because once you push the worker to your user's machine, you would be responsible for keeping it in sync. And as I mentioned, workers are quite tricky, and even if you register the new version, it doesn't mean that the old one will get replaced immediately. And if you don't want to get stuck in this out-of-date MOCs logic, I would probably not recommend doing so. But you can definitely try. I mean, service workers are meant for production. And if you find value in that, in running MSW in production, perhaps, yeah, give it a try. So it's possible, but debugging is maybe a more common use case. Yeah. It mainly focuses on testing and development rather than, like, acting like a gateway.

Okay. Richard says, hi, was just about to ask about WSMOCs. When is that likely to be introduced? Will it support dynamic responses? I'm not sure I follow WSMOCs. That's what was written. Maybe Richard can clarify on that.

In the meanwhile, let's go to Testibyte's question. What are some of the use cases for mocking RPs? Mocking third-party RPs makes sense almost always. Would you mock RPs you own? Well, I guess it depends on the kind of testing level you are on. Because if you're testing your components, like the integration of them, like, let's say some login form or a checkout form, just a little piece of code you wrote. But it already has the multiple components and perhaps it does requests. So, I guess it makes sense to mock your own API here because you don't want to rely on an actual server during this test. And once you bring it up one level up to end-to-end tests, then probably here you want to hit your actual end points and make sure that it works for customers. I would probably split it like that. Yeah. And so probably depends on the size of your project and what level of tests you are doing.

Okay. Have we heard back from Richard yet so that we can answer the question? Oh, web socket mocks. Sorry, he meant web socket mocks. Okay, got it. So, do you want me to repeat the whole question? Yeah, if you don't mind. Hi, was just about to ask about web socket mocks. When is that likely to be introduced? Will it support dynamic responses? Okay, so, yeah, as I mentioned, we are working on WebSocket support as well. I kicked off the prototype a few months back, but I guess it needs still some polishing and I would expect it somewhere this year, but probably closer to summer. It depends on the priorities, but it would definitely be nice to have it there so people can try it out and let us know, because we obviously can't predict all the use cases that you have. So once it's out, try it out and let us know. And in regards to dynamic responses, I'm actually not sure. I don't have much experience with WebSockets myself, like using them. I'm digging through like how they implemented and specification, but I haven't gotten to the part of this feature yet. So perhaps if you're concerned with this, I would suggest you go to GitHub and just raise this in the pull request. So we won't miss it out.

Thanks. Well, but summer is not that far away. So can we try it out soon? We'll do our best. Do you have any plans when the next version is going to come out? I mean, we already talked a little bit about the rest of the roadmap. Yeah, we have a few small developer experience improvements, which I think going to land next week. A lot of people were asking to help them debug scenarios when requests are not intercepted. And this is really common. Sometimes there is a typo, sometimes there's a wrong domain name or something. And we already have some basic logic that is just gonna tell you, hey, this is not intercepted, but we're going to push it further and we're going to use string matching logic that's going to suggest you the handler that you perhaps meant. So whenever you request the resource, we're going to check, hey, maybe you did a typo, or maybe you requested a GRFQL query, but you meant imitation with the same name. So expect that, I think, next week.