Deploy with Speed and Confidence Using Contract Testing and Pact

Well, thanks, everyone, for coming for my talk to deploy with speed and confidence using Contract Testing at PACT. My name is Matt Fellowes. I'm a core maintainer of PACT. I'm also the co-founder of PACTFlow, which is a continuous delivery microservices platform. And you know, if I wasn't working in IT, I'd probably be working in sports and fitness to get away from it all. If you want to contact me after this talk, you can follow me on my handles below.

So the agenda for today, we're gonna be talking about how to release software, and particularly distributed systems. We're gonna start with talking about the problem with end-to-end integration tests. Then we're gonna talk about how PACT works, and the principles of contract testing. And then we're gonna finish off with a bit of a demo.

So the old way, or the current way a lot of companies test their microservices, is to do what we call end-to-end integrated tests. And what that requires is you to stand up your entire platform, something like this, and use a functional API testing tool, like Postman or whatnot, and drive requests for the entire system. So for example, you push it through...maybe not a user interface with Postman, but you push it through the platform, and it's gonna pass through all the layers of the system. So Microservice A, Microservice B, Microservice C, and the request is gonna make its way through all real systems, it's gonna pass over real networks, it's gonna talk to real databases, send emails, whatever it is that your actual application is going to be doing. Now this is great, if the system works, and the tests pass, it does give you some level of confidence that your system's working as expected, but this kind of testing comes at a huge cost.

The first issue is that it's slow, testing to pass with real layers and need to do real things and this can be slow, of course. But also oftentimes they can't be run in parallel, and the reason for that is the stateful nature of these types of interactions.

The second issue with these types of testing is that they're fragile and they can be nondeterministic. So this property of flakiness is very present in these types of testing. So even if they do pass, they may take multiple runs to get there.

For example, you need every version of every service in the system to be lined up. If any of those change, the test could break. If you've got the wrong version of data, the wrong version of the tests or the wrong configuration for your environment or just that somebody has tampered with the environment in advance, it's possible that your test will fail. They're very costly to maintain.

And when you do find issues, or when you do have an issue, finding the actual problem and the source of the issue can actually be really costly to do. So, for example, if you have a failure that's causing Microsoft's B here, well, it may not be visible from the outside why that test actually failed. So you might need to go digging through your log platforms like Splunk or Simulogic, you have to trace some correlation IDs through the system to actually find out what the problem is. And then you need to find the code version for that particular service, go to the repository, and go digging.

It's basically like finding a production issue. So it can be quite costly just to find the bug itself. Often times it fails just because of those flaky reasons I mentioned earlier. Similarly, it's difficult to achieve full coverage this way. So what I mean by that is you've got multiple systems here and you've got a lot of different potential scenarios that can go on. And so running your tests this way, it's very possible that you're not going to get all the tests you want. Because a, they take so long to run. B, they're costly to maintain. And also you just literally cannot run that many tests in the amount of time, because the combinations spiral out of control.

Because you test everything together this way, well, then, you really have to release everything together this way. Because you don't have confidence if I deploy just a single component that things will continue to work at the end of it. So you now need to deploy things together. And doing that means you've got teams coupled with each other at release time. And that means teams are waiting on other teams to get things done. And we know from Agile theory, that's not very good. And so these types of tests don't scale well. They tend to get worse and worse over time.

So let's say you increase the number of teams and components over time in this linear fashion. What you see is this nonlinear relationship with the number of environments or the complexity of environments, that you need to manage. You see that the build time goes up, or the complexity of those builds, and the failure of those builds starts to go up. You also see the risk associated with change, moving up exponentially. And of course, we now have developers idle, we have a lot of queues, we have, you know, cards on the wall, that are referencing other teams waiting for their work to be done, before we can get ours done. And this all has a huge cost associated with it.

Of course, good tests have the exact opposite properties of what we just talked about. They're fast, they're isolated, they're easy to reason about and they're easy to maintain. Could mocks come to the rescue here? Well, as you've probably all written, it's very simple to write two separate unit tests on either side of the service boundary. We can write a unit test for the consumer, mocking out the provider, and we can write a unit test for the provider, simulating the consumer. And these are great. They can run really fast, they can be easy to fix, easy to find bugs with. Of course, they may not represent what actually happens in production, and so it's quite possible for you to put an assumption in there that's not valid, and because it's a unit test, we're not checking that assumption later on. And so, we have this nice property of, you know, all these great properties, but the new problem is that they're not actually representing reality. It's hard to keep both sides in sync.

And this is where we can talk about contracts. So, you're probably familiar with specification first design, where an API producer specifies or creates a contract using Swagger or something else, and they publish that document to all of its consumers. And there's a number of great properties that come with this. But one of the downsides is when you move and change and modify that version of the contract or the specification, it's easy to accidentally break a consumer because you don't know what parts of the API they're using. And it requires a lot of diligence to ensure that you don't push out backwards compatible changes. This is where we can talk about consumer driven contracts, which inverts that relationship. Consumers can specify what they need of the provider. And write those in their own form of a contract and give it to the provider, each consumer having its own potential subset of the API. And then the API just needs to implement the superset of all those contracts, and then it can get its job done. This has some really interesting side effects, or consequences. The first one is that you'll know when you break a consumer, because the consumers are telling you what they use. You get a form of documentation, because the consumers are giving you exactly what they're using every time they push a build, and you can test things independently.

So this is where the tool pact comes in. It combines the idea of fast isolated mocks and unit tests with contracts to ensure that they don't drift. It's an open source, consumer driven contract testing tool, and it's designed to help you test your microservices and distributed systems independently. Its main use cases are in things like JavaScript web applications, native mobile apps, talking to restful services, JSON or XML, or talking of message queues. So think of Kafka or SNS and these kinds of things. Its goals are to reduce or remove entirely the need for those end-to-end integrated tests and reduce reliance on those complicated test environments. Pax benefits are basically, because you've got focus for a single integration point, you're only looking at one thing at a time. You don't need to deploy. So because you don't need to deploy, you don't need a test environment to do this form of testing.

You get fast and reliable feedback because of it. The bug is always going to be found on your machine. You don't need to go digging through logs. This means those tests run really fast and they scale linearly. And last, because you're testing things independently, you can now release them independently.

Okay, let's quickly talk about how Pact works and then we'll show how it works in action. So we have a consumer website... Sorry, we have a website that's talking to a product API. We call the website a consumer and we call the API the provider. And the messages that pass between that, the sum of those, we call that the contract. So it's a consumer-driven contract testing framework and so the first thing we're going to do is we're going to write a test from the consumer side first to define the expectations of the provider. So what Pact will do is mock out the provider. We never make them talk to each other. But Pact will simulate the provider API. And the consumer can say, given that I make a request to get 1, 2, 3, 4 for the product endpoint, I expect to get back some response. And we do this for all the things the consumer needs of the provider. At the end of that session, we're going to record those interactions into what we call a Pact file or a contract. We're going to share that with a tool like the Pact broker or Pact flow, which will help us exchange the contract and version the contract across our ecosystem. And then finally, on the provider side, what we're going to do is we're going to pull down the contracts from Pact flow, we're going to replay them back against the provider. And Pact is now going to simulate the consumer. It's going to replay this request, check the responses, and if they match what the consumer does, we now have symmetry on both sides of this interaction. We have two fast mocks, and we've now got a contract that's ensuring that those two mocks are actually valid.

Okay. So, we're going to get into our demo. We're going to use a React product catalog website that talks on an ExpressJS backend as our example, we're going to test the product forward slash ID endpoint and show how consumers can help drive the API design. We're going to look at the provider workflow, we're also going to look at how we can gate releases with a tool called Can I Deploy? So, let's look at the application. So here it is, it's a very uninteresting website, I'm sorry, it's not the greatest React website of all time. But you can see here, the home page just lists the products and we can drill down into an individual product to get the data for it. And they're going to hit two different endpoints. We're going to test the endpoint for this page.

Okay. Looking at our code for this. We'll jump into the consumer test. So, if we look over here at our product page, this is our React component. You can see here, to populate this page, we need to get some data from an endpoint. Now, instead of actually loading this in, in other way, we're going to talk to an API endpoint and we're going to use a class to do that. So, this API class has got a method called get product, and that's what's going to get the product data for us.

So, from a Pact point of view, we can test this method, this is what we care about, this is the target of our Pact test, we don't need to test anything with React to do this form of testing. And, you can see here, it's just going to hit the product forward slash id endpoint, it's going to send some headers and it's going to then convert the data it comes back into a product class, and the product class looks like this.

On the provider side, we have a similar thing, we've got a product definition over here, we have our routes that deal with the different endpoints, and we have a Pact test over here as well. We're not going to get too much into the pact test here, because it's a lot of config, but basically to run the Pact test on this side, we're just going to stand up the provider, tell Pact how to find the Pact files, and we'll replay them against the provider. Lastly, we're going to share the contract with a Pact broker, in this case, it's a posted Pact flow. And it's going to show us the current state of the interaction over time. So you can see here that the current version of the consumer is in the master, and it's been deployed to production and the provider has also been deployed to production as well. We can drill in to the Pact, and we can see the various interactions that are supported by this contract. In this case, getting a product with ID 10, we get the ID, the type and the name back in the body. But most importantly, we're going to use this to show how we can make changes into the system and then promote them through environments.

Okay, so let's look at the consumer pact test first, because this is where we start. This is our pact test here. We're going to follow the standard arrange, act and assert model just to see how this all works together. So first up, we need to tell Pact, we need to tell our unit test what our code is about to do. As I said, Pact is a marking tool. It's going to validate what we actually do. So given that a product with ID 10 exists, making a call to get that product, using the get verb at this path. We expect to get back an HTTP 200 with some headers and a body that looks like line 19. You can see the expected product is ID 10, type credit card, name 28 degrees. The like matcher here basically says, we don't care about the values here, we just care that the keys exist and they're of the same type. So later on when the provider verifies this, we're not going to fail if different IDs come back or even different products come back. We're not going to get into matchers today either, but we have a flexible library we can use to make this much simpler.

The second thing too is to actually call the API. We configure our API client and rather than talk to the production client, we're going to talk to the PACKT mock service. Then on line 42, we call the method. That method there is going to call a real HTTP endpoint, but in this case, it's going to talk to PACKT instead of the real thing. And then on line 45, all we need to do is write our unit test assertions for this call.

So, let's pretend like PACKT doesn't exist. Well, what should we test in this unit test to make sure our code did what we thought it did? So, this test is already passing, as you saw before. It's already published a PACKT flow. What we want to do is add a new field. What happens when we evolve this API? So, this is a product website. It would be nice to actually display a price for the product, right? So, let's add price into the mix. So, we'll add this new expectation on product here, we'll add it to our product class as well, and then we'll have price available to us to be able to display in the React component. And what I'm going to do is I'm going to check out a new branch, my code, and I'm going to shut down those processes.

Let's create a new field, feet, add price. What I'm going to do here is simulate a CI process as if we're doing a continuous deployment. We're in a branch, so this will be a pull request flow, and what I can do is I can run a fake CI. And what this is going to do, this is going to run the test, it's going to publish the contract to PACKFLOW. We're then going to run a tool called can I deploy and say, is it safe to release this change? And the answer will be no, because this is a new field, the provider has never verified the contract, in fact, the provider hasn't implemented it yet. So we're going to be told it's not safe to release this yet. So back to my terminal, you can see the test passed up there. We've published the packs, and we've said, can we deploy, and we can't deploy. So the build has bailed out with a non-zero exit code. You can see, though, in our code base there's a PACKFLOW that's being created, and you can see it's now got the price property captured in that contract file. We're not going to talk too much about the contract file for now. Just know that it exists, and that's what we use to mediate this whole process. So now that we've added the price property, let's have a look at the PACKFLOW and see what it sees. Cool. So we can see a new contract has been created up top here called FeetAddPrice. It's yet to be verified. So this version of the Consumer can't be deployed anywhere yet because no provider has implemented its features.

So let's go ahead and do that to the provider now. Now I've got some stash changes to avoid Demo Gods. But what we're going to do is we're going to add the price to the property, the Product class, and add it to the repository as well, so there's data.

So there we go. We can see the price has been added. What we can do now is, in theory, we can push this into master. So let's see what it looks like when we commit and push this change. The provider should run the test by pulling the contracts down from PacFlow. It'll share the results of that verification back to PacFlow to say if it passed or succeeded. It will then run the check saying can I deploy this version of the provider to production? The answer should be yes, because it already supports the current version of the consumer. And it's also got the new functionality for this new branch. So we deploy it to production.

Okay, so you can see the provider's run. A whole bunch of assertions have been happening by the PAC framework. It's published the results back to the broker, and it's run the can I deploy check. Can I deploy says yes. This version of the provider satisfies the current production version of the consumer that it would be deploying to, and so it deploys to production. So if I go back to PacFlow and refresh this page, you'll see that this new feature branch has now been verified by the provider. And the version of the provider that satisfies this contract is the same one that satisfies the production contract of the consumer. And so basically it's now safe to merge this change into the mainline and push it to production. So I'm going to pretend like I've merged this. So I'm going to check out back, I check out master. And I'm going to run the CI process as if I've just merged this into master now. And so what should happen is now the consumer will run its tests, it will publish the contract. The contract hasn't changed. So any verifications that have happened before are still valid. It will run a Can I Deploy check. It says, can I move this change to production? And because there is a version of the provider that satisfies these needs, it's safe to do so. And then we deploy it. And then we go, we are now in production.

So refresh this, the prod version of the contract now has the price property in it, as you can see here. What happens if we remove a field from the provider? What happens then? Will we catch a bug? Well, for example, if I remove the ID field, sorry, go to provider, and comment that out, I can now run NPMT locally without pushing any change and find out if I'm going to break any of my consumers. There you go. It correctly found that a consumer's using this property, and if we removed it, it would fail. But if I remove the version property, what's going to happen here? Well, as we know, no consumers are currently using this property, so it's actually safe to release it, to remove it. There we go. That's an interesting property of PACT. PACT is able to pick up changes this way at attribute level to find out compatibility.

Okay. So today we talked about the cost of end-to-end integrated tests. We saw that there's a high cost of maintenance, and that they scale poorly. We saw how contract testing can help with integration testing by combining the approaches of fast and isolated unit tests with contracts to prevent drift. We saw how PACT works and how we can gate releases using Can I Deploy?

So I hope that talk was useful. Really appreciate you coming on. And feel free to contact me on my details there, if you'd like to talk further. Thank you very much.

Hi Matt. Hello everyone. Thanks for having me. Is the poll result surprising to you, or did you expect this? No, that sounds about normal. Look, not everyone's suffering so bad they need to buy bulk coffee and buy themselves a coffee machine like, like I do at home. But, you know, most people find the integration testing, or at least end-to-end integration testing, challenging enough that they have to spend a bit of time on it and generally need something to get through that pain because, you know, the flaky tests, because it takes time to manage because chasing down the issues. You know, there's always an excuse to not want to look into those tests and write those tests and maintain those tests, so it's not entirely surprising, but it's also good to hear that not everyone is in so much pain that they need to, you know, have caffeine hooked into their veins just to better get through the day.

Yeah, but flaky tests are always a good excuse to get another cup of coffee.

There are a few questions. Manuel Zambrano asked, can we use PacFlow with different consumer and provider technologies? I mean, for example, a React frontend and a Python backend serving the API? That's a good question. It's probably the main reason people choose Pact over other alternative sort of contract testing frameworks, so it's worth giving a quick shout out to Spring Cloud Contract. So if you're using Java here, because this is a testing conference for JavaScript, but if you do use Java, Spring Cloud Contract is a decent choice, it's putting the folks out of Spring. They do have support for other languages, but the way it works is you still write groovy scripts, so I guess technically you're probably still writing JVM stuff. But one of the benefits of Pact is that it uses a specification that enables you to work across languages. So really early on in its design, the early maintainers recognized that this would be a challenge, that polyglot architectures were a thing, and that contract testing needed to support polyglot environments from the beginning. So there's actually a specification that governs the way matching happens in a language-independent way. That means that different consumers can be written for different providers and sort of in this environment, but the way the matching happens and the verification happens, so transcends languages. So yes, it's entirely possible to have a JavaScript front-end, an iOS front-end, a Swift front-end, talking to a Java API back-end or Ruby back-end or a .NET back-end. You can mix and match languages kind of as you please.

Amazing. Steve, I'm probably pronouncing that wrong. I'm sorry.

I was asked how different, slash, related is contract testing with applying a JSON schema to every single request you perform using, for example, Postman? Yeah, that's a great question. We actually get that question a lot. I've got some articles. I can point people to afterwards that sort of talk about the difference between schema testing and contract testing. It goes sort of more skin deep. One of the first things that with schema testing, for starters, is that you're only normally looking at their bodies. You're not looking at the request, the other HTTP things. So the verb, the path, the query string, the headers, as an example, they usually don't get covered in your contract test. They're also obviously a very important part of that contract. But obviously, you can do that. The second thing is even if you're using JSON schema on one side of the contract, what's guaranteeing you that that is exactly what the provider needs on the other side. So you do need to make sure there's a way of pinning the schema you use on both sides of the contract and ensuring those schemas are compatible. So for example, let's say you've got version one of the schema on your consumer side, then the provider updates its schema, you need to make sure that those schemas are now in sync. So essentially, you could do that, but you just need to make sure that the schemas are always the same. And then the second challenge with that is if you think about something like contract testing with PACT, the way it wants to work is actually using your intermediates through something like the PACT broker or PACT flow. And what that does is it lets you version and tag your contracts just like you would with Git code. So let's say you've got code, right? You're about to deploy this into production, version two of your code is in Git, as well as version one, and you need to migrate production from version one to version two. So you need a process to make sure that you can smoothly transition from one to two. And so, again, if you're going to use your own schematic testing tool, you're going to need to come up with your own process for evolving that schema from version one to version two. When you've got multiple components in the system. Again, so it's not that you can't do that. But you're gonna need to build a lot of things that the contract testing frameworks have in them already. What I will say, though, is it's a strategy that people seem to be employing and you know, you can probably get 70% of the way there and that might be good enough for most people's use cases.

Thank you so much for this detailed answer. Richard Fordshow is asking, Is contract testing mainly for unit tests or for end to end tests like with Cypress? I guess the question asks a little bit about where would contract testing stand in the testing pyramid? Yeah, so it's a good question. So probably, I'd say there's two questions. One is kind of where it fits in the pyramid. Can you use tools like end to end testing tools, which I think is, I think as a community, we need to fix that, end to end testing means too many things. We gotta fix that. I think we can't have a situation where Cypress means end to end test, and is really just testing the UI layer. And then we also can refer to end to end testers passing through the entire platform. So I'll put that challenge out there for someone to try and fix. But let's talk about the first one. So where contract testing fits in the pyramid, it sort of fits in two places on the consumer side. Pardon me, it's much closer to a unit test. So basically you should be picking a single function as you saw in the talk, and running a unit test for that function. So that's usually pretty straightforward. On the provider side, you've got a few more options about how far up and down the pyramid you go. Usually it sort of sits in the middle. So we would normally overlay the contract test into the middle of that pyramid, the service test or the integration test layer, where basically you would run up your provider as a bit of a black box, you'd start out any third party dependencies, and you start the service up and then PACT would talk to that service, you know, through HTTP mechanisms, passing through probably multiple layers of your application, but usually they still run very quickly. So they're kind of closer to a unit test, but they're not really a unit test, because they've got to pass through a few layers. So that's usually where they fit in the pyramid. And, you know, in addition to that, what we normally do is we, we supplement, you know, in order to remove end-to-end tests, we supplement those contract tests. Or if you think about it, we got to end-to-end integration tests we want to get rid of. Contract testing removes the tests from the end-to-end tests that have the testing for the collaboration and the communication. Can they talk to each other? All the contract bits of that go away. So we shrink the end-to-end tests a bit. We then look in there and go, well, actually, usually what you find is at end-to-end tests, a lot of them is actually the consumer not trusting the provider. And so the consumer is writing end-to-end tests to check the behavior of the provider. So what you do is you take those tests out and go, nope, they shouldn't belong there. They belong in the provider's code base. Their job is to make sure their code works, not the consumer's job. And then you whittle it down to just a few tests and that's when you can go, well, what's the value in having these few tests left behind? Maybe we can put those in production synthetic tests, or we can put them somewhere else.

Or we can remove them altogether because they're not adding value and we can move faster. Now the second part of that question I think is around, could you integrate it with something like Cypress? The short answer is yes, you could integrate it with Cypress. There's some challenges with doing that and we're currently sort of making it less problematic to do that for testing with Cypress. But the short of the answer is, because typically with Cypress tests, you're going to have a lot of overlap. You don't really want to be capturing too many overlapping scenarios in your contract tests because they need to be replayed against the provider. And so that can become a burden on the provider side testing if you add too many. But again, we're doing some work to try and optimize that process to make it easier to use Cypress to write tests and to verify the provider side. And I should mention, we literally at PackFlow, we are writing tests using Cypress now for some of our user interfaces and we're experimenting with generating contracts through that as well. And I suspect in the next couple of months we'll probably release a plugin for Cypress as an example that could be replicated for others.