Safely Handling Dynamic Data with TypeScript

Hello, everybody. My name is Ethan Erewood. I'm a software engineer for Microsoft and today I'm going to be talking to you all about safely handling dynamic data with TypeScript.

So, handling data. What is data? As software developers, we use a lot of it and in a lot of different ways. Some good examples, at least of how I use it is within API routes, when building a back end service, dealing with forms in the front end, and also authentication payloads and all of the surrounding things that go with authentication in an entire full stack application. And this is just a short list. You can only imagine how long this can get when you start dealing with databases or data science and just general, any sort of communication between large systems.

So, let's take a peek at an example of a record of data. In this case, I'm using a JSON object. We got a bunch of keys here. ID, name, employed, company, age, and projects. We're representing a person. Maybe this is an employee directory or maybe it's a user directory for a site such as LinkedIn where we have a user, we want to get their name, we want to know are they employed or not, which is a Boolean value. We want to know what company they work for. We want to know how old they are. And we also might want to list their projects. And as many folks know, JSON is a very verbose way of representing data. There are it has a lot of great primitives that are all based in JavaScript, and it can be quite extensive. In fact, entire APIs are powered by just JSON projects alone through the open schema format.

Talking about backend APIs, let's take a look at a Fastify route. In this case, we're defining a post route. The path is add user, and the request handler here has two arguments, request and response, and we're destructuring the body from that request object. Does anyone know what type body might be? Is it a record, an object, is it any type? Trick question. It's unknown. The body property of that request object, taking a peek again at the code, the Fastify route has no idea what it is because in context of Fastify as a framework, we're not sure what the developer intends to be coming in through their request. And there's no way for Fastify to know that when you're right when the code is being written or even compiled. Well, maybe not when it's compiled. We'll get to that later. So let's take another look here.

Looking at the previous JSON object or take a slice of look at the ID and the name, there are string keys and string values. And then looking at the post route again, we know that body is unknown that we're destructuring from the request object. So what would be body.name? In this case, imagine that the JSON object is being sent to this route as the body in the post request. So what would name be? Probably a string, right? Well, a trick question again. TypeScript will throw an error. Object is of type unknown. Why is that? Well, it's because the body property is coming from that request object. Because it's unknown. No other types can be derived from it in safely, in TypeScript. TypeScript goes, no, no, no, stop here. I don't want you to keep going and using properties on this object because as the TypeScript compiler, I don't know what it is. And I can't provide you the type safety that you're looking for. So even though, as a user, we might think, ah, the name property, it's always going to be a string, there's no if-ans or buts about it. In this case, TypeScript is like, well, you didn't tell me that. I have no way of assuring that. So there's some patterns we can use. You can use basic typecasting where we can say body.name as string, and we will tell TypeScript it's a string. But in that case, there is no verification. There's no way of saying that assuring that that name property is actually a string because TypeScript is a compile time only type safety. During runtime, it's all just JavaScript. There is no type safety at the runtime. So what are some other solutions? That's where I want to introduce JSON Schema. JSON Schema is a super powerful API, lets you define a JSON object with more JSON. Isn't that just wonderful? Kidding aside, JSON Schema is actually incredibly verbose, even more verbose than the JSON object it's probably defining. JSON Schema uses a standard or a specification to allow a developer to define the shape of a JSON object, given things such as the type, listing the properties, saying what properties are required or not, listing if there are additional properties or not, and even being able to define more complex types. As you can see in the code sample, the projects property is of type array and then we get to go even further and say the items of that array are of type string. JSON schema, and this only scratches the surface of JSON schema. You can use regular expressions, you can use references, you can use logical operations like all of or any and or some, and it's just so powerful when you leverage JSON schema to define your JSON objects. So all that said, though, JSON schema, I think it's even in their spec, is intended for validation. The validation in a sense of here's a schema and here's a JSON object.

You give that to a validator and you say make sure that my JSON object is actually what I'm saying it's supposed to be via the schema. Throw an error or give me some special output if it's not. So with that, we're we have our JSON object, that payload from before in one hand. We now know that we can define a schema for it in the other hand.

So going back to the Fastify application and that route, are we able to provide any type safety yet? Not really. All we can do is validate that that incoming body is what we want it to be using JSON schema. But come on now. It's 2021. There's some pretty cool things we can do.

So, let me introduce you to TypeBox. TypeBox is a fantastic library that allows you to not only define JSON schemas using a fluent-like API, but also derive a static type of that JSON schema in your TypeScript code. So, given this very basic example, you can see on the second line, constant T equals type.string. That is going to return type.string, an object with one property, type, that is set to string. This is a valid JSON schema object. We are saying that the variable T is the JSON schema type string. And we can use that JSON schema that is assigned to variable T in anywhere we would use JSON schema, in any validation or serialization level.

But on that last line of the sample, you can see type T equals static and then the generic parameter type of T, which is that constant declared on the second line. And now that type T is a string. The type box library is smart enough to derive the type from a JSON schema into that static shape. And so at a more complex example, let's redefine our body schema using type box. You can see we started with type dot object and then we've listed our six properties, id and name, our type string, and then employed company age and projects are all just type optional. And then within those optional types, we define type Boolean for employed, string for company, number for age. And in projects, you can see we're doing type string inside of type array, which is the type box way of saying an array of strings. And then we're defining a type T body schema is going to equal the static resolution of the type of that type box body schema. And then in the final block of this code, Fastify.post, you can see we're now using a generic parameter. It's a object, a name, I like to call them name generic parameters. And we've assigned that T body schema type to the body property of that generic parameter. The Fastify type system will forward that T body schema type through to the body property of the request object. And so that's great. Now we've typed the body property of the request object.

But we're still missing a key step here, but this code already solves it. Is the validation step. As you can see underneath the add user string, there's another argument added to this route. And it's schema and then the object body and then it's the body schema. And don't get confused. That body schema is the actual JSON defined, returned by type boxes API. Fastify has this schema option which allows the developers to verify the incoming data of a request and the outgoing response shape using JSON schema. Under the hood, we pass that to another JSON validator and we validate the content based on the schemas provided. That validation step happens before that handler function executes. So if the validation step fails, if the incoming request body doesn't match the schema that we've set it to in that second function argument, then the entire route will error and return an error to the user saying invalid body. And then I think it also includes the JSON validator error as well.

And with that all together, inside of our function handler now, by the time Fastify gets there, we can think, okay. Fastify has validated the body property of the request object using the schema we've defined using typebox. We've also provided the type for that body property using the generic parameters that is a type that is derived directly from the same JSON schema used to validate that property. And so, now, inside of our function handler, that body property from the request is going to have the tbody schema type. And accessing something like body.name will, in fact, return a type safe string. And if you're accessing one of the optional properties, it will be that type. In this case, body.number or undefined.

And this is my favorite part about all of this. But it also sort of breaks down a little bit of the whole kind of problem in the first place. Where TypeScript is a compile time only type safety not run time based. And so, we are inside of a route handler and you're taking that body from the request object. If we forget about the fact that Fastify is going to validate our incoming data for us, you have to remember that you can unsafely just say body is this type. And then go about your business and your route. And that's kind of it. You know, there's no actual true type safety there. But with the validation layer that Fastify adds, we can get as close as possible to type safety for highly dynamic data. Because if we put a hard line or if we trust Fastify enough to be like this route handler will not execute if the validation step doesn't succeed, then we can be, then we can tell TypeScript confidently that the type of that property inside of that route handler is going to be what we want it to be, what we think it's going to be. Because we trust in that validation step. If that validation step logically is incorrect and will let through something that we don't, aren't prepared for, then we're going to have a problem. And unfortunately, even a JavaScript application wouldn't be able to sort of respond to that. You know, it will probably have to error out. If you're trying to access a property that doesn't exist, because it somehow gets around the validation step.

So all in all, JSON schema plus TypeBox plus Fastify is super, super powerful. And I briefly want to jump over to VSCode and show you all this in sort of real-time so that you can get an understanding of what it actually will look like in a real Fastify application. So in my Fastify app here, I have this basic run function that's going to create a new Fastify app. We're going to register our create server plugin. We're going to do a wait, app.ready, get the address, and then print some nice things out to the console. Over here on the server, you can see we've imported that type box API. We've defined our body schema. And you can see it has this really interesting shape.

The type of body schema is a type box T object with a generic of this named generic parameters. Where the first property ID is of type T string. Name is of type T string. Employed is of type T Optional with the generic parameter T Boolean and so on and so forth. In fact, the projects one gets cut off, but it's going to be T array, T string. And we can sort of hover over that by seeing what would this projects be. You can see in the IntelliSense it's T Optional, T array, T string. Generics are crazy powerful nowadays.

You can see down here this type, in the type T body schema, it's kind of looking kind of funky. We have empty object and empty object and object, optional employed, optional company, optional age, optional projects, and then and another object ID and name. Going down into our Fastify route, you can see we're passing that body schema that is defined here. It's an actual JSON schema to the body property of our schema parameter here for Fastify. And in this post generic, we're passing that type T body schema. And if we hover over body inside of the function handler you can see we're getting that static object type. And it passes that same generic name generic parameter that comes from body schema. And now when you do body.name, you get type string. And if you do body.age, you're going to get number or undefined. And so inside of this function handler, you can now most confidently rely on the type safety of your dynamic data that is all based on the same JSON schema that is being used to validate it at the same time.

So jumping back over, I want to say a big shoutout to the Undraw Graphics illustrations collection. Without it, my presentation would be way, way more boring. And also this whole thing was built with HighlightJS and the TMCW Big Library. I highly recommend it if you yourself are working on a presentation. I want to say a big thank you for attending my talk today. I hope you learned something. And I hope to answer your questions in a little bit. If you'd like to follow me on Twitter, there is my thing. You can also follow me at GitHub as well. I love to connect with folks. I love talking about Node.js and TypeScript. And yeah, I hope to see you all around.

Thanks again. Hello. Wow. Hey. So what do you think of the results of the poll? I think that was great. I think that was the answer I was kind of hoping for where it seems like a good part of the crowd at least knows what it is, has used it before at different levels of experience. And there's still a slice of folks who have never touched it. And hopefully, after this talk and maybe this Q&A, they'll give it a shot. Yeah. That would be awesome, of course. Well, that's why people come to these events, right, to get to know new stuff and hopefully play around with them and use them at their companies in production.

The first question from one of our audience members is from Walker MAA, and he says, great talk. Is Express also able to validate body against schema as Fastify? Yeah. So, Express, I believe, so I don't use Express as a Fastify maintainer. I tend to just use Fastify. I believe that it's not built into Express, but there are very similar JSON schema validators that will work just the same as Fastify with Express that you can use as like a middleware. And the thing with Fastify is that it's built in, so it's like by default, you don't have to load any special plugin. There's no, there's nothing you have to like pass to the server. It will use a schema as provided to the routes. And under the hood, we use a validator called AJV, which is another JSON validator, and I think there's another question in here that asks about one of the other options there. So most folks are familiar with something like Joy, which comes from the Happy web framework set of tools. But for Fastify we use AJV, it proves to be the fastest, and it was most spec compliant with how fast your JSON schema iterates. Awesome.

Next question is from Christina, is Typebox the TypeScript version of Joy? Yeah, so Joy is a big library. It does a lot of things. Firstly, it does have a similar API to Typebox, where it lets you build your JSON schemas using the Joy API. That part of it is very similar to Typebox, but where they differ is that I believe Joy will still do validation on the...has validation and serialization features for you based on the schemas that are output by the Joy API. Typebox doesn't have any validation, doesn't have any serialization. It's just the JSON schema building part. It will play nicely with any JSON validator.

Okay, awesome. Next question is from Zero Carol. Okay, cool, but why not use Swagger OpenAPI? This is a great question. This comes up frequently all the time. The answer is sort of the same questions. Why not? Go for it. Swagger and OpenAPI, I believe, is all implemented on JSON schema. Everything that they export has their own blops of JSON schemas that you can use and plug those into other things, other tools, such as like the automatic swagger websites that you see. Those are all powered by the JSON schemas and that is the exact kind of thing that you can pass to your routes here, where if you use the OpenAPI format to define your REST API, you could use those same JSON schemas to make your Fastify routes type safe when you're implementing that API server.

Cool. Next talk is after you, but first we have a question from Johnny Gat. Isn't Typebox the same as TypeScript interfaces? Yes, it is. Typebox is a tool that will output TypeScript types as well as the JSON schema. You could do it yourself by hand. You can write your interface that implements your own JSON schema. The idea here is classic programmer like, don't repeat yourself. Why write two schemas? Plus, if you make a change on the schema, you have to make sure you make that change on the interface. There's not a lot of great ways to validate that the interface that you wrote is actually correct. The correct implementation of the JSON schema. Users make errors all the time. So by leaving it to a tool to make that translation and convert the schema into a type or vice versa or both at the same time, that is a lot more reliable. Yeah, it adds a bit of a security layer that things don't go out of date and out of sync. Awesome.

Next question is from Max Zdzadorov. Why should I use JSON schema and not just declare TypeScript interfaces? Well, I think we kind of just touched on that, but do you have some more insights that you want to share? Yeah, I'll share a little bit more. So, the reason why I strongly encourage folks to use JSON schema rather than just passing a type interface to their route is because Fastify will validate the thing that you're providing a type for and it's that validation step that actually provides the type security that you're using TypeScript for in the first place and this kind of gets into the whole ethos of like, what is type safety? Is it actually type safe if it is going to be something that occurs at runtime and it's that level of like, security and safety and if we can get as close to that as possible that is sort of the best case scenario and getting to that point, getting to as close to safety as possible or type safety as possible is having to rely on a tool like this where it's JSON schema for validation and the type is inferred by that same schema. Yeah, cool.

So, another question from David Lime, could TypeBox be used to generate OpenAPI also? Great question. I'm not super familiar with OpenAPI, but I believe so. I believe you could find some level of interop there where you could maybe define your OpenAPI objects or schemas using TypeBox and have that JSON schema to use wherever you might use your OpenAPI schema and have the resulting types. The only thing I'll say though is I believe with OpenAPI, a big part of the goal is to get that JSON object as its own file and then pass that file around. Where TypeBox, it only returns it during run time. It's in the code. With that said, it's JSON in a variable. You can 100% stringify it and write that to a file. But that second operation step might be a little bit different than what many OpenAPI users are used to. Okay. I believe that that is the last question that we have at this moment.

We have another one. This question is by Marnice. Do you know anything like TypeBox but then for the front end? TypeBox or similar APIs should work on the front end. It's all about what other JSON tooling you have. TypeBox is just a JavaScript module that has a very easy to use functional API that returns a JSON object. I don't think there's any special node JS things that it's using. If there are, I believe the interop with the browser would be okay. Off the top of my head, I don't know if TypeBox can just plug and play with a browser. It should be fine. If not, there's definitely alternative tools that could be. Or PRs Welcome. Yeah, exactly. Someone nice. If you have nothing to do over the weekend, you can try it out on your front end. We have a comment from DC boy CM, who has a Batman Avatar.

I think he's a DC fan. Great stuff. Great talk, Ethan, my favorite of the day so far. You can take that home and tell your mom that you had a big fan.

And with that, I would like to end this Q&A session. Thanks a lot for your great talk and this informative Q&A session and hope to see you again soon.

Of course. Thank you all. Have a good day. Don't forget Ethan is going to be in his spatial chat if you have more questions for him. Bye.