Generating types without climbing a tree

Hi, everyone. I'm Matteo Collina, Platformatic Co-founder and CTO, and today I'm going to talk about something that makes me a little bit nervous, generating types. Entries. Why? Let's dig into it. So, the first question for all of you is, how many ropes do you need to climb a tree? This is a fundamental question and we are going to try to answer that question in this talk. So, let's go.

First thing, a couple of things about me. I am Matteo Collina, part of the Node.js Technical Steering Committee, also board member of the Open Source Open JS Foundation. I created a bunch of libraries, help maintaining nodes, probably using some of my software given it's been downloaded 17 billion times last year, so yeah, maybe. So, some of my decisions impact a few people, so hopefully I don't break too many of you during this talk or otherwise.

So anyway, let's go and let's jump. So, I wrote a lot of modules, I maintain a lot of modules on NPM, a lot of downloads, whatever that matters. By the way, if you ever use one of those, if you want to send me a GitHub sponsor, everything is appreciated. I have a little bit of a small confession to make. What? What? Well, before 2022, I almost never used TypeScript. And here I'm speaking at TypeScript Congress. So what is that? It's really fascinating. I completely skimmed and avoid this technology for as long as I could. What happened in 2022? Well, I started a new company, and it was a developer experience for companies. So yeah, I kind of had to dig into TypeScript. In fact, it's all based on something I'd previously done, which is Fastify, which is a project inside the Open Source Foundation too. And it's all about building APIs. I'm a back-end engineer by trade and what I want to help people and how to help people with my tool, you know, to build and use APIs better and make it faster and make them easier to develop, maintain, scale, whatever you name it. So yeah, typically the typical flow of when one of your team members or somebody else is starting writing your APIs, you have an API on one end and you are typically a back-end engineer writing some complex SQL or some complex MongoDB queries, whatever you want to like, then they generate, write some documentation from that API and then your client, your engineer, code their client. So, okay, how does it make to generating the types? Yes, let me introduce the problem first. So, this is more or less what Plasformatic wants to simplify and improve how we create back-end software. Typically what we want to do is we want to move, allow you to move very quickly from A to B using as much rails as possible, guard rails for all of you to move very fast. And then, you know, you can get your old SUV to go where exactly you want to go. So, we want to help developers get rid of all the undifferentiated heavy lifting of building your JS apps. So, as part of that, we, at some point, we started focusing after we worked a lot on APIs and improve on two APIs that had been written.

What we found out from talking to a lot of people was that, you know, oh, but having a client for those APIs was actually fundamental for providing a good experience and streamlining the development. So, what happened was that, you know, most people really like this flow much better. So, instead of having to call the client and prepare documentation for the API, I just want to hit a button and generate a client.

In fact, this has been so popular for ages, and kind of we got it lost a little bit along the way, because, you know, I remember generating clients for my APIs when I was doing WSDL and SOAP a long time ago. So, you know, probably not a great comparison, but it is. So, anyway, this has been extremely popular recently. In fact, this has been popularized a lot by a few new libraries. One is tRPC, which is fantastic, provides a fantastic developer experience. It allows you to expose your... creates an RPC call from the server to the... from the client to the server in a completely type-safe way. It's great, okay? It has only one significant drawback, which is introduced type coupling within your... the code of your frontend with the code of your backend, it also uses a proprietary protocol to communicate. So yeah, it's very tight. It's a very tight coupling between client and server. There is another library called TS Rest, which is also great, fantastic library, but it has another small... still another small problem, which is you need to specify the contract between your client and your server at its own typescape definition. Again, in order to consume the library, you need to have a source code dependency on the server. So again, for me, it provides introduced tight coupling between those two.

In fact, the problem here is the concept of you know, what happens in most teams that I have worked and developed with and so on, it's that you have basically some API that is developed by more or less a team and there is another group of people that you know, creates and consumes or that consumes those APIs. And those groups typically are more or less separated in most companies and in most organizations. So, integrating everything together, create a tight coupling between everything that makes it very hard to break down and scale and keep evolving things. And by the way, there is nothing to be taken by those libraries, which are fantastic. I'm just talking about what I am not, some of the reasons why I think we need a better client experience for APIs. So what, where are we left? Well, something that we want to provide some fantastic developer experience but without any tight coupling. So, without any code being shared, essentially, between the server and the client, so that you can essentially use and consume other APIs. And so, that one can use TypeScript 4.9 and the other one can use TypeScript 6 in the future. And they can evolve and have different versions, different things. Everything is fun and perfect. And so, how can we generate this developer experience? I started this journey, one of my journey types of work with this question.

And as part of this journey was the little bit of a survey that shows that more or less 80% of developers prefer REST to GraphQL, which sounds very much unintuitive to me, but that's what it is. And if you have better data than this, please let me know, because this is actually something that I'm very curious about.

So my journey started where how can we automatically generate a client for our REST APIs? And the generated code should be either minimal and simple. So how can we get that? Okay? Well, something very fantastic was that OpenAPI is a standard, okay? And I built this framework called Fastify, which is a really good framework. It's taking over more or less a lot of it's getting a lot of attention these days on how to build APIs in Node.js. And you can easily get an OpenAPI or swagger, as it was previously called, create generated very, very quickly on top of your routes. So we can get the documentation, but also the spec definition for OpenAPI. So we have all the, we can get these mostly for free using Fastify or using platform.

So the client, in fact, had a few requirements for me. And one was, well, I have my routes defining using OpenAPI, and I want to consume them very easily on the front end. I really don't want any data validation happening at runtime, or maybe it should be disabled, okay, whatever, because I don't want to waste any time on the client. I want my code to be as snappy as possible. But I also want to be fully typed to catch developer mistakes on the editor. And I want to use fetch, so I have no dependency whatsoever at runtime. So I can just get my generated file, and that's it.

So my first try to generate this code was to use. So I started using this code generation. How can I get the code that generated my code generated? And I read a good article on DevTool. I started doing a lot of things and I wanted to do. Let's do an experiment. Let's just generate this very, very simple function and this code on the right. It's what it takes to generate that function. Now look at the code. It's actually complex. You can read what it's doing. I create an identifier, I create the function, it does all the things, but it's very, very hard to... If there is a problem there, it's actually very, very hard to debug. It also takes a lot of code to generate very few lines. So I was not necessarily very... Oh, I was...

Oh, ouch. How can I... I thought I could generate TypeScript easily, but how can I get it? It seems this is a massive tree that I have to climb and I couldn't get it done in my timeline, so I was like, maybe not. I didn't want to climb any trees, but this seems a massive tree to climb, so, ouch, how can I get it done?

So I did another try. I did some scavenging, and I did a second try and this library that I found is called Cold Block Writer, which is absolutely something that it's really, really nice. And I think you should be taking a look at Cold Block Writer if you are trying to generate code dynamically. Why? Because it has a very, very good property here. If you look at the code on the right, it's essentially homomorphic in the sense that the code, in how we write the function, is very, very similar to the generated code that we wanted to have. Because these are homomorphism between those two things, we can actually find bugs and fix them very very easily because, oh, these are the three lines that generate the same code. I can see these three lines and those lines. So, it's actually very useful, very easy to write because it has this very nice block property that allows us to create insert braces automatically. Hey, nice! And it also generates and manages the indentation for us automatically. I really like this library.

It has only one big drawback. The generated code is not guaranteed to be valid. It's also way, way smaller, so that if I wanted to bring down the TypeScript compiler, like 50 something, whatever megabytes, they're reducing it. So, maybe it's now 30. I don't know. Anyway, it's a massive dependency, the TypeScript compiler, but this one is actually very tiny too. So, it's actually very nice and cool. So, it seems like a lazy option here. And I am a lazy developer, so I started going, let's do that. Let's do lazy.

Okay. And let's take a look at the OpenAPI spec and how it generates the code. What code we want to generate. So, look, the OpenAPI spec, it has a path property, and it says that there is a slash movies, for example. And each path has an operation ID. And this operation ID, we can use to generate our function names. So, when we, somebody wants to call a wait get movies, for example, we can call the slash movies endpoint automatically.

Okay? Also, it has defined its parameter and responses. Yeah, it's great. Okay? And then we can do the exact same thing for each of all the other operations that are defined in my OpenAPI specification. In fact, we can even, you know, we can even, oops, sorry. We can even, you know, check and define the types and generate the types for each one of our parameter and responses. If you look at it, it's actually pretty great. Okay? And we have all the parameters defined and then we have all our responses defined. So we can generate interfaces for both of them and use them as part of our API.

So we have create movie and there is a create movie request that returns a response. Seems good. So I think this is, you know, for me, one of the interesting points here is that you see here we have split, oops, sorry. We have split here the types of our APIs with the actual interfaces. So in this way, things can be manipulated better, we think. So anyway.

Some of the good, there's a few good ideas here, but the generated code is actually very, very simple. Like it's you can actually map with one to one to what you need. And I think it was, we could generate this client very quickly. So let's do a little bit of a quick demo. And so that we can see how easy it is to do stuff. So let's create a little bit of a small API, so we can do create platformmatic. And here we go. We want to be here, new SQLite, but we could use everything. Oops, sorry. And yes, yes. And no for now, because I don't want to do NPM install. So I am creating very quickly an API automatically. And you know right now I don't need anything. So when I start this up, I could here do a platformmatic start. And if we open this up, okay, and we go back in here, sorry, here we go, I can see exactly what's happening. So this is what gets loaded and I can open my OpenMPI documentation, and I see that I have movies with all my parameters, which we can automatically get for creating everything. So how do we use, how do we, oop, sorry, how do we generate the client? Okay, oop, sorry, I am… So how do we generate the client? So let's go back cd client, okay, and here we have, I prepared a very simple thing with a package json, vcsnode, and typescript so we cannot waste time with npm install, so now I can do plt so-clear, and plt frontend, and I need to put in the url of my app, and then I want to generate it as typescript, and it has generated us two files api-types.b.ts and api.ts, and we have two of them, okay, and let's take a look. So this client is actually very simple to use, and we have uh we have here our fetch. So I need rts.config.json here, and I need the compiler options okay, and I need the target, and okay, otherwise things will be awkward. Here we go, and comment back, okay, good. So this will be awkward, okay. So anyway, what we have here, we have our our fetch. So it's all fetch based and all standard based. So we have actually no dependencies whatsoever, and then in api.types, as I showed, we have our get movie request here that gets things through.

So yeah, this seems pretty good. So let's try and use this and how the experience would look like. So if I open test yes, I could import from api. And now I could get setBaseURL and for example, create movie and get movies. Now I could do setBaseURL 0.3.0.4.2. And then we want to create here, does not, as you see here, does not exist here, okay?

And we, does not exist in the createMovieRequest. So in fact, if we look at the types here, if we go in createMovie, sorry. Okay, createMovieRequest, you see that it has an ID, optional ID, and then a required title. So here it's fine, but then we need to, we have created our movie and we don't need anything here. So okay, and then we could just get movies here, and do console log, and I need specified some parameters, but in this case are empty. So, and we have generated, we have getting these movies. Okay. And let's try this. Okay. So if I do TS node test. Now it's running. Cool. Okay. I'm running price. Why it's running price? Because these two are, you know, let's be nice. Let's be nice. So function run. Here we go. And I think that they were executed in parallel, right? So we can do await. And here we do Start Wars. And here we do press await. Yes. And then run. Now we can run this again. And here we go.

You see we have the two that we had before. And now we have Star Wars. We can also verify it very quickly, if everything was correctly. And it's here, we could try this out and see if this works correctly. As you can see, we got the exact same result. And this is the code that will be executed and generated. So no, this is, I think this is pretty nice. It's a pretty good way to generate a client. As you said, here I'm just going to let me do a tree. Here we have no dependencies whatsoever apart from ts-node and stuff, like if you can, here we have only ts-node and types. So there's no other dependencies in those files. It's super small and all API based. So the benefit of this approach of using openAPI is that it works with any openAPI compatible with any openAPI system, so you can get a very good developer experience just by generating some code for those openAPI definitions that are already available for a lot of APIs, so you don't need to generate some new ones or create new libraries or anything to consume them. So I don't know, I think this was an approach that was very, very interesting to try and verify if everything would have helped us somehow.

So the question for you is, did we climb the tree? So hey, did we climb the tree? I don't know. Anyway, thank you very much for watching. Try our open source tool at docs.platformatic.dev. And I don't know, thank you very much. And thank everybody for having me at the S-Congress. Thank you. Bye.

What's up, Matteo? How are you doing? Hi everybody. Hello. I love your voice man. You've got such a great presentation style. That's awesome. Why don't we jump to some questions? We've got some real crackers here. We were looking at these just before we came on and we were talking about this one. What should you do if you have internal issues and your front-end team can't trust your back-end team? You mentioned that in your talk you don't think you should be using like validating with Zod on the front-end or anything like Zod. What should you do in those situations? There are two bits, okay.

You don't, you really don't want to do those kinds of validation in production. Mostly because doing any validation does, or having any validation library in the front-end does more or less two effects. First, it makes your code slow down. This is the first one. And the second one, it increases your bundle size. So as small as it is, as small as your library is, it still needs to be parsed, evaluated. Whenever you're going and calling, you're validating that response and it takes quite a lot of effort in it, okay. So it's actually, to be honest, not something good for production usage in validating all responses.

Having said that, adding that some kind of development mode or something to be able to pinpoint your team and your client team and say, look, your response is not what you said it should be. With a little finger, you do something different. You've been in that position, so I'm pretty aware, so everybody. I think we'll be adding that, in fact, maybe we already did, but in the time I recorded the talk. So it's coming, or if it's not there, it's coming soon. It's useful, but not so useful in that sense, because if you have this problem of, you don't want to be running it in production, right? So otherwise, you lose a lot of things. Yeah. I mean, you don't necessarily want your development code to be different from your production code as well. So there's that kind of in the mix, too.

I love what you said at the start. You're kind of famous, kind of recent TypeScript convert. I remember, I've been following you for a while on Twitter, and a few years ago, you said, you know, TypeScript sucks, why are you all doing this? And now you're here. So what happened? What changed your mind? What's the story of how you felt? OK. I realized something very, very interesting about TypeScript. It's TypeScript is code that you run in the developer editor. And that's probably the biggest truth about TypeScript that you can think about. And that's what allows TypeScript to create a certain level of developer experience that are not possible otherwise. And I love that. OK, to be honest, this is pretty phenomenal and amazing. So that's probably the reason. I haven't changed my mind. It took a while to convince you though, right? I mean, you weren't on board straight away.

No, I wasn't. I had a very bad impact at the beginning with the TypeScript community, and they basically completely flushed me out as a person. I received hundreds of issues telling me to write this in TypeScript, and you don't really want anybody to come to your house and tell you to change the paint. Or change the foundations as well. I mean, that's a big deal. Change the foundations, okay? And in some cases, not even possible. Like when this happened, TypeScript was very, very... It was not even a 1-on-1 mapping with JavaScript, and it was pretty bad, to be honest.