Taming Language Models through TypeScript

Hi everyone, my name is Daniel Rosenwasser and I'm the program manager on the TypeScript team at Microsoft. Today, I want to talk to you about TypeChat, another thing I've been working on recently with others on my team.

So, in the last few months, you've probably seen a lot about artificial intelligence and large language models. These language models are very powerful, and you've probably gotten the chance to even use them in the form of something like ChatGBT or a similar sort of chat program where you can ask the model questions, it can give you answers, you can iterate through ideas, and it's just a great way to creatively iterate through concepts and whatnot.

That works really well for people, but these language models are very powerful, and you may have had the idea of trying to bring the smarts of one of these language models into your app, maybe to add a natural language interface in some way. So for an example, let's say we have some sort of app that's supposed to help us plan out plans for a day, plans for the weekend, maybe specific to a location. It doesn't seem impractical given the fact that these models have been trained on so much data in the world, so they might know a lot about Seattle or some other city that you're trying to scout out ideas for.

In this app, maybe we just want to be able to ask a question, that's our user intent, and then get a set of results with a couple of venues and each of their descriptions as well. So that seems all good, but how would you go about trying to get one of these language models to produce data that you can use in this app?

The thing that you may have realized in trying to do something like this is you end up trying to often pamper the AI to give it in a specific format, and even once it's in that format, it comes up with natural language that's just not always going to be easy to parse. So for example, here's a prompt and response that took a couple of tries to get even reasonable, right? One of the things that you might notice is I end up trying to just give it a little bit of guidance on how I expect the answer to be, right?

This isn't so bad, right? But mostly because it's come up with a regular answer in a specific format. And so it's actually given me a format of list items. Each of them is numbered. And then between the venue and the description, I have a colon. Now, is the AI always going to come back with that schema or format? Not necessarily. A lot of these language models are non-deterministic. So you really can't count on this format. But even if you could count on that format, you can't always trust that the data is going to be uniformly parsable, right?

So for example, in this case, I have a list and I have everything in this format. What about trying to split the colon, right? You might try to say, let me just try to shave each of the numbers off and then split by the colon. But what if one of the items in your list has a colon inside of the title or something like that, right? You're basically trying to do natural language parsing at this point, right? And now you have a bug, and now you have to figure out how to be resilient against that. And so this ends up being a little bit, impractical for most people, right? It is very hard to parse natural language.

But many of you probably also realize that you can get the language models to respond in the form of JSON. And that's great, right? Now you actually have something that your app can easily just do a JSON.parse or whatever, and get the data that you need and work off of that. But that only really works for sort of simple examples, right? Here I was able to say, here's an example of the JSON that I want to get back, right? I have a venue and a description, a venue and a description. And the AI is pretty good at figuring that out. But it doesn't tell us about maybe optional properties, maybe the case where you have three different kinds of objects that you expect to be in a specific position, things like that. And so just giving examples would be impractical because you would get into these combinatorial explosions of all the types of things that you would want to actually provide. So examples aren't enough. What you need is something a little bit more.

And it turns out there is a format that does work out pretty well, for the most part, in our experience. And it's something that you're all familiar with here at this TypeScript conference, which is types. TypeScript types are actually a great format for describing the exact format that we want out of a language model and all the sorts of shapes that we're expecting. So types are actually really good at guiding the language model into the thing that we want, right? And you can actually take the types in an application, like the actual text of the types in your program, take a user intent, and craft a prompt that you can send into an AI service, into a language model. And that will provide you with JSON, You would say, give me a response in the form of JSON, here's the type that you should conform to when you provide that response. And so, now you're able to guide the language model.

But like I said, the language model isn't always going to come back with a response that is actually exactly as we expect, right? So maybe it comes back with JSON, but it's not of the format precisely, right? So we can always say, hey, you didn't get me correct in the JSON, try again. But you need something else that pushes an AI so that you can say try again as well. And that's the validation.

And so, actually, once you have the types, you can use the types to validate the response from the language model as well. And so, what we actually just decided to do is, we said, hey, if we have the types and we have the JSON, it's very trivial to construct a small program where you can get TypeScript itself to check each of the properties and each of the values that have come back, because it's very simple at that point. Because JSON is really just roughly a subset of JavaScript. And JavaScript is a subset of TypeScript.

And so, you can actually create a program, have TypeScript type check it, and if it has an error, you feed that error back into the language model, and have the language model try to repair whatever its response was from the last time. That works out pretty dang well. And so, you can use this to actually construct even more complex responses, right? So, you can actually say, hey, maybe I am a kiosk that takes orders for coffee or something like that, right? And so, the orders are effectively in the shape of a cart. A cart has many items. Those items can be one of several different kinds of items. And we give that schema. We give a user intent, like I want two lattes, one tall and one grande. And the language model can or good language models can typically respond with well-formed JSON that conforms to the type. And it depends on the quality of your model, right? You can have it repair a couple of times. But generally speaking, many of the ones that you're available to people today that are broadly available do come back with a good response here.

And so I think that this is a good time to give a quick demo of TypeChat, which is this library that we put together to encapsulate all these ideas and make it easy to get running with them. So I'm going to open up the TypeChat repository, right? It's all open source. You can get it today. We announced it about a month ago or so. And so we want to first look at the readme for the examples, right? So this is the readme at the top level. But to get a sense of TypeChat itself, we want to take a look at the examples. In the examples folder, we have a readme with a table that describes each of the examples that we have. And they basically go from easiest to understand to maybe the most difficult. So the sentiment example that we have is kind of like the hello world of TypeChat. It's a classifier which takes a user intent, right? It takes a sentence from the user in a prompt, sends that to a language model and tries to categorize that sentiment as whether or not the sentence was positive, negative, or neutral. So let's go into that. We have a sentiment schema. And this schema file is what's actually going to be used when we send it over to the language model. We're actually going to take the full text of this schema file. But notice that it's just a very simple response, right? It's just an object with a property on it that gets a string that is either the string negative, neutral, or positive. Also, just something to keep in mind is I mentioned we're going to literally send the entire text of the file over to the language model.

So, that also means that things like comments are also going to be useful for kind of coaxing the language model to give you a response that fits a shape better than just the raw types themselves. So, you can also communicate more than just the types.

So, we're already in the sentiment example. Everything is already built here. So, what I'm going to do is run the example. So, this thing creates a prompt and I'm going to just write hello world. So, the language model that we're using right now, which is GPT35 turbo, is categorizing this sentence as neutral. Every single time I do this, as many times as I put an exclamation mark, it's always neutral for some reason. But if I write something like type chat is pretty cool. That's positive.

So, let's take a look at what the code looks like to actually construct this. So, first off, we have this function called create language model. Create language model actually takes the environment variables that we have in our process, and that's populated by a .n file. So, that .n file, I'm not going to show you the full contents, but you can configure things like the model that you want and some other data. Create language model currently is just there to sort of bootstrap you so you can get an easy model that's accessible that loads up either open AI models or Azure Open AI models. Really, a model is a very general concept. You can bring any model that you want. You can bring something that's trained locally. You can bring something that is a completely different service. Type chat is agnostic to all of that. All that we care about is, can you take a string and make a connection through a language model and eventually return either a success or a failure on the string. We just care about, can you give us a string from a string? Anyways.

So, we create the language model, we infer it from our environment variables and then we actually load up the schema file. This is the actual schema file that we have adjacent to main.ts. Now, there's some subtlety there and I'll talk about that in a second, but notice that this file right here corresponds exactly to this file right here. And then we create what's called a JSON translator. And this thing is going to take the model, take the schema and the name of the type that we want our response to conform to. And this is sort of like the entry point in our types file, right? We might have many types in this file. We may have a whole net or graph of types and we want to make sure that we know the exact one that we're starting off with. And so we're feeding in the name of the type, we're feeding in the actual type that we've loaded up in the program here, and then we have a translator object.

And we also have this process requests function, which is really just there to make it easy to create a command line prompt because Node for some reason makes it somewhat difficult. You give it the prompt that you want to have every single, you know, the text that you want to print at every single prompt, and then a callback every single time you get a user intent or a sentence, right? So this is every line that gets entered at the prompt.

So we run each of the requests through the translator, the translator eventually calls the language model, the language model responds, and then the translator takes that, uses the types, constructs the program and internally uses TypeScript to check the program to make sure that it has succeeded. And if it has not, it possibly does some retries and maybe backs off a little bit as well. And so if we're not successful, we just error and just exit. But if we are, what we end up with is well-typed data. So we get our sentiment, a success of sentiment response, our data is a sentiment response, and then we have that well-typed sentiment. And because TypeScript is type-checking this thing, we know that this thing is strongly and static or strictly typed. And so that is very easy to get going. Altogether, about 22 lines of code. And there is a little bit of subtlety there because in our build, we actually have to do a little bit of extra stuff. So if you're using something like ts-node, this isn't a problem, but because we are actually looking for the TS input file, we're actually using the schema file types, we actually just copy them all over into the applet directory. And it's a little bit rudimentary, but it works pretty well. Right? And so if you don't want to do this, you can always use something like ts-node. And there are probably easy ways of getting this working with something like deno or bon or whatever, right?

So the sentiment example is pretty easy to get going. We also have another example of a coffee shop. So in our coffee shop, we have a slightly more sophisticated schema. I'll close the other files right here. And so this is similar to what I had in the presentation just a little bit ago. Just bear with me for a second. We have a cart, the cart has items, the items are each of these line items, and then you know et cetera, et cetera, et cetera. So we can order apple brand muffins, blueberry muffins, bagels, whatever. And so I can actually exit this prompt, go up, go into the coffee shop example, and then I can just run this thing. And I could say one small latte and a bagel please. And so we look at the output here, we actually get well formed JSON and for each of the items, we have one latte drink, it's a short, and a bagel, right? And the quantity is one, the quantity is one. And you know, depending on your model, you'll actually end up finding that this is this will also this often works consistently well, or it can have occasional issues. So, you know, let's just try this one. Give me one tall latte, and a banana. And so notice that actually fail here. It actually ended up saying, I didn't understand the following, and then listed that thing out.

And that's because of how we formed our schema, right? The interesting thing is that when you create these schemas, it's actually good to give them a degree of uncertainty in a lot of cases, right? In cases where a language model can't quite understand the user intent, it's good to have an out. And so the out here, the degree of freedom, whatever you want to call it that we've added, is we have this thing called unknown text. And so whenever the language model doesn't understand a concept, it tends to put things in this unknown text bucket, right?

And so what will happen is as we make our orders, this thing works basically the same way. But what we say is, whenever we encounter an unknown type, then we say, I couldn't understand the following. And then we just run through the whole thing and explain and we elaborate, right? So I can say I want one tall latte and a banana and a gorilla holding the banana. And it's able to translate each of those into an unknown, which is a banana and a gorilla holding the banana, right? So that works pretty well. Generally speaking, if you feel like a user is going to get something wrong, it's actually pretty good to add these degrees of freedom. So saying something like I permit unknown text, or maybe a Boolean that said the order was not clear or something like that can help with your application and you can be a little bit more robust, maybe flag specific things down. And one of the key things about something like type chat is because you get it all in JSON, you can always preview exactly what a user was going to get when they made one of these requests. So that's like maybe the easiest part of type chat to kind of grok.

One of the things that we also realized was that people are often trying to do some form of planning or scripting or programs with AI, right? Maybe you have an application, you have many, many, many different tasks and they can all be run subsequently in sort of this order. And you can use the basics of type chat, like the type chat approach that I just showed you, to create commands, right? So you can create objects with a command name and maybe some arguments or properties that follow that could be used as arguments in some way.

And it turns out many, many programs like VS code and others use a form of performing actions that's similar to this, right? There's a command name and then like some properties that follow. And you can also sequence them too. But one of the problems with this is that, well, you only get a linear sequence with this, right? And you don't have a way of referring back to an earlier argument in some cases. And so what you wanna be able to do is also validate one that you're getting the right flow of data, right? That each of these steps actually has the right type of data that it produces that can be used in the next step. But also, you wanna be able to refer to each of the individual steps prior because maybe you need to have some intermediate results before you run an action or something like that.

And so, one question that you might ask is, hey, what if you just generated code and you just took a very similar approach to type chat that you had before, right? All you need is types. And so maybe you say, here's my API, generate a program like this. That works well sometimes, but there are a couple of issues, right? First is, we do want type chat to be cross-language over time, right? We do want to be able to bring this to other languages like Python or C Sharp or others. But when you say I'm going to create explicit code, you need to be able to also say I want to sandbox this as well. Or maybe I want to be able to have something that, like, interprets the real code in some way. But then you end up creating a very rich interpreter that really has to interpret a lot of JavaScript as well or whatever language that you create. And then you end up with a lot of other issues like, okay, well, I have APIs that are async versus sync. So now the AI needs to generate code that knows about that as well. And it turns out when you have the real language, it tries to draw outside the lines a lot. So, if you have JavaScript, it's going to try to take full advantage of the JavaScript that's available to it. That's not always ideal.

So, what we actually kind of found was, you can throw the AI off enough by just creating a pseudo-language in a sense. And you say, actually, just respond in JSON. And the JSON should look like this. And it can piece that together really well enough and say, hey, you know, there are these meta properties that each of these properties can refer back or specify that I have a function or whatever. And so, this is basically a fake language or representation of language that's just kind of serialized to JSON. And it works well enough. But where do you get the validation from? How do you know that the inputs from writeFile are actually of the correct type? The trim text creates a string and that string can be used as the output, the input to the writeFile call. Well, we can then take the fake representation, the fake program, take that JSON, turn it into a real program, and then use TypeScript to create a program that actually does need to be type checked. And so, when we do that type checking, then we can validate whether or not each of the inputs and outputs corresponds accordingly. And so, now you do have a well-typed program between each step and you know that the indices correspond correctly. And so, that's another part of type chat is that we have this concept of type chat programs as well. So, I can really quickly show that as well because I'm very short on time at this point.

But I'm going to go into the math example. That's not where it is. And then I'm going to run the program as well. And my terminal doesn't work so well with this. But let's look at the math schema. The math schema is really just a set of methods for add, subtract, multiply, divide, negate. And then some identity stuff and then like I don't know how to handle this right. That degree of freedom that I mentioned before.

And in this program we're actually not using a JSON translator. We're using a program translator. And the key idea here is that it always expects a specific type called API. So I have to export a type called API here. And so what I can do is I can actually just write mathematical expressions and I can say something like add one and 41, right? And so what that does is it actually creates that JSON program before. This is what we construct to actually validate the program. We actually create this thing. And then we have an interpreter that actually runs it. So when you implement this thing, you effectively create an interpreter in the form of like create or handle call. Right? And so you have this evaluate JSON program function and you provided a callback and then it just takes a function name and the arguments that get provided to it. And there's a lot of metaprogramming stuff that you can do to make sure that this is all well-typed.

So at the end, um, jumping back to our math example, I can do, search sort of sophisticated things. Like I can say, um, add one and 41 and then double it after, and I have like a typo there. Um, and in some cases we'll inline this, but in other cases, maybe because of how I've said it, it will create a step and the step can then be referenced in one of the other steps. So it says add, and then it says multiply. And it turns out that the language models that have been trained on both code and pros do pretty well here. Um, so that was our demo. Um, unfortunately I'm short on time, but what we really do encourage people to do is check us out on GitHub. Um, we have discussions, we have issues, we're really looking to hear from you and you can follow us elsewhere on Twitter or Blue Sky or other places as well. Um, so thanks very much for your time and that's a wrap take care.