Meta Framework for Kubernetes: TypeScript Meets Cluster Control

Everyone, it is a huge pleasure to be here. And as TypeScript enthusiasts and JavaScript writers, I want to show you all something new that's going to change the game in the cloud native world, in the hybrid cloud world. But first of all, can I have a show of hands on who's in Kubernetes right now? Who deploys in Kubernetes? Amazing. Everyone, right? A lot of people, right? Okay, that's great. Okay, cool. So here we go.

I'm Case. I have two apps in production. That's kind of a joke, but one of them is the Find My Car app, because I live in a tall building and I don't have intranet. And I barely ever drive because I live in Miami. So I just never remember it. And I go to Find My Car and I'm like, which floor was it? There's like 13 floors. I love infrastructure. I love Kubernetes. I love the hybrid cloud. I build like Kubernetes operators and controllers. I usually speak to one person and one person only. And that's the Kube API server. So if you guys get that one. Go heat and go fence. Hasn't been a good year for the fence. But anyway, a little funny application. I scribbled out my car though. But we can see I was parked on 535 and I had not driven for 41 hours. So I really don't drive. Okay.

All right. So Kubernetes Fluent Client. So I can't really get into Kubernetes Fluent Client without talking about Pepper. Because Kubernetes Fluent Client is the engine behind Pepper.

Long story short, though, it's completely open source. It's battle tested. It's used at scale across many, many clusters. And it is a TypeScript framework. Not Go, but TypeScript designed for simplicity. I should have mentioned like I love Go. And I never really wrote that much TypeScript until like this job, you know, about like three years ago, I started writing a lot of TypeScript.

But first is Pepper. Pepper is a Kubernetes admission controller in an operator builder framework. If you all are familiar in the industry, it's most similar to operator SDK or Kube Builder. It's somewhat similar to like a Kybernaut or an OPA. But I'm just going to give you a little metaphor for what is Kubernetes admission so I can actually jump into the KFC as we call it.

Imagine you are in a club and there is everyone that wants to go into this club. It's a heavily sought after club. And there's basically two types of bouncers. There is a mutating bouncer. So what does the mutating bouncer do? When someone's going into the club and they see that person is, you know, let's say the club requires you to wear a jacket. They will give that person a jacket. So a mutating bouncer can actually change the clothes or give people things, change things, right? Change Kubernetes objects. In Kubernetes admission, there can be as many mutating webhooks as you need. So one mutating bouncer is giving you a jacket. The other mutating bouncer is checking if you have the correct kind of footwear.

After all the mutating webhooks comes the validating webhooks. So what the validating webhooks job is, or the validating bouncers, is to look at your attire and say, fine, does it meet the dress code criteria? One validating bouncer might say, let me check the shoes. Do the shoes work? And then there is another validating bouncer that might be looking for the jacket. So long story short, this is how you decide what can enter into the cluster. And the reason for this is more like, you know, having a global security posture in your cluster. Nothing gets into my cluster that's running as root or running with high permissions. Like I want to make sure that we are completely safe here.

But now what about if the club was opened earlier and there were people there before the bouncers got on work, before the bouncer started working. That's where the informer gets in. The informer is like, imagine like maybe like an informant or a spy that says like, I want to just make sure the club is running smoothly. I want to make sure that, you know, no one's fighting in the club or, you know, there's not shady business so that everyone can just make sure they're having a great time. So the informer is for watching resources that already exist in their cluster that are not going through admission because admission is basically like creates, updates, deletes. Okay. So and honestly, the informer is the KFC part of Pepper.

So what are like the use cases of even considering Kubernetes with TypeScript? Well, you all are pretty much Kubernetes experts from the show of hands I saw. And I didn't know what kind of people I would be chatting with. If you all would have like cloud native experience. But one, and this is going to be so obvious to you all now, but to have more control over your application in production, I want to make sure I'm not running out of resources. I want to basically when something comes in to my cluster, I want to actually replicate it into a side cluster, into another database. Advanced scheduling. We've all heard of AI, right? So the AI is like sometimes there can be like multiple agents, right? You can have your Slack agent that can like, you know, read your Slack messages. You can have your Gmail agent. But all of those take very given resources. And as we know, in Kubernetes, maybe not all nodes have sufficient resources in terms of memory or CPU, but also it's not common that that nodes have GPUs. So we have to schedule those workloads to fall in a place where they can actually run. Simplifying complexity around deployments, like this is like the main reason for an operator. And I'm going to go out on a limb and say like as we dive deeper and deeper into AI with the configurations becoming much more complex, Anna's wanting to like look at things like metrics and scrapes and all these around all of these components of the AI stack. Like I think we're going to have to abstract away that complexity with operators. And so that's basically like simplifying around deployments. Integration tests, right? I have this custom resource called web application.

So I deploy my web application. I want to do an integration test to like look and see if there's a deployment in my cluster. Look to see if I have a service in my cluster, a service account, role binding, whatever. But integration tests.

And at first it was because we were trying to do something very similar to Kiberna, another admission controller, but we wanted the full power of a complete, strongly typed programming language in admission. Before a Kubernetes resource gets into admission, I want to be able to do literally anything to it. There's a webhook timeout, usually about 10 seconds, but if you're strategic, you can do a lot. If worried about the webhook timeout, you can use the informer pattern with no timeout limitations.

In a cloud-native tech stack, we have tools like Prometheus for metrics, Keycloak for authorization, Istio for network encryption. But when one configuration changes, there's often a cascading effect. For example, implementing a network policy to restrict access between services. This situation leads to the need to monitor metrics due to increased traffic on services like websites and databases.

You do not have to be a Kubernetes client go, which is the go library expert to use it because there's a lot of boilerplate. There's a lot of extraction extraction. This literally looks like completely plain English, right? Like when a pod is created or kubectl, apply, a pod, a service, a stapleset. So it's just like plain English. And there's like something called Kubernetes JavaScript client, or Kubernetes JavaScript. I forgot which library that is. But we would have had to really like completely take control of that library and change a lot of the API surface, which I mean, there's a lot of people using it and we weren't going to try to do that. It was just simpler to build in this case.

And this is still talking a lot about the why. But there's a, in a cloud native tech stack, we have things like Prometheus for metrics. We have things like Keycloak for authorization and authentication. We have things like Istio for east to west traffic network encryption. But the thing is, when one piece of configuration changes, it's frequent that there's a cascading effect. So you have a website and a database and your website can hit your database. You put in a network policy because you want your database to be safe that says only the website service can talk to the database service. Everything else is a default deny all.

So only website can talk to database. Well, how about this? You are getting a lot of traffic on your website and a lot of traffic on your database, and you need to start looking at some metrics. So you need to see like maybe your database has crashed before. So now we start, we deploy Prometheus and we deploy these things called service monitors. So we can actually scrape the data from the website and scrape the data from Prometheus. Great. So now what we have to do, though, is we have to open up the network policy. So now Prometheus can hit these two. So now we have website can talk to database. Now Prometheus can talk to both. OK, great. Next, we have our bosses. They come in and they say, like, what are you doing? We're not compliant right now. So we have to encrypt all network traffic.

So we're like, oh, wow, that's great. So now we have to deploy Istio and we're having like a strict peer authentication policy. So all the network traffic is encrypted through mutual TLS. But now our scrape ports have changed for the service monitors for Prometheus. So now we got to make that change. And this is just three components. So this really cascades, and it just is what it is.

Look, if we need these things on the left, right, if we need metrics, distributed tracing, authN through authZ, Istio, the latest and greatest cool stuff out there, we do not want to have to configure things by hand or in a repo. But what about the next application? We want to be able to deploy a custom resource, an abstraction of an application to deploy apps in the exact same way. No human fatigue, no fat fingering. Operators know how to configure Istio, Falco, Loki, and more. This was our goal, and that's why we developed Pepper.

Delegating complex configuration tasks like authorization, authentication, routing, metrics scraping to dedicated tools boosts developer productivity. Applications should focus on their business logic without worrying about low-level networking tasks. Delegate concerns like destination rules, Kubernetes ingress, and network policies to specialized tools like Istio and Keycloak. Boost developer productivity by offloading these configuration tasks to tools specifically designed for them, enhancing the application's core functionality.

My operator, my controller knows how to configure Istio for whatever it is. It knows how to configure Falco. It knows how to do Loki and logs and all that stuff. So this was our goal. And that's why we developed Pepper.

If we think about why we do stuff like this, it's because we want to delegate complex configuration. Should your own application be handling authorization, authentication, routing, creating its own metrics, scraping its own metrics? It's going to be like the biggest application that's going to have so much business logic that has nothing to do with the application itself.

The common thought is that your application should stick to its business logic. It shouldn't be worried about destination rules, the ingress of Kubernetes, or network policy. Delegate that kind of stuff to tools like Istio, network policy, Key Cloak, designed specifically for it to boost developer productivity. This was all about democratizing Kubernetes operator development, keeping it lightweight and fast, and incorporating features like server-side apply for efficiency.

We want to delegate that kind of stuff to things like Istio, network policy, Key Cloak, because these tools are made specifically for it and they are great at it. So we want to basically boost developer productivity by delegating that. And then this was one hundred percent for developer productivity. So what is the desired outcome? One, a quick iteration from an idea like, hey, I think we can do this better and I have an idea on how to do it to a quick POC. We don't want you worried about having to look up syntax. We wanted to be so painfully obvious to production ready. So this is one hundred percent for developer productivity. And it is to democratize Kubernetes operator development for anyone, for anyone.

So it's in TypeScript, but now it's with a very fluent API. Second one is we need it to be very lightweight and very fast because we knew it was going to be run at scale. So a straightforward API, no need for heavy frameworks. And then we didn't want it to just be simple. Like we needed it to have things like server-side apply to reduce the network pressure. So server-side apply is basically you're just sending a small patch to the Kubernetes API server of the control plane. And it doesn't need the complete resource. So if you're sending thousands of resources all the time, you're going to reduce a lot of the network pressure.

We have our own custom Kubernetes informer, which basically watches things in the cluster and reacts to changes. It's like, this was added, this was updated, this part changed. So what's so good about the Kubernetes Fluent Client, other than it's like the engine that powers Pepper, it has the informer, uses these Kubernetes best practices. Well, this one's very interesting. So I really like LoveGo, like I said, and in Go, what you can do is you have these Kubernetes shared informers and these Kubernetes informers that are like part of the Kubernetes Client Go library. Anyway, it was kind of like a joke, but it's nice to be able to delegate things at times. Everyone in my company was like, no, we know we can do better here. Well, our TypeScript informer at one time was failing. It was basically, I'm not going to really go into the informer, but what happens is you do what's called a watch on a resource, and that creates a long live connection. And then the reason why an informer is like very robust is every time it gets an event back, it stores it in a local ephemeral cache, like in the application.

Everyone in my company was like, no, we know we can do better here. Well, our TypeScript informer at one time was failing. It was basically, I'm not going to really go into the informer, but what happens is you do what's called a watch on a resource, and that creates a long live connection. And then the reason why an informer is like very robust is every time it gets an event back, it stores it in a local ephemeral cache, like in the application.

And then every so often, you can say every 10 minutes it does a relist. And so that brings back the list of everything you're watching. And so when you get the list back, you compare the resources, resources version with the resource version in the cache to see if you just brought back something new. That would indicate there was a change that your informer missed. So what we would be doing in arrears is processing the payload or executing the payload.

Like, oh, my gosh, we were 30 seconds late. Long story short, we fixed our TypeScript informer. It was from the node fetch library. There was so many questions. There were so many problems with it. And first, we switched it out for HTTP2, which was more where we wanted to go. It was a little more secure and we could have the connections be long lived. But that had a C memory leak. Back in the folks on delivery and implementation were killing the pod.

And when you kill the pod, you restart the informer and it starts with a list and you get all the latest. So it works like that. So anyway, we had this memory leak. So they're like, hey, what the heck? It's like leaking memory. It's restarting. It's like, yeah, well, now you don't have to kill your pods. It's just going to die when it hits this memory limit.

And then we went to UDG. And UDG is like the best fetch library I've ever seen. It's so powerful. Literally, the moment we got UDG and we were doing nine-hour soak tests, deploying pods every five seconds and checking if they were there like five seconds later, because we programmed Pepper to say, watch a pod. And when a pod is created, delete it.

And then ten seconds later, it's going every five seconds. Ten seconds later, we were seeing if the pod was still around. And it was literally like a watch failure. UDG was like amazing. So amazing. But I was like, I wonder if Go is better. Like I wonder if the Kubernetes client Go is still better. So I did two experiments. First, I put the Go in a sidecar. The Kubernetes share informer.

And I was using DRP streaming. And I basically would program the DRPC stream with a unary call from Pepper and the same container and the other container in the pod. And then I basically just streamed back events. Worked amazing. Right. And then I did a second Go implementation with NATS, just because I really wanted to play with NATS a lot. But I basically would create a topic on a NAT server. NAT servers are external. But I still had my Go sidecar to my Pepper, and then Pepper would subscribe to a NATS topic. And then Go would actually send events through the channel and then Pepper would get them. So it was like a big triangle. So the DRPC streaming worked a little bit better than the NATS, but none of them worked like better than the TypeScript.

I was throwing like 12,000 events at the same time. The thing is, yes, maybe Go is a little bit more efficient. It can compile down. But at that scale, the limiting factor is the network. It's not the programming language. It's like 100% the network. So we realized our TypeScript Informer, just like the Go ones, missed zero events. It had very similar memory profiles. It still performed amazing. So we stayed with TypeScript.

Now I got about a minute left. So I just want to show you all the API surface. I told you all it reads like real English. So look, here's CRUD operations. K8s, kind, namespace, apply, right? So I'm applying something. K8s, kind, dot, namespace, get. K8s, kind, dot, namespace, delete. Best practices, this is server-side apply.

So you have these field managers. You don't have to send over the whole spec. The Kube API server does that. You have these subresources. So isn't that simple? Let me check the logs here. Patching statuses, it's like a controller thing, but all of CRs have a status. And if something goes wrong, you want to at least have a record of that.

Oh, hitting a proxy to a given port on a pod. It's okay, it's a proxy. You can see why this would be super interesting for test environments. Because of this, let me just blindly make sure, just like fail the integration test. And of course, good old finalizers, right? So long story short, a finalizer is like when you say, I don't want you to be able to delete this thing. I don't want anyone to be able to delete this thing because this is critical for my infrastructure.

So you add a finalizer to it. So anything with a finalizer can't just be like fat finger deleted. It protects things. It's like k8s, kind.pod, in namespace, finalize. Remove this finalizer we have. And then evict. Evict respects scheduling way more than delete. We were a bunch of Kubernetes folks, Kubernetes maintainers, Kubernetes contributors, and we had this really smart guy named Mache. He's like, we need to be doing evict instead of delete. So we added it to the KFC.

What I would love it is if you all would star the Kubernetes fluent clients and Pepper on GitHub. That's github.com forward slash defenseunicorns, forward slash kubernetes-fluent-client, and forward slash pepper, P-E-P-R. We are on Kubernetes slack.

And that is all the time I have. We have a few minutes for questions, so don't forget that you can put those in at slido.com. The session we are all in is 1117, and we are on the residence track over in this room, as opposed to the community track in the other room.

So while folks ask their questions, thank you again for such a great talk. I have a probably not technically interesting question, related to the mysterious combination of spices that goes into KFC. Interesting question. Our CTO, he drinks so much Dr. Pepper. Also we call him Pepper Bear sometimes. What's up, Jeff, if you are watching. He drinks a lot of Dr. Pepper.

He loves it. Maybe there is a sponsorship in the works someday. Or maybe you just have to call him Colonel Pepper from now on. We have aspirations to donate Pepper to the CNCF. We're kind of looking into it right now. Dr. Pepper, if you are out there and you want to pump some money into the project, tell us. Awesome, awesome, awesome. I love it.

So you are developing all this in the open with the community, it sounds like. That's fabulous. It's always great to hear about interesting work like this being put back into the community instead of being kept. There's a lot of brilliant solutions people have come up with, but they stay siloed in the organization. So it's very cool to see that.

So it's very cool to see that. We consume so much open source Prometheus, Keyclo, Kubernetes, obviously, SDO. And so we really like one of our huge goals as a company is to give back as much as we can. We have 10 projects out there right now. Great, great.

For beginners who might not have a lot of experience with Kubernetes, with this kind of platform, what are good places for them to start? What's a good toehold that a beginner could get? I think a really... When I want to learn something, I want to have a very clear goal. And with Gen AI, it can really guide you along. But I would say I want to deploy my application. I want to deploy my application and then I want to write integration tests or end to end tests with my application to see how it's working.

Start with the practical. This is what we want it to do in the happy path, maybe. And then start building out the infrastructure I like this idea, too. It's almost like we all need to start thinking more about that fault tolerance, especially some of the other talks today have focused a lot on workflows with AI and all of this kind of non-knowable output of your code. So it's important to be able to handle anything that comes at you, I guess. In the same vein, what if you wanted to cut out the middle company? What if you want complete control? What if you wanted to have so many services that you basically just press enter and it would pull your application?

But first you need to have your baby, your application, so then you can play with your application, seeing if you can increase its resiliency, increase its performance. Maybe you want metrics from it and so you can start deploying things around your application. But have a small goal and a start and just keep going up and then you'll learn so much better. Nice. I like that idea of taking things one step at a time.

Start with the practical. This is what we want it to do in the happy path, maybe. And then start building out the infrastructure I like this idea, too. It's almost like we all need to start thinking more about that fault tolerance, especially some of the other talks today have focused a lot on workflows with AI and all of this kind of non-knowable output of your code. So it's important to be able to handle anything that comes at you, I guess.

In the same vein, what if you wanted to cut out the middle company? What if you want complete control? What if you wanted to have so many services that you basically just press enter and it would pull your application? Maybe I want to be in control of the resiliency of my application and how it performs. Yeah, fabulous. And so another question around this kind of best practices or getting started with this type of work. Is there anything that you suggest folks who are just starting out as beginners with this kind of thinking? Avoid or steer clear of like anti-patterns? Bet worst practices? I think chewing off more than you can swallow or analysis paralysis by over analysis is a really bad thing for beginners.

Is there anything that you suggest folks who are just starting out as beginners with this kind of thinking? Avoid or steer clear of like anti-patterns? Bet worst practices? I think chewing off more than you can swallow or analysis paralysis by over analysis is a really bad thing for beginners. Like, you know, for getting started with Pepper or Kubernetes, go to the Pepper readme, you know, just like start there or, you know, there's so much easy stuff that you can do to get started. And then you're going to naturally gravitate. And if you're interested, you'll want to learn more. I say like, just keep it simple. Like don't, don't try to build Rome in a day. Awesome. Yeah. And I love, I love that idea too of like, you know, starting with the docs and starting with the, if I don't know if there's tutorials and things like that, but I know, I know that I sometimes get too excited and I'm like, I can just jump right ahead to doing the big thing and always end up going back and just reading the intros because there might be some key concepts. Some morsels in there always. Exactly. Yeah.

And so speaking of this open source side of Pepper and KFC, are there any particular areas of KFC that you're looking for contributions on or coming goals for Pepper that maybe the open source community could contribute to? You know, there is, there are. We 100% fully invite and we love our contributors. We have had a few really good contributors. There's a lot of things. So I'd say, first of all, get in there, use it, you find a, find your use case for it, right? Whether that's integration testing or actually building a Kubernetes controller to enforce some kind of a behavior in your cluster, maybe security posture or whatnot, right? Or just to make deployments easy. You have a crazy AI application that has 10 or 20 agents and you know, like that can get hard. But if you can kind of abstract that into like one CR deploy AI app and everything kind of comes up, yeah. So, hey, anything we are, we welcome contributors. We, I'd say get in there, look at the issues, play with it, do feature requests. If you want to knock out a feature request yourself, welcome, just tag us in the repo and that's how, that's how it works. And if it, if it makes, it was like on the roadmap, right? Unless it was like rewrite KFC and Rust. Then you would have to redo the slide. That would be no fair. Great.

Great. Okay. And so probably our last question that we have time for is, how does an auto-scaler help mitigate cluster failures? Could you talk a little bit about that? An auto-scaler. So what an auto-scaler does is you can assign it some resource that says, when my application hits like a 30% memory capacity, I want to create another replica, right? Or another replica. So when there is multiple replicas, only one replica has to be available to serve the request from the Kubernetes API server. So it gives you more resilience and redundancy and the, you said, you said, you said horizontal or you say vertical. Well, I think just in general. Okay. Okay. Making that not a thing. Got it. Yeah. So yeah, a horizontal pod auto-scaler will pump out more pods, more replicas. And then, so you'll have more available to basically serve your request. If you're getting thrown, if you're getting hit by like tons of requests. There's also something called like in place vertical pod auto-scaling. And I believe the resource is called resource policy in Kubernetes, but just look up in place vertical auto-scaling. It's really interesting because it can vertically scale the CPU without re-rolling out your pod. So that's really interesting. A regular vertical pod auto-scaler will re-roll, trigger a re-roll. But this one's like really good. It's kind of new, but it's, it's really awesome. Interesting. All right. Yeah. And, and also a great reminder that the cloud is just other people's computers, right? Yeah. Great. Fabulous. Well, I think that we're, we're just a few seconds away from our allotted time for the day, but everyone, if we could please give another huge thank you to Case for coming here and we'll see you on GitHub, right? Yeah, absolutely. Thank you all. Okay. Thank you so much.