How to Secure Your Node.js Containers on Kubernetes With Best Practices

Hello everyone. Welcome to my talk. Today I'm going to talk about securing your Kubernetes containers, especially for Node.js. If you're a DevOps engineer, there's a good chance that you're maintaining either an on-prem Kubernetes cluster or a PaaS like EKS, AKS or GKE. But regardless of how you run your Kubernetes clusters, you need to make sure that they are secure.

But first, introductions. My name is Deepu K. Sashidharan. I'm the co-lead of jHipster. I also created a nifty dashboard called kdash for Kubernetes. I'm an open-source aficionado, a Polyglot developer and a Java champion. I work as a developer advocate at Okta with a focus on DevOps. I also write frequently about languages and tech on my blog. You can find it on deepu.tech. Please do follow me on Twitter if you are interested in my content. I have written a book about jHipster. If you like this talk, you might like the book as well. So please do check it out.

Before we talk about securing Kubernetes or before we talk about security best practices in Kubernetes, it is important for us to have a basic understanding of Kubernetes security. Like any other complex piece of software, security in Kubernetes is multifold. It can be broadly categorized into four layers. The transport security, authentication, authorization, and admission control.

TLS is used to ensure transport security and authentication and authorization can be done using multiple mechanisms in Kubernetes. There is also a possibility of adding custom admission control modules to add further policies and security in Kubernetes. So these are the things that are available out of the box in Kubernetes.

Kubernetes comes with many security options out of the box, as we saw. But to bulletproof your infrastructure, you need to consider many more security best practices. Today, we'll look into some of the vital security best practices. You can also find a similar blog for me in the link provided in this slide. So please do check that out if you want to read a bit more info about these.

So, first, the first best practice would be using RBAC. Kubernetes supports multiple authorization models for its API server. There is attribution-based access control, or ABAC, role-based access control, or RBAC, node authorization, and the webhook mode. Out of all these, RBAC is the most secure and most widely used, and is ideal for enterprises and medium to large organizations.

With RBAC you can define role-based access control that closely resembles your organization's business roles. RBAC also works great with OIDC authentication. So, with RBAC, you can have much more flexible control of who has access to what, and you can model it based on how your organization is structured, based on your departments and all these things. Most Kubernetes distributions have RBAC enabled by default. You can check this by running the command kubectl clusterinfo down, then look for the authorization mode flag in the output, which will show something like authorization mode equal to RBAC. If not, you can enable it using the authorization mode flag for the API server. It would be hyphen-authorization mode. You can do this when creating a cluster or you can do this by patching a cluster. For example, if you set hyphen-authorization-mode equal to RBAC, command-node, it will enable both RBAC and node authorization on the cluster. You can have multiple authorization also on the same cluster. Once RBAC is enabled, you can create cluster roles and roll bindings and cluster roll bindings to control access to your resources. Here is a small example of a role and role binding that lets users view ports and services. I would recommend you to check out Kubernetes documentation on RBAC and cluster role and role bindings for further learning, as you will see a lot more explanations, a lot more examples on how you can use this flexibly to manage complex authorization requirements.

The next best practice would be to use OpenID Connect or OIDC to secure your cluster. This is for the authentication part. RBAC was for the authorization part and OIDC would be for the authentication part. Kubernetes supports multiple authentication mechanisms. Some of the most common are client certificates, basic authentication tokens which includes service account tokens, bearer tokens, and so on. OpenID Connect or OIDC and also proxy-based authentication. These are all the authentication mechanisms that is supported by Kubernetes out of the box.

Out of all these authentication mechanisms, OIDC is the most secure and scalable solution. It is ideal for clusters accessed by large teams as it provides a single sign-on solution for all the users and makes it easy to onboard and off-board users because once you secure your cluster using OIDC, you wouldn't have to go into the Kubernetes clusters to do any user management. You can do everything from your OpenID Connect provider. You can do user onboarding, off-boarding, everything from there. You don't have to make sure that you don't have to worry about having to clear usernames, passwords or tokens and certificates from user machines and so on. It'll all be taken care of by the OIDC provider, the OIDC mechanism. So it is also way more secure than other mechanisms as you don't have to store any sensitive information on a user's computer like client secrets or passwords with the traditional mechanisms that you would have to do. You can also use features like multi-factor authentication, tokens, Yubi keys, etc., as supported by your OIDC provider, of course, but all these mechanisms would be available to access your Kubernetes cluster as well. So that means you can add additional layers of security which is not possible with any other authentication mechanism supported by Kubernetes. You can also check this blog post I wrote about securing your clusters with OIDC for more details and to see how you can actually do this for your cluster. The blog shows setting this up using Okta, but the steps are similar for any OIDC provider. So you can easily switch out from Okta to Key Clock or whatever OIDC provider you prefer. So do check this blog out to see how you can actually do this.

Here is how OIDC works with Kubernetes. When you run a command with kubectl, it uses kubelogin and opens a browser to authenticate you with the OIDC provider. It then uses the auth response obtained from the OIDC provider to fetch tokens from the auth server and passes the tokens to the Kubernetes API server, which will be used to authenticate the user.

The next best practice would be using secrets. Of course, this one should be a no-brainer. Kubernetes has a secret resource that can be used to store sensitive data. This is a great way to store passwords, keys and other sensitive data. Secrets can be used for storing string data, Docker config, certificates, tokens, files, and so on. Secrets can be mounted as data volumes or exposed as environment variables to be used in containers. Secrets can be plain-text or encoded.

So please don't be that person who uses plain-text for sensitive secrets. Secrets are flexible and native to Kubernetes. So there is no reason for you to not use them. Also, make sure to implement proper RBAC for secrets as well, so that everyone in your organization does not have access to secrets. Because I don't think you would have to expose secrets to everyone who has access to the cluster.

So next would be keeping Kubernetes up to date. Like any other software, Kubernetes also has bugs and issues and CVEs. And from time to time, there might be a high-severity bug that calls for a CVE. Like, for example, memory safety-related issues. Hence, it is an excellent idea to keep the Kubernetes version up to date for the server and the CLI client. You can check the Kubernetes security and disclosure information website to see if there are known security bugs for your Kubernetes version. If you're using a managed pass, it should be pretty easy to upgrade. And for on-prem installations, there are tools like Kops, Kubadmin, and so on. These tools, it makes it easy to upgrade your on-prem clusters.

The next would be restricting admin access. So Kubelet is the primary node agent running on each node. And by default, Kubelet's HTTP endpoints are not secure. This could allow unintended access and hence should be restricted. And furthermore, when someone has access to a Kubernetes cluster, they can access Kubernetes API server and also can SSH into the cluster nodes themselves. This is also not very safe. To limit node access, cluster access should be limited as much as possible. So disable SSH access for non-admin users. Secure your API server using OIDC and RBAC, as we saw earlier, so that only authenticated users with sufficient roles can access the API.

So next would be controlling traffic between pods and clusters. So generally, pods within the same cluster will be able to communicate with each other. And if you have multiple clusters in the same network, there may be traffic between these clusters as well. So do not leave all this open, as it would be increasing the attack surface. And it could lead to a compromised cluster when another in the network is affected. So use Kubernetes network policies to control traffic between pods and clusters and allow only the least necessary traffic, so that even if one cluster or a container is compromised, you don't end up compromising the entire cluster or your network. Next would be to, this is also a bit similar, so next would be to isolate workloads by namespace.

So do not run all your workloads in a single namespace. Isolating workloads in different namespaces based on business needs is more secure and it is also easier to manage with RBAC. So this way you can fine tune RBAC even further to let users access only what they need to see. You can also use Kubernetes network policies to isolate traffic between namespaces if applicable and if required. So isolating your workloads with different namespaces is always a good idea.

As we're securing APIs and the cluster itself, it is also essential to set resource limits on how much CPU, memory and persistent disk space is used by the namespaces and resources. This secures your cluster from denial of service attacks. When a particular container uses up all the resources, it could end up bubbling up and it could end up compromising your entire cluster. So in order to avoid that, use resource quotas and limit ranges to set limits at the namespace level. And you can also use requests and limits at the container level to set some limits as well.

Finally, it is also extremely important to monitor and audit your clusters. So enable audit logging for the cluster and use monitoring tools to keep an eye on the network traffic to, from and within the cluster. So monitoring can be done using open source tools like Prometheus, Grafana or with proprietary tools. So it is very important that you monitor this traffic so that you can prevent breaches before they happen. You can set up alarms and stuff if the tool supports in order to proactively prevent any security breaches. Further more, keep these infrastructure best practices also in mind when securing a Kubernetes cluster. Ensure that all communication is done via TLS. Protect etcd with TLS firewall and encryption and restrict access to it using strong credentials. Setup IAM policies, IAM access policies in a supported environment like, you know, AWS, GKE or Azure. Secure the Kubernetes control plane, don't let it open. Rotate your infrastructure credentials frequently. And if you're using a cloud or PaaS, restrict the cloud-metadata API access because PaaS like AWS, Azure, GCP provide these metadata APIs which could have some information which you don't want to be public. So securing the containers are as important as securing the cluster. So far, we saw what can be done at the cluster level to secure your Kubernetes settings, but it is also very important that we follow best practices at the container levels. So, let's see what they are. So, do not run your containers as root user, as this would give the container unlimited access to the host. In case of a compromised container, this would give the attacker root access and will grant wider attack surface. Always run the containers using a non-root user to limit access and use the least privileged user as much as possible. I mean, use these privileges as much as possible. For Node.js containers specifically, the official base images have a least privileged user called node.

So, we can use that instead of root to be more secure. Next would be to use minimal and up-to-date official base images. So, remove all unwanted dependencies, packages, and debugging tools from the image as it will make it more secure, it will reduce the attack surface and it will also make the image lightweight. And make sure you use official verified images for popular software and prefer LTS versions when possible as they would have longer security patches and updates. And finally, use a trusted registry for non-official images or for non-official images, maybe even consider building the images from source on your own and hosting it on your private registries. And if you're using third-party registries, always verify the image publisher and make sure it is from a reputed publisher. Don't go about trusting every registry or every publisher you come across.

Next would be to prevent loading unwanted kernel modules in containers, especially during development. It might be might end up, you know, using more from the underlying Linux kernels for debugging purpose or testing purposes. But for production, make sure you do not load any unwanted kernel modules. So these can be restricted using rules in ETCModProb or by ETCModProb. of the node or by uninstalling the unwanted modules from the node itself. So this reduces the attack surfaces and also it reduces the resource usage of that particular container.

Next would be to enable container image scanning in your CACD phase. So this would help you to catch known vulnerabilities before they become an attack vector. You can use open source tools like Clio or Anchor for this or commercial computers such as Deluxe or Acid. Or commercial tools like Snyk for this. These are quite easy to set up and they support most CACD solutions out of the box. Another best practice for containers would be to use Docker Bench for security to audit your custom images, custom container images for security best practices. So if you are building container images for your application, it's a good idea to run it through Docker Bench for security to ensure that you're following all security best practices and also to discover additional security best practices that you can do for your particular container. It's a great tool to ensure that custom images you build for production follow all the best practices possible.

Another best practice for containers would be to use Pod Security Admission. So this is the successor to Pod Security Policies. So from Kubernetes version 1.21 onwards, the Pod Security Policies are duplicated and the successor is Pod Security Admission, so you can use this module to limit the container's access to the host. This will help to reduce the attack surface and it will also help to prevent privilege escalation in case of a compromised container.

And since we are talking about securing Kubernetes clusters on a Javascript ecosystem, there are a few things which is specific to Node.js that you have to keep in mind when securing your Node.js containers running on Kubernetes. So only install production dependencies. So in your images, when you are installing a production dependency, in your images, when you are installing dependencies, make sure you pass the hyphen if and only production flag so that you don't end up installing your dev dependencies or other non-production dependencies as this would reduce the attack surface. Optimize for production by setting Node into production. As Node.js applications do have some optimizations when they are run in the production mode and they are much more secure. Safely terminate applications using an init system like dump init. Mostly, we end up just running Node server.js or Node file name.js, but that is not ideal. As in case of application crashes, things are not cleaned up, application is not terminated properly, so use a system like dump init so that all the signals from your host is passed on properly to the application itself. And finally, use a Docker Ignore file to ignore sensitive files like .n, .npmrc and so on, because these files will have sensitive information like passwords or tokens, and we have no reason to expose that to the container, which could also end up in public domains if they are public containers.

So that's it, folks. I hope the talk was worth your time and thank you for attending. You can reach out to me via Twitter and do check out my website for more content. Thank you.

So the question was, what kind of authentication do you use for your Kubernetes clusters? Well, we have a big winner, and I think this is not a surprise with 63% tokens. Yeah, definitely. Were you expecting such a landslide? Yeah, this is something I was expecting. Because that's the default that most people would do. I'm quite surprised by the number that OIDC has. Do we know how many holds are there overall? I can't see that, but, no, we just see percentages, yeah. Okay. Because I wasn't expecting OIDC. No, it's good. It would be the most secure way, of course. So it's good. It's growing. But, yeah, it's... Sorry? It's growing, yeah. From 16 to 19. Yeah. So, yeah, definitely not a surprise that people use tokens as the, you know, default authentication mechanism, because in most of the set up, it's also the default, so it makes sense. So I don't think a lot of people go about changing these unless they have specific requirements and depending on their team and stuff, but, yeah, that's interesting, definitely.

Oh, it's growing. Growing again. Oh, super fast. And even 0% for basic auth. So I'm assuming that would just be username and password. Yeah, yeah, definitely. That's a good trend, definitely. But, yeah. I mean, tokens is not that bad. But, yeah, I mean, for larger organizations, I mean, for smaller teams, I think tokens are perfectly fine. You don't need OADC for everyone. But if you are in a larger team where there is a lot of churn, and especially if you're trying to do R-back, which has to kind of reflect your organization's composition, and whatever, then I think OADC is the best choice.

Otherwise, it's too much of a hassle to manage tokens all time and stuff like that. Yeah. I think we should just stay talking for like five to 10 minutes, and then OADC will be the winner, as it's growing and growing. Probably. Yeah. Right there.

Let's jump into the real audience questions. The first one, what do you say that OADC is the most secure and flexible authentication mechanism for Kubernetes? How does it make onboarding easier? Yeah, that's something that I was mentioning. I think the nicest aspect of OADC, of course, the security and all that apart is the flexibility, because once you have configured your cluster for OADC authentication, and once you have established that, then adding new users and removing them becomes so easy, because you're not going to save anything on a user's computer, because with the certificate-based authentication, you still need to have those tokens and those certificates in your laptop. You have to do that authentication, you have to keep it. And if you want to remove someone from the cluster access, you would have to make sure that they have removed it, or you'd have to change your credentials and stuff. So, that becomes very annoying. I used to work in a company where we were doing a lot of DevOps, we were not a huge team. We were like 6-7 people, but still it was annoying whenever there is a new member to the team or when someone is leaving, you have to ensure that all these are cleared up and stuff like that. So, OADC makes that extremely flexible because you don't have to touch your Kubernetes setup at all. You can do everything from your OADC user management, you can add new users, you can remove them. It just translates to your access. So, if someone is removed and then they are not going to have access anymore, because you can do it from your authentication provider or your OADC app itself. So, that makes it extremely flexible, especially for larger organizations, where there is going to be team changes or if you are going to have churn. And also it makes RBAC easier. Because with OADC and RBAC together becomes so flexible that you can manage access and authorization especially in a very granular level. You can reflect that based on how your company might be set up. So, it makes it very secure and extremely flexible. Yeah, all right. Thank you.

Question from CC Miller. Oh, wait, sorry, that was not a question. That was a note on the poll. But he hasn't asked a question. If you're working with someone who says, why bother, we have security on the app.

What's your answer to that? Sorry, can you repeat the question again? If you are working with someone who says, why bother, we have security on the app, then what's your answer to that? That's an entirely different thing. Security on the app is authentication or whatever for your applications, but if your cluster is not secure, if anyone can access your cluster, then you are in a very bad situation because it doesn't matter if your app has security or not. If I can access your cluster, if I can SSH into your cluster, I can do anything. It doesn't matter if your app is secured or not. You have to secure your cluster first. A Kubernetes cluster is equivalent of your infrastructure. You wouldn't have your infrastructure unsecured because if your app is secure. I would say securing your Kubernetes cluster is more important than the security of your app because it is the key to the kingdom. You can access everything. You can access the database from there depending on your setup. You can do so much if you have access to your Kubernetes cluster. I hope for CC Miller that you don't actually have someone on the team that doesn't care about security. Yes. I would be very happy if someone says that. Let's pray that it's not a real-life scenario.

The next question is from Ravi. What about the security mechanisms for cloud-based Kubernetes clusters? For cloud-based, for example, if you're talking about AWS or AKS or GKE, basically you can do OADC with any of them. They all even make it easier to set up OADC providers. Some of them provide their own. For example, with Google, you can even use Google OADC or you can use a third-party vendor like Octa or even Key Clause or whatever. Whatever you prefer. They provide mechanism to integrate and make it easier. Some of them also provide additional ways of authentication because Kubernetes also lets you, that's where the other thing comes in. You can actually add other ways of authentication like for GCP, for example in GCP, they add their own authentication mechanism when you set up your clusters. By default, it's kind of a token mechanism, but they make it kind of very... I won't say it is transparent but easy to set up kind of thing. You don't have to manually set all these up. You can log in via your Google console, but it translates into kind of a token authentication, but it's a custom authentication. If you look at the way it works, it's kind of custom. It's not the default token authentication, but a custom authentication, they have built. With all these providers, you can also do OADC. I would still say that OADC is the best authentication mechanism with even cloud providers, if you want flexibility, but if you are a small team, one or two person, then probably it doesn't matter. You can go with the default authentication mechanism provided by the cloud provider. Please don't go with basic auth. Anything other than basic auth is still fine, the smaller team. All these mechanisms should still work the same in cloud providers.

What would you say is a turning point in team size, or does it depend on how many changes you have in your team? Yes, it really depends. If you have a stable team of 10 people who are never going to change for a while, then probably not a big deal, But if there are going to be constant changes, then I think it's a good idea also depends on the size of your company. If you're in enterprises, then it really makes sense to go this way, because it really fits the way enterprises operate. But if you're in a startup or small teams, then probably not that much of an issue because you can still manage churn manually, because I hope it doesn't happen like every day or something. But I was in a small team where it was a bit annoying for us, but not a deal breaker, but still it would be annoying. So it really depends. It always depends. I guess it depends on how much time you spend doing stuff manual versus setting it up, right? Yeah, exactly. Because it's like click and install, and it's gone.

Next question, can RBAC be used along with other authentication modes? Yeah, definitely. RBAC can be used with other modes. For example, it's very common to see RBAC being used with node authentication. So ABAC is not exactly duplicated, but it's kind of being not phased out, but the Kubernetes team prefers RBAC, and they are not putting in any resources to further develop ABAC. So it's kind of in a pseudo duplicated mode, I would say. But the other mechanisms like node authentication or webhook, they can be used with RBAC, and sometimes combinations can make your authorization even more flexible or more secure depending on your needs. But yeah, they can be used together.

All right. Thank you. So CC Miller commented on his previous question. I did a long time ago, and unsurprisingly they are not a member of the team anymore. So that's probably on our comment that we hope you don't have a team member that doesn't care about security. Yeah, well, good for you CC Miller. Happy to hear that. Yeah, I can imagine that it is a struggle if you have such different philosophies on security to work with someone on a team. Also an interesting story to see how that worked out or, well, maybe it didn't work out. But yeah, that's also part of working together, right? Dealing with people that have different philosophies. Which is interesting, let's be honest.

So there's no question at the moment, but we have Jessica typing something. Not anymore. She made up her mind. She didn't want to ask a question. Okay. But that is a nice bridge to the next part because Dipu is going to be going to his speaker room on Spatial Chat now. So if you want to... Oh, Jessica's question came in. Are there any setbacks to going for an open source implementation of Sansibar? Sorry? Of what? Of Sansibar. What? What's Sansibar? I don't know. I know it's like an island, I think, but I don't think she's talking about an island. I don't really understand the question.

Well, Jessica, then I want to invite you to answer that so that we can still go over this question. She is typing. So let's give her a minute to type. Google Sansibar. Oh yeah. And she wants to discuss it on the space chat. So let's do that because we don't have a lot of time left.

So, Deepu, thanks a lot for joining us. It's been great fun having you here. Yeah, thank you for having me. And thanks to everyone who joined, and thanks for all the questions. I hope the talk was useful and you could take something from this.