Fine-tuning DevOps for People over Perfection

Hi, my name is Julie. I'm here to talk to you today at DevOps JS about DevOps. And I want to do something a little bit different. I want to focus more on people as opposed to let's just say DevOps in theory and on paper.

So, before we get started, I have to just briefly show a disclaimer that I'm appearing here as myself and much of what I'm going to share with you today is my opinion based on my experience. So, it's also not a complete guide to DevOps for this talk and the time duration. I've decided to pick a couple of examples to kind of illustrate the points I want to make.

So, I've been building for the web for a very long time. A bit older than I look, and I've worked at every place from startups to actually full like corporate. I was self-employed for a long time, so I actually was freelancing at various companies and that was my initial exposure to DevOps.

Then I joined Alianz Germany, which is a multibillion-dollar insurance company, and we moved some projects to the cloud in less than a year. It was a crazy ride, but I learned so much not just about DevOps, the skills, but really at scale. I already knew a lot of those practices and especially around Git and automated deployments, but transferring those across the team is a lot harder than it sounds.

Today, I am an engineer at Microsoft, part of the FastTrack for Azure program, where I help customers onboard to Azure. I specialize in cloud architecture and DevOps automation. We don't just help them with best practice guidance. If they run into a big problem or a challenge, we also help unblock them. Some of the content in here is going to be from those customer scenarios as well as internal Microsoft, kind of like my story, my experience, which is why this is my opinion, not a kind of like here's how you should do it.

The reason is because DevOps is a journey. Every company is going to be a little bit different. They're starting in different places. You always have different people with different preferences and just kind of like how they work. So it doesn't matter if you bring in somebody like me who's been doing it for a decade, it depends on the team, the company processes, and you have to kind of make all of this work together.

This is a little bit different from some of the talks I give. Part of it is that this is year two in COVID. Doing a lot of these things that have to do with transformation and cultural transformation is really hard when everything is remote. Sometimes you've never met your team. I've never met my team. So some of the things I'm going to talk about in terms of best practices actually become much more challenging when you don't have that face time to have that kind of nuance and it's like, okay, is she serious, or is she just being her snarky self? And then some of those rules, especially with security, how can I bend some of them and why? So we're going to look at that.

What I want you to get out of today is kind of a lot of these things that are best practices, even if I'm telling you they are, but you don't have to do them. You don't have to do them today. You don't have to do them next week. You have to eventually get there and also you can get there without following some of those best practices. So it's not about tools. It's going to be about people. And people are going to be the difference between success both in delivering a product and as well actually growing a team, investing in a team that will still be with you in a year or 10. So without further ado, let's get started.

Okay. So the first thing, pull requests. Everybody loves pull requests. Right? It's best practice. But you can also kind of do it wrong. So let's say we have a pull request workflow. Everybody's working on a branch, okay? You might be used to, I'm just going to open the main branch again, merge my workflow and push, and it's going to be like, nope, the server might reject it. So you have to then open a pull request, and you make your commits, do all these things, and then you might have a convention or workflow set up that says, I'm going to put hashtag sign off. A robot should now merge this. It didn't work. Let me do that again. Let me is what I have to deal with at work. Sometimes you just give up, right? And you might not even close the pull request.

There are so many repos with dozens or hundreds of pull requests. It's a bit kind of like really frustrating because you want to help, you want to contribute, but it's not going anywhere, right? It's kind of stuck. Now, even if pull requests are going in and the builds are running, you can eventually get code merged in. They can be really slow, and that's also just as frustrating and as disappointing. So, this is what a pull request might look like. In documentation, it sounds really easy. Open, approve, close, and merge. So, you might have it actually tied to a pipeline and you've got to wait for a build agent. If you have lots of people and you don't have as many build agents, you didn't buy those as you actually need, you'll wait forever. So, then if you're like me, you grab your phone and you're like, let me look at Twitter, see what's going on. You go back and when finally ran, oh, it failed. But it's like half an hour, an hour later, let me go have lunch, let me come back. That's not necessarily super productive either, right? So, what you end up having is like this traffic jam of pull requests. And that's not necessarily helpful either.

So, I love this tweet by Keith and he's talking about pull requests, why you have them, and it's the best way to describe when you should not have them. And that is pull request is like airport security, right? When it was first introduced, it was really designed more for like the open source community and you want to welcome outside contribution, right? Large and small. And you want to introduce a way to collaborate on that.

It's not just about security, right? It's about discussing kind of a code change, oh, what do you mean by this, can you make that change? But if you're on the same team, do you need all those gates? Maybe you don't, right? So let's look at what actually, this workflow looks like and where people get stuck, right?

So this is actually a diagram I use at work, and it's in the Azure documentation. And if you look at that first part, what you'll see is that that pull request is a gate to that protected branch, the main branch or the production branch. And often I will also see people who lock that down entirely, even admins can't push directly to it. So you really are stuck in this pull request workflow. The problem here is that that is all done in code, right? And nobody's perfect. I'm not either. And sometimes those things break for whatever reason. And then what you end up having is actually something like this. So I blanked out everything that you can't see. And I took this screenshot this morning, and it made me really sad, right? 26 days ago, that was the last time somebody contributed to this repository. It's something internal that theoretically as engineers we should be using every day, right? There's stuff in there because Azure changes every day. There's bugs in there. People should go fix typos and broken links, et cetera. But what's happening is you'll see we have about 50 pull requests that are stuck. Like this repository is, in my opinion, kind of like it's run amok. It's just too many. And you also just see like the number of forks we have for team that's actually only a few hundred engineers worldwide. It sounds like a lot. But in the scheme of Microsoft or in the scale of Microsoft, it's not a lot of people.

What you really want to avoid, because this is the worst feeling ever, is that frustration. Because what happens is you stop. And when you stop, you're not delivering business value. And that's really harmful, right? That's when the tooling isn't helping you. In fact, it's hurting you. So this is a great tweet from Mateo Colia, who's on the Node.js core team. And I really loved it. And he was talking about, OK, he helped this company, and they were deploying every two weeks. It was an overnight deployment. Very long. And they're doing it every day now.

So how? How would they do that every day? And the key point is actually you remove those gates. And you trust your team, right? They get to deploy when they're ready. So that sounds really simple, right? But is it super simple? How does that look like? So I pulled out this really old photo from the Allianz, right? So what does it mean when you're ready to deploy? So you think, oh, it's when we have tests. Everything is green. Well, guess what? In real life, not everything is great. Not everything is always great. And one of the things that was very interesting was that sometimes, and I never saw this actually in startups so much, I saw it more actually at this giant corporation.

People said it was too risky to deploy during business hours for an internal line of business application. So let's still work eight hours but shift everything and let's do it when nobody else is using it. And that's okay for the beginning. Trust me, it's not a goal. I think we only did that once with pizza and I have to blank out the faces because, yeah, GDPR. So it's really up to the teams.

In the beginning, you don't need to deploy 10 times a day. It's okay to start with every two weeks. It's okay to start with every month if you're dealing with a legacy application that is, you know, making you a lot of money, it just takes that time.

So another example that's more complicated, especially when you're like, ooh, gatekeeping. Why do we have gatekeeping? We have it for security. So, you know, who doesn't want security? But it sounds too easy. Let's go through this example about how maybe it's too much, or maybe we don't react to it. So I was giving, actually, a webinar last week, and I'm demoing how to use security and whatnot. And nobody actually said, Julie, but look at that. There's a vulnerability. Why aren't you looking at it? I wish they asked me. But they didn't. So let's go through it. And I'll tell you why it's still there, and it's going to stay there for the time being. So I go into the Dependabot alerts. So this is all automated. Hey, Julie, we found these issues. You should go fix them. One of them is really high. So the bottom one, for globparent. Now if we open it up, Dependabot, which was bought by GitHub, is trying to be helpful. Here you go. It's just the version's less than 5.1.2. So upgrade it, and you should be good.

And it just gives you a suggestion. So let's try that. I'm going to stick that in my package JSON. I'm going to do NPM update. This is what I see. NPM warns, all these messages and whatnot. And at the very bottom, it says you have 15 vulnerabilities. I thought I had two before, now I have 15. And then it says run NPM audit for details. And you're like, hmm, okay. Well guess what? I've done this before, I'm going to run NPM audit fix immediately. So lots and lots of code. And if I scroll down to the bottom, I have to do that in that screen. I still have 15 vulnerabilities.

So, like, why? I was hoping it would be gone. So let's do NPM fix force. I don't know if that's a tag. But when you're a developer, that's what you do. You try again. And we still have 15 issues. You can't get past it.

So the hard part is actually now figuring out what should I do. There's no easy answer. You kind of have to just stop, zoom out, and think. Right? So we have all these tools. And they know about a certain fact at a certain time range. Like, oh, 5.1.2. But it doesn't mean anything. It doesn't help you fix it.

So after some Googling and Stack Overflowing, oh, there's this relatively new NPM feature. You can do an override. And you can say I don't care what version is required in my dependency tree. I want this one. So let me put in the latest ones for these. So you know, I'm going to update my node version control manager on my local machine, my Docker file, all these things. I actually have some tests. So I'm going to do a preflight check, which runs all of them, and do a git push. Everything is green. All good. But then the CI build says, nope, it's not going to work. And if you dig into the error, right, it's giving me actually an open SSL error. So I'm like, no, I don't want that. Like, that's even worse, like encryption. Let me just kind of ignore it.

And I'm actually go and revert that change. And notice that in the commit, I also put a link to the Stack Overflow thread a little bit, like, why is it there? So you'll see actually in all my pipelines, I tend to have continue on error. It's like, thank you for the alert, I'm going to keep going. There's also a Azure Defender for Cloud Container Scanner. And it'll go through the code, and it found the exact same thing. And so it's like, okay, let me thank you for the alert, I need to disable you as well.

Now, all my builds are running, right? But that shouldn't be the end of the story. It's just I need to deploy. So, you know, continue on error, I know about the error, let's figure out what we're going to do. The key to that is understanding, what is Glob Parent? What am I using it for, right? And what is the open SSL issue? And how is that going to be used? So, you know, I was like, hmm, I'm not sure about the open SSL. But I actually do know what the Glob Parent is going to do. And so let's first understand why, like, what is this kind of vulnerability? And so it is a denial of service so kind of vulnerability, which basically means your application isn't serving requests anymore because it's going to be bombarded. And so there's a network version, but there's also a, let's say, bad code version where if you give it the right, let's just say inputs, then it could kind of slow to crawl or even crash. Right? Is that going to happen? Like, in my example? So, I didn't show you what this project is. But the answer is no. So, I'm building this application. It's a proof of concept. And what I want to do is give a kind of an assessment. That's not a checklist. Right? So, if you select certain options, then maybe your security will go up, but at the cost of increased complexity. And so, all of the things here, the questions and the factors, all of that is done in markdown. So, it's kind of pulling this all together. It's a headless CMS. I don't need a database. And it's figuring out what goes where and is attached to what based on the file tree. So, that's what it's glopping. It's kind of matching all these things. How do I throw all this together? That is my file system. Right? There is no user input. This application also has a build process.

So, it's doing all that figuring out live, but you just run a build at some point when you're done and when you deploy it, and you're running in production mode or just serving it in static mode or whatever it's called. And you don't have that anymore. That gap isn't there.

So, in this kind of situation, what I kind of tell people is that tools are stupid. I'm just trying to get you to remember that the tool is not the holy grail. Just because it says there's a vulnerability, that doesn't mean, okay, stop everything, drop everything that you're doing and, like, figure that out. Because even if you figure that out, right, it might be, like in this case, it's choosing the lesser of two evils.

So, the OpenSSL issue, whatever it has to do with the encryption that is used for the cookie, where the results, or rather the inputs that you gave, the answers you picked, is saved, right, and it's all encrypted. But for me, I've decided that risk, even though there's no real user data in there, is worse, I think, than kind of, okay, something that could only happen to me in development. So, you won't know any of that until you go through that process. And this was just one particular example that I encountered this month, right, but for every kind of notification you get, you kind of have to go through this process. And it's really, really hard. And then when you suddenly get something like this, we get to that other point of frustration, right, things that are clogged down. Some organizations will measure like, oh, how many alerts are still open, how many vulnerabilities are still there. But it doesn't actually tell you if it is a security, how bad, or like, it says high or moderate, but like, is it really that bad? Like, lesser of two evils. That's what I kind of always say.

Okay. So, you know, we talked about pull requests, right? We talked about how you need to trust your team. We talked about as well that even with security, you also have to trust your team to kind of learn and grow.

And the last thing I want to touch upon today is actually craftsmanship. Because we make DevOps really hard. But I want people to actually want to do it. And the key to that, in my opinion, is loving what you're doing. And being proud of what you do.

So, this is what I see a lot of, right? Like everywhere. On GitHub open source as well, not just at work. You're probably using the GitHub UI if you're clicking something and it says update, read me, update, read me, update, read me. That's okay. It's a place to start. Right? Now, what's interesting is when customers come to me and one of the biggest challenges people face is actually versioning, especially if they're using microservices. And for whatever reason, you have to correlate things. In which case you don't have a microservice. You have a distributed monolith.

Anyway, when you see something like this, it looks super nice and, like, oh, my God, simple and super easy. And how do you do that? Well, the answer is kind of practice and discipline. So, this is a change log that I made using a tool called standard version and it relies on something called conventional commits. And that's what it is. It's just a naming convention. So, when you make a commit message, this is what it looks like. How it figures out if there's a bug fix or a feature, it's just a prefix. You know, throw some stuff between the parentheses and it will try to categorize things for you. And then as well, if you add the hashtag and a number, of course, GitHub links it to the issue. Now, if you're really, really paying close attention, you can tell that some of those I've also adjusted by hand. Because even I am not perfect. Even I will sometimes be in a rush and I'm super pedantic, which is why I say even I. I'll be in a rush and I'll just also submit something.

But when I actually make a release, I will look at the change log and edit it as necessary before committing it. Because at that point, I am publishing it to you, the consumer, and I want you to understand it. Any tiny mistakes I make in between, that's okay. It's just for me. But at the point where I say, okay, it's ready for the public, I want to make it look a little bit nicer.

The other thing too, is that sometimes we're kind of like, you're not doing everything perfectly. And in this case, we had to restructure a bunch of things. Take that extra step to make sure everybody gets credit. This I've seen happen a lot in corporate, where unfortunately, people actually count those commits. And you know that GitHub profile with all those little green squares? So, you need to make sure that not just for you, but also for your colleagues, right, that their GitHub users and usernames are also getting credit for that work. Otherwise, people don't see it. Now, it's also not just about that kind of corporate loophole. It's also about, let's say, appreciation of teams and being grateful for the help that they're giving you. So, this takes 30 seconds, but it goes a really long way for your team culture.

Another thing that I want to tell people to spend time on is actually doing documentation, right? So, I write this both for myself, as well as people who are going to use the software that I publish or samples, right? It's not meant for humans, sorry, it's not meant for your builds. It's meant entirely for humans. This has no purpose in code whatsoever except to help you understand something. And that's really, really important because we almost always have to go back and look at things in the future. And I don't remember what I did last week, let alone six months ago. And if we're working on multiple projects and microservices, et cetera, then, yeah, sometimes you take a long break from a project. So, it's really important to add that documentation.

Now, when I show demos at work, you look at something like this, and this is a GitHub action workflow. And I recently took some time to completely migrate an Azure pipeline workflow. It looks really nice, right? But how long does it take? In reality, years. There's so much experience that comes from just actually creating pipelines that works well for me, not necessarily for everybody, right? And then figuring out kind of, okay, there's a gap here. There's a gap here. This feature works that way. It takes a long time. The simpler and more elegant it looks, the more effort it takes to get there. And it's okay. You don't start there, right? What you want to do is actually invest in yourself, which means good habits. Good habits, little tiny steps that over time actually result in that big goal you're trying to reach. If you try to get to it from day one, what you might end up doing is actually hurting your own motivation and your team's motivation. And that's almost kind of the worst thing you can do, right? I think it's better to have more contributions and stuff you need to clean up than to have no contributions. And you might think, oh, no, no, they just have to update the pull requests or whatever, you know, just have to make the build go green. But there's a human element to that that might be missing in those types of rules that are just all or nothing, don't do that. Really, it's about people. It's about investing in people long term. So, things like documentation, et cetera, it's not a cost, it's not a to do, it's not a chore, it's an investment.

And with that, I am done. So, thank you for coming to this talk. I hope that it will get you thinking critically the next time you read some article about best practices, 10 things you must do. Do you really need to do it? And if you're interested in more types of topics like this, counter intuitive, what is it actually like in real life? You can follow me on Twitter or on YouTube. Dan, thank you very much.

Good to have you here. So, let's look at the results. So, the question was, what does it feel like to deploy to production? And with 57%, we have a winner at passport control and airport security followed with 27% waving at your coach as you step into the field or 17% showing an ID before entering a bar or club. Was this something you were expecting, Julie?

Actually, it surprised me. The airport controls that many people. I thought this audience, this conference, folks who use like JavaScript, I don't know, they would be more like kind of laid back. I think what surprises some people, because my job today helped like Azure customers is like even the Allianz didn't have passport control and airport security. So we have all, we had all those really super strict requirements, right? So my job there, I was a full stack engineer and then I later had like a mentoring role across many teams and we established all the rules, but it got to a point, for example, some of the compliance stuff, the product owner. So a non-technical person would create the pull requests that could deploy to production, for example, or the product owner would be the one to click a button in Jenkins that deploys. Like there weren't any sort of hard controls on everything at the time. It's just, when you have a business that's worth that much money, it's scary anyway. Like, so you're only going to click that button if you're like a hundred percent sure. And because people actually own their repositories, they own repositories, so the most part is like, okay, if I screw up, I'm just going to shoot myself in the face. It might've been that it was really early in the cloud journey and they didn't put all those rules on yet, but I don't think so. I think it was just that there was a good amount of trust, which is like hilarious because I have airport security today in internal repositories and I don't even work on Azure, but it's not necessary, right. I hope it's clear from the top that it's not necessary to have that much control, trust is much more important, I think for long-term growth of the team, team spirit, and then also just like adding more features and business value. I guess also with our audience, most of the people are not actually deploying user-facing code, but they are building the infrastructure, right. So the chef doesn't eat in his own kitchen, kind of thinking of that, they're scared of their own code maybe. But hey, we'll never know unless everyone tells us now in chat. So let's, if you're ready for it, hop into the Q&A, and as a reminder, if you have any questions, you can still do so in the channel DevOps Talk Q&A, so be sure to join that.

First question is from Sissy Miller, how would you handle the scenario where the vulnerability is a valid issue for the app but can't be updated because other things are not compatible with the fix? So if I understood correctly, there is a valid security vulnerability, so it's like a confirmed vulnerability, there's no fix. And how should we handle that, is that was the question? Yeah. Yeah. And there's also compatibility issues. Yeah.

Okay, there's compatibility issues. That's something you have to figure out with your security team. And so I can say that the customers do this, I can say that a couple years ago, I'm pretty sure the Allianz probably still does this. You have to find that balance, right? And often even in large organizations, you'll have basically a contract, okay? It's going to be locked. You're deploying something that has a known vulnerability, and you have however many days or hours to actually fix it, and you have to make that decision, right? There's a vulnerability if the team or the business owner, the product owner says, we go anyway, okay, then the clock starts ticking after you go into production and you fix it by then and that might mean rolling back. So there's always going to be a contract, I think, between all the different stakeholders in your organization, and that includes the security team. And I think the hardest thing is to find the right amount of confidence. The example I said, it's a valid security issue, but what is the chance of it happening? There's no such thing as 100% security. I think you just have to evaluate it, take a little bit of a risk, and figure out what is right for you. Yeah, I agree.

Next question is from Jessica. How often should we revisit what good looks like? We have to start somewhere, and how do we know how often we should optimize for success? I think you know in your gut. So you never reach perfect. And it's funny because I show demos at work all the time and in webinars. And I'm like, oh, we're doing this now. And then like five months later, I would totally do it differently now. And so as you kind of work with what you've built or as other people join your team, as you get new experience, but you might change it. And whether or not you have the time to change it, that's going to be dependent on what's your workload, how much time do you have, do you have to prioritize features, et cetera. So you're probably going to be doing it all the time and you will never reach perfect. And I think if you reach perfect, I'm like, that's an interesting bar. Or you have something that, there are some services, you deploy it and they're done. You don't really, like if it's not a customer facing, let's just say it's an API to consume, some things are kind of done and you don't update them regularly, maybe for some security maintenance. But same thing then with automation, it's kind of done, so. Yeah, I'm thinking of, I don't know where I got this wisdom from, but I once heard, if you're looking at something you built a year ago or two years ago, and you're not ashamed of it, that's a bad thing, right? Cause that means you have not progressed in your knowledge for a year or two years. And you mentioned half a year, half a year is like acceptable, right? But yeah, if you, and of course you don't need to progress if you're fine where you are, then you're fine.

But most people here I'm assuming are here to learn new things. And yeah, I thought it was a nice, nice rule of thumb. Like if you're looking at something from two years ago, you're not ashamed, it's a bad thing. Yeah, it's not just though ashamed or what I've been learning, right? So I'm reading this book right now, it's called Think Again by Adam Grant. So sometimes it's just that it's not, you learn something new, but you realize something and you just change your mind and changing your mind is actually not a bad thing, right? It means you're thinking like a scientist, maybe you have new learning information now and you change your mind. It didn't get better, it's just different. And when I changed it, it works for me, it won't work for you, but that's okay. It's for me, not for you.

That's nice. We have another question from Cece Miller. Is it possible or can you over-document and what is too much, or is that a team thing? You can definitely over-document, I wish I could share so many internal things with you. Documentation, it's really hard to find that right balance and sometimes there's like not much text and actually that takes a lot more time to figure out what is actually necessary. Let me write a paragraph and delete like 10% of it or 20% of it or rewrite it because it doesn't make sense. As much as you need, and not much more. I think one thing I try to tell my colleagues and I'm not really successful at it is you don't need to write a book. You don't need to write whole paragraphs. If you can get away with bullet points, do it because nobody reads it anyway, who has time to read pages? So you will decide or the people you're talking to will decide if it's too much. The couple of times, oh, didn't you read the wiki? And I'll say, no, it's too long. And I'm the rects. And so people are like, you didn't read it? I'm like, no, I didn't read it. I didn't have time to read this much text? No. Yeah. It reminds me of a meme that was going on a few weeks back. Something like, why waste two minutes reading some good documentation when you can just spend a whole day trying to figure things out for yourself? I like that too, that also works. And I felt like personally attacked when I read that. Nah, it's not attacking, it's just it's all good. It's all good. But I guess a lot of people are guilty of that.