Scaling Fast – Engineering Lessons From ~15 Years of Tech Startups

So, we invite you to join me in welcoming one of the speakers to the room. And, of course, I'm sure you're all excited to hear what he's going to be talking about.

Scaling fast, engineering lessons from about 15 years of tech startups. I may be old. So, your users are going... Now what do you do? What happens next? So, what happens at that inflection point where you go from slow growth to a lot of growth? The first thing you've got to remember is that tech is a Red Queen's race. That means that you have to catch that explosive growth right now while it's happening because no mode lasts forever, and if you found something that's worth chasing, there's at least five other companies trying to do the same thing while you're chasing this growth.

So, to keep up with the growth, you're going to have to scale three parts of your company. You're going to have to scale the business, the team and the tech. Now, scaling the business is pretty simple. Forget what all the business people are telling you. There's only two things you can do here. You can either sell more products to more people. That's why everything is a subscription these days. Or you can sell products for a higher price. That's why every company you see starts as B2C, and then they go to B2B and Enterprise, or if you follow any influencers or indie creators, they start with a $10 e-book and then eventually they have a multi-thousand dollar video course. That's the same thing.

But when you're scaling the business, you have to beware of the S-curve. Every business or product hits a saturation point where your churn and your customer acquisition start to balance out and you kind of stop growing, and that's when you have to figure out something new. You have to either release more products, find new audiences, or just, you've got to figure out something new to do when you hit the saturation point. But as an engineer, I think we're all engineers here, why would you even care about any of this? That's because the business side is your fuel. You want the business to grow because fighting the market means you're not fighting each other within the company. When your business is growing, that's where the promotions come from, that's where more engineering challenges come from, your career grows, your CV looks more impressive, all the good stuff comes from growing the business and having those business results. And if you don't have growth, if you're in a stable, low-growth business, your career kind of turns into a zero-sum game. You have to steal your promotion from somebody else who's not getting promoted. You have to, you know, everything becomes a zero-sum game. So the main numbers that you have to care about when you're thinking about the business side is your cost of acquiring a customer and their lifetime value. The delta between those two numbers fuels everything else in your business. Now, while the business people are taking care of that side, you're looking like this as an engineer. You're just trying to build as fast as possible so that you're not the bottleneck for the company.

The only goal you have as an engineering team at a startup going through that inflection point is to not be the reason that the business has to slow down. And remember, engineering is the tool, not the goal. You're building that flower, but what you're selling is those awesome people who can do cool shit that they weren't able to do before.

So one thing, this brings us to scaling the team. Many tech problems, when you squint a little, are actually people problems. Yes, you can write some really amazing code that does wonderful things and makes you feel super smart, but it would be so much easier to just have a five-minute conversation and be like, you know, why is the API not returning the data I need? Because if you can solve it that way, that five-minute conversation can save you a week's worth of work and it's a lot faster. It's a lot faster to solve technical problems that way.

So how you scale the team really impacts everything else in your engineering processes. And the mistake that many companies make at this stage is that they build horizontal teams instead of vertical teams, so you end up with a front-end team and a back-end team, maybe some sysadmin people, and this is a really great way to ensure that everyone is always blocked by somebody else. The front-end team is always waiting on the back-end team to get their work done. The back-end team is always waiting on the front-end team. Somebody is always waiting on the sysadmins. And the only way that those teams wouldn't get blocked by each other is if their roadmaps are carefully in sync and nothing ever takes longer than you planned, because engineering projects always take exactly the amount of time you planned for them, right?

So instead what you really want are vertical teams. They should be organized by the business domain that they're tackling. The idea here is that you organize your teams by what they're doing, not how they're doing it. There are different names for this. Every consultant that talks about teams and stuff has a different name for this concept. Some call them empowered product teams, stream-aligned teams, business-capability-centric teams, but whatever you call it, they always have the same goal. It's how can we have teams that deliver value on their own without being blocked by other teams. So the goal is that each team, or you in a team, should be able to own your destiny, own your mess, and understand your domain. Owning your destiny means that you can ship value from idea to production. Your team comes up with an idea and can get it all the way to production, delivering value to users. Owning your mess aligns incentives and means that you are in charge of keeping your code running in production, making sure it works, and all of that stuff. There's no writing some code and throwing it across the wall to QA and deployment teams and so on. You should be in charge of that because that way you care about it more. You get to feel proud of the stuff you've built because you're there to see the impact it has on your users. With that long-term ownership, what you get is domain expertise where you really understand the users that you're solving problems for and the problems that they have, which then unlocks a really good collaboration between product and engineering where you can work in this product development cycle where you can focus on getting us across the water, not just building a bridge. The idea here is that engineering teams, especially product engineering teams, get involved early in the process, as early as possible, and ideate together with the product to develop the best solution to the problem, not just a solution that somebody else thought of. You can do this because you understand the domain and understand your...

I didn't mean to press that yet. What you get from this is that engineers understand why they're building what they're building, which then lets them prioritize business outcomes over just writing cool code and getting distracted in long debates about what is the best way to write a for loop or whatever. All the stuff that you see on Twitter, most of those debates don't actually matter at all. When you have those engineers that understand their domain, you can start delegating decisions, not just tasks.

And this is probably the biggest lever you can pull when you're scaling a team or scaling a company, is that the people who are closest to the problem, the people who are closest to the domain they're working with, probably know best what is the right solution. Ultimately, you can't have one central decision maker who is doing the thinking for everybody on the team. That's very common early in startups, like the founder or eventually like the head of product comes up with all the ideas and everyone else is just building, and what you get is a very strong bottleneck that has to get involved in every little thing and is blocking the entire company or all of the products and everything you're working on from advancing. And, unfortunately, or unfortunately, this also means that you're going to have to eat your ego when you're dealing with code review, and you're going to have to merge some imperfect code, because it is okay.

It's fine. Code that works always beats code that is perfect. Have short PRs, as short of a code review process as possible, and raise your hand if you measure your code review time in minutes? Two hands. Nice. Hours? Few? Okay. Who has been in a code review that has taken multiple days or weeks to get something merged? I feel sorry for all of you. Work in progress is the fastest way to kill all of your progress. When you have stale code lying around, it becomes harder and harder to merge and resolve that, like, merge conflicts, all of those things, especially when you're in a situation when you have multiple teams working on the same product together, or you have teams working on different microservices that all talk to each other. If you have code that's just lying around in purgatory in code review, it gets harder and harder to get that landed, so one way you can speed that up is to talk to each other before you write the code. Before any code gets ... so what we like to do in my team is to get together for every new project, agree on the design, agree on the solution ahead of time, before any code is being written, so that by the time we're making code reviews, it's just a very quick check. We can do most of our code reviews in two or three minutes. You get two people to look at it, smoke test maybe, and be like, okay, cool, looks good to me, if it's broken in production, we'll just fix it later. That's scaling the team. Now, how do you scale the tech? Don't think too hard about the tech. The tech is kind of the easiest part of scaling a business. For the most part, you're just solving known problems with known solutions. This is in How Big Things Get Done. This is a really good book. It's not about software engineering, but it's about large-scale engineering projects, and it talks about how China is able to ship a bazillion bridges and power plants per year, whereas the USA isn't. The main factor they talk about is that in China, they have like 10 standard designs for everything, and then the engineers come in, look at the situation, and just pick the most appropriate design to apply to that situation, and then here's the thing. They tweak the situation to fit the design.

They don't tweak the design of an off-the-shelf component to fit the situation. It's the other way around. So it fits the solution you already have, and that way you can build much faster, and you're saving your innovation tokens for the things that are actually new and differentiating in your company.

Which brings us to just solve the problem, not a different, more difficult problem. The hardest thing to do in software engineering is to build a generic solution, especially when all you have is a specific problem. It is generally a good idea, or rather, I'll say it this way. The older I get, the more my code looks like a very simple step-by-step process. It doesn't impress anyone. It's super boring to look at, and personally, I think the biggest compliment you can get on your code is when somebody looks at it and goes, wait, that's it? Right, that's it. It is simple, it's easy to understand, and it's easy to maintain.

Because you always have to push back on complexity. Engineering is an endless fight against complexity, and sometimes that means saying no to an architecture astronaut who's trying to build a generic framework to solve a problem forever when you just need, you know, we need to ship this code tomorrow, we don't even know yet what we're building in ten days, and you're trying to build a framework. Just fix the problem, solve it, and move on.

This also means that, again, because you understand your domain, because you have long-term ownership of the stuff you're building, you can push back on your product as well, on your product managers. Very often, you have one requirement that blows up the entire story and makes it way harder than it needs to be. You can usually push back on your PM and be like, hey, this one thing is making our work a lot harder. And they'll go, oh, yeah, shit, no, we don't actually need that, we don't care. Just drop that requirement and now you can use a very simple solution.

And where most complexity comes from in a software engineering project is not actually the code you're writing, it's the architecture you're creating. If you think about your code as a series of interconnected boxes, the boxes are like your code, those are classes, modules, services, whatever you want to call them, and the lines are how they talk to each other. The more connected these lines are, the more intertwined everything gets, the more complex it is.

One lesson that I've learned is that you can always change how a self-contained box works. You can rip out all of the code and write new code that's cleaner or better or whatever, but it's very hard to change those connections. And one way to limit that complexity is to use to organize your code vertically instead of horizontally. So instead of splitting components, hooks, types, utils or whatever you have in your stack, I don't know what stack you use, but if you instead split your code by everything that goes into a dashboard, anything that goes into a user profile, into a to-do list, you then have modules or boxes that aren't as intertwined. And it's those clear APIs that make everything easier.

One trick I like to use is to use type-driven or contract-driven development where we ahead of time, we talk about what are the different moving pieces of a story we're building, and then we talk about how they're going to talk to each other, and then everyone who's writing the code is empowered to build the internal implementation whatever way they want. Unfortunately, or fortunately, using vertical modules may mean that you will have to copy-paste some code. And this is probably one of my most controversial takes is that drying code, do not repeat yourself, is actually bad. And whenever possible, you should lean on separation of concerns more than you do on drying up your code.

Because when you dry up your code, your architecture ends up looking like this if you dry it up too much. Everything is overlapping. You get components or services that take 10 boolean parameters to configure their behavior, and you change one thing over here, and suddenly something completely different that you weren't even thinking about breaks. When that's happening to you, that means that your code is interconnected in a way that causes problems for you.

So, one of the things that I like to focus on, especially when talking about architecture, is to focus on your data, and your data structures, how you organize data in your database, how you organize data access patterns, stuff like that, because it turns out that when you get the data right, your code is so easy to implement that it almost feels like you have nothing to do all day. And when you get your data wrong, when it doesn't fit your domain model, it feels like you have to keep bending over backwards to make things fit together.

And that is why you want to use vertical teams that write vertical code, because it means that you as a team are in control of your API, in control of your data, in control of your UI, and you get to design everything so that it fits together, solves your problems, and not some generic straw man that you find on the internet and people are talking about.

So, that was my talk. I'm turning all of this into a book. If you use that QR code, you can learn more about what I'm doing. And thank you. Thank you. Thank you.

Let's do this one. How do you avoid the imperfect code you merged today making progress slow a couple of years down the road?

That is one of the challenges in software engineering. If you try to build everything perfectly today, you'll end up spending a lot more time. For example, you may spend extra time building a super generic API in anticipation of future clients, only to realize later that your abstraction was wrong and you need to throw away the code anyway.

Instead, focus on getting your data right and getting your architecture right. It's easier to throw away bad code and replace it with better code later, but changing the architecture is much harder if you get it wrong. So prioritize domain modeling and getting that part right rather than obsessing over clean code.

People talk a lot about this, and they worry about it, but programming over time is the job of software engineering. If you try to build it perfect today, you're going to end up spending a lot more time. So a common example is we think we're going to need a generic API to access some data. But we only have one client. So we're going to build our API in a super generic way in case we have more clients in the future. Cool. You do that, you spend a lot of extra time debating your design, debating how to make it generic and all of that. And you ship, and you're very happy. Then six months later or two years later, you actually get that second client. And usually what you realize is that you got your abstraction wrong, and you need to throw away that API anyway, because you made it generic in a way that only fits the client that you had, not the new client that you created. So I would say it's kind of a cop-out answer, but it depends. For the most part, if you focus on getting your data right and getting your architecture right, it's very easy to later throw away the code that is bad and, like, replace it with better code. But it's very hard to change the architecture if you get it wrong. So I would primarily focus on domain modeling and getting that part right rather than on having super clean code.

I think the next question is... My company thinks classical teams, front end, back end, develop deeper expertise of their specific app. How to ensure this also in product teams? So how do you ensure expertise in product teams? I think it's a different kind of expertise. I'm a strong proponent of having product teams. You can have a product team that together is an expert in their domain and their problem set, or their type of user, or whatever. But then in there, you can have people who are focused more on the front end, people who are focused on mobile, people who are focused on back end. So together they have domain expertise and separately they have stack expertise for their specific parts of the stack. And then if you have the vertical organization of your code, you don't actually need to know what the other parts of your app are doing, because you have clean, well-defined APIs so that you don't need to know the details. Yeah, so I think that's my answer there. Domain expertise over specific code expertise and clean APIs so you don't need to know those details as much.

The next question, thanks. How do you avoid the imperfect code you merged today making progress slow a couple years down the road? I think we did that one first. No, I think I closed the first one. It's out of sync. It doesn't work. Do you like something else? Let's do this one. How to balance dry and duplicating your code? I think in terms of balancing dry and duplicating your code, I like to have senior engineers or somebody who gives a shit doing gardening on your codebase, so I like to let patterns develop, encourage people to not worry too much about drying code up, just build what you need to build, and then have a background process that's observing what people are doing, looking for patterns that are coming up, and then turning those into frameworks. So when you have four or five different solutions to a very same thing, it's going to become very obvious what those abstractions need to be and which parts of the code actually can be dried up and which ones aren't. It's very hard to predict that ahead of time, and honestly, the worst, hardest to maintain, shittiest code I've written in my life was always trying to dry something up that looked similar but was actually diverging in different development directions, and then it's very hard to detangle that afterwards. But if you let it go in its own way first and then back it up later, that makes it easier.

Yes, next one, I chose product feature teams without product management included. Good or bad idea? I feel very strongly that product should be part of product engineering teams. So, you have like platform teams where you can have a team whose stakeholders are internal users, like React is a really good example of a platform team at Facebook that's developing a frontend framework and their stakeholders are other engineers. Then you have proper, like, and they actually, I think they do have product managers in the team who are focused on the platform part. So, yeah, I guess what I'm trying to say is that if at all possible, you should have somebody doing the product manager role on every team. They don't have to actually have the product manager title. It can just be a tech lead or an engineering manager, but somebody whose job it is to talk to stakeholders, check that the things you've built are solving their problems, potentially translate domain to engineering, business speak to engineering speak and stuff like that. When building in vertical slices, how do we avoid dependency hell with shared libraries and models requiring changing all slices when one of those changes? So, this is a really good example of drying up your code too early. If you've split something off into a library and you find yourself constantly having to work on both the library and the product at the same time, that means that your library is not actually a library, it's a product concern and should just be merged back into the product. A sign of a successful library is that in the beginning, when you first make it, you have to work on both together, because the library is new, but eventually you should be making fewer and fewer changes to your libraries and more and more just to your product. That's a sign of a good abstraction. And actually splitting them into a different repository is a really good trick to encourage engineers to not leak product concerns into your libraries. That friction is actually helpful, I would say, because it encourages you to avoid putting product concerns in the library.

Thank you very much. There's always questions. Let's applaud Zbizek.