Scaling Fast – Engineering Lessons From ~15 Years of Tech Startups

Hey, I'm Swiss, and we're going to talk about scaling fast and some of the engineering lessons I learned in the last 15 years of doing tech startups.

You've built something that people want, and it's growing super fast. Now what are you going to do? There's three things you're going to need to scale. You're going to have to scale the business, the team, and the tech.

Scaling the business is pretty simple, but it's still hard work. There's really only two ways that you can scale. You can make more sales or you can make bigger sales. And that's why everything is either a subscription or going after enterprise these days. The reason you care about this as an engineer is because without the business, there is no engineering. Engineering is all about building the right thing at the right time for the right situation. And no matter what you try, you can't save a bad business with better technology. And it's also the business is the fuel for everything you do. The numbers you want to know are cost of acquiring customer and their lifetime value. And your goal is to build tech that fits these numbers. But honestly, that's mostly what the business folk are going to be doing. What you're really doing is this. You're building the tracks before the train stops. And you're trying to go, you're basically trying to go as fast as possible to make sure that engineering is never the bottleneck that stops the business. And no matter what you hear on Twitter or the internet, engineering is just a tool that the business uses to achieve its goals. It's not the main thing that for the most part, it's not the main thing that the business is doing. What you're trying to do is you're building that flower. But what you're selling to the users are new users who can do awesome stuff. I know I love my technology, but users don't give a shit about what you're doing. They just want to use your thing and move on with your life. So that's about scaling the business.

Scaling the team is also kind of important, even though we don't get to talk about it that much as technologists, because a lot of tech problems are secretly people problems. Your architecture and the process that you use to build products is going to reflect your team structure. So for example, if you put QA in a separate team, you're going to be spending a lot of time waiting for QA to give you your feedback. And you're going to have engineers that are throwing things over the wall instead of making sure their code works. And then you get a lot of back and forth at the end after the code is written, which slows you down and is really annoying for everyone, for everyone involved.

And the worst thing I've seen startups do is that they organize teams based on the technology instead of based on the goal or the purpose of what you're building. So if you organize your teams like that, you're going to end up spending a lot of time being blocked. You're mostly going to be waiting. You build the thing, you build your part of the thing, and then it goes to another team and then another team and you end up with a very linear, slow process.

Whereas if you can orient your teams based on their purpose, based on their goal, so like an admin team that owns admin users, a buyer team that owns buyers, a growth team that focuses on growing the product, you're going to spend a lot less time being blocked by other teams and being able to build projects or new features, start to finish as part of a single group. The goal of doing that is that you want to deliver value without being blocked. And there's many different names for this sort of team structure. Some people call them empowered teams, extreme aligned teams, business capability-centric teams. It's all consultant capability book, and it just depends on whose book you read.

The main goal, the main point everyone is trying to convey when they talk about this is you want a team that can ship features from start to finish without getting blocked. And part of that, my favorite part of that is that you get to own your own mess. So I used QA as an example. Um, if you have engineers that get that on their own mess and that they're, that are accountable for the things they ship, you're going to have a lot fewer bugs. It feels slower, but you're actually going to have a lot fewer, a lot fewer issues. So for example, a couple of days ago, I got paged at like 3am, kicked out of bed, went to look at the alerts, and it was something incredibly stupid and dumb. Now guess what I've made really sure was fixed the very next day. Um, you know, I got paged, I fixed the bug in one day and the bug, and that's not going to happen anymore, which is a lot faster than how a lot of, a lot of teams get to do it. Also a bit more stressful. Uh, the other part of having vertical teams that can own their domain is that you can focus on a better product engineering partnership where engineers work on getting us across the water, not just building the bridge.

So you end up with this circular feedback cycle instead of a linear process where you get to work together with product to build the, to build those flowers that, um, that customers want to move faster. It lets you, it lets you build faster, um, uh, using feedback cycles, and you can iterate on solutions. I, I personally find it very rewarding as an engineer where, when I can talk to the customers, understand their needs, design, uh, design my code and my products as a way to solve those user needs and then, um, give it back to them and iterate on those solutions. The idea is that you can build stuff that you can build the appropriate solution for your use case, not just something you heard about in the talk. And that's what engineering is really about. What that means is that yes, sometimes you're going to have to merge really bad code that sucks, uh, because at the end of the day, your code doesn't need to be perfect. It just needs to work. So you're going to have to let go of your ego and let other people experiment with different approaches. It's okay if somebody writes code in a way that, uh, in a different way than you would write it because it's fine, as long as it works. And as long as those engineers own their own mess, uh, if it breaks, they're going to fix it. If it works, it's great.

And you didn't spend a bunch of time bickering about little details that didn't end up mattering anyway. Uh, the reason that matters is that work in progress kills your progress. You want to avoid having a bunch of stale PRs and long live branches and things like that lying around because they tend to collect dust. Um, and the longer you have code that hasn't been merged or that's just been sitting there, the harder it gets to merge the more ideas you have. Oh, but we can still improve this or, oh, we can tweak that. You can always tweak, tweak things and improve your code later. Right now you should just finish the, finish the work, get the feature out there, see if users like it and focus on fixing it better, uh, on fixing it later. And the best trick I found to, uh, have slower PR cycles and have an, uh, be making, making it easier to merge quickly is to talk about what you're doing before you start writing the code. I know we're engineers, we hate meetings, but I promise you a five minute conversation about how you're going to approach a problem, how you're going to, uh, write the code, what are the trade-offs, um, making sure everyone understands everything that's involved is much faster and much easier than having a three day long PR discussion where you write the comment, then you go work on your own code, and then you come back three hours later when they reply and, you know, just have that five minute conversation and keep the code review and pull requests as just a quick sanity check at the end.

Okay, so we talked about scaling the business and scaling your team and, uh, making the team work more smoother, if that's a word. Um, so what about scaling the tech? Well, despite what people say online, the tech is really the easy part. Um, you're mostly just solving problems that have known solutions. So you should just pick an off the shelf design, apply it to your domain and move on. What you want to avoid is spending all of your innovation tokens on things that don't really matter. So there's this great book called how big things get done, where they talk about real world projects. And one of my favorite examples is why does China build so many more new power plants, roads, bridges, and all of those things than we do in the West, both in the US and, um, and in Europe. What the authors identified is that China moves fast because they have like 10 standardized design designs. They pick the one that is closest to the situation, to the situation they're dealing with. And then they change the situation to fit the design rather than the way we do it in the West, where we treat every project as a unique little snowflake. We spend a bunch of years innovating on a brand new design, then dealing with all of the repercussions of having unknown unknowns and being surprised by things we didn't predict. And then when it's built, we throw away all of those lessons and start again from scratch next time. I'm not saying you shouldn't innovate. What I'm saying is you should save your innovation for the core parts of your product, where you're actually making a difference. You don't need to like invent a new design system library or invent a new state management framework or figure out all of these things. Just pick one. We're in the, we're in JavaScript. I promise you there's 10 or 20 libraries for every little thing you're trying to do. Just pick the one that works and move on and focus on, you know, focus on solving the problem, not a different, more difficult problem. This is something that gets me a lot. Writing code that does a thing is pretty simple, but it's not that fun.

Writing a generic framework version of that same code that tries to be super flexible, predict everything that's going to happen in the future and scale from your five users right now to 50 million users you hope you'll have in the next few years. That is really hard. And I promise you that you're just wasting time. You're most likely trying to scale earlier than you need. And I promise you, I can promise you that the way you think your code is going to break when you hit more scale is not how it's actually going to break. Scale will surprise you. Things break in weird ways. And also you don't need to do everything the same way that Facebook, Google, Shopify, or whoever do. And even Shopify.

Shopify is actually a great example. They are still to this day at their current scale, they are a Ruby on Rails shop and they use MySQL to drive. I think they like to share numbers around Thanksgiving. A few years ago, they were doing several million queries per second and API, several million API requests and users per second, all on Ruby on Rails and MySQL. So the tech will scale. And when you get there, you can figure it out then. Don't try to scale too quickly because your main job is to push back on complexity. Like I said earlier, you're trying to get us over the water right now. You're not trying to just build a bridge because somebody said, somebody asked for a bridge.

Part of that is that when you're working with your product manager and you have that product engineering partnership because you own a vertical domain of the business, you can push back on their ideas. If there's one requirement that completely blows up a project and makes it take 10 times longer than otherwise, it is okay to look your product manager in the eyes and be like, yo, do we really need this part of the feature? Can we just not do this and then still solve the problem well enough? And what might surprise you is that a lot of the times when they find, when you tell them the actual cost of their ideas or the cost of what it will take to build something, they'll just say, yeah, you're right. That's not worth it. Let's not do that. Because really you understand the code a lot better than your managers, your product managers. You even understand the details of your code way better than your, I think we don't call them architects anymore, than your principal engineers. They don't know the details. If something is way harder than it looks from the outside, you have to tell people and they are very likely going to change requirements. Like I was mentioning earlier from how big things get done, you can modify the situation to fit the design that already exists. That is almost always faster than bending over backwards to move a bunch of code, rewrite it and fit this one requirement that you maybe don't even need. And a lot of what makes some things hard and others not hard is the way you've structured your architecture. The code, like the actual low level code, changes a lot more often than your overall architecture does.

You can think of architecture as like a dependency graph on your code where each box is like a module, function, class, component, whatever you want to call them. And the lines are how those components interact. The more lines you have on this graph, the harder your code will be to work with, the more you have to cross different boundaries, you have to make things work together, the harder this is going to be, and the more you'll find those little, oh, what if we just move this button from here to there? And suddenly everything explodes and breaks apart.

At a high level, your architecture is going to follow your team structure. At a lower level, it kind of depends on what you do. Ideally, you can use vertical models where the idea of vertical models is that you can reduce your architectural complexity, so have fewer of those lines, if you organize your code based on what it does, not based on what it is. So everything you need to build, the dashboard goes together, everything for a team goes together, everything for a to-do list goes at least in the same folder. And then if it gets super complex, you can fractally break it up more. What you want to avoid is to have a folder for components, a folder for hooks, a folder for types, a folder for utils, because then everything ends up, depending on everything else, you'll know you're in this situation if every file starts with like 50 import statements because every file needs to import almost every other file in your code.

On the client, you're going to end up calling these vertical modules components or pages, sometimes groups of pages. On the back end, you're looking for something like a service-oriented architecture. You can do this with a monolith. Monoliths are great. I'm actually a big fan of monoliths, but monoliths can be difficult to keep clean and to actually enforce those separations. So if your team really can't keep it clean, and if you're growing very large, it's okay to use microservices or micro frontends, but those actually bring a lot of complexity of their own that at least in a startup, you can avoid for much longer than you need.

Everyone likes to use Amazon and AWS as an example of, hey, we should only talk to each other via APIs and everything should be a microservice. They only did that when they were, I think, over 500 or 600 engineers working on the same product. I've been in startups where we started splitting everything into microservices when we were like 10 engineers. That is too early. You don't need microservices or micro frontends if there's just 10 of you and you're stepping on each other's toes a little. Use folders. They're fine.

With vertical models, what's really helpful there is it gives you a clear API. Because not everything is crisscrossed and importing other things, you can define a clear API or clear interface. What I like to use for this is contract or type-driven development so that contract or type-driven development basically means that you define the interface first and then you go build the internals of that interface. This automatically forces you into designing vertical models and it also forces you into having those architectural discussions early on when things are still flexible and easy to change, not later in your PRs when you're bickering over code.

The main thing you're trying to avoid is getting hit by Hiram's Law which states that if someone can depend on something in your code, they are going to. And that's how you get those super crisscrossed, that's a big part of how you get those crisscrossed APIs that are unclear and you end up breaking code that you didn't even realize was connected. My controversial take here is that if you get this right you're going to need to write a lot fewer tests than you do if everything is interconnected because the module boundaries are already guaranteeing that you're not breaking code that doesn't feel like it's related to what you're doing.

One of the main sources of this highly interconnected architecturally complex code that I've seen is try do not repeat yourself. It's probably some of the worst code I've written that way. There was actually a really good example of this in my new startup a couple weeks ago where we had a function that looked, oh we're just repeating this same function in three or like this same thing in three different things in three different places. What if we consolidated this and made it easier to use? So we consolidated it and it was fine and then a few days later somebody else came around and was like, oh this function looks really useful and moved it moved it to a different place, changed how the function behaved and ended up breaking completely unrelated things to what they were doing because they were connected. So what happened there was that we were deduplicating code that looked similar but actually had different goals. So we made our architecture diagram look like this and the easiest way to recognize when you have dry instead of separation of concerns, the easiest way you can identify that is if you've ever used a component that for a function that takes like five or six boolean properties just to specify how it behaves or what it's doing in this situation, that was most likely a sign that you're using a component that has been deduplicated incorrectly and it should be split up in multiple components. It's okay to have repeated code. Code is cheap. What you don't want to repeat is the intent and the source of truth of your logic. So one way I like to identify this is if the code looks similar right now but serves different users, it's almost certainly going to evolve in different ways. Or if the code looks similar right now but the users are trying to achieve a different task or a different goal by using this code, it's also going to split up and need to evolve in different ways. So you should not merge that code.

The last thing I want to say about how you can avoid architectural complexity or just complexity in your code is by really focusing on your data structures. If your data is in the wrong shape for what you're trying to do, you're going to have to bend over backwards to get it working. At the worst, you're going to end up building translation layers.

At the worst, you're going to end up building translation layers. So I was talking to a friend once who was solving the traveling salesman problem which is super hard, unsolvable, like a traditional case of NP-complete code that is unsolvable unless you add some constraints. They had brilliant back-end engineers that made all of this code work, built amazing algorithms that were super fast, but then the user experience, despite all of this amazing engineering work on the back-end, the user experience was super slow because the back-end API returned data in completely the wrong shape. So my friend on the client side building a React application to show this data had to do so much computation that they essentially re-implemented all of the hard logic of solving this crazy problem on the front-end where they ended up having to move a bunch of their computation into Wasm just to make it fast enough for the UI to not choke when you click on a button. Because they got to a point where if you clicked something it would take five to ten seconds to re-render the client side.

So React re-rendering would take five to ten seconds just because it basically had to redo all of the work that already happened on the back-end to get the data in the right shape for what they needed. And that my friends is why you want to have vertical teams that are aligned along your vertical code because you can focus on solving user problems quickly, you have full control of the features and the things you're building from start to finish, you get to design how it's stored in the database, how it talks to your... how it's stored in the database, how it gets crunched on the back-end, what the APIs and interfaces are like, you can make those fit really really well for the client side that you're building, make sure you get all the data you need. That makes your app work faster, makes everything aligned, and it also means that you understand the whole domain from start to finish of what you're trying to build, which then lets you move faster and with fewer bugs.

So thanks, I've been writing this book for a really long time but I promise it will get finished eventually. I've expanded all of what I've said today into basically a book. You can find out more at scalingfastbook.com and send me an email if you have questions, I don't really do Twitter DMs.