Vibe Coding at Enterprise Scale: What Happens When AI Joins Your Dev Team

Hello everyone, thanks for having me today here, I'm Wesley, I'm going to introduce myself real quick, but I want to talk about web coding at its highest scale. We did a pretty good experiment at Modus Create a couple months ago, and I would like to share about that. Alright? So, introducing myself, I am Wesley Fisher, I am from Brazil, I work at Modus Create as a senior principal engineer there. Working there for almost 7 years now, and I have 14 years working as a soft engineer. I have done many different things as a soft engineer, I have done front-end, back-end, DevOps, solutions like that as well. And my expertise is around Amazon Web Services, Java, JavaScript, and serverless applications. And lately I have been very focused on understanding how these new AI type of tools we are getting, coding agents, are going to change the way that we do software, right? Have been very focused on that in the last 9 months now. So while I'm going to the community, having conversations with folks, Reddit, Twitter, AXE, those different platforms, I have seen a lot of people doing different comments, having different opinions about VIPE coding, about using coding agents tools for software development, right? And the pattern I have seen is basically two different groups in opposite directions. People thinking that software developers as a profession is now gone, or soon to be gone, and in the other side, people that think that VIPE coding is something that we should not be doing, people think that VIPE coding, or using agents for coding, is not a good thing, So I am here today to try to put a balance on this conversation by bringing some facts, bringing some useful information of learnings we got in the last few months. I do believe that with AI, the way that we build software is changing a lot. I don't think that in the long run, in the next few years, we are going to be coding as we have been coding for the last 50 years, or at least in the last 25 years. I think that AI, and coding agents, GitHub Copilot, Cursor, Wingsurf, those tools, right? They are going to change a bit. But it's not, I don't have an opinion, a strong opinion about being something bad or good. I think we have a balance, and we need to think exactly how those tools can help us. For that, at Modus Create, we decided to VIPE code an application internally, an internal experiment to really understand how AI can help us do better software to build better applications to help our clients.

So, I'm going to talk about that, I'm going to present data from our experiment, I'm going to present the learnings we got, and I don't want to talk about hype, I don't want to talk about the trends, and what people are talking about out there, right? I'm going to focus basically on the learnings we got from that experiment, and what that taught us. We had basically 10 people working in a three month experiment between March and May, so it was a big thing for us, and we learned a lot that I want to share. Basically the structure of the experiment, we decided to build an application, a simple mobile application that helps users to translate magical jargon into common people knowledge, a couple of languages, right? Essentially, you grab your phone, you take a screenshot or a picture from a doctor's report, a lab report, a lab exam report, and that diagnosis you have there, we translate that to common knowledge, of course using AI in the background as well. We built that application twice. We had two teams, one team what we call DIY, essentially do-yourself team, they were not allowed to use any tools for coding, any AI tools for coding. They could use different tools, Stack Overflow, they could use the traditional software development as we are used to, right? But they could not use any tools for generating code using AI. And we had another team, the AI team, that was allowed to use cursor and GitHub Copilot to generate the code, and they were specifically asked to not code at all unless they really needed that, and when they needed to code, they would need to document that so we would understand why they needed to code something. The AI team was only two developers, two engineers, two full stack engineers, and the no AI team were three developers, right? So we had a smallest team working with AI. Both teams were asked to write plenty of documentation and to manage time sheets during the development process. So we would track to the detail of the task, how many hours they are taking, they took in a given particular feature, right? And we used that modern technology to develop the application using BadRock as our LLM in the backend for Amazon services. We used that Fargate, Node.js, the application was built using React, Ionic, and Capacitor, and deployed using GitHub Actions to both Apple and Google Play stores. Of course, we didn't deploy externally, only internally. The most important part of our experiment was in the very beginning we set up hypotheses on the expectations we had about that experiment. We thought that the use of AI, in this case, cursor and GitHub Copilot, they would help us to be at least 50% faster with a smaller team by 30%. So once again, the team that was using AI was one less engineer than the three engineers of the non-AI team, meaning 30% reduced capacity there.

We also thought that using AI, the need of expertise, technical expertise, in a given matter of the software development lifecycle would not be required. Just to give an example, we had, as a full stack engineer working on this project, an engineer that was primarily a backend Python engineer for his entire career. He was asked to work as a full stack writing React code on the front-end with TypeScript, both in the front-end and the backend. So it was a new language, a new tech stack for him. We expected the quality of the code base to be at least the same quality of the non-AI team quality of the code base. So basically, we used Sonar on both code bases to do static analysis on those code bases, and we compared outcomes of Sonar to understand if the code bases were as good as, at least without the use of AI. And of course, we expected to have learnings, to learn a lot on the things we should do and the things we should avoid while using AI for coding.

The results of the experiment were pretty good. We had a 45% productivity gain on using AI, meaning the team that developed using AI delivered the application 40% faster than the team that was working without AI, at that 30% reduced capacity. A team of two people developed the application almost at half of the time of the team with three people that was not using AI. The Python engineer that I mentioned was able to be productive and to deliver applications using TypeScript. The fact that languages are pretty similar between them, right? TypeScript, Python, they have differences in syntax, but the basics of logic and the basics of software element, they are the same. So the developer was not required to code TypeScript during the project. He was required to think on the features, to think on how to implement the application. So AI helped him to be productive using a language that he never worked at before.

The quality delivered by both code bases was pretty good and similar. Both code bases had a couple of issues here, a couple bugs there, normal stuff, nothing very complicated to fix. None of those code bases had major security hotspots. So we could evaluate both code bases and they were pretty much the same in terms of quality. But of course, nothing is perfect. We also noticed that the same Python engineer that I mentioned before, I asked him during the development to act as a DevOps person working on Amazon Web Services, especially around application load balancers set up in the architecture, right? And that was a point he struggled a bit, not because the ability of using AI didn't help much because he was not clear on what he wanted to develop in that moment. He was not aware that he should use an application load balancer versus an API gateway, for example. So the ability of knowing exactly what you want to ask AI to do reduced his performance. But he was able to deliver the feature. It just took longer than a person that was not using AI, the DevOps person of the other team. This proves that AI can do the job. But you as an engineer, you must know exactly what you're asking for AI to work for you. You cannot just, hey, you know, build this for me.

But you as an engineer, you must know exactly what you're asking for AI to work for you. You cannot just, hey, you know, build this for me. You need to be more clear on your output, the output that you are expecting. What is the application you are developing? The decisions you make, your software architecture decisions. Provider, if you're going to use like cloud as a provider, Amazon, the things you want to use on Amazon, those kind of things, you still need to make decisions and provide that as a context to the NLM. And it taught us a pretty important thing.

AI coding or agent coding for us means using AI to speed up the process of writing the code. But the human in the loop must be a person that is skilled enough to make decisions on how AI should work and skilled enough to do code review, to do the review of the AI generated code. While Vibe coding says that you just rely on AI, you ask for something and you don't review the code. We don't believe in that. We don't trust on that. It's not the way that we want to work at MobScript and the way that we want to teach people to work on.

So for us, the human being is still required and the human being is required to be a soft engineer, to be a person skilled on engineering, on writing applications. That person is still the rock star on this workflow. This person is required to do the planning. This person is required to review the code generated by AI. This person is required to execute the process of validating the application and the code that was generated by AI to make sure that that application is ready to go to production. So the end of it is basically AI is not a replacement for architectural clarity or engineering expertise but can be a multiplier of it.

So the end of it is basically AI is not a replacement for architectural clarity or engineering expertise but can be a multiplier of it. So as long as you focus on having pretty good engineers and you still focus on skilling up your engineering team to use AI, those engineers can be better, can do a better job using AI, if not 100% of the time, but pretty much it, most of the time. They can use AI to brainstorm, to make decisions, to write an implementation plan and to ask the AI agentic coding tool, cursor, copilot, whatever the tool you decide to use to make sure that tool is going to speed up the development process for you by the correct and respectful management of the person.

What are the key learnings and the workflows we got there, right? First, for me, the best prompt, the best way you communicate with AI is being clear and concise on what you expect AI to do for you. So going to copilot and being very clear, crystal clear, on what we expect AI to do for you is the way that I see most productivity gains and most high-quality professional results out of it, right? And for me, it starts with having atomic level tasks. Basically imagine you have like a user story on your tier board that you want to use, that you want to implement, right? That story needs to be broke down into smaller units. It can be subtasks and for the subtask, you can create an implementation plan which contains, which is essentially for me, implementation plan is a markdown file on your repository.

And that markdown file, you define atomic level, big step actions that you want AI to go over in an interactive way. So basically you create that implementation plan. You can use such PT, other AI tools to craft that implementation plan. And then you review it and you are sure that's the plan you want to follow. You start asking AI to go over it and working on those bullet points, right? On each bullet point, when AI is done, you review the code base, you run validation steps such as linter, unit tests, tools you have available on your command line to review the quality of the code base. And then if you are happy with the output, you accept.

If you are not happy, you provide feedback. When you are happy, then you commit that to Git and it saves you if in the next prompts, AI hallucinates and started deleting code. So committing often, working in smaller units, smaller baby steps, in iterations way is the way that we found that Mondo's creating that experiment be the most productive way to work with coding agents. Of course, giving context to DLLM is also very important. And by giving context, I mean not a large context, but a rich context with narrow laser focus at information that is related to the task you want to accomplish. Too much context might make your AI to start hallucinating, right?

That's not something we want. So finding ways to feed the DLLM with the correct context for the task is also a skill that the human being, the software developer managing the tool must develop now. And of course, there are different ways to do that. Using instruction files to set up like coding patterns that are agreed upon the team or having MCPs to feed the DLLM with external context from Stack Overflow, a Google search, a perplexity query or, of course, you doing a good job on writing the prompt. And of course, cleaning the context after a couple of interactions because it starts growing and hallucinating.

Some good prompts, of course, being clear and concise on explaining what to expect the AI to accomplish for you, not just writing, hey, do something for me, right? Write sass here or write tasks for that component. You need to be more clear. You need to be narrow and laser focused on what you want the AI to accomplish. For me, I don't use that much of prompts because I really want to use the implementation plan. And implementation plan is essentially my prompt at the end of the day, right? So having an implementation plan with big steps and clear and concise action items there helps a lot because you can go to the DLLM and say, hey, take a look on the implementation plan and let's start working through that. And this validation loop for me is key, is probably the best thing we learned because it keeps me focused, keeps me productive and gets me good results.

So I ask AI to start working on a task. That's the first step there. The AI, DLLM generates some code for me. That task being a narrow laser focused BAPESTEP task helps me to have a smaller batch to review instead of a large batch that it would be like a 100 files pull request that would take me longer time to review and probably start missing stuff after a while. So it writes me some important code, a couple components, a couple functions, right? And then I do my human review. I go there and review the code base and start prompting with feedback. I write a large prompt with all the feedback I want.

AI goes and starts refactoring, changing the code per my feedback. I do my human QA work, my manual test, right? I go there and do manual test to make sure that the application that is being developed is working as expected from the user perspective. I run automatic validations, which is also pretty important, like using unit tests, running tools like Linter, Lizard, Horusec. Those command line tools allow me to feed the AI with some context about what's wrong on my code base.

Let's say I use Lizard to calculate the complexity, the cognitive complexity of a function, and then I use that feedback on the LLM so it knows what's wrong and then fixes it, right? So doing those validations, the automatic validations from the command line, is also pretty important. And also I'm happy I do the git commit. I often commit so I avoid losing code because of AI craziness at some point. Test-driven development is one of the workflows that I really like. It helps a lot because everything that I just said about narrow, laser-focused tasks, that's TDD, right? TDD is based on writing smaller steps, on coding with tasks in smaller units.