Vibe Coding at Enterprise Scale: What Happens When AI Joins Your Dev Team

Hello everyone. Thanks for having me today here. I'm Wesley. I'm going to introduce myself real quick. But I want to talk about Web Coding and Enterprise SQL. We did a pretty good experiment at Modus Create a couple months ago. And I would like to share about that.

Introducing myself, I am Wesley Fisher. I am from Brazil. I work at Modus Create as a Senior Principal Engineer there, working there for almost seven years now. And I have 14 years working as a software engineer. I have done many different things as a software engineer. I have done front-end, back-end, DevOps, solutions like that as well. And my expertise is around Amazon Web Services, Java, JavaScript, and serverless applications.

And lately, I have been very focused on understanding how these new AI type of tools we are getting, coding agents, are going to change the way that we do software. I have been very focused on that in the last nine months now. So, while I'm going to the community, having conversations with folks, Reddit, Twitter, X, those different platforms, I have seen a lot of people doing different comments, having different opinions about VIP coding, about using coding agents tools for software development. And the pattern I have seen is basically two different groups in opposite directions. People thinking that software developers as a profession is now gone or soon to be gone. And in the other side, people that think that VIP coding is something that should not be doing. People think that VIP coding or using agents for coding is not a good thing. So, I am here today to try to put a balance on this conversation by bringing some facts, bringing some useful information of learnings we got in the last few months. I do believe that with AI, the way that we build software is changing a lot. I don't think that in the long run, in the next few years, we are going to be coding as we have been coding for the last 50 years. Or at least in the last 25 years. I think that AI and coding agents, GitHub Copilot, Coursera, Wingsurf, those tools, right? They are going to change a bit. But it's not, I don't have an opinion, a strong opinion about being something bad or good. I think we have a balance and we need to think exactly how those tools can help us. For that, at Modus Create, we decided to VIP code an application internally and an internal experiment to really understand how AI can help us do better software, to build better applications, to help our clients.

For that, at Modus Create, we decided to VIP code an application internally and an internal experiment to really understand how AI can help us do better software, to build better applications, to help our clients. Right? So, I'm going to talk about that. I'm going to present data from our experiment. I'm going to present the learnings we got. And I don't want to talk about hype. I don't want to talk about the trends and what people are talking about out there, right? I'm going to focus basically on the learnings we got from that experiment and what that taught us. We had basically 10 people working on a three-month experiment between March and May.

Basically, the structure of the experiment. We decided to build an application, a simple mobile application that helps users to translate magical jargon into common people knowledge. Essentially, you grab your phone, you take a screenshot or a picture from a doctor's report, a lab report, lab exam report. And that diagnosis you have there, we translate that to common knowledge. Right? Of course, using AI in the background as well. And we built that application twice. We had two teams. One team, what we called DIY, essentially do-it-yourself team. They were not allowed to use any tools for coding. Any AI tools for coding. They could use different tools, Stack Overflow. They could use the traditional software development as we are used to, right? But they could not use any tools for generating code using AI.

And we had another team, the AI team, that was allowed to use Courser and GitHub Copilot to generate the code. And they were specifically asked to not code at all unless they really needed that. And when they needed to code, they would need to document that so we would understand why they needed to code something. The AI team was only two developers, two engineers, two full-stack engineers. And the non-AI team were three developers. So we had a smallest team working with AI. Both teams were asked to write plenty of documentation and to manage timesheets during the development process. So we would track the detail of the task, how many hours they are taking, they took in a given particular feature, right? And we used that modern technology. We developed the application using Bedrock as our LLM in the backend for Amazon services. We used Fargate, Node.js.

We used Fargate, Node.js. The application was built using React, Ionic, and Capacitor and deployed using GitHub Actions to both Apple and Google Play stores. Of course, we didn't deploy externally, only internally.

The most important part of our experiment was in the very beginning when we set up hypotheses on the expectations we had. We anticipated a 50% increase in speed and a 30% reduction in capacity with AI usage. We also expected reduced technical expertise requirements and equivalent code quality to non-AI teams. The experiment results showed a 45% productivity gain with AI, delivering applications 40% faster with a 30% reduced capacity.

The Python engineer successfully worked with TypeScript, a new tech stack for him, showing that AI helped him be productive with a new language. Both codebases maintained good quality and had minor issues to address, with no major security vulnerabilities found. The AI team's productivity gain and rapid application delivery showcased the effectiveness of integrating AI tools like Coursera and GitHub Copilot in software development projects.

So it was a new language, a new tech stack for him. We expected the quality of the codebase to be at least the same quality of the no AI team quality of the codebase. So basically we used Sonar on both codebases to do static analysis on those codebases, and we compared the outcomes of Sonar to understand if the codebases were as good as at least without the use of AI.

And of course, we expected to have learnings, to learn a lot on the things we should do and the things we should avoid while using AI for coding. The results of the experiment were pretty good. We had a 45% productivity gain using AI. The team that developed using AI delivered the application 40% faster than the team that was working without AI, at a 30% reduced capacity. A team of two people developed the application almost at the half of the time of the team with three people that was not using AI.

The Python engineer was able to be productive and deliver applications using TypeScript without coding in TypeScript during the project. AI helped him to be productive using a language that he never worked with before. Both codebases had minor issues to address, with no major security vulnerabilities found. However, the Python engineer faced challenges when working on Amazon Web Services, especially around application load balancers setup in the architecture, indicating that clear understanding of AI's capabilities is crucial for optimal performance.

So the developer was not required to code TypeScript during the project. He was required to think on the features, to think on how to implement the application. So AI helped him to be productive using a language that he never worked at before. That was pretty important to us as well.

And the quality delivered by both codebases were pretty good and they were similar. Both codebases had a couple of issues here, a couple of bugs there. You know, normal stuff, nothing very complicated to fix. None of those codebases had major security hotspots. So we could evaluate both codebases and they were pretty much the same in terms of quality.

But of course nothing is perfect. We also noticed that the same Python engineer that I mentioned before, I asked him during the development to act as a DevOps person working on Amazon Web Services, especially around application load balancers setup in the architecture. And that was a point he struggled a bit. Not because the ability of using AI didn't help much because he was not clear on what he wanted to develop in that moment. He was not aware that he should use a application load balancer versus a API gateway, for example.

So the ability of knowing exactly what you want to ask AI to do, reduced his performance. But he was able to deliver the feature. It just took longer than a person that was not using AI, the DevOps person of the other team. So this proves that AI can do the job, but you as an engineer, you must know exactly what you are asking for AI to work for you. You cannot just, hey, you know, build this for me. You need to be more clear on your output, the output that you are expecting. What is the application you are developing, the decisions you make, your software architecture decisions.

Provider, if you're going to use like cloud as a provider, Amazon, the things you want to use on Amazon, those kind of things, you still need to make decisions and provide that as a context to the LLM. And it taught us a pretty important thing. And I want to be clear on the difference between the vibe, coding hype or the nature of vibe coding, which was defined at early this year. And what we we did there at part of the experiment. Right. So AI, AI coding or agent coding for us means using AI to speed up the process of writing the code. But the human in the loop must be a person that is skilled enough to do to make decisions on how I should work and skill enough to do code review, to do the review of the AI generated code. While vibe coding says that you just rely on AI, you ask for something and you don't review the code. We don't believe on that. We don't trust on that. It's not the way that we want to work at ModuScript and the way that we want to teach people to work on. So for us, the human being is still required.

And the human being is still required to be a soft engineer, to be a person skilled on engineering, on writing applications. And that person is still the rock star on this workflow. This person is required to do the planning. This person is required to review the code generated by AI. This person is required to execute the process of validating the application and the code that was generated by AI to make sure that that application is ready to go to production. So the end of it is basically AI is not a replacement for architectural clarity or engineering expertise, but can be a multiplier of it. So as long as you still focus on having pretty good engineers and you still focus on scaling up your engineering team to use AI, those engineers can do a better job by using AI.

If not 100% of the time, but pretty much most of the time they can use AI to brainstorm, to make decisions, to write an implementation plan and to ask the AI agentic coding tool, cursor, copilot, whatever the tool you decide to use. To make sure that tool is going to speed up the development process for you by the correct and respectful management of the person. So what are the key learnings and the workflows we got there? First, for me, the best prompt, the best way you communicate with AI is being clear and concise on what you expect AI to do for you. So going to copilot and being very clear, crystal clear on what to expect AI to do for you is the way that I see most productivity gains and most high quality professional results out of it. And for me, it starts with having atomic level tasks. Basically, imagine you have a user story on your JIRA board that you want to use, that you want to implement, right? That story needs to be broke down into smaller units. It can be sub-tasks and from the sub-tasks you can create an implementation plan, which contains, which is essentially for me implementation plan is a markdown file on your repository. And that markdown file, you define atomic level, big step actions that you want AI to go over in an interactive way. So basically you create that implementation plan, you can use that to other AI tools to craft that implementation plan. And then you review it and you are sure that's the plan you want to follow. You start asking AI to go over it and working on those bullet points.

On each bullet point, when AI is done, you review the code base, you run validation steps such as linter, unit tests, tools you have available on your command line to review the quality of the code base. And then if you are happy with the output, you accept. If you are not happy, you provide feedback. When you are happy, then you commit that to Git and it saves you if in the next prompt AI hallucinates and start deleting code. So committing often, working in smaller units, smaller baby steps in interactions way is the way that we found that ModuScript in that experiment be the most productive way to work with coding agents.

Of course, giving context to the LLM is also very important. And by giving context, I mean not a large context, but a rich context with narrow laser focus that information that is related to the task you want to accomplish. Too much context might make your AI to start hallucinating. That's not something we want. So finding ways to feed the LLM with the correct context for the task is also a skill that the human being, the software developer managing the tool must develop now.

Now, of course, there are different ways to do that. Using instruction files to set up coding patterns that are agreed upon the team or having MCPs to feed the LLM with external context from Stack Overflow, a Google search, a perplexity query. Or, of course, you doing a good job on writing the prompt. And of course, cleaning the context after a couple of interactions because it starts growing and hallucinating. Some good prompts. Of course, being clear and concise on explaining what you expect the AI to accomplish for you. Not just writing, hey, do something for me, right? Write SAS here or write SAS for that component.

You need to be more clear. You need to be narrow and laser focused on what you want the AI to accomplish. For me, I don't use that much of prompts because I really want to use the implementation plan. And implementation plan is essentially my prompt at the end of the day, right? So having an implementation plan with big steps and clear and concise action items there helps a lot because you can go to the LLM and say, hey, take a look on the implementation plan and let's start working through that.

This validation loop for me is key. It's probably the best thing we learned because it keeps me focused, keeps me productive, and gets me good results. So I ask AI to start working on the task. That's the first step there. The AI, the LLM generates some code for me. That task being a narrow, laser focused, big step task helps me to have a smaller batch to review instead of a large batch. It would be like a 100 files per request. That would take me a longer time to review and probably start missing stuff after a while.

So doing those validations, the automatic validations from the command line, is also pretty important. And also, I'm happy I do the Git commit. I often commit so I avoid losing code because of AI craziness at some point. Test-driven development is one of the workflows that I really like. It helps a lot because everything that I just said about narrow, laser-focused tasks, that's TDD. TDD is based on writing smaller steps, on coding with tasks in smaller units. Debugging is also important.

Let's say I use Lizard to calculate the complexity, the cognitive complexity of a function. And then I use that feedback on the LLM so it knows what's wrong and then fixes it.

So doing those validations, the automatic validations from the command line, is also pretty important. And also, I'm happy I do the Git commit. I often commit so I avoid losing code because of AI craziness at some point. Test-driven development is one of the workflows that I really like. It helps a lot because everything that I just said about narrow, laser-focused tasks, that's TDD. TDD is based on writing smaller steps, on coding with tasks in smaller units.

You write the test, you write the function, and then you try to get the function to pass the test. Those smaller actions are very productive to do with AI as well, especially when you are building complex components with different conditions or building a complicated set of functions of a business model. Debugging is also important. It's also very good using AI. You can use an MCP to grab CloudWatch logs, Grafana logs, external information about a problem you are trying to solve.

And then you ask AI to not code, but to figure out what's wrong in the code base based on those logs. And then once you have a diagnosis on what's going on there, you can ask AI to write an implementation plan to fix it. And then you go over that implementation plan. So for me, AI is not magic. But with clear prompts, narrow-focused tasks, rich context, a validation loop that makes sure your quality is kept, it can be an unfair advantage.