FullStack Monitoring with Open Telemetry: End-to-End Observability for Modern Applications

Hello, Amsterdam! Cês tão bom? We need some energy after lunch, right? This is my fourth time here. I've been traveling a lot and I came all the way from Brazil and I just want to learn some things. Are there Brazilians here? Woo-hoo! Nice! So, I'm going to show you some experiments—not experiments, but some ideas that I got from the field.

So, I've been a performance developer for a long time, and I know everyone is speaking about AI and AI, but I know there are a lot of other stuff that is still important. And I'm going to try mixing both, so I hope you enjoy it. I know it's going to be a lot of content and I won't be able to show everything I want to show you here, but you can access the slides later, the references, and so on, so forth.

Perfect. AI. AI. AI. I've seen people say, well, I'm so bored about this keyword that people are just saying out there. I'm not actually bored, but I'm a little bit worried. It seems we have a new standard, right? What is the standard? The standard is applications crashing everywhere. I have no idea how many times I've seen this page this week and I've seen all the main services are down. And this is not only happening to GitHub. I'm just showing as an example. This seems to be a new normal. I've seen applications crashing, and I don't know about you, but when I started, it was pretty bad to have a downtime. It was pretty irresponsible, I would say. But now it seems like it's okay. So it's not only with GitHub. You can see a lot of other providers. I know that AI is changing a lot. People are just pushing and pushing more code, but still, we can spend more machines or do something, right, to avoid breaking out. The other thing for me, which is really bad, it's the vulnerability. I've never seen so many posts about being vulnerable, data leaks, and more. And I don't know, people used to tell me, well, you are not afraid about your data on the internet. I was like, yeah, Google knows me and knows everything I have. But now with these leaks, I'm not sure anymore. Right.

Blackmails can happen. Scams may happen. And it's a pretty scary moment that I don't see a lot of people talking about. I think one of the biggest things we had and we were blaming NPM and a lot of stuff, but this is happening, it's supply chain attacks. And this actually got into OpenAI stuff and chattpt codecs. So when I was trying to use it these days, I was like, this is a malware. How scary is it, right? But yeah, 2026, we just started, it was like, yeah, AI is going to solve all the problems. This is very nice. But in the reality, it was like, oh, my God, this is fine. Everything is firing down. Sometimes people don't even realize they are down. So the way here in this talk is to show you how to avoid this and how you can use data in your favor without having to spend a dime.

OK, so as I told you, I've been a software developer for over 10 years now, but I've been focusing on performance applications, mostly Node.js. And I started doing some consultancies in Brazil, mostly helping companies to survive Black Friday events and campaigns. And I usually try doing a report for them, pinpoint what they can have for quick wins and much more. So here is an example of what we do. Usually this is actually my YouTube video showing, but normally we can inspect how the application code is going. What is the algorithm? If we have any problem with libraries, in cases that users have an APM, an application performance monitoring platform or tool, we can get a lot of different queries and response times and much more. And this helps a lot with the investigation. And still we know that APMs are generally really expensive. So in my case, I opted to open source saying I have a Kubernetes cluster there running and I'm not a DevOps guy. I'm a software developer who is very curious. But here you can see that we can do everything really for free using SDKs, open source. And this is actually a web application, right? We usually see examples with back end, but this works also in the front end.

Well, when I'm trying to investigate a performance issue with some customer, I think the biggest bottleneck for me is that they have different data sources. Logs I generate in one side, metrics in other, tracing in others. If you haven't known these words, I'm going to explain later too. But see, I was getting all this information about some problem that got there. I get the alerts. So once I have an alert, I know what to tackle first.

And after getting all this data, I have to run the application locally. I can have stress tests, run flame graphs, CPU profiles, and a lot more to finally have the final report to send to the customer. See, this is a work like it would take a week depending on how complex is the problem. And we know that in production, things are not so happily well managed, right? We know that stage environment production and dog food have the same application, but it's not the same because the data is different and sometimes they behave differently. So this is something really hard when trying to observe applications. And some customers don't even have like a APM platform. So we have to do everything by hand. Running a load test locally, we have to actually try getting local metrics, so this is a very good library called Clinic JS that I have some examples there too. So you can detect what your application is doing locally and actually try getting the CPU profile, analyzing how long the function is taking in process and so on and so forth. Nothing related to OpenTelemetry so far, but I'm just showing how it's generally depressed.

So as I told you, I have to gather all this data and then I have some final report and we have to do a lot of assumptions. Nothing is perfect and sometimes we are not 100% sure that the cause of the performance issue was the problem. So after doing all of this, we came up with a report that we can send to a customer. This one I did for Meeture. So Meeture is a framework, the older guys and girls will remember this one, but they're alive so I started helping them to make the platform a little bit faster and reliable. Okay, AI. So all of this I did it manually, but now I'm going to show you how we can do this way faster using AI, but not only AI, but data itself too. So OpenTelemetry is the whole thing here. I think it's one of the best things humanity has created because it was really a community.

You're going to see how many companies are helping on this. And for me, I would say Docker was the best one for my life, I would say. It's one of the biggest ones, but OpenTelemetry is one too. So just to know how many of you have been working with the SDK of OpenTelemetry? Nice, nice, nice. So a lot of you are going to see this first hand. One of the things that I started researching about this cool protocol and library that works in any program languages, any platforms, is how companies are contributing to each other. Look how many of the bigger ones are here. Google, Splunk, New Relic, Datadog, all of them are trying to reach a standard. So any of us can benefit from this. We don't have to spend a dime. It's under a foundation and it's really reliable for us.

So the first topic we can actually get is logs. So I'm going to show you some of these concepts. So logs. What is a log? Here, you usually use a console.log, but the problem is you don't have the full context. So right there, with OpenTelemetry, I can see an error, I can see the stack trace, I can see when it happened, and I can try correlating different events from there. So you can imagine docs are one side, but we have others too.

Then we have metrics. Metrics we usually use to measure time during some periods of events. So here, I'm trying to see what is V8 doing, the Node.js life cycle, and I can see the endpoints as well, where I can see routes, the time for the response, and we can start giving some grasp. From there, this is the one that I use most, I would say. This is the tracing that we call. So we can track since the first event up until the end. Look how amazing is this. We can see here, I'm starting on API, right? It's on backend side, but it could have started on the browser.

So I could click like purchase, and from the purchase, it's going to go to the backend, it's going to go to the database, from the database back to the backend, back to the frontend, and you can see all these three, and you can see, wow, probably my application is not that slow, but I'm using a third party API that is ruining everything. So this can have and can show us a little bit of more and more context, and this is where I spend most of the time. Finally, when you have data, you can build dashboards. So here I have the dashboard for all services, but what I use most with customers is looking at the errors or see the request count, and everything's here. So once you have the data, everything just turns out to be way easier, right?

So first of all, I'm using only open source tools. This is the reality from my own infrastructure that I have over 10,000 students accessing every day. So your app is using the SDK, it's just sending the data to the OpenTelemetry Collector, which is like a service hub, just to send the data to the other databases, because they are optimized for different kind of data. So metrics goes to Prometheus. Let's say logs goes to Grafana Locking. Tracing goes to Grafana Temple. And finally, we use the Grafana as the main hub to access all this data, build dashboards, build alerting, and so on and so forth.

Okay, when you start, or when you think at home, the first thing is to realize what libraries you use in production, and you can try checking the instrumentation for this library. And look how amazing is this protocol. We just add the instrumentation right there, and we can just start stacking up for every library we have. Because we probably want to measure how fast is the database, how fast is the other responses, and we can just start stacking up all those libraries, and we get everything just in place. One thing that is very nice is that OpenTelemetry has a demo there that you can see all the microservices running, front-end, back-end, other different services, and you can see it clearly in the dashboard.

So this is really helpful, but I won't go too far on this, because we are here to talk about AI, AI, AI, right? So once we have the data, once we know what we probably want to see, then we can start using AI to make our lives easier. So for me, doing this investigation, trying to work with customers, for me everything changed after AI, when doing this kind of investigation. One thing I want to tell you guys, I don't know if you've suffered of this too, but has anyone, anybody else got hacked about their React 2.0 shell vulnerability? So you know, this was a vulnerability on React that people were able to send scripts and just execute shell commands from your page.

One thing I want to tell you guys, I don't know if you've suffered of this too, but has anyone, anybody else got hacked about their React 2.0 shell vulnerability? So you know, this was a vulnerability on React that people were able to send scripts and just execute shell commands from your page. So there they can write this very ugly package there, or script, and we just use like a fetch, which goes to your React application, in my case it was a Next.js application, and they were able to execute or download a file from there. It was pretty scary. I just got to know because I got an alert with 90% of users on my cluster. So I use AI for this, very basic one, I just asked on my LLM, it was like, well, why is my application spending so much CPU? And then it did a lot of investigation, ran a lot of commands on my cluster, and I got the report. So in the end, I was like, wow, this is going to be content. I was like, nice. I did a YouTube video showing all this. I was like, no, no, keep there, let's see what they are doing. But in the end, they were trying to mine some crypto on my cluster, but hopefully I was using Kubernetes, and it was all isolated. So they were able to get in, but they were like very lost there. But it was nice. It was for me a realization, like, wow, imagine if I didn't have this, how would be the problem to get and finally understand what is the thing going on there? Okay, so the AI was using the Kubernetes CLI and was able to get a lot of response and back and forth, but see, it's not correlating my data. Logs could have some information there, probably metrics I can get any more and more. So what if we can leverage the skills, MCPs, and more that AI is showing us? So I could use Grafana MCP to try relating all the data. I could use Context-7 to get the most updated code and check and ask the LLM to fix this bug or issue. I can ask the GitHub MCP server to create the issue or even the pull request for me. And in my case, I could finally use the Google Docs MCP to create the final report for me. See, before I was opening a lot of different tabs, now I can do everything from my own editor. So, poof, it's really fast. So in this case, I'm showing Grafana, it's open source, running on locally. So the first thing here is that they have a lot of different tools to access all the data we have. So once we have the tools there, you can see, my God, it's a lot. This must spend a lot of tokens, I think. But with skills, I think this could be nice. But see, we can access logs, metrics, tracing, and we can try correlating all of them from here. So my investigation generally starts looking at the dashboards I have. And now I can see we have a lot of errors for this endpoint. And I can go to my editor and ask, well, I'm having this problem here. Try to investigate.

Once we have the MCP and all their tools configured, this process is going to be way straightforward for me. Look, how amazing is this? So I have here my ID. I'm just saying all the prompts. Now I'm trying to correlate data. So I have a more specific prompt here. But the final one here is for showing the report, creating the report itself for me. So it's asking a lot of questions. But see. Boom. I have the report ready. Imagine I was spending eight hours to make something similar. Now I have every information for different data sources right in my hand. And I even have the solution for debug. So this was a problem that I created on purpose, which was exhausting the database connections. But we can see which line was happening and why, when. And, well, for me, this changed a lot of my daily work when working. So I would say the major point of this is thinking about how you can correlate your data. So data is the most important thing. We're going to go there. So to summarize, I would say OTEL, or Open Telemetry, is the standard. So you can use in any different languages. I've been using a lot with TypeScript, with JavaScript. You just have to check out the libraries you want to install there. But it's way easier to use. It may work with other APMs. Let's say you don't want to go self-hosted, because you're going to probably spend time and money on storage and data. Sometimes it's complicated to manage. But you can use the same protocol, the same library, and just move from one provider to another, because it's a standard and it's an open protocol. And data. Data is good, right? Data is always good.

And try tracking a lot, as much as you can, of data. And finally, observe your apps. Don't be one more AI company that just breaks in the middle of the day that people won't trust. And the data is here. It's pretty straightforward for us. There are a lot of tutorials out there. And we can get a lot from it.

All right. So here, we have all the resources. You can find all the slides. And before we go to the final questioning, I would like to take a selfie with you. What do you think? Woohoo! All right. So, as a tradition, I'm going to start from this side. And we're going to try doing some Ola. What do you think? I need some energy right now. Right? Woohoo! So, I'm going to start from this side. So, one, two, three. Woohoo! I'm going to do once more because I want more energy. Right? So, one, two, three. Thank you so much, Amsterdam!

What I see up on here is the idea of native support, actually, for Otel. So, for open telemetry in browsers, do you think this is going to be, like, standardized or put into some kind of browser APIs or something like that? Have you seen anything like that? No, I wish. I wish it was. But now I think Chrome has a performance API there that you can observe. But what people usually do is try just using the SDK. But it would be a nice one. Okay. So, there's no kind of, like, from any browsers, you haven't seen any signals on that? No. Okay. Yeah. That would be an interesting thing.

And then the other thing is, do you believe that introducing LLMs to analyze the open telemetry, do you think it's going to create, for example, more work? Or because there's some kind of problems or inconsistencies in the data? You know, like, AI can make mistakes all the time. I would say this is a very nice question because when I was doing by hand, I usually get the first trace ID, and I try moving around and try gathering this one-to-one. When I'm using some LLM that has context, has data from other places, in my mind, I was more accurate because I'm not just analyzing a single trace. I can analyze a group. I'm gonna spend tokens. That's fine. But in the end, I can have different perspective, because sometimes I'm just running a load and I cannot just rely on a single ID, for example, right? So I don't think it's more work, but still, we have to check by hand. You have to verify. Yeah. What kind of steps do you go through? Like, just as a follow-up to this, what kind of steps do you go through to verify? I try executing the response. Like in this example, oh, you have a problem creating connections. Just change the number. I was like, okay, I'm gonna try changing. Run once again and see how the results are, and if the number ever change. If change, okay, I believe because I have a proof in the end, right? Yeah, yeah, okay. And one thing here is, how can you hook up, like what are the actual steps that you take to hook up LLMs to your OTEL data? Oh, it's really nice. In this case, I only had the Grafana MCP, which is running on my Docker Compose, so it could be on Kubernetes and everything else. And because Grafana is the hub for those tools, which they own too, it's open source, but they own, so Grafana is just the initial point for it. So I didn't have to do much. I just configured the MCP and asked questions and it's able to go there. Okay, nice, yeah. Still, you could use skills and other better approaches to save tokens, but I was trying to go the easy way. Okay, okay, nice. Let's see, what else do we have here? Also, you demonstrated the use of this LGTM stack from Grafana. And what are your thoughts on other open source stacks like this Cygnos? I've seen like Open Observability, I've seen like ClickHouse you can use, that is a bunch But in my case, most of the investigation, as it's free, as it's open source, is the major tool I've been using a lot. I actually, I would say the past six months I was working for an APM company and the base was OpenTelemetry 2.0. We had our own dashboards, but even there I was using Grafana because I had more control of the data for some reason, I don't know. Right, right, nice, okay. Did you, like this is maybe a harder question to answer, but it says can we get the prompt that you use for the AI analysis? Like how would you recommend prompting about those types of analysis things? I think like as any LLM processes, I actually start typing what I will do. So in my manual steps, I was like go to the logs, go to the metrics, go here and try getting all the data.

So those are the steps. I think I put in the resources the project if I'm not mistaken, but we have the GitHub open there, we have the prompt, but honestly it's like it's harder to get a result when it's inconclusive. So sometimes the error is intermittent, and sometimes it happens now and it's going to happen just in a few hours and you're not sure if it's a delaying problem, if it's a third party problem. So this kind of stuff is more tricky, but generally it's not that hard.

Okay, so for coming up with the, like what I'm hearing is for coming up with these prompts, you have to have some kind of background knowledge of what steps to go through and everything. Which I think it's the same for developers, right? People are saying, oh we're going to be replaced. No, no, no. We are just using our mind, trying to download our brain, because we know the steps and we just tell the machine how to do the steps. Right, right. Yeah, yeah, that makes sense.

What about the browser integration for OTEL? How mature would you say that is? Well, they say it's experimental, but it's for years right now. I'm not sure if there's a lot of people there. But for this demo, I was trying to get the, I could get the same metrics that I got from Web Standards. So it's the first time to page, to download the page, so you can have the downloads, but I'm not sure if it replaces the analytics. So this is something that I will try out. To be honest, I worked mostly in the back end side for this. But still, I could get basic metrics that work for me, and I can have something near to the Web Standards one that shows your SEO speed and so much more.

Yeah, yeah. Well, I mean, if it's been around for years and it's still in this same stage, then probably a lot of people and companies are using it, and so there's some kind of level of battle tested there. Like the npm package for the web, I thought I saw like 16 million downloads per month. Right. Even though the doc says it's experimental, everyone is using for a while the demonstration too, so I think it's safe. So this is a good package for them to compromise, right? Exactly. They can take this one over, and then 16 million downloads, right? Yeah. Yeah. Yeah, what else do we have here?

So also, when you go through, so just a little bit deeper into the kind of review that you do of the AI reporting, do you review like all the details? Like you go through each thing kind of like in detail? Because then the question here says, because it would take a while to review everything. So yeah, what would be your workflow there? I would say, like working with customers, they have to believe me by seeing in practice. So usually when I write the report, I try creating reproducible examples. So I try doing like a test, like a unit test for my proof there.

So okay, if you have this, let's say, if you have this synchronous function, when you run the stress test, you're going to see that this function is going to keep on CPU for x seconds, and the value is going to be this, this, and that. So this kind of approach, it's what helps out. However, I would say to you like 99% of the performance issues I solved was very silly. It's like, oh, AVG was just outdated. When we updated, the speed was just crazy amounts. So I think the whole topic about this talk is about finding quick wins. So you don't have to change a lot of applications, but quick wins, you can see, well, where's the most used endpoint in this company? What's the slowest query? If you can change, you can extract a lot of value from that. Yeah, low-hanging fruit, right? Yeah, yeah. Yeah, those unit tests kind of sound a little bit like AI evals a little bit, like to evaluate what the AI has been doing. To be honest, I was doing manually. I was like, this is the best script, let me run the best script. Yeah, well, any kind of testing is good, yeah, yeah. Yeah, and maybe one last one here.

How do you know if you are over-logging, like logging too much data? You don't. Okay. What I've done with some customers was like trying to find... We deliver the first one and they complain about the storage, and then we start optimizing. And this is like always a cycle because you can never finish optimizing. There's always room to optimize, optimize, and you can get crazy because in my case, I was like, wow, I can change this, but some changes, the final customer don't even realize the change. So this is some balance in the middle, but still, modeling the data on logs, I think there should be some special people from data that can say, well, this is important, but like the environment is not important. I cannot do this in that. So more experts and more experts. Yeah, yeah. That makes sense. Yeah, I mean, so the answer to the question is when your hard drive gets full or if you have some expert that can help you? Just a feeling around this. Usually your disk is not going to be full because we have like the TTL. So it's like, well, the data is going to survive for just a week. After a week, it's removed and we can have the cycle. Because usually logs and telemetry, we can see now and we can compare now. Yeah, OK. Cool. Well, hey, thanks. Thanks so much for the talk. Yeah. Give another hand for Eric. Thank you.