FullStack Monitoring with Open Telemetry: End-to-End Observability for Modern Applications

Hello, Amsterdam! Cês tão bom? We need some energy after lunch, right? This is my fourth time here. I've been traveling a lot and I came all the way from Brazil and I just want to learn Are there Brazilians here? Nice! So I'm going to show you some experiments, not experiments, but some ideas that I got from the field. So I've been a performance developer for a long time and I know everyone is speaking about AI and AI, but I know there are a lot of other stuff that is still important and I'm going to try mixing both.

So I hope you enjoy it. I know it's going to be a lot of content and I won't be able to show everything I want to show you here, but you can access the slides later, the references, and so on and so forth. Perfect. AI! AI! AI! I've seen people say, well, I'm so bored about this keyword that people are just saying out there and I'm not actually bored, but I'm a little bit worried. It seems we have a new standard, right? What is the standard? The standard is applications crashing everywhere. I have no idea how many times I've seen this page this week and I've seen all the main services are down.

And this is not only happening to GitHub. I'm just showing as an example. This seems to be a new normal. I've seen applications crashing and I don't know about you, but when I started, it was pretty bad to have a downtime, was pretty irresponsible I would say. But now it seems like it's OK. So it's not only with GitHub. You can see a lot of other providers. I know that AI is changing a lot. People are just pushing and pushing more code, but still, we can spend more machines or do something to avoid breaking out. The other thing for me, which is really bad, it's the vulnerability.

I've never seen so many posts about being vulnerable, data leaks and more. And I don't know, people used to tell me, well, you're not afraid about your data on the Internet. I was like, yeah, Google knows me and knows everything I have. But now with these leaks, I'm not sure anymore. Right. Blackmails can happen. Scams may happen. And it's a pretty scary moment that I don't see a lot of people talking about.

I think one of the biggest things we had and we were blaming NPM and a lot of stuff, but this is happening. It's supply chain attacks. And this actually got into OpenAI stuff and chatAPT codecs. So when I was trying to use it these days, I was like, this is a malware. How scary is it? Right. But yeah, 2026, we just started. It was like, yeah, AI is going to solve all the problems. This is very nice. But in the reality, it was like, oh my God, this is fine. Everything is firing down. Sometimes people don't even realize they are down.

But so the way here in this talk is to show you how to avoid this and how you can use data in your favor without having to spend a dime. OK, so as I told you, I've been a software developer for over ten years now, but I've been focusing on performance applications, mostly Node.js, and I started doing some consultancies in Brazil, mostly helping companies to survive Black Friday events and campaigns. And I usually try doing a report for them to pinpoint what they can have for quick wins and much more. So here is an example of what we do. Usually this is actually my YouTube video showing, but normally we can inspect how the application code is going, what is the algorithm.

If we have any problem with libraries, in cases that users have an APM, an application performance monitoring platform or tool, we can get a lot of different queries in response times and much more. And this helps a lot with the investigation. And still we know that APMs are generally really expensive. So in my case, I opted to open source and I have a Kubernetes cluster there running and I'm not a DevOps guy. I'm a software developer who is very curious. But here you can see that we can do everything really for free using SDKs, open source. And this is actually a web application. We usually see examples with backend, but this works also in the frontend.

Well, when I'm trying to investigate a performance issue with some customer, I think the biggest bottleneck for me is that they have different data sources. Logs I generate in one site, metrics in other, tracing in others. If you haven't known these words, I'm going to explain later too. But see, I was getting all this information about some problem that got there. I get the alerts.

So once I have an alert, I know what to tackle first. And after getting all this data, I have to run the application locally. I can have like stress tests, run flame graphs, CPU profiles and a lot more to finally have the final report to send to the customer. See, this is a work like would take a week depending on how complex is the problem. And we know that in production, things are not so happily well managed, right? We know that stage environment production and dogfood have the same application, but it's not the same because the data is different and sometimes they behave differently. So this is something really hard when trying to observe applications. And some customers doesn't even have like a APM platform.

So we have to do everything by hand, run your load tests locally. We have to actually try getting local metrics. So this is a very good library called ClinicJS that I have some examples there too. So you can detect what your application is doing locally and actually try and getting the CPU profile, analyzing how long the function is taking in process and so on and so forth. Nothing related to OpenTelemetry so far, but I'm just showing how is generally the process.

So as I told you, I have to gather all this data and then I have some final report and we have to do a lot of assumptions. Nothing is perfect and sometimes we are not a hundred percent sure that the cause of the performance issue was the problem. So after doing all of this, we came up with a report that we can send to a customer. So this one I did for Meeture. So Meeture is a framework, the older guys and girls will remember this one, but they're alive. So I started helping them to make the platform a little bit faster and reliable.

OK, AI, AI. So all of this I did it manually, but now I'm going to show you how we can do this way faster using AI, but not only AI, but data itself too. So OpenTelemetry is the whole thing here. I think it's one of the best things humanity has created because it was really community. You're going to see how many companies are helping on this. And for me, I would say Docker was the best one for my life. I would say it's one of the biggest ones. But OpenTelemetry is one too. So just, you know, how many of you have been working with the SDK of OpenTelemetry? Nice, nice, nice. So a lot of you are going to see this first hand. So one of the things that I start researching about this the school protocol and library that works in any program languages, any platforms, it's how companies are contributing to each other. Look how many of the bigger ones are here. Google, Splunk, New Relic, Datadog, all of them are trying to reach a standard. So any of us can be benefit about this. We don't have to spend a dime. It's under a foundation and it's really reliable for us. So the first topic we can actually get is logs. So I'm going to show about some of these concepts. So logs, what is a log? Here, you usually use a console dot log, but the problem is you don't have the full context.

So right there with OpenTelemetry, I can see an error. I can see this stack trace. I can see when it happened. And I can try correlating different events from there. So you can imagine docs are one side, but we have others too. Then we have metrics. Metrics we usually use to measure time during some periods of events. So here I'm trying to see what is V8 doing, the Node.js lifecycle, and I can see the endpoints as well. Where I can see routes, the time for the response, and we can start giving some grasp.

From there, this is the one that I use most, I would say. This is that tracing that we call. So we can track since the first event up until the end. Look how amazing this is. We can see here I'm starting on API, right? It's on the backend side, but it could have started on the browser. So I could click like purchase. And from the purchase, it's going to go to the backend. It's going to go to the database. From the database back to the backend, back to the frontend. And we can see all these three, and you can see, wow, probably my application is not that slow. But I'm using a third party API that is ruining everything.

So this can have and can show us a little bit of more and more context. And this is where I spend most of the time. Finally, when you have data, you can build dashboards. So here I have the dashboard for all services. But what I use most with customers is looking at the errors or see the request count, and everything is here. So once you have the data, everything just turns out to be way easier, right? So to get an idea of how it works, I won't show you how to install it, but shortcuts should get up front on this subject. So first of all, I'm using only open source tools. This is the reality from my own infrastructure that I have over 10,000 students accessing every day. So I have your app is using the SDK. It's just sending the data to the open telemetry collector, which is like a service hub, just to send the data to the other databases because they are optimized for different kind of data.

So metrics goes to Prometheus. Let's say logs goes to Grafana locking, tracing goes to Grafana Tempo. And finally, we use the Grafana as the main hub to access all this data, build dashboards, build alerting and so on, so forth.

OK, when you start and or when you think at home, the first thing is to realize what libraries you use in production. And you can try checking the instrumentation for this library. And look how amazing is this protocol. We just add the instrumentation right there and we can just start stacking up for every library we have because we probably want to measure how is how fast is the database? How fast is the other responses? And we can just start stacking up all those libraries and we get everything just in place.

One thing that is very nice is that open telemetry has a demo there that you can see all the microservices running front end, back end, other different services. And you can see it in the dashboard. So this is really helpful. But I won't go too far on this because we are here to talk about AI, AI, AI, right? So once we have the data, once we know what we probably want to see, then we can start using AI to make our lives easier.

So for me, doing this investigation, trying to work with customers, for me, everything changed after AI when doing this kind of investigation. One thing I want to tell you guys, I don't know if you suffer of this, too, but has anybody else got hacked about the React2Shell vulnerability? Just so you know, this was a vulnerability on React that people were able to send scripts and just execute shell commands from your page. So there they can write this very ugly package there or script, and we just use like a fetch which goes to your React application. In my case, it was a Next.js application. And they were able to execute or download a file from there. It was pretty scary. I just got to know because I got an alert with 90% of users of my cluster. So I use AI for this very basic one. I just asked all my LLM, I was like, well, why is my application spending so much CPU? And then it did a lot of investigation, ran a lot of commands on my cluster, and I got the report. So in the end, I was like, wow, this is going to be content. I was like, nice. I did a YouTube video showing all this. I was like, no, no, keep there. Let's see what they are doing. But in the end, they were trying to mine some crypto on my cluster, but hopefully, I was using Kubernetes and it was all isolated. So they were able to get in, but they were like very lost there. But it was nice. For me, it was a realization like, wow, imagine if I didn't have this, how would be the problem to get and finally understand what is the thing going on there? OK, so the AI was using the Kubernetes CLI and was able to get a lot of response and back and forth. But see, it's not correlating my data. Logs could have some information there, probably metrics I can get any more and more. So what if we can leverage the skills, MCPs, and more that AI is showing us? So I could use Grafana MCP to try relating all the data. I could use Context 7 to get the most updated code and check and ask the LLM to fix this bug or issue. I can ask the GitHub MCP server to create the issue or even the pull request for me. And in my case, I could finally use the Google Docs MCP to create the final report for me. See, before I was opening a lot of different tabs. Now I can do everything from my own editor. So it's really fast. So in this case, I'm showing Grafana. It's open source, running on locally. So the first thing here is that they have a lot of different tools to access all the data we have.

So once we have the tools there, you can see my god, it's a lot. This must spend a lot of tokens, I think. But with skills, I think this could be nice. But see, we can access logs, metrics, tracing, and we can try correlating all of them from here. So my investigation generally starts looking at the dashboards I have. And now I can see we have a lot of errors for this endpoint.

And I can go to my editor and ask, wow, I'm having this problem here. Try to investigate. As we have the MCP and all their tools configured, this process is going to be way straightforward for me. Look, how amazing is this? So I have here my ID. I'm just saying all the prompts now I'm trying to correlate data. So I have a more specific prompt here. But the final one here is for showing the report, creating the report itself for me.

So it's asking a lot of questions. But see, boom, I have the report ready. Imagine I was spending eight hours to make something similar. Now I have every information for different data sources right in my hand, and I even have the solution for the bug. So this was a problem that I created on purpose, which was exhausting the database connections. But we can see which line was happening and why, when. And well, for me, this changed a lot my daily work when working.

So I would say the major point of this is thinking about how you can correlate your data. So data is the most important thing. We're going to go there. So to summarize, I would say OTEL or OpenTelemetry is the standard. So you can use in any different languages. I've been using a lot with TypeScript, with JavaScript. You just have to check out the libraries you want to install there. But it's way easier to use.

It may work with other APMs. Let's say you don't want to go self-hosted because you're going to probably spend time and money on storage and data. Sometimes it's complicated to manage. But you can use the same protocol, the same library, and just move from one provider to another because it's a standard and it's an open protocol. Data is good, right? Data is always good. And try tracking a lot, as much as you can, of data.

And finally, observe your apps. Don't be one more AI company that just breaks in the middle of the day that people won't trust. And the data is here. It's pretty straightforward for us. There are a lot of tutorials out there and we can get a lot from it. All right. So here we have all the resources. You can find all the slides.

And before we go to the final questioning, I would like to take a selfie with you. What do you think? All right. So, as a tradition, I'm going to start from this side and we're going to try doing some ola. What do you think? I need some energy right now, right? So I'm going to start from this side. So one, two, three. I'm going to do it once more because I want more energy, right? So one, two, three. Thank you so much, Amsterdam.

What I see up on here is the idea of native support, actually, for OTEL. So for OpenTelemetry in browsers. Do you think this is going to be like standardized or put into kind of some kind of browser APIs or something like that? Have you seen anything like that? No, I wish. I wish it was. But now I think Chrome has like a performance API there that you can observe. But what people usually do is try and just using the SDK. But it would be a nice one. OK, so there's no there's no kind of like from any browsers. You haven't seen any signals on that. OK. Yeah, that would be that would be an interesting thing.

Another thing is, do you believe that introducing LLMs to analyze the OpenTelemetry, do you think it's going to create, for example, more work or because there's some kind of problems or inconsistencies in the data? You know, like I can make mistakes all the time. I would say this is a very nice question, because when I was doing by hand, I usually get the first trace ID and I try moving around and try gathering this one to one. When I'm using some LLM that has context, has data from other places, I in my mind, that was more accurate because I'm not just analyzing a single trace. I can analyze a group. I'm going to spend tokens. That's fine. But in the end, I can have different perspective because sometimes I'm just running a load test and I cannot just rely on a single ID, for example. Right. Right. So I don't think it's more work, but still we have to check by hand. You have to verify. Yeah.

Yeah. What kind of steps do you go through? Like, just as a follow up to this, what kind of steps do you go through to verify? I try executing the response like in this example. Oh, you have a problem creating connections. Just change the number. I was like, OK, I'm going to try changing run once again and see how the results are. And if the number of errors change, if changed. OK, I believe because I have a proof in the end, right? Yeah. Yeah.

OK. And one thing here is how can you hook up? Like how what are the actual steps that you take to hook up LLMs to your to your hotel data? Oh, it's really nice. Like, in this case, I only had the Grafana MCP, which is running on my Docker Compose. So it could be on Kubernetes and everything else. And because Grafana is the hub for those tools, which they own too, is open source, but they own. So Grafana is just the initial point for it. So I didn't have to do much. I just configured MCP and ask questions and it's able to go there. OK, now you could use skills and other better approach to save tokens. But I was trying to go the easy way. OK, OK, nice.

Also, you demonstrated the use of this LGTM stack from Grafana. And what are your thoughts on other open source stacks like this Cygnos? I've seen like open observability. I've seen like Clickhouse you can use. That has a bunch of... But in my case, most of the investigation, as it's free, as it's open source, is the major tool I've been using a lot. I actually, I would say the past six months, I was working for an APM company and the base was OpenTelemetry too. We had our own dashboards, but even there I was using Grafana. OK, because I had more control of the data for some reason. I don't know.

OK, because I had more control of the data for some reason. I don't know. Right, right. Nice.

OK. And did you... Like, this is maybe a harder question to answer, but it says, can we get the prompt that you use to do for the AI analysis? Like, how would you how would you recommend prompting about those types of analysis things? I think like, as any LLM processes, I actually start typing what I will do. So in my manual steps, I was like, go to the logs, go to the metrics, go here and try getting all the data. So those are the steps. I think I put in the resources, the project, if I'm not mistaken. But we have the GitHub open there. We have the prompt. But honestly, it's like it's harder to get a result when it's inconclusive. So sometimes the error is intermittent.

Oh, my God. You got it right. Yeah. And sometimes it happens now and it's going to happen just in a few hours. And you're not sure if it's a delaying problem, if it's a third party problem. So this kind of stuff is more tricky. But generally, it's not that hard. OK, so for coming up with like what I'm hearing is for coming up with these prompts, you have to have kind of some kind of background knowledge of what steps to go through and and everything. Which I think is the same for developers. Right. People are saying, oh, we're going to be replaced. No, no, no. We are just using our mind, trying to download our brain and because we know the steps and we just tell the machine how to do the steps. Right, right. Yeah, yeah. That makes sense.

What about the browser integration for hotel? How mature would you say that is? Well, they say it's experimental, but it's four years right now. I'm not sure if there's a lot of people there. But for this demo, I was trying to get the I could get the same metrics that I got from Web Standards. So it's the first time to page to download the page. So you can have the downloads. But I'm not sure if it replaces the analytics. So this is something that I would try out. Be honest. I work mostly in the back end side for this. But still, I can go. I could get basic metrics that work for me and I can have something near to the Web Standards one that shows like your SEO speed and so much more. Yeah. Yeah. Well, I mean, if it's been around for years and it's still like in this same stage, then probably a lot of people are people and companies are using it. And so there's some kind of level of battle tested. Yeah. Like the NPM package for the web. I thought I saw like 60 million downloads per month. Right. Even though the doc says it's experimental, everyone is using for a while. Yeah. The demonstration to. So I think so. So this is a this is a good package for them to compromise. Right. Like they can they can take this one over and 16 million downloads. Yeah. Get. Yeah. Yeah.

What else do we have here? So also when you when you go through. So just a little bit deeper into the kind of review that you do of the AI reporting. Do you review like all the all the details like you go through each each thing kind of like in detail? Because then the question here says because it would take a while to review everything. So, yeah. What would be your workflow there?

I would say like work with customers. They have to believe me by seeing in practice. So usually when I write the report, I try creating reproducible examples. So I try doing like a test, like a unit test for my my my proof there. So, OK, if you have this the let's say if you have this synchronous function, when you run the stress test, you're going to see that this function is going to keep on CPU for six seconds and the value is going to be this, this and that. So this kind of approach is it's what helps out.

However, I would say to you, like 90, 90 percent of the performance issues I saw was very silly. It's like all AVG was just outdated. When we updated, the speed was just crazy amounts. So I think the whole topic about this this talk is about finding quick wins. So you don't have to change a lot of application, but quick wins. You can see, well, where's the most use endpoint in this company? Yeah. What's the slowest query? If you can change, you can extract a lot of value from that. Yeah. Low hanging fruit. Right. Yeah. Yeah. Yeah.

Those unit tests kind of sound a little bit like A.I. evaluations, a little bit like evaluating with the A.I. To be honest, I was doing it manually. I was like, 'This is the best script. Let me run the best.' Any kind of testing is good. Maybe one last one here. How do you know if you are over-logging, like logging too much data? You don't. What I've done with some customers was trying to find when we deliver the first one and they complain about the storage, then we start optimizing. It's always a cycle because you can never finish optimizing. There's always room for more optimization, and you can get carried away.

However, in my case, I was like, 'Wow, I can change this.' But some changes, the final customer doesn't even realize the change. So, there is some balance in the middle. Still, modeling the data on logs, I think there should be some special people for data who can distinguish what's important and what's not. More experts. That makes sense. The answer to the question is when your hard drive gets full, or if you have some expert who can help you. It's usually not going to be full because we have the TTL. The data survives for just a week. After a week, it's removed. We have this cycle because usually logs and telemetry, we can see and compare easily.

There's always a room for optimization in logging and data storage. It's important to find a balance between optimizing and ensuring the data is relevant and useful. Having experts who understand the importance of data and its storage can greatly help in managing logging efficiently. The answer to over-logging lies in monitoring storage capacity and involving experts to determine the significance of the data being logged. Balancing the need for data with efficient storage practices is crucial for maintaining an optimal logging system.