Tools for better Observability in NodeJS Serverless IoT Applications

This ad is not shown to multipass and full ticket holders
JSNation US
JSNation US 2025
November 17 - 20, 2025
New York, US & Online
See JS stars in the US biggest planetarium
Learn More
In partnership with Focus Reactive
Upcoming event
JSNation US 2025
JSNation US 2025
November 17 - 20, 2025. New York, US & Online
Learn more
Bookmark
Rate this content

Observability is crucial for successfully operating large IoT fleets. IoT incorporates different components, including hardware, network, on-device software, and cloud. Devices operate under unreliable conditions and constraints, and need to be monitored remotely. Cloud applications become complex and costly, as they are built to handle device activity at scale. Answering questions such as:
-    Do I have a problem in my IoT application?
-    Where is the problem?
-    What is causing the problem?
-    How much of my fleet is affected?
-    Is my code expensive to run and if yes, how can I fix that? ,
can be challenging. Logging, monitoring and tracing are fundamental observability pillars. However, they are often viewed as non-functionals in IoT applications, and can fall off the radar, or are not standardized during development.

This session will show how to leverage Open Source tools, such as AWS Lambda PowerTools, in a fully functional Serverless IoT application, to ease adoption of best practices for modern application development, and integrate services such as Amazon X-Ray, Amazon CloudWatch and AWS IoT Core features, to achieve end-to-end observability.

This talk has been presented at Node Congress 2023, check out the latest edition of this JavaScript Conference.

FAQ

Many people think that working in the IoT space is all about cool devices, prototyping, and learning new protocols. However, the reality is different, especially when moving to production, where constant challenges and issues arise.

When moving IoT projects to production, you may encounter issues such as devices going offline, missing data, inconsistent data, and poor dashboard performance. This necessitates robust observability, including metrics, logging, and tracing.

Observability is crucial in IoT applications because it helps teams understand what is happening within the system. It involves implementing metrics, logging, and tracing in a standardized way to aid the operations team in diagnosing issues.

Tools such as AWS IoT rules engine, Lambda functions, and AWS X-Ray can be used to build observability in serverless IoT applications. These tools provide structured logging, metrics, and tracing capabilities.

AWS Lambda Power Tools is an open-source library available for TypeScript, Python, and other languages. It provides utilities for structured logging, metrics, and tracing, making it easier to monitor and debug serverless applications.

The AWS IoT rules engine integrates with Lambda functions asynchronously. It sends messages to the Lambda service, which then queues the function for execution. The results and any exceptions are logged asynchronously.

By default, AWS Lambda functions configured for asynchronous invocation will retry twice if the initial execution fails.

Lambda Power Tools can be installed using Lambda layers or NPM. It can be instrumented using middleware libraries like MIDI, decorators, or manually in the code to enable logging, metrics, and tracing.

AWS X-Ray is a service that provides tracing capabilities for applications, allowing developers to see the flow of requests and diagnose issues. In IoT applications, it helps track the execution of Lambda functions and other components.

Common issues include missing data despite using protocols like MQTT with quality of service settings, data inconsistency, and performance problems in aggregating metrics for multiple devices.

Alina Dima
Alina Dima
8 min
14 Apr, 2023

Comments

Sign in or register to post your comment.
Video Summary and Transcription
The Talk discusses the challenges of IoT development, including issues with fleet offline, data missing, alerts not working, inconsistent data, and slow loading dashboards. It explores how to build observability in IoT applications using metrics, logging, and tracing. The integration between the rules engine and Lambda is explained, highlighting the use of tools like Lambda Power Tools and X-Ray for logging, monitoring, and tracing. The Lambda invocation process and the tracing capabilities of X-Ray are also mentioned.

1. Introduction to IoT Challenges

Short description:

Everybody starts in the IoT space thinking it's all sunshine and butterflies, but when you go to production, it becomes a maze. Prototyping and testing in the lab may go well, but then issues arise. Fleet offline, data missing, alerts not working, inconsistent data, and slow loading dashboards.

All right, so everybody who starts in the IoT space kind of thinks that this is how the IoT journey looks like. You know, everything is sunshine and butterflies, and you work with devices and they're very cool. You prototype with devices. You learn new protocols and so on, so you think it looks like that.

But actually, it doesn't look like that. When you're going to production with an IoT solution, it more looks like that. So you're constantly putting off fires of one kind or another. So what's really going on? So when you're working in IoT, you are actually part of an ecosystem. You are working on the device side, or you're working with teams who work on the device side. You have backends on the cloud side. You're working with cloud teams. You're working with data teams. So it's all just really relatively crazy, and it can get even crazier really fast. So it is a maze.

You suddenly are prototyping or testing with your devices in the lab, and it all works, and everything's fine, everybody's happy. And then you go to production, and suddenly 50% of your fleet is offline from one day to another. And you try to investigate why, and you don't know why. Then you've got data missing. You're sending data. You're using MQTT. You've done all the right things. Ideally, you've used quality of service one. Ideally, you've actually used local storage at the edge as well, but still data's gone missing. So your data team is complaining. You don't know where the problem is. What about the alerts you built in? Well, you're not seeing any of them. Have you actually built them? Well, I don't know. It would be a good idea if you did. Data is inconsistent. Your users are basically complaining that loading a dashboard for, I don't know, 50 devices as an aggregate, to see aggregate metrics just takes too long.

2. Building Observability in IoT

Short description:

To build observability in an IoT application, you need metrics, logging, and tracing in a standardized way. Let's explore how to achieve this in a serverless backend scenario, where an IoT device sends data over MQTT, picked up by an AWS IoT rule and pushed into a Lambda function.

So it's all crazy. So what do you do about all of this? So clearly, you actually need to build observability in your application. So you need metrics. You need logging. You need tracing. And, ideally, you need all of this in a standardized way, so an operations team who is actually looking at this stuff, looking at this data can actually understand what's going on. So let's see how we can build observability in an IoT application.

I'm going to make the assumption that the back end here is mostly serverless. So I'm imagining a situation where you've got an IoT device sending some data over MQTT and you've got an IoT rule, you know, an AWS IoT rule, picking up this data and pushing it into a Lambda function. You're using this amazing cool integration that AWS IoT has with the rules engine. And you think everything is perfect, right? So if you scan that QR code, you can actually look at the code for what I'm going to show you. You can do that. I have it linked at the end as well. So I'll give, like, two seconds for people to look at that.

3. Integration between Rules Engine and Lambda

Short description:

The integration between the rules engine and Lambda works asynchronously, with the Lambda function being put into a queue and executing asynchronously. To enable logging and monitoring in your applications, you can use tools like the Lambda Power Tools, an open source library that provides utilities for structured logging, metrics, and tracing. The Lambda Power Tools can be installed using Lambda layers or NPM, and can be instrumented using middleware libraries like MIDI. By injecting the tracer and logger into your Lambda function, you can send traces to X-ray for observability. In this case, an IoT device simulator sends a message to the rules engine, which interacts with the Lambda function through the Lambda service.

All right, so what I'm going to show you right now is what you might not be expecting about the integration between the rules engine and Lambda. So the Lambda function – let's skip this for now. So the Lambda function that I'm using looks a little bit like that, right? So you are setting – you have a tracing library. I will show you what the tracing library is and talk about it later. But you've actually – your Lambda function just does something and throws an exception, right? And that's basically it. And so if you look at this – you know, if you look at this architecture here, you'd expect to see the exception right away.

Well, the funny thing is that the way the rules engine actually integrates with Lambda is asynchronously, right? So rule engine sends the message to the Lambda service. Lambda service says, great, I've got it, 202. And then your Lambda function is put into a queue and it executes asynchronously. And only then, when that execution is done, somewhere, ideally in some log file, you will see the result of your Lambda execution, right? So that's why – of course, when this is happening with one device, you know, you think, yeah, I've got it under control. I can go to the log of the Lambda function, and I can look there, and I can see that actually the Lambda function failed. But ideally, you would actually use some tracing tools and, you know, some tools that you may enable you to do logging and monitoring in your applications, so you can see this stuff relatively easily.

And so one of these tools is the Lambda Power Tools, and Lambda Power Tools is actually an open source library. It's available for Typescript. It's also available for other languages like Python, for example. And what it does is it provides you with a set of utilities that you can integrate in your, you know, in your JavaScript application, so that you can create easily structured logging, you can create metrics, you can even build your custom metrics, and you can also somehow, you know, see the traces in a service that is called X-ray, right? So, I mean, of course, this works if you're integrating with AWS Services. If you're integrating with other types of services, you might want to identify different observability tools that you can use, right?

So the way you would install the Lambda Power Tools is basically using either Lambda layers or you can use NPM. And you can instrument using MIDI, which is quite a famous middleware library for Lambda, or you can do it with decorators or you can do it manually. So it's actually looking quite neat if you look at the TypeScript code, right? So here I'm just using Node modules. I'm not going to go into details on that. But then you can create your tracer and logger, and then you can just use MIDI to literally inject them into your Lambda function. Right? So with what I have here, basically all your traces are going to go from your Lambda function invocation are going to go into x-ray, right? So let's see. This is actually not looking very good. So maybe I'm just gonna switch and show it to you really quickly. I still have 56 minutes. That would be nice. Right. So basically what I've done here, I've sent a message from an IoT device simulator, which is using MQTT.js as a library. So this is my client application and this is the... In this case, it's the rules engine, and this is the Lambda context from the Lambda service.

4. Lambda Invocation and X-Ray

Short description:

This is the Lambda invocation. The Lambda service accepted the function but executed it twice due to the default retry configuration in AWS Lambda. X-Ray provides tracing capabilities. Check the GitHub repositories for more information on Lambda power tools and X-Ray.

And this is actually the Lambda invocation. So when you look at this on a high level, this is actually looking really good, right? So you don't see an error at all. But when you look down here, you actually see that the Lambda service accepted the Lambda function, but then you see that it actually executed or attempted to execute this twice. So that's interesting. And that's the default configuration of AWS Lambda. It's going to retry by default if it's an asynchronous mode. So when you use SAM and you create your Lambda functions, the default is always twice. So think about that. You know, asynchronous invocation twice. Right.

So this is kind of how X-Ray looks like, and here you can see the tracing and so on. We don't have more time today, but you can always just have a look at the GitHub repositories that I have linked and learn a little bit more about Lambda power tools and about X-Ray and so on. Thank you very much.

Check out more articles and videos

We constantly think of articles and videos that might spark Git people interest / skill us up or help building a stellar career

You Don’t Know How to SSR
DevOps.js Conf 2024DevOps.js Conf 2024
23 min
You Don’t Know How to SSR
The Talk covers the speaker's personal journey into server-side rendering (SSR) and the evolution of web development frameworks. It explores the use of jQuery for animations in SSR, the challenges faced in integrating React with Umbraco, and the creation of a custom SSR framework. The Talk also discusses the benefits of Next.js and the use of serverless artifacts for deployment. Finally, it highlights the features of Astro, including its function per route capability.
AWS Lambda under the hood
Node Congress 2023Node Congress 2023
22 min
AWS Lambda under the hood
Top Content
In this Talk, key characteristics of AWS Lambda functions are covered, including service architecture, composition, and optimization of Node.js code. The two operational models of Lambda, asynchronous and synchronous invocation, are explained, highlighting the scalability and availability of the service. The features of Lambda functions, such as retries and event source mapping, are discussed, along with the micro VM lifecycle and the three stages of a Lambda function. Code optimization techniques, including reducing bundle size and using caching options, are explained, and tools like webpack and Lambda Power Tuning are recommended for optimization. Overall, Lambda is a powerful service for handling scalability and traffic spikes while enabling developers to focus on business logic.
AWS Lambda Performance Tuning
Node Congress 2024Node Congress 2024
25 min
AWS Lambda Performance Tuning
Top Content
This Talk covers various optimization techniques for Lambda functions, including parameter fetching, code minification and bundling, observability with Power Tools and X-Ray, baseline testing with load testing tools, caching with Elastic Cache and Redis, and optimizing code size and memory usage. The importance of library choices, power tuning for cost and performance, leveraging subprocesses and sandboxes, and adjusting concurrency limits are also discussed. Overall, these techniques can significantly improve Lambda function performance.
Advanced GraphQL Architectures: Serverless Event Sourcing and CQRS
React Summit 2023React Summit 2023
28 min
Advanced GraphQL Architectures: Serverless Event Sourcing and CQRS
Watch video: Advanced GraphQL Architectures: Serverless Event Sourcing and CQRS
GraphQL is a strongly typed, version-free query language that allows you to ask for specific data and get it in JSON format. It simplifies data retrieval and modification by allowing the server to handle all necessary operations. Serverless architectures, such as AWS Lambda, are scalable, cost-effective, and good for event-driven applications. Event sourcing and CQRS are techniques that ensure consistency and separate reading and writing parts of an application. Building a GraphQL API with commands and queries can be achieved using AWS AppSync and DynamoDB. This approach offers low latency, scalability, and supports multiple languages. Challenges include application complexity, data modeling, and tracing, but starting with simplicity and making something work first can lead to success.
Demystify the DX for Lambda functions
DevOps.js Conf 2024DevOps.js Conf 2024
30 min
Demystify the DX for Lambda functions
Premium
Welcome to this session on Lambda Developer Experience. Learn about using AWS Cloud Development Kit (CDK) to write code in your favorite language and automatically generate CloudFormation templates. Test locally with the SAM CLI and deploy with CDK. Accelerate testing and updates with CDK flags. Use the AWS Toolkit to invoke Lambda functions, analyze logs, and generate code with Application Composer.
Building Dapps with React
React Advanced 2021React Advanced 2021
30 min
Building Dapps with React
The Talk discusses building decentralized applications (DApps) with React and explores the benefits of smart contract technology. It highlights the characteristics and infrastructure of Web 3 applications, including decentralized indexing, off-chain data storage, and decentralized file storage. The Talk also covers identity in Web 3, with a focus on self-sovereign identity and the use of blockchain for identity verification. The process of building a DApp with React and Hard Hat is explained, along with deploying contracts and interacting with them. Overall, the Talk provides insights into the world of DApps and the technologies involved.

Workshops on related topic

AI on Demand: Serverless AI
DevOps.js Conf 2024DevOps.js Conf 2024
163 min
AI on Demand: Serverless AI
Top Content
Featured WorkshopFree
Nathan Disidore
Nathan Disidore
In this workshop, we discuss the merits of serverless architecture and how it can be applied to the AI space. We'll explore options around building serverless RAG applications for a more lambda-esque approach to AI. Next, we'll get hands on and build a sample CRUD app that allows you to store information and query it using an LLM with Workers AI, Vectorize, D1, and Cloudflare Workers.
Building Serverless Applications on AWS with TypeScript
Node Congress 2021Node Congress 2021
245 min
Building Serverless Applications on AWS with TypeScript
Workshop
Slobodan Stojanović
Slobodan Stojanović
This workshop teaches you the basics of serverless application development with TypeScript. We'll start with a simple Lambda function, set up the project and the infrastructure-as-a-code (AWS CDK), and learn how to organize, test, and debug a more complex serverless application.
Table of contents:        - How to set up a serverless project with TypeScript and CDK        - How to write a testable Lambda function with hexagonal architecture        - How to connect a function to a DynamoDB table        - How to create a serverless API        - How to debug and test a serverless function        - How to organize and grow a serverless application


Materials referred to in the workshop:
https://excalidraw.com/#room=57b84e0df9bdb7ea5675,HYgVepLIpfxrK4EQNclQ9w
DynamoDB blog Alex DeBrie: https://www.dynamodbguide.com/
Excellent book for the DynamoDB: https://www.dynamodbbook.com/
https://slobodan.me/workshops/nodecongress/prerequisites.html
Serverless for React Developers
React Summit 2022React Summit 2022
107 min
Serverless for React Developers
Workshop
Tejas Kumar
Tejas Kumar
Intro to serverlessPrior Art: Docker, Containers, and KubernetesActivity: Build a Dockerized application and deploy it to a cloud providerAnalysis: What is good/bad about this approach?Why Serverless is Needed/BetterActivity: Build the same application with serverlessAnalysis: What is good/bad about this approach?
Frontend to the Cloud Made Easy - A ReactJS + AWS Workshop
DevOps.js Conf 2024DevOps.js Conf 2024
59 min
Frontend to the Cloud Made Easy - A ReactJS + AWS Workshop
Workshop
Eyal Keren
Eyal Keren
This workshop enables you to learn how to develop React applications, and then deploy them to the cloud (or building them to the console) coupled with a backend, fully abstracted, with no complex backend configuration, simplifying the building and deployment of frontend & web apps to the cloud.
Building a GraphQL-native serverless backend with Fauna
GraphQL Galaxy 2021GraphQL Galaxy 2021
143 min
Building a GraphQL-native serverless backend with Fauna
Workshop
Rob Sutter
Shadid Haque
2 authors
Welcome to Fauna! This workshop helps GraphQL developers build performant applications with Fauna that scale to any size userbase. You start with the basics, using only the GraphQL playground in the Fauna dashboard, then build a complete full-stack application with Next.js, adding functionality as you go along.

In the first section, Getting started with Fauna, you learn how Fauna automatically creates queries, mutations, and other resources based on your GraphQL schema. You learn how to accomplish common tasks with GraphQL, how to use the Fauna Query Language (FQL) to perform more advanced tasks.

In the second section, Building with Fauna, you learn how Fauna automatically creates queries, mutations, and other resources based on your GraphQL schema. You learn how to accomplish common tasks with GraphQL, how to use the Fauna Query Language (FQL) to perform more advanced tasks.