Bull-Ish Your Queues With BullMQ

Hello, and welcome to this talk where I talk about queues and Boolean Queue. Today I want to tell you a story of a team working on a client project that needed to handle a simple queue. They approached the platform team for help, but were only given one queue, which wasn't enough. The team had to find a solution using the tools they already had, such as Posgas SQL and Elastic Cache. The first possible solution was pgboss. If you don't know pgboss, it is a simple npm package that you can install in your node application and basically create a sort of queue system on top of Posgas SQL. And expose an API to call the queues and handle the queues inside of your application. The only solution left is Elastic Cache and Redis. And in this particular scenario, what shines is BoolMQ. BoolMQ is a simple npm library that implements a fast and robust queue system on top of Redis. It can be used to decouple your code and create a microservices architecture. You can create a flow by depending on different queues and creating the result based on each queue's result. Out of the box, BoolMQ provides parallelism and concurrency. Additionally, you can have OpenTelemetry set up and a UI dashboard for queue monitoring. To create a PullMQ queue, provide a name and a connection string to Redis. The code includes a run method that continuously pushes new temperatures into the queue. The consumer uses a worker to handle data and execute queries. Close the connection when closing the Node.js process. Run the publisher to push data and the consumer to receive data. Create multiple publishers or consumers as needed. The second demo shows how to set up OpenTelemetry in your Node.js environment to track what happens inside your queues. Use BullMQ Hotel and the BullHotel instance to track telemetry for your queues. The last demo demonstrates the use of Fastify with BullMQ to create a UI for managing queues. You can do whatever you want with OpenTelemetry. The last demo showcases a Fastify application within the UI. Fastify is a simple framework for handling Node.js APIs. Set up Fastify using the Fastify adapter exposed by BullMQ, BullBoard. Expose the '/UI' API to show the UI in the server. The UI allows you to interact with the queues and view completed items. BullMQ is a straightforward solution for handling queues and can be easily scaled using Redis. For most cases, using a queue is necessary and well-documented. However, it relies on Redis, which means data loss if Redis goes down. Additionally, you'll need to learn another library. Thank you for listening!