Automate the Browser With Workers Browser Rendering API

Hi, everyone. I am really excited to be here at JS Nation. I'm going to be talking about how to automate the browser with the workers browser rendering API. Now, that's a mouthful. From now on, I'm going to refer to this as just rendering API or browser rendering. I'm really sad I couldn't make it to the conference in person, but I hope you still enjoy my session.

In this session, I'm going to show you how you can perform browser automation tasks using the Cloudflare Developer Platform. And at the end, I'll show you some cool use cases and examples of some tasks that you can automate with the browser. Again, I am Gift Iguenu. I work as a developer advocate at Cloudflare. I'm really excited about the Developer Platform at Cloudflare, and I talk about it on Twitter slash my personal blog. So, if you'd like to follow me there, that would be great.

All right. So, I have a short personal story to tell before going into this talk, and it's likely based on my experience as an expat living in the Netherlands. So, I feel like this is going to be relatable if you live in the Netherlands, or if you live in any major city that has housing crisis. It's super difficult to find an apartment. Usually takes around, what, one to three months, if you just relocated or moved to a new city, to get a nice apartment, right? And it's also quite expensive. In the Netherlands, last year, there was, like, house shortage over 390,000 homes were had shortage. So it was very difficult to find an apartment.

And at the time, I was looking for an apartment in the Netherlands, specifically in Amsterdam, because I was living in a different city, and I wanted to move to Amsterdam. And I had a specific apartment in mind that I wanted to move to. It's like a building complex with several types of apartments. Now the interesting bit about this specific place is called our domain. Basically how it works is you just need to go to the website, and once there is an available apartment, you pay for it, and it means that once you are able to pay, it means that you have the apartment.

Now, the problem is, for every apartment that is available in that building, it's essentially already booked, right? So every week, if maybe someone moves out, or there's an availability, they would update that every week on Wednesday at 12 p.m. So for the next one month, I would go to the website on Wednesday at 12 p.m. to check if there is an availability, right? So I was looking for a two-bedroom apartment. And of course, it took me a while to be able to find this. I was manually checking the websites every week, which was not nice. But thinking about the subject in question, browser automation, at the time, if I had known about this, this is something I would have done.

But just so if anyone has this problem, this is how I solved it, using browser automation. So the Worker's Browser Rendering API basically allows you to, you know, run automation within your browser. You could do things like, you know, navigate to a specific website, take a screenshot of the website, create a PDF out of the website, and so on.

So for my specific use case, instead of going to the website on Wednesday at 12 p.m. every single week, I wanted to automate this process, right? I wanted to allow my browser do that for me, without me having to do it myself. So what did I do? I essentially wrote the browser rendering worker that would navigate to the specific website at the specific time. Honestly, I just would run this every single time, just because I wanted to be sure that I could find an apartment, right? So what this does is, it navigates to the websites, takes a screenshot of the specific availability page. So if there is an availability at a time, the website is going to tell me that, oh, you're lucky. There is currently an apartment available. In fact, at this specific time, there was three different apartments available. So I was able to, you know, go to the website and pay for the apartment that I wanted to do.

Of course, this is just a cool example that, you know, browser automation can help you with. There's so much more that you can do. For example, with this demo, I could as well go one step further to make a Chrome job that I would make this worker run at a specific day every week. Or I could also add like email automation where if there's an apartment, instead of like returning the value of the apartment to the website, I could send myself an email and I would be automatically informed of the availability, right? So that's an actual use case that I think browser automation can help with.

So with that, I want to talk about underlying technology within that, you know, setup I just showed you, which is Cloudflare Workers. If you're not familiar, Cloudflare Workers is a serverless environment that allows you to create new applications or augment existing ones with, you know, languages you're already used to like JavaScript, TypeScript, or even Python. So this is an example Cloudflare Worker. Basically what this does is it's returning a string or an array of conference data, in this case, GSNation, and I'm returning that to my browser. It's essentially written in JavaScript. This example I'm showing you, but you can as well write it in TypeScript or in Python if you want.

Now moving forward, just a few slides. Yeah. Browser rendering API is one of the services that we offer in the Cloudflare developer platform. Essentially what this is, is it allows you as a developer to control and interact with a headless browser instance. And from that, you can then create automatic flows, for example, navigate to a website, take a screenshot, and so much more. And the underlying technology here, if you ever explored the topic of browser automation, you possibly have heard of Puppeteer. So Puppeteer is a Node.js library from Chrome DevTools team. It essentially provides an abstraction of the DevTools protocol to help you control Chrome or Chromium. Now the worker's browser rendering API is a fork of Puppeteer.

Essentially what we did at Cloudflare is we created a fork and patched it to connect to the worker's browser rendering API instead, so that within your Cloudflare worker, you can directly use the Puppeteer library. This was not possible before, by the way. So we are allowing you to use Puppeteer within workers with this integration.

And how did this come up? This actually was born as a internal need within the team. So within Cloudflare, we actually needed to do browser automation, but it wasn't possible within workers at the time. Many teams wanted to do tools like maybe take automated screenshots or create automated PDFs. For example, one of the tools within Cloudflare called Cloudflare Radar, which is a product for monitoring internet trends, uses the worker's browser rendering API to implement the URL scanner. So basically what this does is it takes screenshots of any URL you input and it will generate like a report of the performance and the security of the URL.

How does this work under the hood? I'll try to explain this to you on how we're using Puppeteer under the hood to allow you to use it within a worker. So the rendering API combines the power of something called the remote browser isolation to help you simplify the process of running Puppeteer within a worker. So remote browser isolation basically allows you to interact with a web browser in a remote environment instead on your own device. So we've wrapped the Puppeteer library so that it works directly inside your project. So this is how we're able to allow you to use Puppeteer within your worker. So once a web socket connection is established, our remote browser isolation will handle all the incoming requests to your worker so that you can perform tasks like take a screenshot or do the different other automation tasks you'd like to perform.

And also, within your Cloudflare worker, you would need to set up something called a browser binding. So essentially, this is what will give a worker an authenticated endpoint to be able to interact with a remote group, your browser instance. This is an example of a worker script that essentially is running the Cloudflare browser rendering API within the script. So essentially what's happening here is you need to import the Cloudflare Puppeteer library, and then you would need to set up the instance by calling the Puppeteer.launch function. Essentially what I'm doing here is I'm launching Puppeteer, and I'm also navigating to a specific page in the browser. All of this is available on GitHub. Like I mentioned, we made a fork of Puppeteer. So the specific library that we're using within Cloudflare workers is called the Cloudflare Puppeteer library and made it open source. And the team working on this always tries to maintain feature parity with the actual fork, which is the Puppeteer library from Google.

At the time of this talk, browser rendering is currently generally available. So if you're interested in checking it out, you can use it as from today to automate your tasks within the browser. We also have released additional features. For example, we have an additional feature called browser management, where you can reuse browser sessions. So an interesting thing that happened or from my experience using it at the time was when I needed to use or perform like two or more tasks within a specific worker, I found it difficult because of the limits that comes with the browser rendering API. So we've made it easier for you to reuse browser session. And I'm going to show you an example of how that works.

We've also added analytics and logs so you can see the amount of browsers that has been spawned during the time of your worker integration with the rendering API.

So what are some use cases, like why would you use this within your application? I already showed you a few, but I'll also share a few more just so that you're familiar with some good use cases to use the browser rendering API. So obviously, I already mentioned taking a screenshot. So if you ever wanted to do, for example, you are running a test and you need to take a screenshot or you need to make a screenshot of a specific page, or you don't want to have to do that manually, you can use it as an example use case. You can use it to generate PDF from a web page. You can use a text web application. Of course, also use it for gathering performance metrics.

I have a few minutes left, so I'll quickly show you some examples of how you would typically want to use the worker's browser rendering API within your application. The first example I'm going to show you is how to take a screenshot and also store this screenshot to Cloudflare R2. So R2 is a blob storage that we offer within Cloudflare that allows you to store things like images and so much more. So I'm going to take a screenshot of a page and store the image in R2. So basically, what I need to do here is I already have a lot of this set up. So I'll quickly show you the code. I have my worker here where I've already gone ahead to install the Puppeteer library. I've also installed additional libraries that allow me to perform the tasks that I want to perform here. The things that I'll call out is first, I'm launching the browser. So basically, I want to launch a browser and then open a new tab within the browser. And for the specific URL I'm going to pass, I want to take a screenshot, which is what I'm doing here. And then after taking a screenshot, I'm then uploading the image to R2, which is what I do in this code block. So let me quickly test this out and show you.

So I'm going to run npm run dev. And once this is running, I will quickly turn something off and try this again. So once this is running, I will navigate to my URL here and I need to also pass in the params of the URL I want to pass in. So for this specific example, I will be using the Hacker News websites for my demo. So hackernews.com. And once I put that in, I should get a screenshot of the current hacker news websites. Obviously, this has been stored to R2. So what I'm pulling in is the public URL from R2. And you see the image generated here. So this is an example of what you can do with the browser rendering API.

Here's another example. I'll just quickly go through them so I don't waste a lot of time. The other thing you can do is also generate a PDF. So for this example, I am doing something similar, right, where I have created a new browser instance. I'm navigating to a specific URL on line one here. And for this specific URL, I'm using the GS Nation schedule. So I want to quickly take a screenshot of the schedule and turn that into a PDF, right? So what I've done here is I'm using the await page.goto to navigate to the URL. Excuse me, I had to pass a wait until here. Why? Because the page will take a while to load, meaning that if you take a screenshot or make a PDF before waiting for the page to load, you might just end up with a blank screen. I'm also waiting for a specific selector within the page to also enable me, you know, fetch or wait for the DOM to load, fetch as much of the data within the page before I make the PDF. And then on line six to nine, I'm finally creating the PDF and the results is what you have here. This is missing a lot of CSS. I am aware of that. A lot of CSS from the page is not showing. But what I could do to fix that is to set a timeout, like a timeout that would take like maybe four or five seconds so that the CSS in the page is completely loaded before I create the PDF. But I didn't do that in this demo. So that's also OK because this page has tons of resources that would need to get downloaded.

Another demo I'd like to show is how you can use Durable Objects to persist browser sessions. So I mentioned that if you would want to perform more than one task within your worker, sometimes it might be difficult and it would also decrease performance. Well, we are now allowing you to reuse browser sessions by either using Durable Objects or by using the session management I mentioned earlier. So I'll show you a quick example of how to use Durable Objects. Mind you, you have a limit of two new browsers that you can open within an account and those will stay up for one minute. Meaning you have 60 seconds to do whatever task you want to do before the session is closed. But you can use Durable Objects to extend that a bit.

In this example, I'm using a Durable Object to create an alarm that would extend my session for another 60 seconds so that I can do what I want to do. So for example, I want to take a screenshot like I did in the effects example. But I also want to take a screenshot of every possible dimension within the screen. So that would essentially run for more than 60 seconds. So I'm adding an additional 60 seconds so that the browser session is extended. I'm doing that using Durable Objects. Secondly, one of the additional features that you could use within the Workers browser rendering is called Session Management where you can reduce a session. Essentially, the difference between doing this and using it ordinarily is that you can now pass in something called a puppeteer that connects. And instead of ending your browser session, you can just disconnect it. And then if you need to reuse the session for the next possible task you have, it's then possible within your worker.

I'll skip over this last demo just because I don't have so much time to go through it. But another thing you could do within your workers browser rendering API is to perform to take performance metrics from an application. And this is an example of what I'm doing here with Hacker News. Those are different things that you could do with the browser rendering API. I know I've briefed through a lot of them, but I recommend that you check out the docs to learn more and also see some examples and try it out for yourself as well. If you have a use case that you would like to automate with the browser rendering API, I would advise that you check out the docs. And also, we have blog posts on the Cloudflare blog that goes into a bit more detail about how all this works. So I recommend you also look into that as well. Thank you so much for listening to my session. I hope you had a great time and also took something out from this. If you'd like to get access to the slides, you can scan the QR code on screen. Thank you. And have a wonderful rest of the conference.