Using the Gamepad API for a Better Gaming Experience on the Web

Hello, and welcome to my talk for this year's JS Game Dev Summit. My name is Arun. And today, I'll be talking about the web gamepad API and how we can extend it and use it effectively for making games on the web.

First, a bit about myself. I'm a senior engineer at Bloomberg, and I mostly work with web technologies, building enterprise web software and browser-based tools. I'm also an open-source enthusiast. I've contributed to several open-source projects, and some of my own projects, mostly extension libraries and modules for Node, React, Angular, etc. You can check out my website if you want to learn more about me and my work.

All right, so without any further ado, let's get straight into it. Let's talk about the web platform for a moment before jumping into the web gamepad API. I guess we first need to acknowledge the fact that the web has grown a lot over the last half decade or so, and it continues to grow at a rapid pace. We all know that the browser has become more than just a renderer or document viewer, and it's because of this growth and the continuous addition of feature-rich APIs that has enabled developers to create much more interactive and efficient web applications. Enhancements such as ambient light events, GPU-accelerated graphics, file system access, and lower-level APIs that provide native control and hardware access not only allow developers to cater to a wider variety of users but it also allows them to support various types of use cases. Which would have not been possible without these enhancements to the web platform. Even though some of these APIs are in very early stages at the moment, they seem to be really, really promising and tend to provide an idea on how web development in the coming years might look like.

Browsers have become extremely powerful and capable enough to handle complex graphics computations, especially for playing video games. From 2D site scrollers, retro emulation, to even high end 3D games. It's possible to play almost any type of game on the web today.

And for a majority of these games that we see and play on the web, the primary form of input has always been keyboard or mouse based. And for touch screen devices, you've seen on screen control layouts like these, which at times can hinder the game experience. And using these types of controls can turn out to be quite a daunting task for gamers who prefer more natural controls, you know, like a directional pad or a joystick for character movements.

Now, desktop games provide this flexibility of using external devices like game pads and joysticks, maybe because they have native support for this built right in. But when it comes to the web, for a long time, there has been no native support for this at all. Developers had to design these complex mouse or keyboard based interfaces for game controls that can be quite tricky to operate.

So, a few years back, the W3C introduced the GamePaths specification, which defines a low level interface to represent gaming devices. Which means using this API, developers will be able to connect GamePaths and similar devices to the browser and then be able to use them in their gaming applications. Which means you'll be able to use devices such as the DualShock, Joy-Con and even next gen devices like the DualSense and Xbox Series X controllers.

Now, as of today, the GamePaths specification is still a work in progress. It's published as a working draft. The specification is intended to become a W3C recommendation. For now we can see that the API itself is supported by all modern browsers, which is definitely a good sign. So let's take a look at the details of the API itself. So the API exposes a method on the Navigator interface which returns a list of GamePad objects and each GamePad object represents a GamePad that is connected to the browser. Now this GamePad interface returns useful information about the connected GamePad. Information related to the buttons, the axis, which are the joysticks basically. The current states and other information like the device ID, status, et cetera. If we look closely at the GamePad buttons, we see that every button is an instance of the GamePad button interface. This represents an individual button on the GamePad and allows access to the current state of button. Using this, we can determine whether the button was pressed or not and carry out the appropriate action in the game. Now, every GamePad manufacturer creates many different kinds of products and each might have a unique style or a layout of buttons and axis. But it's the job of the user agent, of course, to support as many of these as possible, which is why the GamePad API recommends a de facto standard layout, which is called a Standard GamePad Layout. And this layout has been made popular by many gaming consoles. And in this layout, buttons and axis are laid out in a particular order that we'll see. So this is how the Standard GamePad Layout looks. And in this layout, button locations are placed out in a left cluster of four buttons, a right cluster of four buttons, a center cluster of three buttons.

Some controllers might have four. A pair of front-facing buttons, or shoulder buttons, as they are called, on the left and right side of the GamePad. And a pair of joysticks.

Please do note that, since the GamePad API is in early stages, the standard GamePad button layout might differ from one browser to the other. The image here shows the default button mappings for Google Chrome.

So how exactly can we track these button state changes? Are there any events exposed by the GamePad API that we can use? Let's see.

So the GamePad API provides the GamePad Connected Event, which is emitted whenever a new GamePad is connected to the browser. If the GamePad is already connected, when the page loads and gains focus, then the event is emitted whenever there is any activity on the GamePad, you know, a button press or a movement on a joystick. Similarly, there's a GamePad Disconnected Event, which is emitted whenever a GamePad is disconnected from the browser.

Suddenly there is no standardized way to detect GamePad button presses or joystick movements. It's seen already that the GamePad interface does return useful information about the button access states, but there is no actual event that is emitted when these actions are performed by the user. Currently the GamePad API only supports the GamePad Connected and Disconnected events.

So let's try to explore what we can do to track the GamePad button, presses and access movements. We might have a GamePad connected to the browser and in order to track the button and access states, we can continuously poll for the changes in something like a set interval call, and this is pretty much the simplest thing that we can do. If you look at the code here, we are logging out the information of the first button of the first GamePad that is connected to the browser. That's why we use the 0 to X. This works fine. We are able to catch the button state as it updates, but this might not be the best way to capture this information. Let's find out why.

We all know that in JavaScript, everything runs on a single thread and the timer functions are no exception. To execute code using setInterval, we specify a callback that is executed every X milliseconds. The delays provided to these timer functions are sometimes not honored due to priority execution of resource intensive tasks, if there are any in the task queue. And this leads to inconsistent delay intervals. You might have noticed a check while using setInterval to animate elements of the page. And this is caused due to unnecessary and forceful refresh of the page elements. Even before the user screen is ready to process and render those updates. And this is called layout thrashing and it should be avoided at all costs.

So what else can we do here? We can use request animation frame. Now the callback provided to request animation frame is guaranteed to be executed at the start of the frame. Which means that the specified callback will be called before the next repaint operation happens.

The number of times the callback function is requested is around 60 times a second or 60 FPS. But it generally matches the screen's refresh rate. That's why for most operations or animations that are required to remain in sync with the rendering, it is recommended to use request animation frame for best performance.

Now coming back to the Gamepad API, we can see that querying the Gamepad objects and accessing the button states can be done effectively using request animation frame. In the context of a video game, a game loop is something that continuously checks for user input, updates the game state and then renders the scene. Request animation frame is suitable for this because we can perform all of these operations in its callback. And it would remain in sync with the repaint tasks in each frame.

I think that's how user input should be pulled in a gaming application. So just to recap, we are able to detect whether a game pad is connected or disconnected from the browser. That is something that the game pad supports already. We're able to access the button or access state changes using the request animation frame. Now what we ideally would want to do is provide this functionality in the form of an API so that it can be reused in any gaming application. So let's see how we can extend the game pad API to achieve this. Let's start with the game loop implementation. Here we have a start method. And in this method, what we're basically doing is that we're fetching all of the game pads that are connected to the browser. And then we loop through them to check for button and access state changes. And for each game pad that is connected to the browser, we want to listen to any state changes happening on any of the buttons or access. That's why we have like separate event handlers for both of these. The path to start method to request animation frames so that it gets executed recursively and we can track the button and access state changes on every frame.

One thing to note here is the return value of the request animation frame method. So it returns a non-integer value, which is the request ID that uniquely identifies the callback function. And we can use this ID to request the direction and we can use this ID to cancel the frame callback request. We might need to make sure to stop the loop, cancel all the callback requests and free the memory when it's no longer required. For example, when all of the game paths have been disconnected and there is no need for polling the button or access state changes.

Now let's take a look at the implementation of the ButtonEventListener. So for tracking the button events, we loop through all the GamePad buttons, check for the button state, whether it has been pressed or not. And depending on that, we use the DispatchEvent to dispatch any event on the window object. And we use the custom events API to define the event itself, which is called GamePadButtonPressed, and pass in the event data to this custom event, which contains the information about pressed button. And using this, we can listen to the GamePadButtonPressed event on the window object and track state changes on any of the GamePad buttons.

One tip is to maintain an internal key value mapping of all of the buttons and the states, and discard these values as and when the state changes. This way we'll be able to dispatch the custom events only when the state changes positive.

Now for tracking joystick movements, we first determine the total number of sticks on the GamePad. So in the standard GamePad layout, we have two joysticks. And each stick has two axes that represent the direction of movement. And each axis is represented by an index.

We loop through all the axes and determine which stick was moved. So axis 0 and 1 represent the left stick. And axis 2 and 3 represent the right stick. So if the current index is less than the total number of sticks, then it's the left stick, otherwise it's the right stick. We also calculate the direction of the movement based on the axis indices and the values. So a negative value represents a movement in the left or top directions, and a positive value represents a movement in the right or bottom directions. We again use the dispatch event method to dispatch an event on the window object with the required event data. The event itself is, again, a custom event, which is named GamePadAxisMoved, which is in which we can track state changes on any of the GamePad axis.

Now, listening to the GamePad button pressed, they went on a window object could return the info of the pressed button. And the GamePadAxisMoved event would return the info of the axis that was moved. That's pretty much it. We've been able to extend the GamePad API, the GamePad interface in particular, to track button presses and axis movements.

Now, the thing is that you don't have to do this entire implementation all by yourself, right? So, there's an existing open-source library that you can simply use in your games that I created, by the way. And this library is called JoypadJS. It's a library that lets you connect and use various gaming controllers, two browsers that support the GamePad API. It's super small, has like zero dependencies, and support for button presses, axis movement, events, and much more. You can check it out at bit.ly slash joypad underscore JS. It's a library that I open-sourced some time back under the MIT license. It exposes an event-based API that can be used to subscribe to certain events that are detected and dispatched internally by the library.

So this is just an example. So you can subscribe to events by simply specifying an event name and a callback. And this callback is fired whenever the specified event is triggered. So it supports events for handling gamepad connection, disconnection, button presses, and access movements. It also exposes a vibrate method that can be used to provide haptic feedback to the user. So basically you can set the duration and the intensity and customize the vibration of the gamepad. It is something that is supported by all modern browsers except Firefox, as far as I know. The library also provides a set method that can be used to configure global settings for vibration, access movement threshold, which is like the rigidness of the movement, and the custom button mapping option, which can be used to set custom button mappings for improved cross-browser button state tracking. Like I mentioned earlier, different browsers might implement their own standard button mapping for gamepads, right? So using this option, you can make sure that the button mappings are accurate and all of the buttons work as expected on all browsers. The browser support of the library is pretty good, JoypadJS works on all modern browsers that have support for the Gamepad API, of course. Developers have been using the library in all different sorts of projects, games, interactive applications, IOT, which is fantastic. I also learned that RPG.js uses this library for gamepad support, so RPG.js is like a web-based RPG creation framework. So yeah, feel free to give it a shot on sharing feedback that you might have. I would like to point out again that the Gamepad API is in early stages. So the details and the API specifics might change in the future. Again, you can check it out at bit.ly slash joypad underscore JS. As you might have already guessed, I have been doing a lot of research on the topic and have published some blogs that might be really helpful. You can check them out here. And that is it. If you have any questions, suggestions, or you'd just like to say hi, feel free to reach out to me at amdisuser92 on Twitter. Thank you so much for joining in and listening to me and enjoy the rest of the conference.