a11y & Interactive Canvases

Thank you, everyone, for coming here today to watch my talk about accessibility and interactive canvases. So, just a real quick introduction. My name is Oli. I'm a former game developer. I have worked with LarCraft and Sonic the Hedgehog in the past. Great people, by the way. I love them. Nice people. And so now, I work at a company you may have heard of called AG Grid, and I work on the team which does the charting component.

So here, this is basically our website. I'll just quickly show you what it is so that we're all on the same page. So essentially, yeah, it's a framework agnostic library for drawing charts, such as bar charts like this. We also can draw financial charts. We also have some map charts here as well. All different types of charts. One of the key features of our library is that it is rendered using a HTML5 canvas as opposed to an SVG, which is what most other charting libraries use. There are a few exceptions.

So as you can see, like this toolbar here, those are just regular HTML5 buttons. But specifically for this talk, I'm going to focus on the canvas rendering part of AG charts. So the problem space that we're dealing with is when you're rendering a chart, the big question is, should I be using an SVG image or should I be using a canvas? I'm sure most of you are already familiar with the concept of these two, but just to make sure everyone understands, let's quickly go over the pros and cons. So an advantage of the SVG is that, so that's basically a vector-based rendering system. It's rendered by your layout engine, like Chrome or Safari. And it has an XML hierarchy where all of the shapes and everything are defined by some XML tags. Canvas, on the other hand, that's just a single HTML element. The JavaScript developer is responsible for writing all the rendering logic for actually drawing the code. And so you don't have a hierarchy. You just have that single element. The two are basically very, very similar to one another. So as you can see, these two images, they're exactly the same image. One of them is rendered by an SVG where you say, OK, I want a circle here, I want a rectangle here, and I want a polygon.

Same thing here with the canvas, but it's written in JavaScript as opposed to XML where I want a circle, a rectangle, a triangle, and so forth. And so the main advantage of an SVG is that, because you have these XML tags inside your DOM, you can basically, as I'm going to show you right here, if you inspect here, you can set some additional attributes onto the elements. So for those who aren't familiar, there is this concept called ARIA where you can basically say, for example, raw button. This will tell the screen reader to treat this element just like if it was a regular button.

And let's try to see what happens now if we turn on the screen reader. VoiceOver on Safari. Comparison. Button. Less than. Button. Three items. Button. Button. Button. You are currently muted. So as you can see, because now I've set those rules on those XML attributes, the screen reader is able to pick it up and says, OK, these shapes are actually buttons, so I'm just going to treat them as buttons. Then you have to add other handlers like the keyboard and mouse input handling. But you can basically tell the screen reader, yeah, this is what the buttons are.

We cannot do that with the canvas because it's just that single element. So how did I get around this? Well, basically, the solution that I found was to just quite simply hide the canvas. Well, not completely hide it, obviously. We still draw it on the screen. But there is this attribute called aria-hidden. We set that to true to tell the screen reader, just don't pay attention to this canvas. Nothing important here. And then what we have is we've got this concept called proxy elements. And that's essentially just some HTML elements on top of the canvas, but they have a transparent background. Basically, they're not drawn. They're there in the tree. They're there in the DOM.

But they're not drawn. If I were to show you here, I can basically mimic the same result I had before with the SVG. If I were to go here, here we can do the same thing we had before. Let's set the role to a button. Now let's do the same thing as well with the other ones. Yes, here we go. And then here as well with the triangle for completeness. Cool. Now let's go back into full screen and try this out. So there we go. That is the core of ODEON. That is not what I wanted to do. Voiceover off. Okay. Cool. Yeah. So basically, that's the core of how I've implemented accessibility on a canvas. I just basically completely hide it and reimplement the entire tree and say this is where all the stuff is, basically. So let's talk a little bit more about these proxy elements. What are the key elements of it? So the most important thing is you must not set the display to none and you must not set the visibility to hidden. Because these two properties actually remove it completely from the DOM and they remove from the accessibility tree and then your screen reader is just going to ignore that completely. So we don't want that. Instead, what we want to do is we want to set a transparent background. So like opacity zero or you can set background to just transparent and make it a transparent color. Then another thing you want to do is you want to make sure that your proxy elements will have a focus indicator. This is important because some of your users may not necessarily be using a screen reader, but they still require to use the keyboard. You can think of someone, for example, who has perfect vision, but they've got some sort of mobility impairments. And so it's important to make sure that we also style it to make sure that we have some focus indicator to tell the user what they're currently using. And then the main positioning we want to use is position absolute with a container that's common with the canvas. So this is how we can make sure everything aligns up properly.

So here I'll show you an example. I've set here this property here, focus visible, and that tells it to show it when I'm using a keyboard, but not when I'm using the mouse. So here if I were to click on here, you see it says I've clicked a square, I've clicked a triangle, I've clicked a circle. But if I then press the tab key here, my browser automatically detects, oh, now you are a keyboard user. So it started showing the focus indicator around it. And if I press the space key, then yeah, then it basically clicks as I want. And then if I turn on the screen reader, it also properly tells me triangle, square, circle. In this example, I also added some text to these buttons so that it actually tells me what it is that's going on.

Let's have a quick look with an AG chart. So here is a very, very simple chart. I say very, very simple because it's not actually a chart. It doesn't have any data. But the reason we don't have any data is that we don't really use this technique for the data. We use a slightly different technique that I'm going to get into. So we do use this technique for the legend. So that's what I'm going to demonstrate. So as you can see, if I press, I can basically click these and something happens. Well, nothing really happens. It just fades in. But yeah, I can click these buttons here. I can also use it with the keyboard. As you can see, there's a focus indicator that's telling me which element I'm currently using. And it works very well with the screen reader as well.

VoiceOver, next legend page button. Now search text fields. You are currently trying to download. Thank you. That's confidential. Search text. You are currently on a mission.

Close your eyes. Close your eyes. Close your eyes. Dialogue entering legend. Leaving next page. But search text field. You are currently on search. Okay. I think it's got someone. Okay. Basically, if you press page button you are next legend. Previous legend page. There you go. There it says. Dim. As you can see. This one, it says not dimmed. Previously entering. And then you go here. Every time that I press. Off. On. Off. I'm not sure. Let me talk. Okay. Yeah. Basically, as you can see. I think you have sort of seen the concept. Every time you move around the screen reader talks very, very, assertively.

This thing is going on. This thing is going on. But that's actually very good. Because if you have ever seen a blind person using a computer. This is the madness that they have to deal with. Just like. So that's good. That's nice. We have got a very responsive screen reader that responds to the input and output.

So that's very good. Like I said. That chart didn't have any data. And the reason for that is that these proxy elements, they work well for UI components that don't update much and they don't move around so much. And they work very well for UI elements that already have some sort of area predefined semantic in the spec. So that example that I showed you. The legend items. Those are just switches. Which is a button that also has an on off state. And then the pagination button, left, right. Those are just regular buttons. So it works very well for that.

Wait. Someone's sending me a Slack message. Okay. You know that person. That's not important. So it doesn't work very well if you want to have a completely customized control. That's one of the advantages of a canvas. You can kind of draw whatever and make up new controls. So how do so let me show you a very simple example.

So here I've got this new UI control that I've invented. It's called the ball. It goes to the right and it goes to the left as well. But let's and so that's all it does, basically. It's a very, very simple control. But let's see what happens when I turn on the screen reader. Dialogue. Interactive canvas. Position 2. Image. Position one. Image. Voice over. Okay. It's a very, very straightforward example. But it's pretty cool. Every time the ball moves, it tells me position one. Position two. Position three. Position four. That's nice. That's nice. Cool, cool. It's like every time the ball moves, immediately the screen reader is telling me what's going on. That's nice. That's nice. So what's going on here? So basically the trick that I'm using here is that I'm treating the screen reader announcements just like a video stream. Like I said before, I'm a former game developer. And so if you're not familiar with how, like, video rendering logic works, we typically will use a data structure called swap chain, which is basically just two images. So one image is drawn onto the screen, onto the display, and another image is hidden somewhere in the back, and the graphics card will be drawing on that image.

And then when the next frame is ready, you swap around the purpose of those two images. That new image comes shown in the front. Then the old frame, the old image can be reused, and the graphics card can then draw the next frame. That's the exact same concept that I'm using here.

So I've got two proxy elements. I've got one that's always visible to the screen reader and another one that's always hidden, but it's there, it's ready, I can update the text value and make that one visible whenever I want. And the method that I use for swapping these buffers is the focus method.

The reason why this works very well is that, first of all, a canvas, you can kind of conceptually think of it like a video feed, because basically you draw something, and after a certain amount of time, maybe after a few milliseconds or after a few hours or whatever, it draws another image. So it's kind of like a video feed in a way. This mechanism we're using syncs up very well with the update pattern of the canvas.

And another thing that I noticed is that assistive technologies such as screen readers, they handle focus changes very, very, very well. Like I had tried different approaches such as using an ARIA alert or having an ARIA label that you update occasionally, but it was always that odd one out, that odd browser that wouldn't work, or that odd screen reader or that platform, like there was always something somewhere not quite working, but the only thing I found that works very, very reliably is focus changes. This works on every single browser, on every single screen reader, and on every single everything that I have ever tested.

So let's kind of have a look at this demo a bit to understand what's actually going on. So let me just walk you through the code. Here we've just got some input handling, and we draw the ball. The rendering logic is important, so I'm not showing that. And then what we do here is the actual swapping. So we swap around the purpose of these two proxy elements that we've got. And then what I do is the new one that's about to be shown, I update the text value, the ARIA label of the announcer, and I set it to visible. And then the other one, I set that to invisible.

Let's make it hidden, and then we do the focus, which is actually the swap. And so let's have a look at what happens when that moves around. The main thing, the first thing you want to look out for is this style property. You'll see one of them is always on display block, which means visible, and the other one is on display none, hidden. As you see, as I move left and right, these keep switching back and forth. Block, none, block, none, block, none, block, none, block, none, block, none. The next thing you want to look out for is this text value. So I always have to update the text value of the next one I'm about to show. So here, if I want to go to the right, I need to update this to position two.

Now that becomes visible. I want to go to position three. I have to update this one, position three. But, oh, I pressed the wrong button. That's why. Okay. Position three, then you press again. That one goes to position four, and so forth. And so that's basically the mechanism we've got. We've got a stream of announcements that we keep swapping back and forth, and so it will assertively keep announcing things.

That's very cool. I like that. That's nice. We've got a really good product here. But we can do better. As I've said before, it's very, very important to make sure we have a focus indicator. So we can use that focus visible property that I was mentioning a while earlier. Just a quick reminder. So focus visible class, what it means is if you read the spec, it's like loads and loads of lots of words. But roughly what it means is hide all of this stuff for a mouse user, but show it for a keyboard user. More or less. That's more or less what it means.

So we're going to use this pseudo class and add some stuff in there to show a focus indicator. And we can use any kind of shape we want. A square, a rectangle, a circle, a Pikachu, anything we want. It's amazing. It's great. Let's have a look at that. So here we go. Same demo as before.

The main difference is you see I've set my announcers to be invisible. Opacity 1. However, when focus visible is enabled, I want that to be visible. Then I've got some styling here for a focus indicator. And what I've done here with my announcers, the only difference is I've added this child element, which is the focus indicator. Which means when one of these announcers comes into focus, if necessary, we will also show a focus indicator. So basically if I were to click this with the mouse and move left and right, this is the same thing as before, but nothing really is happening because I'm using the mouse.

But you will notice that here these bounds are being updated. And that's because I'm updating the bounds of a focus indicator that's currently not shown. But if I click here and then I press the tab key, wow! Look at that! It's a circle! And another circle. Wow, that's cool. So I move this around and then it tells me exactly what it is that I'm manipulating. That's very, very cool. I mean, here it's kind of pointless because we've only got one ball.

But if you were to use your imagination, what if we had like thousands and thousands of balls everywhere? Like how would you know which ball you're moving? You would have no idea. But now we know. Because we've got that little focus indicator around the ball. And it's not a square. It's a circle. That's so cool. I didn't even know we could do that, but we can. So yeah. Let's have some quick examples. Real-life examples, not just some boring ball. Let's see what we can do. So here let's turn on the screen reader as well. VoiceOver on Safari. So wait, let me first show you how it would be without a screen reader. So you can click here to enable, disable some series. If I click on a thing, it tells me this thing has been clicked.

Yes. Okay. And with the screen reader as well, I can do the same thing with the keyboard. If I come here, then I can press the space key, and it updates the clicked value. And I can come here and hide these things. And every time I move it's going. As I said, the thing that's nice is that the screen reader is very, very assertive and responsive. There's never a case where it's not announcing something, or it's waiting for some other announcement to end.

As soon as I press anything on the keyboard, it will cut off what it's doing and announce what's the most recent state. And that's good, because that's exactly how you want to make it as accessible as possible, because like I said, someone using a screen reader, they'll just be like going all over the place. You've got to be quick. We don't have time to wait. All right? So yeah, that's very, very cool.

Let's try another example. So here, we all like a good pie or a donut, pastries. I'm more of a croissant guy. I'm Belgian. We say cocoa chocolat as well for little bread croissants. No, wait, not bread croissants. Bread, whatever. That's not important. That's not important. But basically, the main thing I want to show you here is that we really can make a focus indicator any shape we want, right? So if I were to hover over these things here, you can see, OK, the canvas is being updated. We're updating the frames, because you can see the highlight is changing.

OK, that's cool. That's cool. But if I come in here with a keyboard, whoa, cool, look at what it's doing. Look at that focus. Oh, that's like a little sector thing. That's like a little triangle. That's so cool. And we do this as a... No, no, no, no, no, no, no. Shut up, shut up, shut up.

Thanks. We do this as well with the map. So if you've got a map of the USA, the focus indicator will go zhoosh, zhoosh, zhoosh, everywhere. And it will do the same thing with Brazil and then Africa and everything. It's so cool. Any shape you want, you can do that, because we're just using SVG image. We set that path.

So that is essentially, in a nutshell, that's kind of how we have got a canvas that's accessible. There's a lot of more details and like little nitpicks here and there, but that is the gist of how it works. So I'd be happy to answer any questions you have. That's the QR code that you can or cannot scan. Up to you. It's the source code for this specific slide deck that will show you exactly what's going on. The code is a little bit messy. But who cares? Whatever. We're learning here, right?

So, yes, that's all. Please feel free to ask me any questions. So, the first question here for you is why do we need an extra HTML element for the focus indicator? Can't we draw the shape with canvas? Yes. You can draw the focus indicator with the canvas, but the advantage of using an extra HTML element is that you get that focus visible CSS pseudo class. That means you leave it up to the browser to decide whether or not you need to draw the canvas. Otherwise, you need to write a lot of logic yourself. The best way to actually know whether or not you're supposed to use a focus indicator is to use the built-in pseudo class, because then the browser can tell you, no, this is for a keyboard user or this is for a mouse user.

Why shouldn't we use HTML for relatively static UI components like the example stock chart? Why shouldn't we use HTML for relatively static UI components? So, you could. In theory, you could. So, I think for some charts, in theory, you could have a single HTML element for all of those bars, rather than just one that constantly updates. The most general purpose way to do it was to just have that kind of announcer stream that mimics a video feed. Because we're trying to handle different series types, so, for example, if you've got a line with lots and lots of lines, that's what works best for us. I think I've understood the question well, but if not, feel free to ask a follow-up question.

Why are you using two elements with block and none? Ah, that's a good question. Yes. So, basically, the reason we're using two elements is that we want to be able to call the focus function, because when the screen reader detects that the focus has changed, that forces it to announce something, which is why we need two elements. Because if you just have one element and you recall focus on something that's already in focus, nothing's going to happen. So, that's why you've got to have two of them and you've got to toggle back and forth the focus between those two elements. All right. Have you tested this with really real vision-impaired folks? You know, I love this question and I really wanted to test this on real blind people, but sadly, no. So, I had to basically read a lot of the documentation of like Raria guidelines and things of what are the best things and basically just religiously follow all of the guideline rules to make sure that at least we are compliant with the guidelines and the recommendations to hopefully make it as usable as possible. What are your thoughts on adding an AI screen reader to summarize what users can see in the chart? I think that's a great idea, actually, to have some AI that can summarize the charts. It's maybe a little bit out of scope for our library because we try to keep our library as lightweight as possible. So, I'm not entirely sure if it would be trivial to integrate that within our library, but it's an interesting idea and thank you for the feedback. Yeah. And let's see.

What are your thoughts on adding an AI screen reader to summarize what users can see in the chart? I think that's a great idea, actually, to have some AI that can summarize the charts. It's maybe a little bit out of scope for our library because we try to keep our library as lightweight as possible. So, I'm not entirely sure if it would be trivial to integrate that within our library, but it's an interesting idea and thank you for the feedback. Yeah. And let's see.

If you render everything in both Canvas and SVG for accessibility, why not just use SVG? That's a very good question. I actually forgot to say why we use Canvases. Thank you very much for that question. So, the main advantage of using a Canvas is that it gives you control of the rendering logic. And so, that means you can really optimize the rendering specifically for your use case. If you were to use an SVG, you are sort of at the mercy of the browser, which has a more general-purpose rendering logic. So, that's the main advantage of using a Canvas. Another advantage is that if you've got lots and lots of data points, with SVG you need to create an XML tag for each of those data points, and that can put some churn on the web browser as well. Whereas with a Canvas, because there's just one element and you're managing that yourself, you can optimize the performance much, much better. Yeah. And that actually kind of answers one of the questions is, if there's anything you want to add here, is how to make the line chart accessible if it has 300-plus data points?

How to make the line of a line chart accessible? Okay. Right. Okay. That's an interesting question. How to make the line of a line chart accessible? We make the specific markers for the specific data points. And if we've got 300 data points, there could potentially be some improvement that we can do here, because essentially, we have a bit of a naive approach where we start on the first element, and every time you press write, it just goes to the next. Perhaps in the future, at some point, we could add some options to skip a couple and jump by a hundred or something like that. So we do have a bit of a naive approach that works for if you've got 300 data points plus. But I appreciate there's still probably some work to do on that. Yeah. And wouldn't ARIA Live with Assertive work the same way as switching focus? That's a great suggestion. That's actually the initial POC implementation that we had was using ARIA Live Assertive. And then we found out that ARIA Live with Assertive is actually not as assertive as you want it to be. So switching the focus is actually much more reliable than using an ARIA Live Assertive element. Any idea of how the EU tests applications for accessibility? Do they use similar tools? I don't actually have any idea what an EU test application is.

How does the EU test for accessibility? Oh, right. Okay. I suspect that EU will probably have some sort of compliance regulations to check against. I'm not familiar with all of them, but the main one I was using when checking was the WCAG compliance. And that's the main one that I would have. And usually that one covers enough criteria to comply with other standards as well. This one was kind of answered already.

What are the benefits of Canvas over SVG for graphs? And last question here. What about something highly interactive like a Canvas signature pad? A Canvas signature pad. Ooh. That's a tricky one. Yeah. So you could in theory use the same techniques for a signature pad. The real question for me would be... So this is probably more of a design decision rather than an engineering decision. What kind of text announcements do you do for a signature pad? Because you could curl in arbitrary ways. So you could use the same techniques, but I don't know if it would really deliver good results.

Don't we have any direct screen reader APIs that we can directly trigger from the keyboard or mouse events without using elements? No. Quite simply, no. I wish we did, but we don't. I'm not sure if you can answer this one, but we'll see here. How does this ARIA development stand against us using...

All right. All right. All right. Don't we have any direct screen reader APIs that we can directly trigger from the keyboard or mouse events without using elements? No. Quite simply, no. I wish we did, but we don't. I'm not sure if you can answer this one, but we'll see here. How does this ARIA development stand against us using... How does this stand against us building a Chrome extension that uses TransformersJS to do image to speech? And I can also talk to... I'm assuming they can talk to the UI via Transformers. Right. So I'm not familiar with what Transformers.js is. The last talk. But basically it's in regards to being able to use AI models in the browser. Right. Okay. I haven't really developed this with AI in mind. I don't think there's anything stopping you from, in theory, just getting the bitmap data from the canvas and feeding that to an AI. There's nothing that I've written that prevents you from doing that. Nice. Thank you so much. Thank you.