Browser, API and Assistive Technology: A Love Triangle

Hello, everyone. Wow. Another conference talk. It is so fun tagging along as a presentation assistant. But I wonder where we are today. Oh, let me check my context here. Don't worry about that. We are at JS Nation in Amsterdam.

I have been looking forward to being on this stage. Isn't it great to see everyone here? Yes, it is. Everyone looks ready to know more about all the different layers involved in the accessibility journey. Sorry, it is lagging. The video is lagging.

Does anybody want to be my assistant? Please? You just have to read the script. Yes. Thank you so much! Sorry. Sorry about this. Okay. You will read here. You are the assistant. Yes. But we've done that part. Okay. So, yeah. Everyone looks ready. Yes, it is. Everyone looks ready to know more about all the different layers that are involved in the accessibility journey. That's the spirit. So positive. Thank you.

There. That's what I'm here for. This talk needed a lot of research and focus. Remember, when we didn't understand all of the systems between the software developer and the assistive technology. Yep. You're right. A lot of research indeed. But once we discovered that the systems could be compared to a good old fashioned love triangle, things started to make sense.

Yeah. And we keep on learning, right? Because the systems are also constantly changing. A few years ago, some APIs that we will discuss today were not even supported. Whoa, whoa, whoa. Spoiler alert. Please only mention what we need to know right now. Of course. Sorry. Dear audience, we will walk you through how each company contributes to their accessibility relationship. Where the strength and where the drama.

Perfect. And while we know that the world is mobile first, in the interest of time and because the history of assistive technology has been on desktops, we will focus just on the accessibility journey on desktops today. Did I forget something? No. But you're running out of tokens. If you're great for more tokens, I can introduce you. Would you like for me to do that? No, I think I can do that myself. Thank you. Thank you. That is hello world in the name in the language that has given me the name. My name is and I am South African. I live now in Sweden and work as a software engineer and accessibility champion at the beloved home furnishing company IKEA. I'm also a conference committee member at one of Sweden's largest developer conferences already.

And I volunteer and advocate for digital accessibility as well as bridging the digital divide. I welcome you to this talk. It's great to finally be here. My goal is that at the end of this talk, we will make accessibility testing a little less frustrating. I know I've had my fair share of frustrations. Sometimes, just sometimes, it's actually not your code.

Let's get into it. I'm going to start us off with a case. A few weeks ago, I led an accessibility test session with my team. One of my colleagues had to test keyboard navigation. Along the way, he found a focus management issue. He could navigate on the elements behind the modal. This is obviously a failure because a keyboard-only user would have to tab through all the irrelevant content behind the modal to get back to the modal. So we asked another colleague to check.

Same browser, same operating system, but different results. How was that possible? Luckily, they had me. But we will collect clues along this talk and see if we can solve this. Our first stop is the browser. As developers, the browser is the closest relationship we have to the accessibility journey. When we look at our DevTools, we have the possibility to see that our HTML has translated into an accessibility tree. The accessibility tree is an extraction of our roles, names, and properties of our elements. These are meaningful semantics that are required to be communicated by the screen reader to the users and sometimes to other assistive technology like Braille readers.

These are meaningful semantics that are required to be communicated by the screen reader to the users and sometimes to other assistive technology like Braille readers. So when we map the accessibility tree, we actually have guidance from the W3C. We find the accessible name and description computation as well as the accessibility mapping specs. These specs only have one job. Given this HTML element, what should the browser put in the accessibility tree? So if we consider a button, the HTML accessibility mapping spec will put in the role, state, and properties of the button. The accessible name and description computation will then fill in the name and the description slot. The specs define what should be in the accessibility tree, but they don't talk about the structure of it. Each browser has the flexibility on how they can apply them.

And what does this mean? It means that each browser creates its own accessibility tree. An internal accessibility tree. For example, some browsers flatten some data points more than others. So unlike the DOM tree, the accessibility tree is not standardized. This presents the first layer of discrepancy in our data. The next point is going to be our platform APIs. I'll start with some surveys from the screen reader survey number 10 by WebAIM and then the state of the frontend report by Software House. On one hand, we have the majority of screen reader users using Mac, using Windows as their primary operating system. And then on the other hand, we have frontend developers using Mac as their primary system when they're working. This already presents a mismatch in the accessibility journey.

And I say this because operating systems have their own accessibility APIs. The APIs provide protocols for conveying necessary information from the accessibility tree to the assistive technology. The APIs don't coordinate across operating systems. Our first API is the one from Mac and it's known as the AX API. VoiceOver has deep integration with this API. And then following that, we have the modern API for Windows is the UI automation and Microsoft Active Accessibility is a legacy one and it's been superseded by UI automation. Narrator, JAWS, and NVDA are able to support both of these Windows APIs. These APIs sit in the operating system and the browser and any application as well as your custom app can talk to them. But we also have something special. We have iAccessible2 and this is another accessibility API but it's actually embedded into the browser, namely Chromium-based browsers as well as Firefox. So, it doesn't sit in the operating system. Before the adoption of UI automation, it was the primary API for these browsers.

So far, we've discussed the browser and accessibility APIs, each providing different instructions on handling the accessibility tree. This tree isn't a fixed structure but a model reshaped to suit each platform API, introducing variability in its representation. Screen readers play a crucial role in converting digital text to speech, benefiting visually impaired users. From the initial IBM screen reader in 1986 to popular options like JAWS, NVDA, and VoiceOver, screen readers have evolved significantly, with ongoing AI advancements enhancing features like detailed image descriptions.

Combinations of browsers, screen readers, and platform APIs create intricate systems, impacting accessibility. For Windows, Edge with Narrator utilizing UI automation API is common, while Safari on Mac interacts with the AX API for VoiceOver. Although streamlined stacks owned by one company appear straightforward, real-world usage shows a diversity of complex combinations. Notably, Chrome and Firefox present challenges due to their interactions with both operating system APIs and browser-embedded APIs.

Since then, we have had a few more invented and used. The most popular of screen readers according to the survey is JAWS. And JAWS is a paid screen reader for Windows. We have NVDA followed closely in popularity and it's the free and open source option. VoiceOver is the native option for Apple devices. There have been some many interesting AI development and innovations for screen reader technology. For example, Apple recently announced some new features in VoiceOver. They are promising contextualized and more detailed AI descriptions for images.

Okay. So, those were our components and their qualities. But they don't act alone, right? We can have like several combinations within the system. I think we should walk through some of the browsers and screen reader combinations and see what's at play. It might be easier to start with the combinations that were intended for Windows and Mac. For Windows, the browser would be Edge and then narrator would be the screen reader and Windows would have exposed the UI automation platform API. For Mac, we would have Safari and that would talk to the Mac OS and the Mac OS would then expose the AX API to the VoiceOver screen reader. When the whole stack is owned by one company, the pipeline looks clean. But life is hardly that simple.

Only 7% of users from the screen reader survey use the VoiceOver and Safari combination. In reality, there are numerous and often far more complicated combinations to consider. Chrome and Firefox are the browsers that probably account for the most complexities. Let's explore their paths. In this journey, we have both operating system level APIs and the browser embedded APIs. UI automation is the accessibility API for Windows and browsers and screen readers are working towards supporting it as the default instead of IAccessible2. Chrome has been actively increasing its UIA support. So, UIA is the API that's primarily exposed on Chrome instead of IA2, but IA2 is now has-is supported for historic compatibility. For Firefox, it has a slow adoption for UI automation and so it primarily exposes IAccessible2. When it comes to the screen readers, it becomes even more complicated.

So, JAWS has comprehensive support for both UI automation and IAccessible2, while NVDA is increasing its support for UI automation but primarily works with IAccessible2. Narrator exclusively supports UI automation. JAWS and NVDA can inject their own code into browsers to create a virtual buffer for quicker processing. This ability to make educated guesses on HTML errors can mask bugs, emphasizing the importance of bug reporting for accessibility improvement.

Various repositories host bug listings to aid developers in addressing accessibility issues. A centralized platform for testing different browser and assistive technology combinations, like A11y support, can provide valuable insights. Projects such as the Accessibility Compact Data Project aim to integrate accessibility information directly into development tools and browsers.

The speaker highlights the complexity in the accessibility journey, involving internal accessibility trees, platform APIs, and screen reader compatibility. Providing a mental model for debugging accessibility challenges, the speaker aims to streamline the accessibility testing process for developers.

NVDA and JAWS have the ability to make educated guesses when HTML is wrong. This means that the bugs can be masked. Your code is actually broken, and if somebody is using it without an assistive technology, it won't work for them. If you do find bugs, the responsible thing to do is to report them. We find several repositories where you can find bugs that have been listed. This also helps us when we are unsure to go into these repositories and see if there's anything that relates to something that we're struggling with.

But wouldn't it be great if there was one place where we could find what works and what doesn't work for different combinations? While we have some options, you can go to A11y support and find the results of different tests that have been done related to screen readers. Here's a test conducted in 2021. I don't mean to pick on VoiceOver, but it's quite easy. We can see that for VoiceOver, when you have an area described by attribute and you also have a reference for role alert, it doesn't work. So they've given the result of the test here. Sometimes, these tests can be outdated, but I think it's a good start.

Another project I would like to highlight is the Accessibility Compact Data Project by Lola Odelola. They want to make it easier for developers to know if a feature that works in the browser will also work on assistive technology. They want to take it a step further and actually integrate that information into the tools and browsers that we already use as developers, so we don't have to go to yet another platform to find this information. My name is Tepisa Luhutjo, and I thank you for the privilege of your time. It has been an absolute pleasure talking to you.

This is the combinations of all the accessibility, the journey. We have an internal accessibility tree by the browser, platform APIs treating accessibility trees differently, and screen readers with varied compatibility to platform APIs. Debugging accessibility can be frustrating without the right setup. Understanding these complexities can lead to a smoother process. My name is Tepisa Luhutjo, and I appreciate your time and engagement.

For developers needing to support multiple operating systems, the setup preference varies. Some utilize virtual machines to test across systems. Operating on Mac can present challenges for cross-system testing. Virtual machines offer options, but effectiveness may vary. Consider using dedicated devices for comprehensive testing.

Libraries like React and ARIA pose challenges. ARIA attributes in HTML can complicate screen reader outputs. A significant number of A11y support tests relate to ARIA issues. ARIA's label computation algorithms and priorities can lead to complexity and best practices recommend caution in its usage.

Maybe even have a dedicated device or multiple devices. Definitely. Yeah.

This first question is, what's your take on libraries like React, ARIA? Oh my. I don't know React, ARIA. Are you familiar with just the ARIA attributes in HTML? Yes, of course. OK, OK. So ARIA in general, I actually consulted a native screen reader user about this talk. And they do say that ARIA is actually the biggest—one of the biggest problems in the screen reader output. And if you actually look at the A11y support website, 80% of the tests are related to ARIA issues. So there is something problematic with that. And I think it's also because when you look at the accessible name and description computation, it has a way of—it has its own algorithm of finding out how we're going to name things. And with the ARIA thing, there's different priorities that it takes over different things depending on how you put the labels on. So ARIA makes things complicated. And I understand why they say best practice is not to use it. But I understand that sometimes it's necessary. Yeah. But ARIA just definitely complicates things for screen readers. I don't—we have to look into that. And I would think this carryover is to React, ARIA. Because React, ARIA probably makes those available to React components. So you're going to see the same issues as you would with the regular ARIA.

This next question, is it possible or worse to overlabel HTML with ARIA attributes using both ARIA description and ARIA described by? Yeah, I do think so. And as I've na—as I've—as I've mentioned, right, the naming and compu—naming and description computation, it has its own sort of process on how it—how it will pick what the name will be and how it will pick what the description will be. But again, each browser makes that decision as well. So you might find that what Chrome puts as a description, another browser has put as another property. So it does complicate things, yeah. Wow.

So it does complicate things, yeah. Wow. I do believe that the W3C, when I look at the guidelines, they are trying to move towards standardizing this, because it would help a lot if all the browsers sort of spoke the same language when it came to the accessibility tree. Right. I mean, that seems super important. Yes. But hopefully it happens soon.

Next question, is there an explanation for some screen readers being more popular than the native OS options? Yeah, I mean, and that's really surprising, right? Like if you're on Mac, you would think you'd use like primarily voiceover. And if you're on Windows, Windows especially, I would think you'd use narrator because that's the one that's built in. However, JAWS has a history of being the go-to supplier for government agencies for sort of public health. So when one does become blind and needs support from the government, JAWS is like where they hand you over to and where they teach you about screen readers and all of that stuff, so it then becomes a default. I see. Yes.

And I guess those are usually like paid products or proprietary. Yes. Exactly. So they have good marketing. I see. Yeah. And then the next one, accessibility always felt like a secondary priority for companies. How do we push the narrative of its importance? What has worked for you? What has worked for me is that I work at a company that has a lot of users and consumers. And of course, it's going to be a company that they're going to like watch, right? But besides that, if you are a smaller company, I don't know. I think empathy is increasing in society. And it is about resources, and I think that it does take to be, I mean, even when companies prioritize it, some people have to champion it.

Some people have to do it outside of the usual time. And when I look at some very influential and, you know, progressive people in the accessibility community, most of them say that they started first doing it themselves and then had to bring it to the company and be like, this is it. I think that one of the biggest problems is that there's no structure for accessibility programs. So it's very difficult to know what the company wants to ask you for. But if you start with something, you try it out and it works. It's much easier than for a company to like, oh, OK, well, you've got the blueprint, let's go for it. But it would take somebody to like have their own initiative. And that's not easy.

I found from personal experience because I've suffered from carpal tunnel before. And every now and then I can't use my keyboard. But it's so funny that I don't think about how accessible my apps are until I can't even use them. Yes, exactly. I've also at work, I've also run what I call accessibility empathy labs. And I stole that idea from Skyscanner. I saw a video by them. And you basically just sit your team down for 30 minutes and you challenge them. And like, use the keyboard only, you know, for like 10 minutes or you put some apps that you can use that blur the screen or cover the screen and say, well, try and navigate our website and see how it feels.

I think if people actually feel it themselves, it might be something that they then remember. Like, instead of just go look at the guide, like that thing is very difficult to read. I don't like to read it myself. But when you start telling people, OK, well, the fact that you can't get through this thing is how everybody lives every day. Maybe then we start to realize that, yeah, this is not fun anymore. Nice. Have you ever done like blindfolds in those sessions? Not blindfolds, but I'm able to like make the screen black. Yeah. And I found a really cool game actually that does like a simulation of some of these things. It's really fun. Yeah. So make it fun.

So make it fun. I know it sucks. Like, why do we have to make some people's struggles fun? But it's a good way to sort of start the conversation and not make it feel like a burden. Definitely. Yeah.

What's the single most important action we can take as developers to improve accessibility? I think get to know native screen reader users or people that use assistive technology. And I mean, you don't have to have one by your side yourself. But like, really, there's tons of videos. There's people talking about it online.

Do you have any experience with, I think I've seen like Playwright has like an accessibility plugin. Right. Yes. Would you recommend using something like that as well? Playwright is I think one of the most popular and effective ones.

There'll be a fire and they won't be able to hear it. So I think like start following people that are creating the content so that you can like get this empathy thing going. That is for me, the one thing. I know it's not a technical answer, but I don't think our solution like hinges on technicality when it comes to accessibility. It really starts with a human first approach. Definitely.

Got time for probably one more question. I think I'll pick this one about, yes, what is the cheapest and or automated ways to test for accessibility that doesn't have resources for more detailed or manual testing? Well, in the age of AI. But I think, look, if it doesn't have resources, more detailed and manual testing, I think it's enough to have two different operating systems and just run that with the respective screen readers, you will actually get very far. If you just even just do two separate operating systems. Mac with Safari, Windows with Chrome or Firefox. All right. And I think it will work.

Do you have any experience with, I think I've seen like Playwright has like an accessibility plugin. Right. Yes. Would you recommend using something like that as well? Playwright is I think one of the most popular and effective ones. I myself haven't gone into automated just yet. And it's because like I really like to get a sense of the things. But I think Playwright is a good option. Awesome. Thank you for that. Yeah, of course. That's all the time we have for questions. Thank you for your talk. Thank you. Yeah.