Browser, API and Assistive Technology: A Love Triangle

Hello, everyone. Wow, another conference talk. It is so fun tagging along as a presentation assistant. But I wonder where we are today. Oh, let me check my context here. Don't worry about that. We are at JS Nation in Amsterdam.

I have been looking forward to being on this stage. Isn't it great to see everyone here? Yes, it is! Everyone looks ready to know more about all the different layers involved in the accessibility journey.

Sorry. It is lagging. The video is lagging. Does anybody want to be my assistant? Please. You just have to read the script. Yes, thank you so much! All right. Hi, everyone. Sorry about this. Okay. We will read here. You are the assistant. We have done that part. Okay. Everyone looks ready. Yes, it is! Everyone looks ready to know more about all the different layers involved in the accessibility journey. That's the spirit. So positive. Thank you. That's what I'm here for. This talk needed a lot of research and focus. Remember, when we didn't understand all of the systems between the software developer and the assistive technology.

Yep, you're right. A lot of research indeed. But once we discovered that the systems could be compared to a good old fashioned love triangle, things started to make sense. And we keep on learning, right? Because the systems are also constantly changing.

A few years ago, some APIs that we will discuss today were not even supported. Of course. Sorry. Dear audience, we will walk you through how each company contributes to the accessibility relationship. Where the strengths and where the drama. And while we know that the world is mobile first, in the interest of time and because the history of assistive technology has been on desktops, we will focus just on the accessibility journey on desktops today.

Did I forget something? No. But you're running out of tokens. If you're great for more tokens, I can introduce you. Would you like for me to do that? No, I think I can do that myself. Thank you. Dumela Lefati. That is hello world in the name in the language that has given me the name Tepiso. My name is Tepiso Luhujo, and I am South African. I live now in Sweden and work as a software engineer and accessibility champion at the beloved home furnishing company, Ikea.

Dumela Lefati. That is hello world in the name in the language that has given me the name Tepiso. My name is Tepiso Luhujo, and I am South African. I live now in Sweden and work as a software engineer and accessibility champion at the beloved home furnishing company, Ikea. I'm also a conference committee member at one of Sweden's largest developer conferences, Orodiv, and I volunteer and advocate for digital accessibility as well as bridging the digital divide. I welcome you to this talk. It's great to finally be here.

My goal is that at the end of this talk, we will make accessibility testing a little less frustrating. I know I've had my fair share of frustrations. Sometimes, just sometimes, it's actually not your code. Let's get into it. I'm going to start us off with a case. A few weeks ago, I led an accessibility test session with my team. One of my colleagues had to test keyboard navigation. He didn't need a screen reader. Along the way, he found a focus management issue. He could navigate on the elements behind the model. This is obviously a failure because a keyboard-only user would have to tab through all the irrelevant content behind the model to get back to the model.

So, we asked another colleague to check. Same browser, same operating system, but different results. How was that possible? Luckily, they had me. But we will collect clues along this talk and see if we can solve this. Our first stop is the browser. As developers, the browser is the closest relationship we have to the accessibility journey. When we look at our dev tools, we have the possibility to see that our HTML has translated into an accessibility tree. The accessibility tree is an extraction of our roles, names, and properties of our elements. These are meaningful semantics that are required to be communicated by the screen reader to the users and sometimes to other assistive technology like braille readers. So, when we map the accessibility tree, we actually have guidance from the W3C. We find the accessible name and description computation as well as the accessibility mapping specs. These specs only have one job.

Given this HTML element, what should the browser put in the accessibility tree? If we consider a button, the HTML accessibility mapping spec will put in the role, state, and properties of the button. The accessible name and description computation will fill in the name and the description slot. The specs define what should be in the accessibility tree, but they don't talk about the structure of it. Each browser has the flexibility on how they can apply them. And what does this mean? It means that each browser creates its own accessibility tree, an internal accessibility tree. For example, some browsers flatten some data points more than others. So, unlike the DOM tree, the accessibility tree is not standardized. This presents the first layer of discrepancy in our data.

The next point is going to be our platform APIs. I'll start with some surveys from the screen reader survey number 10 by WebAIM, and then the state of the frontend report by Software House. On one hand, we have the majority of screen reader users using Mac, using Windows as their primary operating system. And then on the other hand, we have frontend developers using Mac as their primary system when they're working. This already presents a mismatch in the accessibility journey. And I say this because operating systems have their own accessibility APIs.

The APIs provide protocols for conveying necessary information from the accessibility tree to their system technology. The APIs don't coordinate across operating systems. Our first API is the one from Mac, and it's known as the AX API. VoiceOver has deep integration with this API. And then following that, we have the modern API for Windows is the UI Automation. And Microsoft Active Accessibility is a legacy one and it's been superseded by UI Automation. Narrator, JAWS, and NVDA are able to support both of these Windows APIs. These APIs sit in the operating system, and the browser and any application as well as your custom app can talk to them. But we also have something special. We have IAccessible2. And this is another accessibility API, but it's actually embedded into the browser, namely Chromium-based browsers as well as Firefox. So it doesn't sit in the operating system. Before the adoption of UI Automation, it was the primary API for these browsers. So, so far we've talked about the browser and we've talked about the accessibility APIs. Each API has different instructions on how to handle the accessibility tree. So then the accessibility tree is not a single fixed structure.

It's a semantic model that gets reshaped to fit each platform API. This is what my assistant calls more drama, right? Let's move on then to screen readers. Our screen readers convert digital text into synthesized speech and they are an integral tool for many individuals who are blind or visually impaired. The first true screen reader was called the IBM Screen Reader and this was released in 1986. Since then, we have had a few more invented and used. The most popular of screen readers according to the survey is JAWS. And JAWS is a paid screen reader for Windows.

We have NVIDIA follow closely in popularity and it's the free and open source option. VoiceOver is the native option for Apple devices. There have been so many interesting AI development and innovations for screen reader technology. For example, Apple recently announced some new features in VoiceOver. They are promising contextualized and more detailed AI descriptions for images. Okay. So those were our components and their qualities. But they don't act alone, right?

We can have, like, several combinations within the system. I think we should walk through some of the browsers and screen reader combinations and see what's at play. It might be easier to start with the combinations that were intended for Windows and Mac. For Windows, the browser would be Edge and then narrator would be the screen reader and Windows would have exposed the UI automation platform API. For Mac, we would have Safari and that would talk to the MacOS and the MacOS would then expose the AX API to the VoiceOver screen reader.

When the whole stack is owned by one company, the pipeline looks clean. But life is hardly that simple. Only 7% of users from the screen reader survey use the VoiceOver and Safari combination. In reality, there are numerous and often far more complicated combinations to consider.

Chrome and Firefox are the browsers that probably account for the most complexities. Let's explore their paths. In this journey, we have both operating system level APIs and the browser embedded APIs. UI automation is the accessibility API for Windows. And browsers and screen readers are working towards supporting it as the default instead of IAccessible2.

For Firefox, it has a slow adoption for UI automation and so it primarily exposes IAccessible2. JAWS has comprehensive support for both UI automation and IAccessible2. NVDA is steadily increasing its support for UI automation. Narrator supports only UI automation. JAWS and NVDA are actually able to inject their own code directly into the browser, creating a virtual buffer for quicker processing of webpages without constant OS queries.

JAWS has comprehensive support for both UI automation and IAccessible2. NVDA is steadily increasing its support for IAccessible2 and primarily works with it currently. Narrator supports only UI automation. The ability of JAWS and NVDA to inject their own code into the browser creates a virtual buffer for faster webpage processing without constant OS queries. These screen readers can make educated guesses to mask bugs caused by incorrect HTML, emphasizing the importance of reporting bugs for accessibility improvement.

Repositories provide a platform to list and address bugs, aiding developers in troubleshooting. Efforts like the Accessibility Compact Data project aim to integrate accessibility information into developer tools and browsers seamlessly. Tests conducted on screen readers, such as VoiceOver, reveal specific compatibility issues with certain webpage elements. Projects like Accessibility Compact Data and accessible test results in support resources strive to simplify the process of ensuring browser features work across different assistive technologies.

Centralized resources showcasing test results of assistive technology compatibility can benefit developers in understanding what works and what doesn't across various browser-screen reader combinations. Initiatives like Accessibility Compact Data by Lola Odelola aim to streamline accessibility information integration into existing developer tools and browsers. The combination of accessibility features and technologies is crucial in enhancing the overall user experience for individuals with disabilities.

NVDA and JAWS have the ability to make educated guesses when HTML is wrong, masking bugs that could affect users without assistive technology. It is crucial to report bugs to enhance accessibility and troubleshoot code issues. Repositories listing bugs provide valuable resources for developers struggling with specific issues. Consolidating information on what works and what doesn't for different combinations of technologies can streamline the testing process and improve accessibility.

Accessibility testing results reveal specific compatibility issues, such as VoiceOver's challenges with certain webpage elements. Projects like the Accessibility Compact Data by Lola Odelola aim to integrate accessibility information seamlessly into developer tools. The goal is to simplify the process of ensuring browser features work across various assistive technologies. Efforts to centralize test results offer developers insights into improving accessibility and user experience.

The integration of accessibility features and technologies, as demonstrated by the Accessibility Compact Data project, is essential for enhancing user experience. Understanding the complexities of the accessibility journey, from browser trees to platform APIs and screen reader compatibility, can aid developers in debugging and improving accessibility. By providing a smoother process and mental model, developers can navigate accessibility challenges more efficiently. Tepisa Luhujo expresses gratitude for the interaction and highlights the importance of addressing accessibility complexities.

Sometimes these tests can be outdated, but I think it's a good start. Another project I would like to highlight is the Accessibility Compact Data project by Lola Odelola. They want to make it easier for developers to know if a feature that works in the browser will also work on assistive technology. They want to take it a step further and actually integrate that information into the tools and browsers that we already use as developers so we don't have to go to yet another platform to find this information.

I'm going to then show you a slide that brings everything together. So this is the combinations of all the accessibility, the journey. We have an internal accessibility tree by the browser. We have platform APIs that treat the accessibility trees differently. And then we have screen readers that have their own compatibility to the platform APIs. I believe that the frustration sometimes when we're trying to debug our accessibility is that we don't have the right setup. I hope that this mental model and some of the complexities that I've discussed with you here today will give you a much smoother process.

My name is Tepisa Luhujo and I thank you for the privilege of your time. It has been an absolute pleasure talking to you. We have a few coming in, but I have one question. First of all, thank you for enlightening me. Like, I didn't know there were so many incompatibilities in so many places that things could go wrong in that flow. Yes. Yeah. I guess for developers that want to or probably need to support multiple operating systems, like, do you have a preferred setup? Like, are you running virtual machines to try to make sure you can test across all Yeah, I think for me, I'm one of the developers that use Mac, right? So I'm already on the outs of this whole process. And that was what was frustrating for me. It felt like every time I needed to test, I needed to go borrow a Windows computer.

Yeah. But we do have systems and products, albeit paid, that you can do virtual machines on so that you can test with your NVDA and your JAWS. But I don't know how effective they are, actually, because you can see that there are a lot of places where there could be some mishaps depending on what system you're using.

Yeah. Yeah. But I think it's worth it. Okay. Yeah. It's tricky. Maybe even have a dedicated device or multiple devices. Definitely. Yeah. Yeah.

All right. This first question is, what's your take on libraries like React-Aria? Oh, my. I don't know React-Aria. Are you familiar with just the Aria attributes? Yes, of course. Okay. Yeah. So Aria in general, I actually consulted a native screen reader user about this talk. And they do say that Aria is actually the biggest, one of the biggest problems in the screen reader output. And if you actually look at the A11y support website, 80% of the tests are related to Aria issues.

So there is something problematic with that. And I think it's also because when you look at the accessible name and description computation, it has a way of it has its own algorithm of finding out how we're going to name things. And with the Aria thing, there's different priorities that it takes over different things depending on how you put the labels on. So Aria makes things complicated. And I understand why they say best practice is not to use it, but I understand that sometimes it's necessary. Yeah. Yeah. But Aria just definitely complicates things for screen readers.

I don't we have to look into that. And I would think this carry over is to React Aria because React Aria probably makes those available to React components. So you're going to see the same issues as you would with the regular Aria.

Is it possible or worse to over label HTML with Aria attributes using both Aria description and Aria described by. Yeah. I do things. And as I've mentioned, right, the naming and description computation, it has its own sort of process on how it will pick what the name will be and how it will pick what the description will be. But again, each browser makes that decision as well. So you might find that what Chrome puts as a description, another browser has put as another property. So it does complicate things, yeah.

Wow. I do believe that the W3C, when I look at the guidelines, they are trying to move towards standardizing this because it would help a lot if all the browsers sort of spoke the same language when it came to the accessibility tree. Right. Yeah. I mean, that seems super important. Yes. Hopefully, it happens soon.

Is there an explanation for some screen readers being more popular than the native OS options? Yeah. I mean, and that's really surprising, right? Like if you're on the Mac, you would think you'd use like primarily voiceover. And if you're on Windows, Windows especially, I think you'd use narrator because that's the one that's built in. However, JAWS has a history of being the go-to supplier for government agencies for sort of public health. So when you are, when one does become blind and needs support from the government, JAWS is like where they hand you over to and where they teach you about screen readers and all of that stuff. So it then becomes a default. Yes. And I guess those are usually like paid products or proprietary. Yes. Exactly. So they have good marketing.

Exactly. So they have good marketing. I see. Yeah. All right. Next one, accessibility always felt like a secondary priority for companies. How do we push the narrative of its importance? What has worked for you? What has worked for me is that I work at a company that has a lot of users and consumers. And of course like it's going to be a company that they're going to like watch, right? But besides that, if you are a smaller company, I don't know. I think empathy is increasing in society and it is about resources and I think that it does take to be, I mean, even when companies prioritize it, some people have to champion it. Some people have to do it outside of their usual time.

And when I look at some very influential and, you know, progressive people in the accessibility community, most of them say that they started first doing it themselves and then had to bring it to the company and be like, this is it. I think that one of the biggest problems is that there's no structure for accessibility programs. So it's very difficult to know what the company wants to ask you for. But if you start with something, you try it out and it works, it's much easier than for a company to like, oh, okay, well, you've got the blueprint, let's go for it. But it would take somebody to like have their own initiative. And that's not easy.

Yeah. I found from personal experience, because I've suffered from carpal tunnel before. And so every now and then like I can't use my keyboard. But yeah, it's so funny that like I don't think about how accessible my apps are until I can't even use them. Exactly. I've also at work, I've also run what I call accessibility empathy labs, and I stole that idea from Skyscanner. I saw a video by them. And you basically just sit your team down for 30 minutes, and you challenge them, like, use the keyboard only, you know, for like 10 minutes. Or there's some apps that you can use that blur the screen or cover the screen and say, well, try and navigate our website and see how it feels. I think if people actually feel it themselves, it might be something that they then remember, like, instead of just go look at the guide.

Like, that thing is very difficult to read. I don't like to read it myself. But when you start telling people, okay, well, the fact that you can't get through this thing is how everybody lives every day. Maybe then we start to realize that, yeah, this is not fun anymore. Have you ever done like blindfolds in those sessions? Not blindfolds, but I'm able to like make the screen black. Yeah. And I found a really cool game, actually, that does a simulation of some of these things. It's really fun. Nice. Yeah. So make it fun. I know it sucks. Like, why do we have to make some people's struggles fun? But it's a good way to sort of start the conversation and not make it feel like a burden. Definitely. Yeah, this next one along the same lines. What's the single most important action we can take as developers to improve accessibility? I think, get to know native screen reader users or people that use assistive technology. And I mean, you don't have to have one by your side yourself, but like, really, there's tons of videos, there's people talking about it online. I've come across several things on LinkedIn where somebody's explaining like, even a hotel room. I didn't know there was such things like accessible hotel rooms, you know, like a fire alarm. I rely on the sound. But some people actually need the fire alarm to have light, because they can't hear it. There'll be a fire and they won't be able to hear it. So I think like start following people that are creating the content so that you can like get this empathy thing going. That is for me, the one thing. I know it's not a technical answer. But I don't think our solution hinges on technicality when it comes to accessibility. It really starts with a human first approach. Definitely. Yeah. One more question.

I think I'll pick this one about, yes, what is the cheapest and or automated ways to test for accessibility that doesn't have resources for more detailed or manual testing? Well, in the age of AI. But I think, look, if it doesn't have resources, more detailed and manual testing, I think it's enough to have two different operating systems and just run that with the respective screen readers. You will actually get very far. If you just even just do two separate operating systems, Mac with Safari, Windows with Chrome or Firefox. All right.

And I think it will work. Do you have any experience with I think I've seen like Playwright has like an accessibility plugin. Right. Yes. Would you recommend using something like that as well? Playwright is I think one of the most popular and effective ones. I myself haven't gone into automated just yet. And it's because like I really like to get a sense of the things. But I think Playwright is a good option. Awesome. Thank you for that. Yeah, of course.

That's all the time we have for questions. Thank you for your talk. Thank you.