a11y Testing Is Broken: How Lighthouse and Axe Fail in Real Projects

As Phil told you, I came from Spain and lived there for six months already. And the main knowledge you should learn like starting living there is una cerveza, por favor. I'm very happy to be here and we're going to talk about what Alba told us. I have no idea. I didn't hear. Yeah.

So the first thing I need to warn you, this talk won't be kind of a solution because the things I'm going to talk today about, they are depending on the solutions you already have in your systems, but I will give you at least an idea how you can test your applications for web accessibility. Also, mobile accessibility, why not? Even like the approach is okay for everything. And the wind is low. And also the idea of it, like you should understand the idea of what to check if you don't know what to check after you already checked everything. So yeah, let's go.

But first let me introduce myself. My name is Gulasha. I prefer this short, sweet, no one can like in Sweden, I am Gulash. If you know, you know. Yeah. I work at SEMrush as an accessibility project manager. I also work for Web Technologies, VTM ambassador, like a lot of labels. If you have any questions you can come to me. I am good at things like accessibility 100%. So yeah, just come to me and ask. You can join me on LinkedIn and Twitter. Ask there everything you want to. A little bit about myself. Like this is actually the picture of some people, like in our Minsk, in our Belarus community. I'm originally from Belarus who were doing this conference at the very beginning. So it's really cool to be here and to support their ideas. Yes.

So I started in 2008 as a computer science student. Then I went to my first job like right away after the college. And then in 2018, community got me, front end got me and accessibility also. I was the only person in Belarus, I guess, who knew something about accessibility. So I started working on the front end accessibility project. It was like a huge journey, a lot of inspiration there. So really happy to be here. And in 2024, I joined SEMrush as an accessibility project manager.

Oh my god, it's really difficult to know the web design and all this stuff. It's like really difficult not to go into the details and just watch everything. Yeah. SEMrush is a leading online visibility management and content marketing platform. We have a lot of offices around the world, a lot of people, and only one accessibility expert there. A lot of pressure, but still we're like started caring about accessibility a few years ago. So I am very happy to join this way in our company.

And yeah, a few weeks ago, I was required to pass the medical, how to say, coaching something like this. So it was a requirement from Spain. So as a worker in Spain, I should pass this training. It was six hours of information. I was like really overloaded with everything. And also I am not an English is not my first language. So it's really difficult to understand sometimes what they want from me. So I was really hoping for translate some things during going through this interesting stuff.

So I was really hoping for translate some things during going through this interesting stuff. So I tried to copy the text and put past it into the Google Translate. You can imagine what happened. Nothing happened. I couldn't even copy it. Because the whole text here is a picture, like a big picture with a text on it. So you can't copy it. You can do anything with it. Like only AI can help you to recognize all the text, but it's not the case for you if you don't know how to do it.

Accessibility here is broken. I couldn't access the content. So if I would be a blind person, I wouldn't get this information at all. So medical things are not accessible. It's really fun. That's why I decided to talk about it. I couldn't even test it with accessibility technologies with some plugins because they don't have this AI feature to recognize texts from your pictures.

Just in case, what is accessibility about? First of all, you'll meet this abbreviation like A-11-Y. It's the same as with internalization and all this stuff. So accessibility is a thing. According to the W3C initiative, it's a practice that ensures that your content is available for everyone. It doesn't matter if the person is blind or they just, I don't know, missed something or they are just distracted with something. So it doesn't mean all individuals should access your content.

And yeah. According to the W3C initiative, it's a practice that ensures that your content is available for everyone. It doesn't matter if the person is blind or they just, I don't know, missed something or they are just distracted with something. So it doesn't mean all individuals should access your content. So of course, it's more about people who are not really able to do it because of their health issues. But yeah.

So why do we care? First of all, we care because of the users. A lot of people in the world have real disabilities, medical conditions. 15% of the world population, that's a lot. But also we should have in mind that there are people who are not included in this 15%. So we should think about us who broke their arm or something like this. I'm pretty sure something like this happens to someone of you. I am left-handed. I can't use scissors at all. So it's like, why?

And also some situational stuff, like really high lightness, I don't know, around you. Like really light sun. So you can't see anything from your screen because of the poor contrast. Or you are driving or you are carrying a child, something like this. Like you have one arm now. So yeah. It's really important to remember these people are not included in this number. And also here, we live in Europe. Europe is about accessibility. It's really cool. But this year, the new European Accessibility Act is coming. So all the companies who provide people with some web interfaces should be accessible by July 28, 2025. In a month from today, everyone should be accessible.

So this is the date when Europe will be able to fine you, to take your money, to tell you you have three months to fix everything on you, or you'll be fined with some big monies. And it's not really about this act, because it's already working in the European countries. For example, like a recent case, the airline called Vueling. I fly here with the Vueling. They were fined for 90,000 euros because of their inaccessible website. So it's already happening. Yeah. So that's why I decided it's time to talk about you. About you, not about you. About the accessibility with you. Yeah. And what can we do today? Like in a month, the act is starting working. So the first attempt is to use some automated testing to test everything you have for accessibility. And automated testing is cool. It will say, like, you are 100% accessible with, like, a really small effort. It's really, really cool. We don't judge this approach at all. We're happy that you will use it. Do it. And W3C already updated the list of the automated tools. Like they collected this list. There are many, many tools in this world. You can use, like, 70 for automated testing. That's a lot. And I really love that this website shows how they when they were updated. So you can choose the recent ones, the new ones, to for your cases, just try them. Yeah.

So today we're going to talk about Google Lighthouse and Agile DevTools. Because they are the most popular on the market. The first thing you will hear if you are going to test your websites for accessibility is these two tools. Lighthouse is easy because it's already in your browser. And Agile DevTools is, like, it's a really cool tool. Because they provide you with a lot of information. Lighthouse also provides you with a lot of information. But Lighthouse feels more core. But the funny thing here is that Lighthouse and Agile, they use the same accessibility core. Like, the same library, which is checking accessibility. But they, for some reason, use it differently. So I don't know why. That's why I decided to compare them. Because I really want you to see that not all the tools are working the same way. Even if they use the same code under the hood. So yeah. Lighthouse and Agile DevTools really recommend. Agile DevTools is just a plugin for your browser. So install it. And use the free version of it. That's enough for now. So these tools are usually claiming that they are... They have a huge coverage.

Like coverage of all the accessibility issues. Like about 60% of the issues will be caught on your website when you test it first time with Lighthouse. Or actually, the same. Like Agile core. Agile DevTools Lighthouse, the same. But the experts are not so sure. They are showing us... The experts... Like really huge audit companies in America, who are having these numbers. Like 25% of the issues on your website could be caught automatically. Only 25. Like true or false things. This is green. This is yellow. Is it color contrast enough? No. Yes. So what they can count, kinda. There are 35% of the issues which should be reviewed by the person who is testing. Like manual review. And also there are 40% of the issues which are... Which can't be caught automatically at all. And the kind of the issues are about the visual stuff. The text understanding, the labels. Like are they clear enough for the user? Also some technical details I will show you a bit later. So it's really... I show you this resource where they collect all the types of the issues which can be tested automatically. Or there are some ideas how to test them, etc. So just go there and check. Maybe give some ideas. It's open source. So yeah.

Let's test real life examples. And first, of course, I want to move back to the text alternatives for graphics. For images, like I couldn't copy it. So if the image is a text, if it lacks the text presentation, you will face the problems like you can't access the content with the assistive technologies. You can't copy the text. And you can't automatically translate tools. And this is because I'm a web developer. Of course I open DevTools every time. So this is because you don't have any alternative text for this image. Like it's empty. But it's there. So it couldn't be tested as a problem with Lighthouse. Because it shows like everything is really good. You have alt. It's empty. It seems... It means that it's a decorative image. You don't need to put anything there. So it's not an issue at all.

I even couldn't test this website because it's iFrame. And not all the tools can test iFrame. So just be careful. But some tools can. In SEMrush, we do... It's really important to understand that there is a complex graphic. You forget about it. But it's really important to give an explanation for this kind of graphics. For example, we give the explanation for our charts, for our data in some way for the users who can't read it or see it.

So it's really cool to do some things. Keyboard interactions. I love this example. It's like forever. It stays there forever. V3C Schools, of course, it's kind of an outdated resource. But they have this custom select box example. So how to style it? So I checked the website with Aux Dev Tools for accessibility. And what I see? I see that there is no issues. There are six issues for color contrast. And one issue for the select. The native one. But there is no issues for this blue select. It's accessible. It means it's accessible. So okay. Cool.

And Lighthouse is the same. It finds the same issues, like background and foreground color contrast. And also the select doesn't have the accessible name. So cool. We are covered. But why? Why? Because diff is everywhere. So this component can be caught by the testing tools. Because it's just diffs with text. So it's nothing. Like for the Aux Dev Tools, it doesn't have any type. It's not a field. It's not a select. It's just diffs with text. So no. No testing for this component. Unfortunately. Yeah. So you should like... You even don't know about it. There is no issue. It's okay. The other thing with the keyboard is focus visibility. And I show you the case. The initial state is two buttons.

Left and right button. Then I press tab key. And I see the focus state, like beautiful border around one button. And then I press tab key to see the same border on the right button. What I see, actually, is a small line between these elements appears. So it's not having the same way of the bordering. We always in front and we always have issues with the focus visibility. Keyboard focus or mouse focus doesn't really matter. Yeah.

So this is an issue which is visual. The Lighthouse, the Aux Core, they can't find this kind of issues. So you need to check it manually with your eyes. Also, maybe some kind of testing. Like the... I don't know. Like the screenshot testing or something like this. Yeah. And there are no issues. If you test this website for accessibility, you find just some unlabeled button and just one. So it's a great result. And there is no issues. And also, Lighthouse and Aux show the same result.

This is the thing I really want you to show, is the voice control tool. Like every operational system has this tool which is called voice control. So you can control, obviously, the operational system with your voice. You say some comments, like... I hope I switched it off. Like start listening, stop listening, and so on. So I decided to test the website. So you can't move your arms. You can only operate with your voice. Yeah. So the website of the transportation company, really unrecognizable. So I have tested first this website for accessibility with Lighthouse. It shows like there are some links which are not labeled. Okay.

Then it's like a really great result. No issues. Then I tested with Aux. Aux gives us different results for some reason. Like there are some links, there are some images which are not labeled, and also some SVG which is not labeled. So Aux kind of gets more information. So okay. And these are links. So we can like really fix them really fast and we'll get like 100% of the result.

Let me show you a small video how I operate with my voice with the system, what I see. I hope you can hear it. I'm not sure if I can hear it. Click search. So I'm asking to click search. Show names. Nothing happens. I assume that the labels are different. So for search, it's search trips, actually. Click change departure date today. So my experience became like longer. Click change departure date today. It's not just a departure field. That's something I don't see. Click search trips. Now I ask to click search trips and it works. It works and goes to the different page.

Go back. I go back to the previous page and yeah. Hide names. Ask to hide names and just try to. Click search. Do it once again. Click search. Nothing happens. Click search trips. And search trips happens, like works well. This is because of the accessibility labels which are different from what you see. Start listing. Just a second. Yeah. When I check these buttons, I see that the name, accessible name, is search trips. And the content was search. So for people who don't, they just see your content. They rely on the labels. So it's really important for them to understand, like to click directly.

They don't want to speak more or find some places to speak to where the real label exists. Same for the continue buttons on the second page. They have an accessible name. So select this trip and continue. So it's how should I guess? I only ask for click continue and it won't work. Because the label is much longer. So yeah. We don't overuse real labels, por favor.

Page zooming is a huge thing. Like this is the website before I zoomed the page. I checked it for accessibility. It's already funny. Because like when I opened DevTools, it became like smaller. And I already see the issues with the content, which goes out of the screen. But there is no issues, like zero issues. Yeah. So right. It's accessible, fully accessible. And the same. These results are different from ActiveTools.

Like now Lighthouse gives us more information about this. So yeah. I zoom up to 200% in the browser. And I see that content goes out of the screen. There is no like scroll. So I can't get this content already. So it's inaccessible. Also, some labels. Like 100% they are OK. 200% they are hiding under the other content. So it's really something you can't check automatically. And yeah. So such things like we have some new feature. We add it to our website. And we never check how it works with the existing functionality.

So here the menu is hidden under the banner about the country change. What I really want us to get from this talk is the impact of the Accessibility Act on all developers, emphasizing the importance of checking interfaces for accessibility. Although tools are helpful, human understanding is crucial as tools may not catch everything accurately. Retesting is essential, and insufficient testing can exclude users from the product, affecting everyone, not just those with disabilities. Remember, thorough testing is key to inclusivity. Testing Europe's applications for accessibility involves a mix of automated tools like Aux Dev tools and manual testing, focusing on screen reader and assistive technology usage. These auditors use a combination of free and paid tools, ensuring comprehensive testing.

Join me on LinkedIn and connect with me. Testing in Europe includes companies using similar tools for accessibility testing, starting with Aux Dev tools and performing manual testing for screen readers and assistive technologies. These companies emerge rapidly, creating a robust testing environment. The core approach involves using a variety of tools for comprehensive testing. The testing process for the Accessibility Act will likely involve a blend of automated and manual testing, ensuring a thorough evaluation of accessibility features.

But you should remember that they won't check everything, actually. They are not humans. So they can't understand the content properly. So you should be that human who will understand this content. And also, don't hesitate to retest things. Please. Well, I live in Spain, so por favor is just por favor. Si. Yes. So yeah. Please test everything. And remember that insufficient testing will exclude people from your product. And it's not only about people who have some disabilities. It's about everyone. I give you some list of the links from this talk. So you can directly use it. There is also the link to the slide. So you can get them right after the talk. Yeah. So thank you so much. Please contact me everywhere. I'm moving to the next one. Join me on LinkedIn. And I'm really happy you were here with me. Thank you.

The first one is, any idea how Europe tests applications for accessibility? Do they use similar tools? I have the idea. Because now there are companies appearing like mushrooms, you know? Like, every second, new companies who will be kind of auditors. So they test with the same tools. They usually use Aux Dev tools, or they go for more, like, heavy things. But they all use this core. So it will be definitely like this. So Aux is a good tool to start with. And also, they will do a manual testing, of course, because it's really important to test the screen reader, like how the blind people use it, and all the other assistive technologies. But they will do it really fast. So I don't expect anything, like, really hard for the first stage of the testing for the act. So we'll see how it goes. But yeah, they will definitely use these tools, because they are kind of free or even paid. And there is nothing really new on the market. So it's really good. So there's only these ones. Yeah, yeah. So you open the W3C, open the most popular tools, take them, and check everything with them. So it's like this. OK. So at least if we test with those tools, we are good to go. OK. So next question. What is the best way to ensure that your product is accessible to all demographics globally? What can I say? First of all, you need to understand where you are presented. Like, are you presented in the countries which are requiring really hard, like in America in the United States, they require accessibility? So it's really important to have this kind of certification. So there are WCAG standards we have for accessibility.

It's a list of requirements. So you need to be accessible by all the requirements, like for this particular country or something like this. So you go to this list. You check the requirements. You test your application according to the requirements, because this is what will do the auditor. And also, it's really important to get this certification. If you need to be accessible in these countries, you need to get the certification from some kind of well-known company. It's not a real certification. We don't have such things in accessibility. So yeah, some really well-known company who is kind of trusted. So you do it. And then you make an audit. You find all the issues. You're scoping them. You are giving them the, how is that called, priorities. And then you fix them. So I would say, in terms of the requirements, it should be the auditor first who will find all the issues. Or you do it yourself. List all the issues. But if it's not required for you, so just start with automated testing. Please do it. Please find all the issues you can fix automatically. Then start learning how users use the websites, how they use the assistive technologies. And fix these things you find. It's difficult. It's a huge way. It's an ongoing process. Yeah, exactly, exactly. Over it without your users being there, specifically.

So the next question is, any tips on how to advocate for accessibility in our team's companies? So this is kind of how you explain accessibility to your boss. Yeah, exactly. Exactly, exactly. So like, any tips? You already are working on this. It depends, it depends. Yeah, I am working. I was hired for the accessibility. I was hired twice. And both times, actually, thrice. And all three times, it was like, we have a project on accessibility. We need an accessibility expert. So please do it for us. So they were already motivated. But in case you don't have anyone, I definitely can share the link to the latest Accessibility Club Summit where a really huge person explains how to advocate for accessibility through really good examples with the, how to say, the seeds bombs. Like you put the seeds into the ground, and they will grow like beautiful flowers. And you do it there and there. And yeah, so the approach is really fun. But yeah, it's difficult. You can start from yourself. You can start from a small place. We also recommend to start like, really like under, I'm not sure how to explain, but like not announcing it like a huge something.

Because everyone will really try to put you somewhere in a dark room. Yeah, so just start slowly and try to involve more people. Try to give them some advice. Try to learn together. So it's more about the really long process of insisting on some practices in all the spheres. Like designers should know about it. And it's really good to start from designers because the requirements are not that big as for the web developers, for example. Web developers are the most difficult part of it.

Yeah, Android developers are like, yes, we can do it, but we don't want to do it. But wow, results. Oh my God, yeah, yes. Finally, we can do it. And iOS developers are like, we are really cool. We already have accessibility. Yeah, so it's different, but it's a process. And you need to really bring everyone there one by one, I would say. Well, at least now with the European Act, we have something to fight. Yeah, 100%. You can bring this to your boss. And present the find that will go next.

Yeah, well, yeah, we also have a question. Which tool are you using for voice control? It was a default Mac OS tool called Voice Control. And also in Windows there is the same tool. And also the users are using more advanced tools like Dragon voice controlling tools and so on. So also on your phone, the same tools like they are calling, they are all called the same voice controller kind of close to it. But this default one, so just find it in your settings, switch it on and try to do it. But first, Google how to do it, because... Yeah, configure it, for me, it speaks all the time and it's really confusing because I don't know... It's a different one. It's a different one. It's a screen reader.