Stop Guessing, Start Measuring: Quantifying Accessibility

Hello everybody, I'm Giammi. Today, I would like to share with you a case study about how quantifying accessibility have played a crucial role to improve the accessibility of our products at Stack Overflow.

But first, a couple of words about myself. I'm pretty sure most of you here are familiar with Stack Overflow. What you might not know though is that Stack Overflow have an open-source design system we call Stacks. I work as a staff engineer in that team, where we are driving a lot of front-end-related initiatives across our organization.

Little disclaimer, I don't consider myself an accessibility expert, but I had the opportunity to lead the initiative I'm about to tell you. What follows has worked well for our organization, but as anything in software development, your mileage may vary.

Let me start by giving you some context about the accessibility challenges we face at Stack Overflow. Our product teams lack clear accessibility targets. They often fix issues reactively, waiting for problems to be reported instead of proactively preventing them. Our leadership was understandably hesitant to fully invest in accessibility efforts. Without clear visibility into our progress, it was hard to justify prioritizing it alongside other pressing initiatives. And finally, our communities and customers, some of whom with legal requirements for accessibility towards their employees, were struggling to see our commitment to accessibility.

So we knew we needed to do better, both to meet expectations of our clients and communities, and to build more inclusive products in general. So together with our leadership, we started our journey by setting a clear outcome to establish trust with our communities and customers by demonstrating a strong commitment to accessibility. We wanted to go beyond simply meeting legal requirements, we aimed to proactively improve accessibility across our products, making it a core part of how we build. This is how we approach finding our strategy to get to our outcome.

We quickly realized that our design system team, my team, had the most accessibility expertise across the organization. We were in the best position to lead this initiative given our strong understanding of both design and technical aspects of accessibility. Then leadership provided us with a clear outcome to improve accessibility and build trust, but they also gave us the freedom to explore solutions. This empowered us to think creatively and take ownership of how we could achieve this goal. To organize our effort, we use an opportunity solution tree. This is a product management tool that allows us to brainstorm and visualize how each action we consider will contribute to our ultimate goal. With our opportunity solution tree in place, we were able to boil down our strategy to these four main points.

First, we needed to establish a clear accessibility target. Without defined goals, it would be impossible to measure progress or know when we were truly making an impact. Next, we decided to measure the accessibility of our products consistently. We wanted data to back up our effort and ensure we weren't just guessing but actually improving accessibility over time. We also knew we had to lead by example with our design system. Since our design system team had the most accessibility expertise, it was crucial that our design components set the standard for accessibility across the organization. Finally, we wanted to start shifting accessibility left. This means embedding accessibility consideration earlier in the development process so it becomes part of how we do things, not an afterthought. Let's dive deeper into the first part of our strategy.

The first thing we did was to set clear and documented accessibility targets based off the Web Content Accessibility Guidelines, often pronounced WCAG. Having these guidelines gave us concrete, industry-recognized benchmarks to measure our products against. It also created clarity for our teams knowing exactly what accessibility meant in practice. One challenge we faced though was Stack Overflow's signature orange color. Under WCAG guidelines it didn't meet contrast requirements. The reason is that the current WCAG color contrast algorithm is too simple to account for the complexity of human perception. So we made an exception here by adopting the Advanced Perceptual Contrast Algorithm or APCA in short. APCA might become part of the upcoming WCAG 3 specification and allow us to work with the orange while still having a mean to check for color contrast in an automated way. And finally to secure long-term commitment across the organization we formalized these targets through an Architectural Decision Record or in short ADR. This ADR made accessibility a non-negotiable part of our development process, ensuring that every team was aligned and accountable for meeting this target.

So if you're curious about the specifics of this target and how they are documented you can scan this QR code to dive deeper like into our design system documentation about accessibility. Once we had our accessibility targets in place the next critical step was to measure our progress consistently and reliably. We set out to create a synthetic signal to track how we were doing against our accessibility target and give us a clear indication of whether we were moving in the right direction.

For automated check we used a tool called Axecore developed by Deque. It's a powerful tool using many products across the industry that help us to calculate accessibility in an automated way and calculate accessibility score for our product pages. However automated tools like this typically only catch about 50-57 percent of weaker issues in average. So we knew this was just one part of the picture. To fill the gaps we establish also manual scores from various sources including third-party auditors, feedback from the community and client reported issues. This gave us a more comprehensive view of our products accessibility covering things that automation alone couldn't detect.

Finally to make all this data easily accessible and actionable we built an accessibility dashboard. This became a central hub where teams, our community and even clients could track our progress and hold us accountable. Let's now go a bit more in detail into how our accessibility reports both automated and manual contribute to create our accessibility scores. For the automated ones we have GitHub workflows triggered daily that check several weaker rules via Axe Engine and Playwright. Those rules run against a selection of the most important pages of our different products. The workflow results are then transformed into metrics we upload to our accessibility platform.

For the manual issue we have a centralized accessibility board where anybody across the org can log issues. Issues get then triaged by a group of engineers and designers proficient on accessibility. We call this group accessibility champions. Triage issues have a severity field associated to them and they affect the manual accessibility score. We then have an accessibility service that is responsible to pull data from our observability platform and our accessibility central board. Calculate the score and then render them in our publicly available dashboard. And this is how the final dashboard looks like that everybody can consult at accessibility.stackoverflow.net. We're still iterating on this dashboard but even on its current state it provides actionable insights for our team to improve accessibility in our products. In addition to displaying score trends the dashboard highlights accessibility score by page and details all issues identified by the automated runs. We had our accessibility goals in place and a clear way to track progress over time. Next it was time to roll up our sleeves and start tracking some of the accessibility issues in our product. We knew that starting with our design system would have had the biggest impact since our libraries are used to build user interfaces across pretty much all the products. So the design system is where we focus our effort first. As we made improvements we documented and shared WCAG techniques with the rest of the engineering organization ensuring the best practices were widely adopted. Some of the key issues we addressed include improving focus style to ensure clear visual indicators especially for our keyboard only users and we also had the component light skip links allowing users to bypass repeated blocks of content across our pages enhancing navigation for everyone.

Probably the most impactful accessibility improvement we made was rolling out a brand new color palette developed closely with our design team. We used the accessible perceptual contrast algorithm APCA as our guide.

As I mentioned earlier the current WCAG color contrast algorithm is too simple to account for the complexity of human perception. It's especially unreliable when testing certain colors like stucco overflows orange. This is also where we contributed back to open source. APCA being a cutting edge algorithm was not yet supported by the AXE core engine so we created APCA compliance rule for the engine which gave us an automated way to test color contrast across our pages.

The last part of our strategy was to get our organization to think about accessibility early in the development process. We did this through a few key initiatives. First we established accessibility champions within each team. They're responsible for spreading accessibility knowledge and meeting regularly with other champions across the organization. We also work with product and engineering leadership to make sure teams had the space to prioritize and improve their accessibility scores. We created a lean accessibility checklist that automatically shows up as a comment when UI changes are detected in pull requests. And we developed a short internal video training for engineers and designers with practical accessibility tips specific to our needs. We call these accessibility bites.

Perhaps one of the most important things we did to safeguard our accessibility standards was to establish accessibility service level objectives. We essentially use the automated accessibility score as a service level indicator. Each product was assigned an objective. If there is a drop in the score a priority tree incident is automatically created and the on-call accessibility champion is notified. From there there is a seven-day window to identify and fix the root cause of the accessibility regression. By quantifying accessibility we were able to integrate it into processes familiar to our engineers and leadership. Accessibility is no longer a vague intangible concept. It has become as concrete as any other metric we track.

Here are some key learnings from our journey I want to leave you with. First, setting clear accessibility targets for the organization. Without defined goals there is no way to measure success. Equally important is securing organizational commitment on those targets from leadership and from the teams so that accessibility becomes a core priority to everyone. Second, making sure to measure and track product accessibility. When teams can see real progress it drives motivation and it also gives visibility to leadership and customer. It demonstrates commitment and helps quantify the returns on accessibility investment.

It demonstrates commitment and helps quantify the returns on accessibility investment.

Third, shift accessibility left. This means integrating accessibility early in your development process. Invest in educating teams and creating guardrails like automated checks or process-driven steps to prevent regression. It's about setting engineers up for success ensuring that accessibility is part of the process and not an afterthought.

And finally, remember that accessibility work never ends. There is always more we can do. Keep refining processes and invest in user testing to ensure we are truly meeting user needs. Accessibility isn't and shouldn't be a one-time project. It's a continuous journey of improvement.

That's all I have for today. I hope our approach to improving accessibility at Stack Overflow has given you a few ideas on how to prioritize accessibility at your own company.

I want to thank my teammates Dan and Caro, the Stack Overflow's Accessibility Champions Group and the Stack Overflow's Front-end Guild. They all play a crucial role to make what I just presented a reality for our organization.

If you have questions, here you can find my email and my social media handle. Thank you for listening.