This article highlights the practical challenges and solutions in integrating automated accessibility testing into existing frontend development workflows. It provides valuable insights for engineers looking to enhance their testing strategies without disrupting core framework functionalities.
At Slack, customer love is our first priority and accessibility is a core tenet of customer trust. We have our own Slack Accessibility Standards that product teams follow to guarantee their features are compliant with Web Content Accessibility Guidelines (WCAG). Our dedicated accessibility team supports developers in following these guidelines throughout the development process. We also frequently collaborate with external manual testers that specialize in accessibility.
In 2022, we started to supplement Slack’s accessibility strategy by setting up automated accessibility tests for desktop to catch a subset of accessibility violations throughout the development process. At Slack, we see automated accessibility testing as a valuable addition to our broader testing strategy. This broader strategy also includes involving people with disabilities early in the design process, conducting design and prototype review with these users, and performing manual testing across all of the assistive technologies we support. Automated tools can overlook nuanced accessibility issues that require human judgment, such as screen reader usability. Additionally, these tools can also flag issues that don’t align with the product’s specific design considerations.
Despite that, we still felt there would be value in integrating an accessibility testing tool into our test frameworks as part of the overall, comprehensive testing strategy. Ideally, we were hoping to add another layer of support by integrating the accessibility validation directly into our existing frameworks so test owners could easily add checks, or better yet, not have to think about adding checks at all.
Unexpected Complexities: Axe, Jest, and React Testing Library (RTL)
We chose to work with Axe, a popular and easily configurable accessibility testing tool, for its extensive capabilities and compatibility with our current end-to-end (E2E) test frameworks. Axe checks against a wide variety of accessibility guidelines, most of which correspond to specific success criteria from WCAG, and it does so in a way that minimizes false positives.
Initially we explored the possibility of embedding Axe accessibility checks directly into our React Testing Library (RTL) framework. By wrapping RTL’s render method with a custom render function that included the Axe check, we could remove a lot of friction from the developer workflow. However, we immediately encountered an issue related to the way we’ve customized our Jest set up at Slack. Running accessibility checks through a separate Jest configuration worked, but would require developers to write tests specifically for accessibility, which we wanted to avoid. Reworking our custom Jest setup was deemed too tricky and not worth the time and resource investment, so we pivoted to focus on our Playwright framework.
The Best Solution for Axe Checks: Playwright
With Jest ruled out as a candidate for Axe, we turned to Playwright, the E2E test framework utilized at Slack. Playwright supports accessibility testing with Axe through the @axe-core/playwright package. Axe Core provides most of what you’ll need to filter and customize accessibility checks. It provides an exclusion method right out of the box, to prevent certain rules and selectors from being analyzed. It also comes with a set of accessibility tags to further specify the type of analysis to conduct (‘wcag2a‘, ‘wcag2aa‘, etc.).
Our initial goal was to “bake” accessibility checks directly into Playwright’s interaction methods, such as clicks and navigation, to automatically run Axe without requiring test authors to explicitly call it.
In working towards that goal, we found that the main challenge with this approach stems from Playwright’s Locator object. The Locator object is designed to simplify interaction with page elements by managing auto-waiting, loading, and ensuring the element is fully interactable before any action is performed. This automatic behavior is integral to Playwright’s ability to maintain stable tests, but it complicated our attempts to embed Axe into the framework.
Accessibility checks should run when the entire page or key components are fully rendered, but Playwright’s Locator only ensures the readiness of individual elements, not the overall page. Modifying the Locator could lead to unreliable audits because accessibility issues might go undetected if checks were run at the wrong time.
Another option, using deprecated methods like <span style="font-weight: 400">waitForElement </span>to control when accessibility checks are triggered, was also problematic. These older methods are less optimized, causing performance degradation, potential duplication of errors, and conflicts with the abstraction model that Playwright follows.
So while embedding Axe checks into Playwright’s core interaction methods seemed ideal, the complexity of Playwright’s internal mechanisms required us to explore some further solutions.
To circumvent the roadblocks we encountered with embedding accessibility checks into the frameworks, we decided to make some concessions while still prioritizing a simplified developer workflow. We continued to focus on Playwright because it offered more flexibility in how we could selectively hide or apply accessibility checks, allowing us to more easily manage when and where these checks were run. Additionally, Axe Core came with some great customization features, such as filtering rules and using specific accessibility tags.
Using the @axe-core/playwright package, we can describe the flow of our accessibility check:
First, we set up our main function, <span style="font-weight: 400">runAxeAndSaveViolations</span>, and customized the scope using what the AxeBuilder class provides.
constructor(page: Page) {
this.page = page;
this.defaultTags = ['wcag2a', 'wcag2aa', 'wcag21a', 'wcag21aa'];
this.bodyText = '';
this.baseFileName = `${test.info().title}-violations`.replace(/\//g, '-');
A11y.filenameCounter = 0;
}// Exclude selectors for known bugs and elements that we do not consider accessibility issues
constants.ACCESSIBILITY.AXE_EXCLUDED_SELECTORS.forEach((excludedSelector) => {
axe.exclude(excludedSelector);
});Critical according to the WCAG. Serious, Moderate, and Mild are other possible severity levels that we may add in the future./**
* Filter violations based on criticality, then ensure we
* are removing any duplicate violations within a single test file
* Please note: this only removes duplicates on a single test, not the entire run
*/
private filterAndRemoveDuplicateViolations(violations: Violation[]) {
return violations
.filter((violation) => ['critical'].includes(violation.impact))
.map(this.mapViolation)
.filter(this.isUniqueViolation.bind(this));
}<span style="font-weight: 400">slack</span> which provides access to all of our API calls, UI views, workflows and utilities related to Slack. Using this fixture, we can access all of these resources directly in our tests without having to go through the setup process every time.<span style="font-weight: 400">slack</span> fixture. This allowed us to call it directly in the test spec, minimizing some of the overhead for our test authors.// Run accessibility checks and get the violations
await slack.utils.a11y.runAxeAndSaveViolations();<span style="font-weight: 400">test.step</span>. We added the custom label “Running accessibility checks in runAxeAndSaveViolations” to make it easier to detect where an accessibility violation has occurred:Test Steps
- Before Hooks
- apiResponse.json— ../support/api/api.ts:137
- browserContext.waitForEvent— ../support/workflows/login.workflow.ts:273
- Running accessibility checks in runAxeAndSaveViolations— ../support/utils/accessibility.ts:54
To kick the project off, we set up a test suite that mirrored our suite for testing critical functionality at Slack. We renamed the suite to make it clear it was for accessibility tests, and we set it to run as non-blocking. This meant developers would see the test results, but a failure or violation would not prevent them from merging their code to production. This initial suite encompassed 91 tests in total.
Strategically, we considered the placement of accessibility checks within these critical flow tests. In general, we aimed to add an accessibility check for each new view, page, or flow covered in the test. In most cases, this meant placing a check directly after a button click, for example, or a link that leads to navigation. In other scenarios, our accessibility check needed to be placed after signing in as a second user or after a redirect.
It was important to make sure the same view wasn’t being analyzed twice in one test, or potentially twice across multiple tests with the same UI flow. Duplication like this would result in unnecessary error messages and saved artifacts, and slow down our tests. We were also careful to place our Axe calls only after the page or view had fully loaded and all content had rendered.
With this approach, we needed to be deeply familiar with the application and the context of each test case.
We spent some time iterating on our accessibility violations report. Initially, we created a simple text file to save the results of a local run, storing it in an artifacts folder. A few developers gave us early feedback and requested screenshots of the pages where accessibility violations occurred. To achieve this, we integrated Playwright’s screenshot functionality and began saving these screenshots alongside our text report in the same artifact folder.
To make our reports more coherent and readable, we leveraged the Playwright HTML Reporter. This tool not only aggregates test results but also allows us to attach artifacts such as screenshots and violation reports to the HTML output. By configuring the HTML reporter, we were able to display all of our accessibility artifacts, including screenshots and detailed violation reports, in a single test report.
Lastly, we wanted our violation error message to be helpful and easy to understand, so we wrote some code to pull out key pieces of information from the violation. We also customized how the violations were displayed in the reports and on the console, by parsing and condensing the error message.
Error - [A11Y]: CRITICAL
Description: Ensures an element's role supports its ARIA attributes
Help: Elements must only use supported ARIA attributes Target selector: #add-channel-tab
Fix all of the following:
ARIA attribute is not allowed: aria-selected="false"
HTML: <button class="c-button-unstyled addTab__brBMy c-tabs__tab js-tab" data-qa="unstyled-button"
Once we had integrated our Axe checks and set up our test suite, we needed to determine how and when developers should run them. To streamline the process for developers, we introduced an environment flag, <span style="font-weight: 400">A11Y_ENABLE</span>, to control the activation of accessibility checks within our framework. By default, we set the flag to false, preventing unnecessary runs.
This setup allowed us to offer developers the following options:
Ownership of individual tests and test suites is often a hot topic when it comes to maintaining tests. Once we had added Axe calls to the critical flows in our Playwright E2E tests, we needed to decide who would be responsible for triaging accessibility issues discovered via our automation and who would own the test maintenance for existing tests.
At Slack, we enable developers to own test creation and maintenance for their tests. To support developers to better understand the framework changes and new accessibility automation, we created documentation and partnered with the internal Slack accessibility team to come up with a comprehensive triage process that would fit into their existing workflow for triaging accessibility issues.
The internal accessibility team at Slack had already established a process for triaging and labeling incoming accessibility issues, using the internal Slack Accessibility Standards as a guideline. To enhance the process, we created a new label for “automated accessibility” so we could track the issues discovered via our automation.
To make cataloging these issues easier, we set up a Jira workflow in our alerts channel that would spin up a Jira ticket with a pre-populated template. The ticket is created via the workflow and automatically labeled with automated accessibility and placed in a Jira Epic for triaging.
A11Y Automation Bug Ticket Creator -
Automatically create JIRA bug tickets for A11Y automation violations
Hi there, Would you like to create a new JIRA defect?
Button clicked.
A new JIRA bug ticket, A11YAUTO-37, was created.
What to do next:
1. Please fill out all of the necessary information listed here:
https://jira.tinyspeck.com/browse/A11YAUTO-37.
2. Please add this locator to the list of known issues
and include the new JIRA bug ticket in the comment.We perform regular audits of our accessibility Playwright calls to check for duplication of Axe calls, and ensure proper coverage of accessibility checks across tests and test suites.
We developed a script and an environment flag specifically to facilitate the auditing process. Audits can be performed either through sandbox test runs (ideal for suite-wide audits) or locally (for specific tests or subsets). When performing an audit, running the script allows us to take a screenshot of every page that performs an Axe call. The screenshots are then saved to a folder and can be easily compared to spot duplicates.
This process is more manual than we like, and we are looking into ways to eliminate this step, potentially leaning on AI assistance to perform the audit for us – or have AI add our accessibility calls to each new page/view, thereby eliminating the need to perform any kind of audit at all.
We plan to continue partnering with the internal accessibility team at Slack to design a small blocking test suite. These tests will be dedicated to the flows of core features within Slack, with a focus on keyboard navigation.
We’d also like to explore AI-driven approaches to the post-processing of accessibility test results and look into the option of having AI assistants audit our suites to determine the placement of our accessibility checks, further reducing the manual effort for developers.
Closing Thoughts
We had to make some unexpected trade-offs in this project, balancing the practical limitations of automated testing tools with the goal of reducing the burden on developers. While we couldn’t integrate accessibility checks completely into our frontend frameworks, we made significant strides towards that goal. We simplified the process for developers to add accessibility checks, ensured test results were easy to interpret, provided clear documentation, and streamlined triage through Slack workflows. In the end, we were able to add test coverage for accessibility in the Slack product, ensuring that our customers that require accessibility features have a consistent experience.
Our automated Axe checks have reduced our reliance on manual testing and now complement other essential forms of testing—like manual testing and usability studies. At the moment, developers need to manually add checks, but we’ve laid the groundwork to make the process as straightforward as possible with the possibility for AI-driven creation of accessibility tests.
Roadblocks like framework complexity or setup difficulties shouldn’t discourage you from pursuing automation as part of a broader accessibility strategy. Even when it’s not feasible to hide the automated checks completely behind the scenes of the framework, there are ways to make the work impactful by focusing on the developer experience. This project has not only strengthened our overall accessibility testing approach, it’s also reinforced the culture of accessibility that has always been central to Slack. We hope it inspires others to look more closely at how automated accessibility might fit into your testing strategy, even if it requires navigating a few technical hurdles along the way.
Thank you to everyone who spent significant time on the editing and revision of this blog post – Courtney Anderson-Clark, Lucy Cheng, Miriam Holland and Sergii Gorbachov.
And a massive thank you to the Accessibility Team at Slack, Chanan Walia, Yura Zenevich, Chris Xu and Hye Jung Choi, for your help with everything related to this project, including editing this blog post!
Continue reading on the original blog to support the author
Read full articleEnsuring mobile accessibility is critical for legal compliance and inclusive user experiences. This post provides practical implementation details for common Android a11y hurdles, like custom actions and semantic announcements, helping engineers build more robust, accessible apps.
This article demonstrates how applying core software engineering principles like caching and parallelization to build systems can drastically improve developer experience and delivery speed, transforming slow pipelines into agile ones.
This article details Slack's successful Deploy Safety Program, which drastically cut customer impact from deployments. It provides a practical framework for improving reliability, incident response, and development velocity in complex, distributed systems.
This article demonstrates a practical approach to significantly improve CI/CD pipeline efficiency and developer experience. By intelligently caching and reusing build artifacts, engineering teams can drastically reduce build times and infrastructure costs.