Preparing for an accessibility test

This is the first article in a series about how to run an accessibility test. The companion video and information on performing an accessibility test are available from the Accessibility Testing project page.

If you’re new to accessibility, check out the many resources at the W3C Web Accessibility Initiative (WAI). A good, free foundational course is the W3C Introduction to Web Accessibility.

Introduction

When I started learning about testing for web accessibility, I came across a lot of checklists of what to test for but I struggled with two things. One, I didn’t understand the Web Content Accessibility Guidelines (WCAG) well enough to know if I was covering everything. Two, I didn’t know where on the page to start testing and how much I should test at a time.

The answer to both of these is to start somewhere. A little bit of accessibility testing of what you understand is better than no testing. I like to think of accessibility as a spectrum from less accessible to more accessible. Our goal is to improve access to our websites—a progressive and ongoing task. Accessibility is a huge discipline with many facets that takes time to do well. I hope this project helps ease you into the process.

Methodology

At the beginning of each test, we must establish what we are testing for given the different versions and levels of WCAG. Industry standard is to test WCAG 2.1 Level A and AA, 50 discrete success criteria (SC); that is what this methodology follows. What I found overwhelming was the best order in which to test each SC. Do I start at the top with 1.1.1 Non-text Contrast? What if an SC isn’t applicable to the webpage I’m testing?

I’ve created an accessibility testing spreadsheet to help with this.

Download the accessibility testing spreadsheet

The Overview sheet is a place to list the project name, date, WCAG version being tested, the environment used for testing (browser/screen reader combinations) and any tools used for testing.

The Scope sheet is a place to track what you’ll be testing. Later on in the Scope section, I address how to break up a page into components, or chunks of content, for testing. Provide each component with an ID, name, description of how to locate the component and the URL of test page. Update the “Status” column to reflect if the component is in testing, under review or completed.

Below I outline the columns in the Component sheet. For each component tested, create a clone of the Component sheet and rename it. The workbook includes two components to start.

Type

Each of the 50 SC are assigned a “Type” category. The spreadsheet is sorted by “Type” initially, grouping related SC into the following 12 categories:

  • Active Controls
  • Adaptability
  • Contrast
  • Dynamic Changes
  • Errors
  • Interactivity
  • Keyboard
  • Multimedia
  • Order and Focus
  • Page Level
  • Semantics
  • Sensory

Let’s look at the SC under “Active Controls” to understand how grouping SC makes testing easier:

  • 1.3.5: Identify Input Purpose
  • 2.4.4: Link Purpose (In Context)
  • 2.5.3: Label in Name
  • 3.2.1: On Focus
  • 3.2.2: On Input
  • 3.3.2: Labels or Instructions

We can see from the SC numbers that these are spread across a number of guidelines with gaps in the numerical order. While there’s nothing wrong with starting at 1.1.1, I find it easier to test related SC at the same time.

WCAG success criterion

This spreadsheet lists all 50 SC (rows 2-51). Each success criterion is linked to its W3C “Understanding” page which lists use cases and remediation resources. You can sort the “WCAG Success Criterion” column to reorder the SC from 1.1.1 to 4.1.3 instead of using the “Type” grouping.

Level

Each success criterion displays its WCAG level, A or AA. You can filter the “Level” column to display just A or just AA.

WCAG criterion description

The quick reference description of each success criterion is listed in the “WCAG Criterion Description” column. In part two, I discuss what each success criterion covers from a testing perspective and what to look for. In some cases, I’ve added examples or code snippets for reference. Take some time to read these over and understand what WCAG covers.

Status

The “Status” column has a pick list with three values for tracking your testing progress: N/A, Pass and Fail. I like to go through the 50 SC and mark any that don’t apply to the content I’m testing as “N/A”. For example, if there is no video or audio content on your testing page, you can eliminate six of the “Multimedia” SC from testing right away.

If you do not find any issues for a success criterion, mark it as “Pass”. If you find any issues at all for a success criterion, mark it as “Fail”. Filter the “Status” column to see only “Pass” or “Fail” results.

Issues

This is where you will write up any accessibility issues you find while testing the component. Let’s look at an example. You’re testing 2.4.4: Link Purpose (In Context). The test page lists several articles with “Read More” links. You test with a screen reader and they are all announced the same: read more. This means an assistive technology user cannot distinguish one link from the next.

In the “Issues” column, write a concise description explaining

  • What content has the issue (“Read More” links)
  • What the issue is (links don’t provide context)
  • Why it’s an issue (users of assistive technology can’t distinguish between “Read More” links)

For simplicity, I suggest writing up all issues you find with each success criterion in the same “Issues” cell instead of creating a new row for each issue.

Use the browser inspector (F12) to determine if there are issues with the HTML.

The Inspector tab displaying HTML and CSS code in Firefox Devtools
Firefox Devtools – Inspector tab

Recommendations

In addition to identifying issues, we should provide advice on how to remediate the issues. This takes time. It’s only through the experience of testing different kinds of content that you learn the best way to solve accessibility issues. You might want to make recommendations after completing all testing if you need to look up examples and research how to fix problems. Writing solid remediation advice comes with patience and practice.

You need to have a good understanding of how HTML and CSS work to display content. You need to have a basic idea of how JavaScript controls page behavior, for example, enabling a button to open a menu. You don’t necessarily need to be able to code JavaScript but you do need to be able to discuss expected behavior and how to achieve it programmatically, like adding keystroke events for ENTER and SPACE keys to make a custom button keyboard accessible.

Use the browser inspector (F12) to test proposed solutions by modifying the HTML and CSS. I address this further in part two.

Source code

The browser inspector enables you to copy a source code snippet from the HTML or CSS to paste into the “Source Code” column of the spreadsheet. This enables developers to locate the issue more easily in their code base for remediation.

<a href="article.html">Read More</a>

Environment

Industry standard is to test with two or more of these combinations:

  • Chrome browser and JAWS screen reader on Windows
  • Firefox browser and NVDA screen reader on Windows (this demonstration)
  • Safari browser and VoiceOver screen reader on Mac

Below are the most-used browser/screen reader combinations in use according to the WebAIM Screen Reader Survey #9 (May-June 2021).

Most common screen reader and browser combinations
Screen Reader & Browser# of Respondents% of Respondents
JAWS with Chrome50032.5%
NVDA with Chrome24616.0%
JAWS with Edge19412.6%
NVDA with Firefox1499.7%
JAWS with Firefox744.8%
VoiceOver with Safari724.7%
NVDA with Edge553.6%
ZoomText/Fusion with Chrome332.1%
JAWS with Internet Explorer301.9%
VoiceOver with Chrome241.6%
ZoomText/Fusion with Edge181.2%
Other combinations1449.4%

Tools

I’m using a group of freely available tools for this demonstration but I am not endorsing any one in particular. It’s important to test with different tools to find out what works best for your situation.

A good way to learn about accessibility problems is to use an automated accessibility checker. These tools will scan the source code and outline certain issues that can be tested for automatically, like color contrast. It’s important to double-check these flagged issues for yourself and add issues to the spreadsheet only if the flagged issue is valid.

For this demonstration, I’m using the axe devtools Firefox extension. Once installed, it will add an “axe DevTools” tab to your browser devtools (F12). In part two, we’ll run the scan and explore the results.

Browser devtools panel in Firefox with the axe DevTools tab displayed
axe DevTools tab in Firefox

I use the following tools to help me test for other issues:

Scope

It’s important to assess what webpage or website you’re going to be testing so you can break up repeated elements into smaller components. This eliminates the problem of testing the same thing on multiple pages, e.g. navigation. If you’re testing a single webpage, that can often be treated as a single component. But if you’re testing a website with multiple pages, try to save yourself some work by breaking up the content:

  • Header
  • Navigation (global and local)
  • Search
  • Sidebar
  • Footer
  • Main content

Within the main content, you might want to break up lengthy pages into smaller components like carousel, image gallery, form, video player, etc. depending on what content you’re testing. There’s no sense in testing the same kind of content over and over; you want a good sample of the different types of content found within the website. It’s a judgement call by you as the tester as to how large or small a component should be.

Conclusion

You should now have a better idea of what to test and how to test it. In part two, I perform an accessibility test and track my results in the accessibility testing spreadsheet.

View the video: Preparing for an accessibility test

Read part two: Performing an accessibility test

Performing an accessibility test

View the companion video: Performing an accessibility test

In part one, we looked at setting up our environment for accessibility testing, including configuring the accessibility testing spreadsheet. We scoped the homepage of the test site, BoomerangTags.com, into the following eight components to make testing easier:

  1. [Site] Header
  2. [Home page] Hero
  3. [Home page] Pet tags
  4. [Home page] Ratings
  5. [Home page] Video
  6. [Home page] Business information
  7. [Site] Footer
  8. [Site] Cookies settings

The [Site] components of Header, Footer and Cookie settings appear on every page of the site and need to be tested only once. Additionally, we have a “Common Issues” component for tracking any site-wide problems affecting multiple pages, like 2.4.1: Bypass Blocks.

Explore the component

We begin our testing with the 01 [Site] Header component.

screen shot of boomerangtags.com site header with horizontal navigation, dropdown menus, a search widget with autocomplete
BoomerangTags.com homepage site header

Using a mouse, keyboard and screen reader, explore the component. We want to get an idea of what kinds of controls and widgets the header contains. We can also do a first pass for success criteria like 2.4.7: Focus Visible when we’re tabbing through the focusable elements with the keyboard. We can discover hidden menus that display on hover or widgets that expand by activating a button.

The [Site] Header contains the following content:

  • Utility navigation with two links
  • Navigation landmark with a list of 10 items with links
    • Two links have menus
    • The last link in the list opens a search widget
  • Search field with autocomplete and a button
    • Autocomplete list contains several links
  • Shopping cart link

Test the success criteria

Do a first pass through the 50 success criteria (SC) to identify which ones are applicable to the component you’re testing. For [Site] Header, we’re able to mark 19 SC as not applicable either because the content doesn’t exist (video/audio) or the content is out of scope (page level). In part two, we tested 16 of the remaining 31 SC.

Download the BoomerangTags.com testing spreadsheet to explore the results.

Fails

  1. 2.4.4: Link Purpose (In Context)
  2. 1.4.11: Non-text Contrast
  3. 1.4.3: Contrast (Minimum)
  4. 2.4.3: Focus Order
  5. 2.4.7: Focus Visible
  6. 4.1.2: Name, Role, Value

Passes

  1. 2.5.3: Label in Name
  2. 3.2.1: On Focus
  3. 3.2.2: On Input
  4. 3.3.2: Labels or Instructions
  5. 1.3.4: Orientation
  6. 1.4.10: Reflow
  7. 1.4.12: Text Spacing
  8. 1.4.4: Resize text
  9. 1.4.5: Images of Text
  10. 2.3.1: Three Flashes or Below Threshold

Conclusion

While we found failures of six of the SC so far, we found more than six total issues. For 1.4.3: Contrast (Minimum), we have five different examples of foreground text that does not have at least 4.5:1 contrast with the background color. For 4.1.2: Name, Role, Value, we have at least three different issues and will likely have more to log.

In our next session, we will test the remaining 15 SC. Coming soon!

The web is inherently accessible

Originally posted 7 February 2021. Updated 13 April 2022.

It all started when I built the Accessible Web tool which removes CSS and JS from a webpage, leaving plain HTML. I built it as a sarcastic response to a developer who changed his website to require visitors to disable JavaScript before they could view the content. The reactions to this gatekeeping on Twitter were hilarious! So many angry developers didn’t get the joke that they put up similar barriers when they require that people have JavaScript enabled to view anything on their websites.

I created an About page for the project that requires the user to disable CSS to view a message about web accessibility and the importance of semantic markup. The following is a blog article version of that webpage with code examples.

Accessible Web - please disable CSS to view this website

The web is inherently accessible

Your weekly reminder that the web is accessible by default and it’s our design decisions that stop it being accessible #a11y

k.mar (@Kevmarmol_CT) on Twitter December 7, 2020

Why did you have to disable CSS to view this webpage? No reason other than a design choice that excludes sighted people.

Did you know? This is how many visitors “view” webpages already:

  • Search engines
  • Bots
  • Site crawlers
  • Analytics
  • Blind people

This webpage is fully accessible to people with screen readers and Braille displays. But many websites are not due to poor design choices that exclude some people.

What is web accessibility?

When a page is accessible, it was developed with the intention of working for as many people with disabilities as possible. A good place to start learning is the W3C’s Introduction to Web Accessibility. Find out the different ways people with disabilities interact with the web.

Web Content Accessibility Guidelines (WCAG)

WCAG is a set of success criteria for determining if a page is accessible, led by four guiding principles:

  • Perceivable: Information and user interface components must be presentable to users in ways they can perceive.
  • Operable: User interface components and navigation must be operable.
  • Understandable: Information and the operation of the user interface must be understandable.
  • Robust: Content must be robust enough that it can be interpreted by a wide variety of user agents, including assistive technologies.

These contain guidelines and a hierarchy of success criteria from Level A to Level AAA. Many accessibility laws, and current best practice, point to WCAG 2.1 Level AA compliance. There are 50 discrete success criteria to evaluate, though many are not applicable to all pages. For example, if a page doesn’t contain video, you don’t have to evaluate against success criteria for captions or audio descriptions.

See the full list of success criteria

Did you know? The WCAG guidelines were first published in 1999. Web accessibility is not a new concept but a lot of people are learning about it only now.

Semantics

So what’s the point? The point is to develop accessible pages from the bottom up, starting with semantic HTML. A whole lot of developers think they know HTML but are actually pretty sloppy about it. Many don’t think it matters if they use a link or a button, but it does. Every semantic mistake introduces accessibility issues into your code. If you’ve never really “learned” HTML, check out this beginner’s guide to writing good HTML.

By far, CSS color contrast issues are the most frequent accessibility issues I encounter, but the HTML ones are problematic too. Outlined below are the top HTML-related accessibility issues I encounter.

Headings

If you visually scan this page, you can quickly see how it is broken up into sections. That’s due to using headings or the h1-h6 elements. It’s important that every page have at least one h1 so people and search engines know what the topic of the page is. From there, cascade down to h2, h3, and so on.

Did you know? People using screen readers can navigate by headings in much the same way that sighted people can visually scan the page for items of importance.

Buttons and links

Buttons and links may seem similar but they have very different semantic uses. If you use them interchangeably, people can get confused about what a button or link is going to do when activated.

Buttons

Buttons are used for controlling actions on the page, such as a form submit button or toggle button.

It’s preferable to use a native button element instead of creating a custom one:

  • It already has the button role
  • It’s keyboard focusable
  • It’s activated with ENTER and SPACE keys

The button element can have various states depending on its purpose:

Collapsed

<button aria-expanded="false">Menu</button>

Expanded

<button aria-expanded="true">Search</button>

Not pressed

<button aria-pressed="false">Light mode</button>

Pressed

<button aria-pressed="true">Dark mode</button>

Links

Links are used for navigation, literally for linking to another page or place on the same page. When you use a link, people expect to go somewhere new when it’s activated.

It’s important that the link text properly convey the link purpose. Avoid generic text like “click here”.

Same page link

<a href="#headings">Headings</a>

New page link

<a href="/site-map/">Site map</a>

External link

<a href="https://eff.org">EFF</a>

Email link

<a href="mailto:info@eff.org">info@eff.org</a>

Images

The one thing about accessibility most people know is that images need alt text. This may seem straightforward but let’s look at three examples.

1) Alt text

Semantically speaking, every image needs an alt="" attribute. This alone will pass an automated accessibility checker. If an image is purely decorative, you can even leave the alt value empty. But if an image provides context to the content, the alt text must accurately describe the content of the image for people who cannot see it.

<img src="cats.jpg" alt="two cats on an easy chair under a blanket">
two cats on an easy chair under a blanket

2) SVG

Inline SVG doesn’t support the alt attribute. Instead, add role="img" to the svg element to identify it as an image. Give it an accessible name using the aria-label attribute on the svg.

<svg role="img" aria-label="Bar chart showing the years 2016 to 2021 as the percentage of US consumers with a streaming video service increased from 52% to 78%." xmlns:xlink="http://www.w3.org/1999/xlink" version="1.1" xmlns="http://www.w3.org/2000/svg" viewBox="0 0 650 400">...</svg>
US Consumers 52% 64% 69% 74% 78% 78% 2016 2017 2018 2019 2020 2021 0 20% 40% 60% 80% 100%

Go to the Tables section for information on providing access to all data points in the graph.

3) Icons

Any time you display icons that convey meaning to the user, you must provide a text equivalent for users of assistive technology. For example, it’s common to insert an external link icon after link text that opens in a new tab or window.

Icon fonts are not announced reliably by assistive technology. First, hide the icon using the aria-hidden attribute. Then provide a text equivalent with visually-hidden text.

Exposed icon

☎ 1-800-799-SAFE

<a href="tel:1-800-799-7233">☎ 1-800-799-SAFE</a>

Hidden icon

Call 1-800-799-SAFE

<p><a href="tel:1-800-799-7233"><span class="visually-hidden">Call</span><span aria-hidden="true">☎</span> 1-800-799-SAFE</a></p>

Tables

There are many legitimate uses for tables on the web but they are often coded incorrectly. Table data cells need corresponding table headers. This allows the applicable table header to be read by a screen reader before the table data cell contents. Additionally, tables need captions to provide a description of the table to people using screen readers.

In the SVG example under Images, we looked at a svg bar chart where we provided a snapshot of the data using aria-label. We need to provide all the data in a format that assistive technology can navigate and an easy way to do that is to provide all data points in a table.

US consumers with a streaming video subscription
Year US Consumers
2016 52%
2017 64%
2018 69%
2019 74%
2020 78%
2021 78%
<table>
  <caption>US consumers with a streaming video subscription</caption>
  <thead>
    <tr>
      <th>Year</th>
      <th>US Consumers</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <td>2016</td>
      <td>52%</td>
    </tr>
    <tr>
      <td>2017</td>
      <td>64%</td>
    </tr>
    <tr>
      <td>2018</td>
      <td>69%</td>
    </tr>
    <tr>
      <td>2019</td>
      <td>74%</td>
    </tr>
    <tr>
      <td>2020</td>
      <td>78%</td>
    </tr>
    <tr>
      <td>2021</td>
      <td>78%</td>
    </tr>
  </tbody>
</table>

Forms

Every input needs a label. It’s really that simple. The label needs to be visible and persistent (avoid using placeholder). This helps people remember what information they’ve entered. Programmatically link each pair matching the for attribute on the label with the id attribute on the input.

This enables a couple things:

  • People can now click or tap on the label to give focus to the input. This is especially useful for checkboxes and radio buttons that often have small hit areas.
  • People using screen readers will now hear the label announced when the input is in focus.

When you have a group of related form fields, like checkboxes or radio buttons, group them with a fieldset element and provide an accessible name for the group with a legend element.

<fieldset>
  <legend>Cat's Colors</legend>
  <p><label for="black"><input type="checkbox" id="black"> Black</label></p>
  <p><label for="white"><input type="checkbox" id="white"> White</label></p>
  <p><label for="brown-tabby"><input type="checkbox" id="brown-tabby"> Brown tabby</label></p>
  <p><label for="orange-tabby"><input type="checkbox" id="orange-tabby"> Orange tabby</label></p>
  <p><label for="torbie"><input type="checkbox" id="torbie"> Torbie</label></p>
  <p><label for="gray"><input type="checkbox" id="gray"> Gray</label></p>
</fieldset>

Many modern browsers do cursory form field validation with the required attribute though error messages are not always accessible. Let assistive technology users know when a field is required by adding the aria-required="true" attribute to inputs.

<label for="email">Email<span aria-hidden="true"> *</span></label>
<input type="email" id="email" autocomplete="email" name="email" aria-required="true">

For a deeper dive into forms, check out my blog article and talk Structuring accessible forms.

In summary

  • Web accessibility is an ongoing process
  • Use semantic markup: Native HTML + ARIA
  • Headings help people navigate
  • Buttons and links do different things
  • Images need text equivalents
  • Use tables for data
  • Structure forms in accessible ways