Comparing Level Access automated tools to manual accessibility testing

Last updated: 27 January 2023

This article is in response to Adrian Roselli’s article Comparing Manual and Free Automated WCAG Reviews. Go read it first for background.

Automated accessibility testing tools cannot test all of the Web Content Accessibility Guidelines (WCAG) success criteria. Adrian tested four free tools and compared them to his manual testing results. My intent is to add to the body of knowledge by providing results from the Level Access AMP accessibility testing tool. Level Access also provides a free browser extension that runs the same tests called Access Assistant.

I wanted to compare Level Access’s tools to Adrian’s manual testing findings because these tools are what we use at my work and I share his concerns that too many stakeholders lean on automated testing when it uncovers only a portion of potential problems.

Process

I tested the same page, web.dev, and performed a review with the following tools against WCAG 2.1 Level A and Level AA:

I performed the tests on 19 January 2023 with a live version of the site. It does not seem to have changed since Adrian’s testing on 14 January 2023 of these four automated tools:

All references to manual testing results are Adrian’s data. I did not find anything additional in my manual testing.

a screenshot of the web.dev page with a navigation bar across the top with a search icon. Below is some introductory text, a looping video, a decorative image and an image with the text case study.
web.dev webpage – dark mode

Highlights

Number of success criteria failed
Tool Total A AA
Manual 18 11 7
AMP 5 5 0
axe 2 2 0
ARC 3 3 0
WAVE 0 0 0
EAAC 3 3 0

We compare this to the total number of unique failures as some issues have multiple instances but are counted only once and some success criteria have multiple issues.

Number of success criteria failures
Tool Total A AA
Manual 37 24 11
AMP 4 4 0
axe 2 2 0
ARC 3 3 0
WAVE 0 0 0
EAAC 5 5 0

Some tools provide alerts for issues to check manually, including AMP.

Number of alerts
Tool Alerts
Manual 0
AMP 10
axe 0
ARC 7
WAVE 6
EAAC 20

Raw results

This section provides the output as provided by AMP for the various success criteria. It also includes alerts or what AMP calls “Needs Review” issues.

WCAG failures

The following table compares WCAG Level A failures between manual testing and AMP automated results.

Comparing WCAG Level A manual and automated test results
WCAG 2.1 SCs at Level A Manual AMP
1.1.1 Non-text Content Fail
Fail
  1. This svg element does not have a mechanism that allows an accessible name value to be calculated.
    • Rule: Provide alternative text for images
    • 1 instance
    • <svg viewBox="0 0 238 36" fill="currentColor" height="36" width="238" xmlns="http://www.w3.org/2000/svg">...</svg>
1.2.1 Audio-only and Video-only (Prerecorded) Pass
1.2.2 Captions (Prerecorded) N/A
1.2.3 Audio Description or Media Alternative (Prerecorded) N/A
1.3.1 Info and Relationships Fail
Fail
  1. This A does not have a ul element (without an ARIA-assigned role); ol element (without an ARIA-assigned role); an element with a role set to the value: list as a parent; or a ul element (without an ARIA-assigned role), ol element (without an ARIA-assigned role) or element with a role set to the value ‘list’ with an aria-owns attribute set to the ID of the element in the same DOM
    • Rule: Ensure list items are found in a list container
    • 28 instances
    • <a role="listitem" href="/new-patterns-july-2022/" data-category="web.dev" data-action="click" class="card card-vertical"a>...</a>
1.3.2 Meaningful Sequence Pass
1.3.3 Sensory Characteristics N/A
1.4.1 Use of Color Fail
1.4.2 Audio Control N/A
2.1.1 Keyboard Pass
Fail
  1. This A is focusable and has an aria-hidden attribute set to true
    • Rule: Avoid placing inactive elements in the focus order
    • 2 instances
    • <a aria-hidden="true" href="/interop-2022-wrapup/"></a>
    • <a aria-hidden="true" href="/web-platform-12-2022/"></a>
2.1.2 No Keyboard Trap Pass
2.1.4 Character Key Shortcuts N/A
2.2.1 Timing Adjustable N/A
2.2.2 Pause, Stop, Hide Fail
2.3.1 Three Flashes or Below Threshold Pass
2.4.1 Bypass Blocks Pass
2.4.2 Page Titled Fail
2.4.3 Focus Order Fail
2.4.4 Link Purpose (In Context) Pass
2.5.1 Pointer Gestures N/A
2.5.2 Pointer Cancellation Pass
2.5.3 Label in Name Fail
2.5.4 Motion Actuation N/A
3.1.1 Language of Page Pass
3.2.1 On Focus Pass
3.2.2 On Input Fail
3.3.1 Error Identification Fail
3.3.2 Labels or Instructions Fail
4.1.1 Parsing N/A
4.1.2 Name, Role, Value Fail
Fail
  1. The role attribute value of ‘listitem’ given to this A is not allowed. The element’s role attribute should be set to one of the following text values: button | checkbox | menuitem | menuitemcheckbox | menuitemradio | radio | tab | switch | treeitem; or the role attribute can be removed
    • Rule: Ensure ARIA roles, states, and properties are valid
    • 28 instances
    • <a role="listitem" href="/new-patterns-july-2022/" data-category="web.dev" data-action="click" class="card card-vertical">...</a>

NOTE: The rule “Ensure list items are found in a list container” failed in AMP for two success criteria, both 1.3.1 Info and Relationships and 4.1.1 Parsing.

AMP did not fail any WCAG Level AA success criteria so I am not including that table of results. Manual testing found 11 unique failures.

Alerts

AMP provides additional potential issues as a list of “Needs Review” items. These alerts may or may not be WCAG failures and require manual review to determine if there is an accessibility issue.

  1. Avoid inappropriate use of ARIA roles, states, and properties, cites 4.1.2 Name, Role, Value. The A element has an aria-hidden attribute set to the value: true. [2 instances]
  2. Provide valid, concise, and meaningful alternative text for image buttons, cites 1.1.1 Non-text Content and 4.1.2 Name, Role, Value. This button element has a suspicious accessible name value of: all. [1 instance]
  3. Ensure link text is meaningful within context, cites 2.4.4 Link Purpose (In Context). This A element has a suspicious (i.e. lacks purpose or is >150 characters) calculated accessible name value of: css. [1 instance]
  4. Provide synchronized captions for video (which includes audio) or other multimedia, cites 1.2.2 Captions (Prerecorded) and 1.2.4 Captions (Live). This video element does not have a track with kind=captions. [1 instance]
  5. Ensure heading level matches the heading’s visual importance/level, cites 1.3.1 Info and Relationships. [4 instances]
    • This article element contains an incorrect or missing heading level which may cause improper nesting within the document heading hierarchy.
    • This H3 creates an inappropriate jump in heading levels within the document heading hierarchy.
    • This H5 creates an inappropriate jump in heading levels within the document heading hierarchy.
    • This H1 creates an inappropriate jump in heading levels within the document heading hierarchy.
  6. Provide an informative context-sensitive page title, cites 2.4.2 Page Titled. This title has a suspicious value. [1 instance]

Access Assistant

The Access Assistant browser extension Quick Test returned nearly the same results as the AMP test but as a list of issues with code snippets. Accessing the link for each issue displays an explanation of the issue but does not reference any rules or standards.

Screen shot of the Access Assistant browser extension window that displays quick test results for the URL web.dev in a list, the first issue being provide alternative text for images with a code example.
Access Assistant extension for Chrome

* denotes violations flagged by AMP:

  • Provide alternative text for images.*
  • Ensure text and images of text provide sufficient contrast.
  • Provide valid, concise, and meaningful alternative text for image buttons.
  • Ensure heading level matches the heading’s visual importance/level. [4 instances]
  • Ensure list items are found in a list container. [28 instances]*
  • Ensure all active elements receive keyboard focus or can be activated with the keyboard.
  • Avoid placing inactive elements in the focus order. [2 instances]*
  • Provide an informative, context-sensitive page title.
  • Ensure link text is meaningful within context.
  • Ensure ARIA roles, states, and properties are valid. [28 instances]*
  • Avoid inappropriate use of ARIA roles, states, and properties. [2 instances]
  • Provide synchronized captions for video (which includes audio) or other multimedia.

The Quick Test found one issue to check that was not flagged by AMP as either a violation or alert: Ensure text and images of text provide sufficient contrast.

Conclusion

The automated testing results from the Level Access tools are comparable with the other automated tools Adrian tested with manual testing finding more than 9x the unique success criteria issues. Use automated testing tools in tandem with manual testing to find the most potential accessibility issues. Relying on any automated testing alone will leave you with access gaps for your users.

Update: 27 January 2023

I’ve updated the Highlights data tables to reflect Adrian’s findings for the four automated tools he tested. For the sample, Access Assistant found more issues than WAVE Evaluation Tool, axe DevTools, and ARC Toolkit, and fewer than Equal Access Accessibility Checker.

HTML stickers are here

This retro sticker features the <html> element in block letters printed on a holographic background. As the light moves, the reflected gradient on the sticker shifts through the color spectrum. Just 2 inches wide and half an inch tall.

Contact me with your address to get one.

Structuring accessible forms

Last updated: 27 January 2023

Forms have a strong semantic structure from labeling controls to ensuring the states of controls are announced by assistive technology. We look at the basics of building a form with different input types, adding in the necessary HTML attributes to link our form elements semantically, sprinkling in ARIA attributes and data validation and errors.

Below is the Code Pen for my accessible form example. Access the code in the GitHub repository.

See the Pen Accessible Forms by Rachele (@racheleditullio) on CodePen.

Creating our form fields

The two basic parts of a form field are its <label> and <input> elements. We semantically link the two by

  • Using the <label for="uniqueID"> attribute which matches the <input id="uniqueID"> attribute; or
  • Making the <input> element a child of the <label> element

Doing so allows screens readers to announce the field’s <label> when the <input> gets focus. It also enables the <label> as an additional hit area when using the mouse to focus on an <input>, which is especially useful for checkboxes and radio buttons.

We also need to specify the type of input using the <input type=""> attribute. Putting those together, we have a basic form with one text field and two radio buttons.

<form>
  <label for="name">Cat's name</label>
  <input type="text" id="name" name="name">

  <label><input type="radio" id="yes"> Yes</label>
  <label><input type="radio" id="no"> No</label>

  <button type="submit">Send</button>
</form>

A note on comboboxes

As much as possible, use native HTML <select> and <option> elements to create an expandable list of single-select options.

...
<label for="markings">Cat's markings</label>
<select name="markings" id="markings">
  <option value=""> select an option </option>
  <option value="solid">Solid</option>
  <option value="bi">Bi-color</option>
  ...
</select>
...

autocomplete attribute

Part of structuring an accessible form is providing information about the type of data required by the field and providing autocomplete options if available. We do this by adding the autocomplete attribute to form fields asking for personal information.

<input type="email" id="email" name="email" autocomplete="email">

Below are some common examples of fields asking for personal information and their autocomplete values. Reference the full list of autocomplete values.

  • First name: given-name
  • Last name: family-name
  • Email: email
  • Username: username
  • Company: organization
  • Zip code: postal-code

If autocomplete information is available in the browser, fields with the autocomplete attributes will offer suggestions.

Autocomplete suggestion for an email field in Chrome

Grouping similar fields

Our radio buttons are labeled ‘yes’ and ‘no,’ but what are they for? We need to use the <fieldset> and <legend> elements to provide that semantic meaning. The <fieldset> groups the related form fields while the <legend> provides a name for the grouping. These are most often used for checkboxes and radio buttons but they can be useful for grouping other related fields, e.g. Shipping address.

Since radio button options may have similar names, group like radio buttons with the name="groupName" attribute. Keyboard users move between radio buttons in a group with the arrow keys.

...
<fieldset>
  <legend>Is your cat altered?</legend>
    <label><input type="radio" name="altered" id="yes"> Yes</label>
    <label><input type="radio" name="altered" id="no"> No</label>
</fieldset>
...

Marking required fields

Provide a visible indication of the required fields for sighted users, such as an asterisk. Include text before the form indicating how users can recognize required fields. See Kitty Giraudel’s article The required fault in our stars for a discussion on ways to denote required fields. Research from Baymard suggests e-commerce sites need to mark both required and optional fields.

Then there are two attributes we can use to programmatically define required fields: aria-required="true" and required. Use one (not both) of these attributes to ensure required fields are conveyed appropriately to assistive technology. Screen readers will say required when announcing the form field name, role and state.

...
<label for="name">Cat's name*</label>
<input type="text" id="name" name="name" required>

<label for="email">Your email*</label>
<input type="email" id="email" name="email" autocomplete="email" aria-required="true">
...

The required attribute provides a first layer of form validation on submit. Many modern browsers will not submit a form with empty required fields and will even provide a visible error message.

Required field in Chrome

However, user agent error handling is not dependable as these error messages are not always announced or available to assistive technology; they can’t be styled or resized; and they are not persistent. To disable browser validation testing, include the novalidate attribute in the <form> element.

Verifying data

In addition to marking fields as required for assistive technology, we also have to indicate if the data entered into each field is valid. All required fields should also include the aria-invalid="true" attribute added with JavaScript once the form has been submitted. Remove this attribute once valid data is entered.

We can further enforce data validation in HTML for fields that aren’t necessarily required by using the pattern attribute that accepts a regular expression. Let’s create a field for collecting birthday with a required format of mm/dd/yyyy.

...
<input type="text" id="birthday" name="birthday" pattern="^\s*(1[012]|0?[1-9])\/(3[01]|[12][0-9]|0?[1-9])\/((?:19|20)\d{2})\s*$" aria-describedby="error4">
<p id="error4">Birthday format mm/dd/yyyy</p>
...

This pattern matching prevents the form from submitting if the data entered does not match the specified pattern.

A form field that fails pattern validation in Chrome

Ensure users of assistive technology are aware of the required pattern by programmatically linking the help text with the form field using the aria-describedby attribute.

inputmode attribute

We can make it easier for people entering the date into the form field on mobile devices by adding the inputmode attribute. This attribute allows us to specify which virtual keyboard is displayed in supported browsers. Since dates are numeric, we can use inputmode="numeric" to display the numbers keyboard which provides fewer options and larger hit areas for each key.

screen shot of a mobile website form with the cursor in a field called Cat's birthday while the numeric virtual keyboard displays below.
Numeric virtual keyboard on iOS

Note: In my demo, I use JavaScript to add the slash characters to the date automatically as the slash character is not available on the numeric virtual keyboard.

Handling errors

It’s important to provide a list of all errors at the top of the form or provide them in-context of the field in error. We can programmatically link the error message with the form field using the aria-describedby attribute. This is only necessary when an error message provides information that helps the user enter the correct data.

...
<input type="email" id="email" name="email" autocomplete="email" aria-required="true" aria-describedby="error-email">
<p id="error-email">Email format email@domain.com</p>
...

When assistive technology users focus on the email field, the error message is announced along with the name, role and state of the field.

Error messages must be visible, persistent and close to the field in error. Don’t rely on color alone to convey errors. Include an icon with “error” as the text. Move focus to the error messages at the top of the form or to the first field in error on form submit.

A form field in error using JavaScript validation

Conclusion

I hope you found something to make your forms more accessible. I like forms because they have such a strong semantic structure. There are a lot of things to get right but it’s pretty straightforward what “right” means in this context: Make sure anything that is conveyed visually is also conveyed to assistive technology, including the states of form fields.