Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Automated checks for accessibility rules and standards #12018

Open
2 tasks
saw-jan opened this issue Dec 6, 2024 · 6 comments
Open
2 tasks

Automated checks for accessibility rules and standards #12018

saw-jan opened this issue Dec 6, 2024 · 6 comments
Assignees

Comments

@saw-jan
Copy link
Member

saw-jan commented Dec 6, 2024

Automated Checks for Accessibility Rules

Accessibility issues like these (but not all) could have an automated checks:

Why automate?

  • Checking a11y rules is easy and quicker than manual
  • Checks can be brought to the CI pipeline
  • Early accessibility assessment

Why not depend on automated tests ONLY?

  • Automated tests can only check if certain rules have been applied to certain elements or not
  • Cannot guarantee the accessibility of a complete feature

Available Tools

Manual

Automation

POC

Reports

Lighthouse Accessibility Insights
Screenshot from 2024-12-06 10-50-33 Screenshot from 2024-12-06 10-50-55

Automated (axe-core):

{
  "description": "Ensure buttons have discernible text",
  "help": "Buttons must have discernible text",
  "helpUrl": "https://dequeuniversity.com/rules/axe/4.10/button-name?application=playwright",
  "id": "button-name",
  "impact": "critical",
  "failureSummary": "Fix any of the following:
    Element does not have inner text that is visible to screen readers
    aria-label attribute does not exist or is empty
    aria-labelledby attribute does not exist, references elements that do not exist or references elements that are empty
    Element has no title attribute
    Element does not have an implicit (wrapped) <label>
    Element does not have an explicit <label>
    Element's default semantics were not overridden with role=\"none\" or role=\"presentation\"",
  "html": "<button data-v-babade49=\"\" type=\"button\" class=\"oc-button oc-rounded oc-button-m oc-button-justify-content-center oc-button-gap-m oc-button-passive oc-button-passive-raw oc-ml-xs\" id=\"users-filter-confirm\">",
  "none": [],
  "target": ["#users-filter-confirm"],
  "tags": [
    "cat.name-role-value",
    "wcag2a",
    "wcag412",
    "section508",
    "section508.22.a",
    "TTv5",
    "TT6.a",
    "EN-301-549",
    "EN-9.4.1.2",
    "ACT"
  ]
}

Proposal

Add automated accessibility checks:

  • Rules checks - in ONE of the following forms
    • Component level
    • Separate e2e (Playwright)
    • Integrate with existing e2e (Playwright - Cucumber)
  • Keyboard only navigation - e2e (?)
@saw-jan
Copy link
Member Author

saw-jan commented Dec 6, 2024

@saw-jan
Copy link
Member Author

saw-jan commented Dec 16, 2024

  • Rules checks - in ONE of the following forms
    • Component level
    • Separate e2e (Playwright)
    • Integrate with existing e2e (Playwright - Cucumber)

@LukasHirt Could you recommend/suggest at which level we should have the automated accessibility checks?
e2e would be more closer one to the end user experience. Also, if we use the existing playwright-cucumber setup (checking page accessibility using gherkin), it would be very quick to get started.

@LukasHirt
Copy link
Collaborator

Could you recommend/suggest at which level we should have the automated accessibility checks?

How would it look like if it would be implemented as part of e2e tests? Would it go through every page one-by-one? Would that mean we need to handle setup of different scenarios to cover all the screens?

Generally speaking I am a bit hesitant to have this as part of e2e tests. Having to spin up the whole setup to check a11y roles, etc. feels like an overkill. Since this feels more like a "linter" (please correct me if I am wrong here) it looks more to me like a test that should be run outside of playwright.

Could you give me please some better understanding what's the difference in effort between these two approaches? I guess if you say that having this as part of e2e tests would be 10x faster to develop as POC than having it at the component level, I would say go for it. If the difference wouldn't be that drastic I would personally prefer trying the component level first. But that's my 2 cents. If you still believe that playwright for this is the way to go, I'm not going to block you.

@saw-jan
Copy link
Member Author

saw-jan commented Dec 16, 2024

How would it look like if it would be implemented as part of e2e tests? Would it go through every page one-by-one? Would that mean we need to handle setup of different scenarios to cover all the screens?

yeah exactly. e2e means accessing web via url, navigating to the page and do the accessibility checks. that means it tests the accessibility of integrated components.

As far as I can tell, with the component level, effort wise, we probably need to add a lot tests as there are large number of components and there will involve mocking and tests with state changes. This can do the element-level isolated checks.

But with e2e and since we already have working setup for e2e, we just have to implement one or two test steps and write an accessibility feature then run the test using the existing e2e test run command. Also, checking the whole page or certain part of the page would be closer to how/what the end users will experience.
And if we integrate to existing e2e tests, maybe we can also have some scenarios or journey for keyboard only navigations later on.

I would say, it would be easier to get started with e2e test because of the existing setup. But with the component level, we might get lost in the large volume of available components in different packages. 😅

@LukasHirt
Copy link
Collaborator

As far as I can tell, with the component level, effort wise, we probably need to add a lot tests as there are large number of components and there will involve mocking and tests with state changes. This can do the element-level isolated checks.

I see. I misunderstood than how this would work. I was imagining something alongside of a scan of codebase and just checking whether the roles are applied in the code. Then yes, POC with e2e tests seems more reasonable to me as well.

@saw-jan
Copy link
Member Author

saw-jan commented Dec 17, 2024

I was imagining something alongside of a scan of codebase and just checking whether the roles are applied in the code

This is also valid but would be good to start with rendered html.
Thanks Lukas, we will add checks for some important pages first.

@koebel koebel self-assigned this Dec 18, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

3 participants