Accessibility requires flexibility and adjustment to plan user research for all user segments and their needs.
Research for accessibility follows the same principles used for usability testing with some adjustment to accommodate specific user segments. The two largest user segments are and (or with cognitive impairment) people. But every user segment has a specific need and topic to be observed.
- Blind: they need to navigate, orient themselves and interact using screen readers and keyboard.
- Low vision: they need to increase the font size, use magnifiers and related information closely grouped.
- Motor impairment: they need keyboard support, large hitting targets.
- Cognitive Impairment: they need structure, repetition and consistency.
We we learned that observing Blind people we can collect enough information to meet the needs of the other user segments groups, more specifically the neudiverse population.
Find past inclusive research results and reports on MS Teams Research for Accessibility.
Artefact to be tested
Any artifact can be used to test accessibility if you plan and adjust the techniques used to collect data.
Implemented solution
Design concept in images
Design concept as prototype
Protocols, Techniques and Methodologies
The proposed recommendations are based on testing sessions already conducted by SFSF. We compiled the best practices to enable any designer to plan and execute testing sessions with disabled users from the four main groups cited above.
Protocols
A protocols is a or pre-requisites that describe the plan and execution of a testing session. It can include purpose of the test, the research goal and several combined techniques to collect data like observing behaviors during the tasks, survey, questionnaire, etc.
Testing for different user segments may require adjustments when using common techniques. A session planned to validate a design with blind users requires a script to describe the design. Or a session to test an implemented feature with blind users will require listening to the screen reading announcements. People with cognition deficiencies, motor impairment or low vision may require a pre-session survey to understand their needs and expectations to interact with digital products.
These are some protocols used to do accessibility moderated testing sessions:
- Research Goals. Define goals, pain paints to be discovered or strengths to be validated.
- Recruiting. Plan how to recruit participants with disabilities.
- Scripts. Prepare every piece of information to run a smooth session: Welcome, data protection, goals, instructions, tasks, questions, next steps, appreciation note. The script may include also the design description if the session is to validate a design with blind users.
- Pre-session survey. Allows to collect demographic information regarding the needs related specific disabilities when using digital products or devices.
- Questionnaires. enable a conversation and structure the data collection to answer the research questions.
- Collect data. Prepare a collaborative place where notes can be collected.
- Report. Prepare a report so that other can benefit from the learnings and findings.
- Share results. Publish the report on the channel A11Y Research on MS teams.
Techniques and Methodologies
It's well known that is the most to test digital products (Usability Testing 101 by NN Group). We believe that this same methodology is the best one to conduct an accessibility testing session because most of the needs from people with disabilities are unknown to traditional design guidelines. Hence, the opportunity to observe, ask questions and learn from them to translate their needs to the design and create inclusive experiences.
The team to run an accessibility testing session is similar to that used to conduct a usability testing:
- Host Researcher (facilitators or moderator)
- Note taker
- Other observers when appropriate
- Participant who performs the tasks.
While the participant conducts tasks, the research team observes the participant’s behavior, the host uses scripts or create opportunities to ask question, and everyone benefits from listening the feedback from the participant. After the session a Survey like the ACC SUS can complement data collected on the session.
Techniques
Here are some techniques adapted for accessibility that could be used for moderated testing sessions:
- Pre-session Survey: Prepare a questionnaire to collect demographic data from the participant regarding their disabilities and needs.
- Observation on a Virtual Session: For sighted users a screen share works normally. For blind users testing and implemented solutions while using screen readers is best to request the screen share with sound to listen the announcements from the screen reader.
- Observation with Think Aloud: For sighted users this works normally. For blind users testing and implemented solutions you will need to request the participant to create white spaces (without screen reading announcements) to speak.
- Semi structured interviews & Questionnaire: Prepare questions to help you collect information to respond the research question. You may need to adjust some questions to the needs of specific user segments or disabilities.
Qualitative vs. Quantitative data
- Qualitative data focus on insights and findings from observations on how people perform and behave while executing a task. For instance preferences from blind users about exploration and navigation using screen readers or clear flows for neurotypical users.
- Quantitative data focus on collecting metrics like successful task completion or time used to complete a task. successful completion is an important data point to identify accessibility violations. For instance the task cannot be completed using keyboard or screen reading.
Representation and User segments
Usability studies recommend using five participants. But it is not always easy to recruit participants with disabilities.
For this reason the number of participants is not relevant when we are running moderated testing sessions. The feedback from at least one participant with disability contributes to more inclusive experiences.
There are four user groups with disabilities addressed on past researches at SFSF.
- Blind people
- Low vision people
Different from a unmoderated usability testing where it is expected a large number of respondents, an accessibility testing session will observe a person with disability closely using a moderated session.
Testing with Blind people
It is possible to validate a design or test an implemented solution with blind users.
Plan a session using an implemented solution
Testing implemented solutions is the best approach to learn how users explore and interact using their preferred assistive tool. In fact blind participants prefer to test an implemented solution because they can control exploration with their screen reading tool.
When:
To test an implemented solution and observe blind users exploring and interacting with it using screen reading.
Goals:
Define the goal of your research. It can be validate navigation using screen reading, interaction using keyboard, orientation using navigational elements, task completion, find information, or any other other topic.
Research team
Gather people who will support you during the session:
- Host reads scripts and direct the session.
- Note taker will take notes (can be more than one).
- Observers or support: Developer, Quality and UA can provide unexpected support during the session and learn and learn from the session.
Recruitment
SAP Employees can be available upon request to help on testing sessions. We do not have an external pool of candidates at this moment. Maybe we can explore UserTesting to recruit blind users to run asynchronous tests.
System
Implemented solution URL and credentials to be shared with participant.
Script
Prepare a script to start the session with presentations, welcome note, data protections information and consentient to record session.
Contextualization Script
Script with contextualization of the role of the participant: "You are an employee/manager/admin who works with x and wants to do y".
Think aloud
It can be used when participants test an implemented solution. Tip: tell participant to speak when screen reading announcements stopped to avoid speech overlap.
Shared Screen and Sound
To understand how a blind user explore a digital product using screen reading, it is important to request the participant to share their screen and sound from the assistive tool. This allows the host and note taker to listen together the announcements from the screen reading. Hopefully you will be able to understand their navigation strategy making connections (when possible) of visible focus and announcements from the screen reading.
Tasks
Define the job to be done intended to be shared with participant. Try to break multiple tasks into smaller bites to simplify execution and notes.
Interview
Prepare a questionnaire to assess the participants experience during the task execution. Address the common navigation and interaction strategies that would be used by users with their assistive tools and keyboard. This will help elaborate meaningful follow up questions and collect notes.
Collect notes
Use Mural, FigJam, Excel or other tool to collect immediate user feedback.
Sample
It is better to test with only one blind user than not testing at all. The data collected will be more reliable with more than 2 participants.
Insufficient time to go over planned tasks
it is normal to plan three tasks for a one hour session and not be able to complete all of them. When this happen with an internal SAP participant or a participant from the ADA group (employees from our customers), it is ok to request a follow up session to finalize the study.
Plan a session using a design concept
When designing a concept it is normal to ask question about its impact during exploration, navigation and interaction using screen reading or keyboard. If you wan to plan a testing session to validate the design concept to confirm strengths and address issues before sending it to implementation, follow the same steps described above and make these one adjustments: create an script to inform the designer to a blind person.
When:
To validate a concept design with blind users. It requires a few extra task to prepare the session and alignment with what would be the appropriate exploration using Screen Readers.
Design Script
The script will describe the UI elements. It is useful to use some technical terms expected to be used by screen readers like Links, Buttons and non visible content like invisible labels or messages.
Templates to plan, run and generate reports
Do you want to start an accessibility study?
Start here: Accessibility Studies: How to? Tips to plan, execute and generate reports
What you will learn:
1. Why Research for Accessibility 2. Testing with participants
- Planning a study for Accessibility with participants
- Running testing sessions with various segments
- Generating an Accessibility report
3. Testing with Heuristics
- Expert evaluation using heuristics: Plan, run, and create a repo