AI Design Quality Checklist

Intro

Purpose

It is important to establish standards for SAP AI experiences quality. To get started, we are providing a first set of guiding questions and mandatory checkpoints to enable design teams to assess AI Design quality confidently and identify important follow-ups with technical experts. This AI Design Quality Checklist is also tied to the Design Intent Reviews and Design Excellence Reviews hosted by Chief Design Officer Arin Bhowmick.

AI Design Quality Checklist Questions

Checkmarks

Guiding Questions

Resources to help

1. Value

1.1
Is AI necessary to uniquely solve a problem or need – ideally as identified by user research?
AI Design Thinking Resources coming in Q2, 2024
1.2
Does the AI use case deliver value-adding and brand-differentiating experience to a significant number of users and customers?
Product Management, Sales, Marketing
1.3
Will the delivered AI use case add enough value to offset the build and maintenance costs of the AI model?
Product Management, Dev, DevOps

2. SAP AI Standards

2.1
Did you ensure consistency of your AI user experience design with the SAP design system design patterns and guidelines for AI use cases?
2.2
Does your AI use case comply with SAP Fiori’s UI text guidelines for all conversational and generative text interactions?
2.3
Does your AI use case design comply with SAP’s Accessibility Product Standard?
2.4
Does your AI use case comply with SAP’s AI Ethics Policy and product standards?
2.5
Did you do user research to inform and test your AI use case design?
2.6
Does the user experience allow users to see the organization’s ethical criteria for using AI tools and data?

3. AI Excellence

3.1
Does the user know where AI is being used to enhance product value and experience?
3.2
Does the user experience enable users across diverse levels of literacy and technical expertise to easily use the AI feature?
3.3
Do AI recommendations maintain contextual awareness of user needs and priorities?
3.4
Does the design provide controls for customizing the AI’s parameters?
3.5

Does the design allow the user to validate AI recommendations before committing to action by providing transparency on the following?

  • What the AI’s recommendation is optimizing for
  • Why the AI is making a recommendation
  • How the AI is making a recommendation, through progressive disclosure of the model and data being used
  • The call to action: Provide clear messaging on the call to action, or when no action is recommended.
3.6
Are admins and users in control of reviewing, applying, and declining the AI’s automations and recommendations?
3.7

Does the user experience capture user feedback that will be used to continuously improve AI results through

  • Direct feedback patterns
  • UI instrumentation
  • RLHF (Reinforcement Learning through Human Feedback)
3.8
Have you considered ways to provide in-context AI guidance to assist users in understanding what the AI is doing, including best practices for responsible use?
3.9

Have you thought about whether an admin or user should be able to

  • Take over control of the system and overrule all its actions
  • Turn AI enhancements off entirely

4. Generative AI and Joule Embed

4.1

Does the use case call to the user’s attention

  • All AI generated data
  • The limitations and risks of generative AI
4.2
Does the user experience make it easy to leverage output from generative AI into a workflow?
4.3
Have you considered ways to allow users to flag bias in the AI system?
4.4
Have you considered enabling users to capture, save, edit, and share their prompt history and embed or summarize it into exported results?
4.5
Have you considered providing an easily searchable history of interactions with the AI to allow for easy auditing?

SAP AI Ethics UX Considerations

* Additions to item 2.4

SAP has committed to our customers and users to deliver only Business AI that is responsible. The AI Ethics UX considerations listed below serve as general guidance for designers and entire product teams on delivering responsible AI experiences. Overall, all teams at SAP working on AI use cases must adhere to the SAP AI Ethics Policy and follow the mandatory process, which includes, for example, the AI impact assessment for all AI use cases and the AI Ethics Steering Committee approval process for high-risk use cases.

Human-centered AI
  • AI software is user-centric, addressing the widest possible range of applicable end-users, and following relevant accessibility standards, regardless of the users’ age, gender, abilities, or characteristics.
  • Make AI systems identifiable as such to appropriate end users, where applicable, when interacting directly with humans (including via Conversational AI or ‘chatbots’).
  • Develop AI systems in a way that it doesn’t encourage humans to develop users’ attachment and/or empathy towards the AI system. Clearly signal to end users that the social interaction of AI systems is simulated.
  • Where feasible, apply a fairness function to test AI systems for unbiased output.
Human in control
  • AI systems are subject to appropriate human oversight. The rights and freedoms of a human exceed that of AI systems.
  • In situations where humans may be directly impacted by a decision made by SAP’s AI system, introduce human oversight to safeguard that the AI system doesn’t undermine human autonomy or introduce unintended consequences.
Transparency and explainability
  • Provide a clear and simple explanation as to how decisions were made by an AI system used in automated decision processes, as far as is practical.
  • AI systems that engage in profiling or automated decision-making must be able to provide explanations to data subjects upon request. They must be able to describe the data segment the subject was placed into and the reasons they were placed there. This is in alignment and compliance with applicable data protection and privacy law.
  • In addition, the reasons as to why the decision was made must be provided if requested by the data subject. Present the reasoning in a way that the data subject can challenge it.

Role of the Designer:

  • Provides the human-centered understanding of the product by developing a user need strategy including stakeholders, end users, and product roadmap. Conducts: user research, flows, usability evaluation, including voice response and conversational user interaction.

Methods:

  • Methods of user research, design thinking, and correctly defining user stories are mandatory aspects of our development.
style
column-section-1-1, no-footprint