Explainable AI (XAI) – Overview

Foundations / AI and Joule Design / Guidelines / Explainable AI (XAI)

Intro

To help users trust AI, it’s important to provide enough information about the model behind it and explain how and why it produces certain results.

This guideline outlines our approach to explainable AI and is organized into three separate pages:

Overview
Understand why explainable AI matters, its main goals, and the terminology we use.
Guiding Principles
Learn the fundamentals for designing explanations. This section also provides a list of resources where you can find more details.
Building Explanations
Learn how to apply progressive disclosure levels for explanations. You’ll also find examples, as well as top tips that summarize all the important points.

A screenshot of the Accrual Proposals applications, showing a list of six accrual proposals on the left. The details of the one selected are shown on the right, showing the recommended accrual amount, the proposed line items and a section “Basis of AI Recommendation” with a natural language explanation along with a bar chart and a forecast line for the trend per month.

Explanation in SAP SuccessFactors (work in progress)

Why Explainable AI?

Explainable AI (XAI) isn’t just a nice-to-have design feature. Laws require you to provide human oversight for AI outputs, especially in high-risk situations. You can find more details about risk and legal requirements in the SAP AI Ethics Handbook on SAP.com and in SAP’s AI Ethics Policy on SharePoint.

An explanation doesn’t just lay out technical processes. It also needs to make sense to users and work in their daily context. SAP’s approach to XAI is human-centered (HCXAI), which means you should design explanations with the end user in mind. For more about HCXAI, see SAP’s research findings on explainability in the SAP user research library.

In short, explanations should:

Be human-centric, concise, glanceable, and tailored to the user’s role where possible.
Enable data transparency by providing explanations with an appropriate level of detail.
Use global explanations to prioritize ethics and privacy, which SAP business users highlight as top concerns.
For the full list of ethical and responsible AI principles, see SAP’s AI Ethics Policy on SharePoint.
Follow the Agentic AI Ethics Guidelines on SharePoint.
Example: “Human-in-the-loop” (HITL) as a core ethical safeguard.

Although these guidelines focus on design best practices and policies, you should also use the inclusive research methods described in the Inclusive Research Handbook on SAP.com. This ensures human-centered XAI reflects the needs of all users and promotes equity. As a designer, you’ll benefit from this foundation when creating explanations for diverse audiences.

XAI Terminology

These guidelines use the terms below to describe the explainable AI framework.

<div> <div>Term</div> <div>Description</div> </div> <div> <div>Explainable artificial intelligence (XAI)</div> <div>Provide users with enough information about the model, including how it reached a result, what actions it took, why its insights or recommendations matter for the user’s task, and what measurable impact is expected. This also includes details about the model, reasoning, data, APIs, and other data sources used.</div> </div> <div> <div>Human-centered explainable artificial intelligence (HCXAI)</div> <div> <p>A holistic approach to explainability that centers around people and considers the broader social context – such as politics, the economy, and social values.</p> <p>See: <a href="https%3A%2F%2Farxiv.org%2Fpdf%2F2408.05345v2">Explainable AI Reloaded: Challenging the XAI Status Quo in the Era of Large Language Models</a>.</p> </div> </div> <div> <div>Explainable</div> <div>Describes the technical side of an explanation – <strong>what happened and how</strong>.</div> </div> <div> <div>Interpretable</div> <div>Focuses on what the explanation means in the user’s context – <strong>why it matters</strong>.</div> </div> <div> <div>Understandable</div> <div>Helps users easily grasp and <strong>make sense of an explanation</strong>.</div> </div> <div> <div>Intervenable</div> <div>Shows users <strong>what actions they can take</strong> or how to change things if needed.</div> </div> <div> <div>Explanation level: Local (single)</div> <div>Describes a single AI output or decision, such as a specific graph or chat response.</div> </div> <div> <div>Explanation level: Global</div> <div>Describes explanations that apply to the entire dataset, such as data privacy, model training data, or tool stakeholders.</div> </div> <div> <div>Role: Decision maker</div> <div>Institutions and people who decide whether to adopt AI systems, as well as people who use AI to make decisions for or about others.</div> </div> <div> <div>Role: Producer of explanations</div> <div>People who design, build, or monitor AI systems directly.</div> </div> <div> <div>Role: Ethicists</div> <div>People who develop guidelines and policies for fairness, transparency, accountability, and ethical concerns surrounding AI tools, and ensure compliance with relevant standards.</div> </div> <div> <div>Role: Decision subjects</div> <div>People affected by AI-powered decisions, even if they don’t interact with the AI tools themselves.</div> </div>