Explainable AI (XAI) – Building Explanations

Foundations / AI and Joule Design / Guidelines / Explainable AI / Building Explanations

Intro

This page explores how you can create local and global explanations for different progressive disclosure levels.

At the end of the page, you’ll find a series of work-in-progress examples and a checklist that highlights the key takeaways for explainable AI.

Before you start designing explanations, review the explainable AI overview to get familiar with the concepts and terms we use. Make sure you also understand the guiding principles before moving forward.

Explanation Levels

We distinguish between global explanations, which cover the entire dataset, and local explanations, which focus on individual AI outputs.

Keep in mind that “global” and “local” have different meanings here than in some other design patterns.

When you group multiple AI outputs in a table, graph, or similar format, this still counts as a local explanation.
A global explanation applies to the entire model, no matter which output instance you look at. As users become more familiar with SAP AI policies, their need for global explanations usually goes down, since they no longer require as much high-level context.

Local and Global Explanation Scenarios

<div> <div>Explanation Level</div> <div>Explanation Scenario</div> <div>Example Explanation</div> </div> <div> <div>Local (Single)</div> <div>“Why did <strong>this candidate</strong> get rejected?”</div> <div>“The candidate was not shortlisted because their experience didn’t match the required programming skills.”</div> </div> <div> <div>Local (Group)</div> <div>“Why were <strong>these three candidates</strong> rejected, but the other two shortlisted?”</div> <div>Show a table with candidate scores for skills, years of experience, and cultural fit. Highlight the factors that led to rejection for each candidate, and the attributes that contributed to selection for the others.</div> </div> <div> <div>Global</div> <div>“What’s the <strong>hiring policy</strong> overall?”</div> <div>“In general, the AI model ranks candidates based on skills, years of experience, and education level.”</div> </div>

guideline

Plan for global explanations when users need to understand the overall system behavior, not just a single AI output.

Progressive Disclosure Levels – Overview

Progressive disclosure helps you present explanations that match your users’ needs without overwhelming them. Depending on the situation, users may require different amounts of information to understand an AI output. The right level of detail depends on the type of AI, the usage context, the user’s task, and their role.

We define up to four progressive disclosure levels, from no explanation (Level 0) to an extended, detailed report (Level 3):

<div> <div>Level</div> <div>What’s Provided?</div> <div>Purpose</div> <div>Typical UI Elements</div> </div> <div> <div>Level 0</div> <div>No explanation</div> <div>Users don’t expect or require an explanation for an AI output.</div> <div>None</div> </div> <div> <div>Level 1</div> <div>Explanation indicator (<strong>What?</strong>)</div> <div>The minimum explanation level required whenever AI (ML or LLM) outputs are provided. The indicator also acts as an access point to further explanation layers.</div> <div>Icons, badges, or other markers</div> </div> <div> <div>Level 2</div> <div>Simple explanation (<strong>Why?</strong>)</div> <div>A summary that provides the main reasons behind the output. The simple explanation can include visual or textual highlights, as well as links to other explanation levels.</div> <div>Popover or inline summary</div> </div> <div> <div>Level 3</div> <div>Detailed explanation (<strong>How?</strong>)</div> <div>A comprehensive view designed for advanced users, high-risk scenarios, or edge cases. This level covers all aspects processed by the intelligent system, including AI performance, rationale, supporting data, and audit-related information.</div> <div>Overlay, dedicated page or app</div> </div>

Progressive disclosure works best for local explanations, where users start with a simple indicator and can drill down for more detail.

How It Works

Level 1
Explanation indicator

A low-fidelity table showing local explanation indicators in pink on individual line items

Local explanation indicator in lists and tables

Local indicator for cards highlighted in pink in the top right corner of each card

Local explanation indicator for cards

Level 2
Simple explanation

Multiple explanation popovers highlighted in pink indicating a simple information hierarchy

Local explanation popover, conversation items

Overlay highlighted in pink on top of the original cards, greying out the background

Overlay

Level 3
Extended explanation

Explanation page as an overlay on top of the original table, the full page is highlighted in pink to indicate the whole area can be dedicated to explanations. Flexible column layout.

Local explanation page

Dedicated explanation application that appears like a new web/application page

Dedicated explanation application

Level 1: Users encounter the explanation indicator wherever AI outputs are presented, such as in lists or cards.

Level 2: By selecting the indicator, users can open a condensed explanation, usually as a popover, overlay, or inline expand. This view should help clarify the AI proposal with relevant context and highlights.

Level 3: For users who need full transparency – such as experts conducting audits or users operating in a high-risk domain – a further link gives access to the most detailed level. Level 3 provides extended information and may use overlays or open in a separate page or flexible column layout.

guideline

Always include at least Level 1 for any AI-generated result.
Adjust the initial explanation level based on risk, user role, and context. In high-risk scenarios, consider linking directly from Level 1 to Level 3.
Users should decide how much detail to access at each step – don’t force deeper disclosure unless necessary for compliance or risk management.
For global explanations, provide a summary of the overall system or model behavior – typically as a Level 3 explanation. Over time, as users become more familiar with AI policies, demand for these global explanations usually decreases.
Explanations should stay attached to the relevant conversational turn for as long as the thread remains accessible, ensuring clarity and traceability.
In contexts with higher risk, it is advisable to also include timestamps to provide additional accountability.
Use seamful design to empower users.

The sections below look at each of these explanation levels in more detail.

Disclosure Level 1: Explanation Indicator

Use the explanation indicator to mark elements or combinations of elements created or manipulated with the help of AI. It gives users context about the nature and quality of the service. When users click the indicator, show a more detailed explanation (Level 2).

An explanation indicator is a form of AI notice. For more information, see AI Notice User Research in the User Research Library.

The following sections describe the explanation indicator variants and when to use them.

Explanation Type

Local explanation indicators can apply for a group of items or a single item. This affects both the type of information they contain and their placement on the UI.

<p><div> <div>Local Group Indicator</div> <div>Local Single Indicator</div> </div> <div> <div>Explains a collection of AI outputs.</div> <div>Explains the AI output for a single value or item.</div> </div> <div> <div> <p> <picture> <source type="image/webp" srcset="/content/dam/sapcom/design-system/fiori-design-web/v1-139/foundations/ai-and-joule-design/guidelines/media1_1b50ff019caa4d571ba5493881db821f033dfb1a7.png?width=2000&format=webply&optimize=medium" media="(min-width: 600px)"> <source type="image/webp" srcset="./media_1b50ff019caa4d571ba5493881db821f033dfb1a7.png?width=750&format=webply&optimize=medium"> <source type="image/png" srcset="./media_1b50ff019caa4d571ba5493881db821f033dfb1a7.png?width=2000&format=png&optimize=medium" media="(min-width: 600px)"> <img loading="lazy" alt="Local group indicator in pink sits above a table or list as an explanation for the individual outputs" src="<a href="https://www.sap.com/design-system/fiori-design-web/v1-145/foundations/ai-and-joule-design/guidelines/media%5C_1b50ff019caa4d571ba5493881db821f033dfb1a7.png?width=750%5C&format=png%5C&optimize=medium">https://www.sap.com/design-system/fiori-design-web/v1-145/foundations/ai-and-joule-design/guidelines/media_1b50ff019caa4d571ba5493881db821f033dfb1a7.png?width=750&format=png&optimize=medium</a>" width="403" height="288"> </picture> </p> <p><em>Example: Use a local group indicator above a table or list to explain the individual outputs in that collection.</em></p> </div> <div> <picture> <source type="image/webp" srcset="/content/dam/sapcom/design-system/fiori-design-web/v1-139/foundations/ai-and-joule-design/guidelines/media1_102b6681c9c92db14c5758850ead6c2794d03cd9c.png?width=2000&format=webply&optimize=medium" media="(min-width: 600px)"> <source type="image/webp" srcset="./media_102b6681c9c92db14c5758850ead6c2794d03cd9c.png?width=750&format=webply&optimize=medium"> <source type="image/png" srcset="./media_102b6681c9c92db14c5758850ead6c2794d03cd9c.png?width=2000&format=png&optimize=medium" media="(min-width: 600px)"> <img loading="lazy" alt="" src="<a href="https://www.sap.com/design-system/fiori-design-web/v1-145/foundations/ai-and-joule-design/guidelines/media%5C_102b6681c9c92db14c5758850ead6c2794d03cd9c.png?width=750%5C&format=png%5C&optimize=medium">https://www.sap.com/design-system/fiori-design-web/v1-145/foundations/ai-and-joule-design/guidelines/media_102b6681c9c92db14c5758850ead6c2794d03cd9c.png?width=750&format=png&optimize=medium</a>" width="399" height="288"> </picture><em>Example: Use a local single indicator on individual cards to explain each output directly.</em> </div> </div></p>

Local group indicator in pink sits above a table or list as an explanation for the individual outputs

Poorly worded or overly technical explanations damage trust. Ensure that local explanations follow human-centered principles and are tailored to the user’s role. For more details on voice and language, see the user assistance guidelines.

Indicator Style

The styling of the indicator influences how the user perceives and interprets the displayed output. We distinguish between a default style and an extended style.

<div> <div>Default Style</div> <div>Extended Style</div> </div> <div> <div> <p>The default style provides the minimum indication about the type of AI output. It uses explicit, descriptive labels and blue coloring for clickable explanations.</p> <p>Use this style for non-critical AI applications.</p> </div> <div> <p>The extended style uses semantic colors and offers more contextual insight at Level 1. It can also include other semantic elements, such as an icon to indicate a trend.</p> <p>Use this style when you need stronger visual affordance or want to emphasize the importance of the information.</p> </div> </div>

guideline

If users need to be fully aware of the nature and quality of the output from the start, use the extended style with semantic colors for the explanation indicator.
Don’t use decorative AI icons (such as robot heads or brain symbols) to show that the result came from machine learning or a large language model. They don’t improve understanding. Instead, use the SAP Design System to create accessible explanation indicators.

Explanation Indicator Patterns / Examples

To help you understand XAI concepts in practice, this section shows illustrative scenarios and example explanation indicators. These are conceptual examples, not technical specifications. They show how explanations can be represented in a design.

For visual design specifications – colors, spacing, typography, or component usage – see the official Visual Core guidelines. This ensures consistency with the SAP Design System and avoids duplicating technical guidance.

Note: This version of the guidelines does not include equivalent design patterns for global explanations. Future iterations will add them.

<div> <div>Scenario</div> <div>Pattern</div> <div>Examples</div> </div> <div> <div>Local group indicator</div> <div>Ranking for table items</div> <div>A “View Ranking Criteria” link in the toolbar shows the factors used to sort the items, such as price and delivery time.</div> </div> <div> <div>Local single indicator</div> <div>Predicted priority of an item</div> <div>A table column labeled “Recommended Priority” contains values like “Very High” or “Low.” Can include a numeric score to support the label.</div> </div> <div> <div>Local single indicator</div> <div>Quality classification</div> <div>A product has a tag indicating the predicted quality (such as “Premium”), based on the model output.</div> </div> <div> <div>Local single indicator</div> <div>Price prediction</div> <div>A column labeled “Predicted Achievable Price”<br>Example: “100 EUR (+/- 10%).”</div> </div>

Styling Conditions

The length of the explanation indicator and whether you display a link to a simple explanation (Level 2) depends on your use case:

<div> <div>Form Factor</div> <div>Length</div> <div>Action</div> <div>Conditions</div> </div> <div> <div><strong>All form factors</strong><br>S, M, L, XL, larger than XL</div> <div><strong>Short</strong> (default)</div> <div>Display only</div> <div> <p>The user’s display permission is limited to Level 1.</p> <p><strong>or</strong></p> <p>Level 2 explanation is missing.</p> </div> </div> <div> <div><strong>Large</strong><br>L, XL, larger than XL</div> <div><strong>Long</strong> (optional)</div> <div>Clickable</div> <div> <p>The user has display permission for Levels 1 and 2.</p> <p><strong>and</strong></p> <p>Level 2 explanation is available.</p> </div> </div>

guideline

Display clickable explanation indicators if:

The context is explained in more detail in Level 2.
The user needs access to the Level 2 explanations and has the appropriate authorization.

Ensure that the length of the explanation indicator is suitable for the screen size. To prevent truncated explanation indicators, apply the following rules:

If there is sufficient space and no collision with other UI elements, use the optional long style.
If you are listing criteria, show 3 criteria and use the ellipsis to indicate that there are more. If there is not enough space to show 3 criteria, use a short explanation indicator or allow the user to open Joule in full screen mode.
Ranked By: Price, De…
Ranked By: Price, Delivery Time, Minimum Order Quantity, …
If you use the long style, always offer a fallback to the default short indicator for smaller screens or form factors.
If you have no other options to free up space on the UI (for example, by moving other actions into the overflow), wrap the text.

For more information about responsive and adaptive design, see Multi-Device Support.

Disclosure Level 2: Simple Explanation

Accessing Explanations

Users must be able to access an explanation directly from the explanation indicator. The interaction model (such as a popover, walkthrough, or assistant) is still under review as part of ongoing research and design work. No matter which interaction you use, make sure the explanation space offers contextual information that helps users understand the AI output at the right level – local or global. Include key factors, inputs, or criteria that influenced the result.

Explanation Elements

Explanation popovers, screen overlays, or conversational items like Joule can include the following elements:

<div> <div>Explanation Element</div> <div>Content</div> </div> <div> <div>Header title</div> <div>Self-explanatory title.</div> </div> <div> <div>Minimum content</div> <div> <p><strong>Short explanation</strong> in natural language describing what the AI did.</p> <p>For AI outputs shown on a scale (such as risk, priority, or probability), a <strong>key or legend</strong> that clarifies how to interpret the current value.</p> </div> </div> <div> <div>Recommended content</div> <div> <p>Cited <strong>sources</strong> with links to the original material.</p> <p><strong>Timestamp</strong> indicating when the result or prediction was computed; for time-critical outputs, an indication of the last update.</p> <p>List of <strong>the three to five most important parameters</strong> influencing the result, which may include a comparison with averages or reference values.</p> </div> </div> <div> <div>Optional content</div> <div> <p><strong>Short explanation in natural language</strong> comparing the AI’s performance to a reference, such as historical cases, given data, or average prediction accuracy. This may be combined with performance indicators or a trend.</p> <p><strong>In-place feedback components</strong> for collecting explicit user feedback.</p> </div> </div>

Pop-up Example with header, initial information and main explanation section

Level 2 – explanation elements

Filled out pop-up example showing product image, a colour coded key, and explanation information

Level 2 – example of explanation elements (1)

Filled out pop-up example showing product details, a colour coded key, and explanation information

Level 2 – example of explanation elements (2)

guideline

Structure

Limit the breakdown to three to five factors, unless there’s a critical reason to include more, such as in a high-risk use case. More detail can make content harder to navigate and may prompt additional questions or cross-checks.

For a consistent experience, follow the suggested structure above.

Content

Make it easy to identify the most relevant information:

Choose the most suitable format for conveying information – language, visual elements, or both.
In longer texts, emphasize important information so users can scan content quickly.
Use text formatting and visuals purposefully, and not just for decoration.
Restrict text explanations to no more than three short sentences.

Reveal information progressively:

Add value with each disclosure and avoid simply repeating information from previous levels.
Make sure that the information on each level is self-contained and that all elements form a natural flow.

For general guidance on wording, see the UI Text Guidelines for SAP Fiori Apps in the SAP design system.

Natural Language Explanations

Natural language explanations are a human-centered approach to AI explanations that apply conversational interaction principles as they are used in conversational interfaces. Reasoning and relationships appear as sentences in non-technical language, making explanations easier for users to interpret (see The Two Main Goals of Explainability).

The use of natural language explanations is limited to explanation popovers and screen overlays, explanation pages (see progressive disclosure), recommendation items, and the Joule AI copilot.

The table below provides examples of natural language explanations.

<div> <div>Objective</div> <div>Examples</div> </div> <div> <div>Add value at each disclosure level</div> <div> <p><strong>Level 1:</strong><br><em>Ranked By: Price, Delivery Time, Minimum Order Quantity, …</em></p> <p><strong>Level 2:</strong><br><span class="icon icon-accept"></span><em>The price has the biggest impact on the score, followed by the delivery time and then the minimum order quantity.</em><br><span class="icon icon-decline"></span><em>Suppliers are ranked by price, delivery time, and minimum order quantity.</em></p> </div> </div> <div> <div>Specific information</div> <div><span class="icon icon-accept"></span><em>“Polaris Lights” offers the best price (470 USD) for the product “Taillight”, with the lowest minimum order quantity (300 PC). Delivery (14 days) takes longer than average.</em><br><span class="icon icon-decline"></span><em>Best price with lowest minimum order quantity. Delivery takes longer than average.</em></div> </div> <div> <div>Just enough formatting</div> <div><span class="icon icon-accept"></span><em>This is high priority because the net payment is due soon (<strong>in 3 days</strong>) and the amount is small (<strong>1.200,00 USD</strong>). Customer Electronics usually pays on time.</em><br><span class="icon icon-decline"></span><em>This is <strong>high priority</strong> because the <strong>net payment</strong> is due soon (<strong>in 3 days</strong>) and the <strong>amount</strong> is <strong>small</strong> (<strong>1.200,00 USD</strong>). Customer Electronics usually pays <strong>on time</strong>.</em></div> </div>

guideline

Use natural language to summarize the overall situation and to explain dependencies, origins, or principles.
To display the simple description, we recommend using plain text.

Disclosure Level 3: Extended Explanation

You can offer users a more in-depth analysis by providing an extended explanation page, tailored to their role and the risk level of the use case. You can direct users to a separate, dedicated application or add an extra page within your own app. The decision depends on how much control and insight you want to provide for the technology in use. In most cases, extended explanations serve a unique use case and require a separate implementation.

Explanation Page

Explanation page as an overlay on top of the original table, the full page is highlighted in pink to indicate the whole area can be dedicated to explanations.

Detail explanation page in a flexible column layout

Dedicated Explanation App

Dedicated explanation application that appears like a new web/application page

Detail explanations in a separate app

guideline

Provide links from explanation level 2 to the extended explanation page or app only to users with appropriate roles (role-based access). AI analysis is usually relevant only for roles like data scientists or system owners, so offer detailed explanations on a need-to-know basis.
Limit exposure for general users. Don’t give regular business users access to these links, since detailed AI analysis is rarely needed for their tasks.
In high-risk scenarios or edge cases, be aware that any user might need to move directly from level 1 to level 3.
Account for exceptions. Certain roles, such as developers, may not be decision makers but could still require detailed explanations for troubleshooting or integration purposes.

Examples

The examples below show different explanation scenarios.

These images are illustrative only, early-stage / beta / work-in-progress examples to support understanding, not final design specifications.

SAP SuccessFactors

A screenshot of the Accrual Proposals applications, showing a list of six accrual proposals on the left. The details of the one selected are shown on the right, showing the recommended accrual amount, the proposed line items and a section “Basis of AI Recommendation” with a natural language explanation along with a bar chart and a forecast line for the trend per month.

Explanation in SAP SuccessFactors

Explanation level: Local
Explanation type: Rationale
Explanation content: Source
Progressive disclosure level: 2 – simple explanation
User role: Impacted users

Read the full article on this use case in the SAP Community Blog.

Joule Chat Interface

Drop down menu within Joule containing output sources and AI reasoning

Explanation in Joule chat interface (work in progress)

Explanation level: Local
Explanation type: Rationale
Explanation content: Source
Progressive disclosure level: 2 – simple explanation
User role: Impacted users

SAP Deep Research and Planning Agent (1)

Joule interface within the deep research agent initializing orchestration of 4 different research tasks

Initializing orchestration – information in Joule (work in progress)

SAP Deep Research and Planning Agent (2)

Joule interface within the deep research agent showing initial details on task completion

Task updates – information in Joule (work in progress)

SAP Deep Research and Planning Agent (3)

Joule interface within the deep research agent reporting how many tasks in total have been completed

Task completion – information in Joule (work in progress)

SAP Deep Research and Planning Agent (4)

Joule interface within the deep research agent showing a breakdown of which tasks were completed with drop down arrows on each task

Task breakdown – information in Joule (work in progress)

SAP Deep Research and Planning Agent (5)

Joule interface within the deep research agent showing an expanded task and a preview of the code used to create the results

Task detail – explanation in Joule (work in progress)

Explanation level: Local
Explanation type: Data and rationale
Explanation content: Source
Progressive disclosure level: 3 – extended explanation
User role: Producers of explanations, ethicists and impacted users

SAP Deep Research and Planning Agent (6)

Joule interface within the deep research agent reporting the conclusion and result of the research task

Completed task – information in Joule

Top Tips

This section summarizes the most important XAI principles and guidelines.

Before you start with your design, conduct research to figure out the answers to the questions below. If user research is not an option, use a heuristic evaluation method as outlined in the SAP User Research Portal SharePoint to test your design. This should be the minimum investment.

Keep explanations human-centered

Write in plain language and avoid jargon.
Tailor explanations to the user’s role (decision maker, impacted user, producer, ethicist).

Use progressive disclosure

Start with simple explanations (Level 1 – indicator).
Offer additional context when needed (Level 2 – popover or overlay).
Provide detailed analysis only for advanced users or high-risk cases (Level 3 – extended page or app).

Match explanation level to risk

Low risk: Lightweight explanations (indicators, optional details)
Standard risk: Add rationale and key influencing factors
High risk: Offer extended explanations including transparency, sources, and timestamps

Support both local and global perspectives

Local: Explains why a specific output happened (for example, candidate rejection).
Global: Describes how the system generally works (for example, hiring policy).
Grouped local explanations, such as tables or charts, are still local rather than global.

Design for understandability and intervenability

Explanations should help users make sense of outputs (understandability).
Explanations should empower users to act, contest, or adjust (intervenability).

Use accessible, consistent indicators

Don’t rely on color alone – combine text, icons, or labels for clarity.
Avoid decorative “AI” icons, such as robot heads or brain symbols.
Follow the SAP design system for styling and components.

Apply seamful design when needed

Show limitations instead of hiding them.
Add friction for high-stakes actions (for example, by adding pauses or requiring confirmation).
Reveal seams in ways that help build calibrated trust without overwhelming users.

Always respect privacy and ethics

Follow the SAP AI Ethics Policy and Agentic AI Ethics Guidelines.
Ensure explanations don't expose sensitive data unnecessarily.
Make fairness and accountability visible when relevant.

XAI Guidelines

Specifications

Visual Design System (Horizon Visual Core wiki)

Explainable AI (XAI) – Building Explanations

Foundations / AI and Joule Design / Guidelines / Explainable AI / Building Explanations

Intro

Explanation Levels

Local and Global Explanation Scenarios

Progressive Disclosure Levels – Overview

How It Works

Disclosure Level 1: Explanation Indicator

Explanation Type

Indicator Style

Explanation Indicator Patterns / Examples

Styling Conditions

Disclosure Level 2: Simple Explanation

Accessing Explanations

Explanation Elements

Structure

Content

Natural Language Explanations

Disclosure Level 3: Extended Explanation

Examples

SAP SuccessFactors

Joule Chat Interface

SAP Deep Research and Planning Agent (1)

SAP Deep Research and Planning Agent (2)

SAP Deep Research and Planning Agent (3)

SAP Deep Research and Planning Agent (4)

SAP Deep Research and Planning Agent (5)

SAP Deep Research and Planning Agent (6)

Top Tips

Related Links

XAI Guidelines

Related Guidelines

Research Studies

Products

SAP Community

Specifications