Foundations / AI and Joule Design / Guidelines / Explainable AI / Guiding Principles
Intro
This page covers core principles and widely accepted best practices for explainable AI, or XAI. These concepts form the foundation for crafting your own explanations.
Building trust is essential when designing AI-powered tools. In this section, you’ll learn how to assess and support the right level of user trust in your application by using a calibrated trust scale and following practical guidelines.
Calibrated Trust Scale
A calibrated trust scale in explainable AI helps you understand how users perceive and respond to system recommendations or decisions. The scale lets you identify if users place too little, too much, or just the right amount of trust in AI – especially after they see its explanations.
The scale usually includes three levels:
Distrust: Users don’t trust the AI’s outputs and may ignore useful suggestions.
Calibrated trust: Users’ trust matches the AI’s real capabilities, so they rely on it appropriately.
Over trust: Users put too much confidence in the AI and might follow its recommendations even when they aren’t correct.
Calibrated trust scale. Distrust quotation from BBC News. Over trust quotation from SAP research on AI Notice in the user research library.
Building Trust: “How Might We” Checklist
To help users develop the right level of trust in your AI tools, use the “how might we” checklist below. These guidelines reflect the current academic consensus on building effective and trustworthy AI explanations.
<div> <div>Guideline</div> <div>Why It Matters</div> </div> <div> <div>Evaluate trade-offs</div> <div>Some high-performing models, such as deep learning, offer less transparency than simpler models.</div> </div> <div> <div>Complement with XAI methods</div> <div>Use techniques like SHapley Additive exPlanations (SHAP) or Local Interpretable Model-agnostic Explanations (LIME) to improve interpretability when the base model lacks explainability.</div> </div> <div> <div>Understand dependencies</div> <div>XAI considerations may differ, depending on the product or application.</div> </div> <div> <div>Align explanations with the user’s goal</div> <div>Assess whether the final explanation supports what the user wants to achieve.</div> </div> <div> <div>Provide meaningful explanations</div> <div>Clear explanations help users avoid confusion and cognitive overload.</div> </div> <div> <div>Calibrate user trust</div> <div>XAI should help users avoid over trust and distrust.</div> </div> <div> <div>Empower and educate users</div> <div>Explanations should strengthen confidence and knowledge.</div> </div> <div> <div>Structure explanations</div> <div> <p>Answer in the order:</p> <p>1) What happened?<br>2) Why did this happen?<br>3) How did this happen?<br>4) What if…?</p> </div> </div> <div> <div>Tailor to the user’s role</div> <div>Different roles may need different explanation details.</div> </div> <div> <div>Ensure understandability</div> <div>Users need to grasp explanations without extra effort.</div> </div> <div> <div>Enable user intervention</div> <div>Users must be able to step in and adjust decisions as needed.</div> </div> <div> <div>Make explanations easy to access</div> <div>Issuers and recipients should be able to retrieve explanations quickly.</div> </div> <div> <div>Protect privacy and security</div> <div>XAI must not compromise private or sensitive information.</div> </div> <div> <div>Be transparent about gaps or uncertainty</div> <div>Don’t hide what the tool doesn’t know – users need to be aware of limitations. See <a href="https%3A%2F%2Fwww.sap.com%2Fdesign-system%2Ffiori-design-web%2Ffoundations%2Fai-and-joule-design%2Fguidelines%2Fexplainable-ai-principles%23seamful-design">Seamful Design</a>.</div> </div>
The scope and content of the explanations you need to provide depend on the inherent risk level of the use case. Use the AI Use Case Impact Assessment to determine the risk level for each use case.
<div> <div>Risk Level</div> <div>SAP AI Ethics definition</div> <div>Why It’s important</div> <div>Examples</div> </div> <div> <div>Redline/prohibited</div> <div>Use of AI for this purpose is highly unethical. If your use case is built for a redline purpose, you must <strong>immediately</strong> stop developing, deploying, or selling it.<br><br>The solution involves inherent risks that could negatively impact society or is designed with this in mind.</div> <div> <p>AI Ethics Policy, p. 11: “AI systems must not have any harmful impacts on users.”</p> <p>AI Ethics Policy, p. 13: “Negative social, economic, and environmental impacts must be avoided.”</p> <p>For full details, see the <a href="https%3A%2F%2Fsap.sharepoint.com%2Fteams%2FSAPOneAssets%2FLibrary%2Fcompany_policies_and_guidelines%2FGlobal_AI_Ethics_Policy_English.pdf">SAP AI Ethics Policy</a> on SharePoint.</p> </div> <div> <p>Human surveillance</p> <p>Social scoring</p> <p>Deliberate discrimination</p> <p>De-anonymizing data that was previously anonymized</p> <p>Foreseeable harm or development of tools to manipulate individuals or groups</p> <p>Undermining public debate or democratic elections</p> <p>Environmental harm</p> </div> </div> <div> <div>High1)</div> <div> <p>AI used in a similar way to AI solutions that have led to negative consequences for individuals or whole populations in the past.</p> <p>Your use case has potential consequences for people, whether negative or positive.</p> </div> <div> <p>High risk use cases require a high degree of transparency for accountability and oversight.</p> <p>Requires a continuous cycle of internal and external auditing.</p> </div> <div> <p>Processing personal information</p> <p>Automated decision making</p> <p>Consequences for people, either direct or indirect</p> <p>Categorizing people</p> <p>Management and operation of critical infrastructure</p> <p>Employment and HR</p> <p>Healthcare</p> <p>Private services, public services and benefits</p> <p>Law enforcement</p> <p>Migration</p> <p>Democratic processes</p> </div> </div> <div> <div>Standard/Low</div> <div>Use cases that involve minimal potential for harm or adverse consequences.</div> <div> <p>Consider the user roles and scenarios present in the use case and the types of local explanations users will expect.</p> <p>Set up a feedback system to identify which types of explanations are most useful, based on <a href="https%3A%2F%2Fsap.sharepoint.com%2Fsites%2F206415%2FLists%2FUser%2520Studies%2FDispForm.aspx%3FID%3D315%26e%3DFu5mmo">SAP multi-agent system research findings</a> in the user research library.</p> <p>Use <em>Principle 6: Human oversight and determination</em> from the <a href="https%3A%2F%2Fsap.sharepoint.com%2Fteams%2FSAPOneAssets%2FLibrary%2Fcompany_policies_and_guidelines%2FGlobal_AI_Ethics_Policy_English.pdf">SAP Global AI Ethics Policy</a> as the guiding principle for global explanations.</p> </div> <div> <p>Chatbots answering frequently asked questions</p> <p>Automatic content summarization</p> <p>Internal knowledge base assistants</p> <p>Personalized learning suggestions</p> </div> </div>
1) High-risk use cases aren’t prohibited within SAP. However, you must first submit them to the AI Ethics steering committee for assessment before you develop, deploy, or sell them. The AI Ethics team uses sub-classifications of high-risk use cases to determine the evaluation cycle your use case will follow.
XAI User Roles
The literature on explainable AI describes various user roles in relation to explainability. For these guidelines, we’ve combined several taxonomies to create an aggregated list of user roles that impact or are impacted by XAI. Treat these XAI-specific roles as an extra layer on top of your existing line of business personas. Real-world deployments will often reveal additional roles as you define edge cases.
As a high-level guide, here are the main XAI user roles you should consider:
<div> <div>User Role</div> <div>Definition</div> <div>Examples</div> </div> <div> <div data-valign="middle">Decision Maker</div> <div>Institutions that deploy automated decision pipelines and who are under an obligation to explain the output to end-users.</div> <div>SAP, Salesforce, Meta, Google</div> </div> <div> <div></div> <div>People who decide whether and how AI systems are adopted, integrated, evaluated, purchased, deployed, and monitored.</div> <div>CTO, IT manager</div> </div> <div> <div></div> <div>People who use AI to make decisions for or about others.</div> <div>Recruiter, doctor</div> </div> <div> <div>Producer of Explanations (Builders)</div> <div>People who directly design and build AI systems and their explanation mechanisms.</div> <div>AI development, Joule development, knowledge graph, LoB team, AI practitioner, user researcher, conversation designer, user assistance developer</div> </div> <div> <div>Ethicists</div> <div>People with domain expertise who focus primarily on fairness, transparency, accountability, and ethical concerns surrounding AI tools.</div> <div>Policymaker, auditor, consultant to a decision maker</div> </div> <div> <div>Impacted Users (Consumers): End Users</div> <div>People who interact directly with AI outputs and explanations for their own purposes.</div> <div>-</div> </div> <div> <div>Impacted Users (Consumers): Decision Subjects</div> <div>People who don’t interact directly with AI systems but are still affected by the outputs.</div> <div>People receiving a medical diagnosis produced by AI, people whose loan requests are approved or denied based on AI analysis, business users who receive reports generated by AI</div> </div>
The third type of decision maker – such as a doctor or recruiter – might seem different from an institution or an IT manager. However, this person also makes decisions on behalf of others, just on a smaller scale. This sets a decision maker apart from end users, who rely on AI for their own purposes. For an end user, the consequences of an AI-supported decision mainly affect that user directly, while a decision maker’s choices impact other people.
<div> <div>Role</div> <div>Definition</div> <div>Key Impact</div> </div> <div> <div>Decision Maker</div> <div>Someone who uses AI outputs to make decisions for or about others.</div> <div>The use of AI by this person has <strong>consequences for other people</strong>.</div> </div> <div> <div>End User</div> <div>Someone who uses AI outputs to make decisions for themselves to support their work</div> <div>The use of AI by this person has <strong>consequences for themselves</strong>.</div> </div>
The Anatomy of an Explanation
When an explanation is both explainable and interpretable, it becomes understandable. Making explanations understandable is a key goal of XAI. The diagram below comes from a presentation to the SAP engineering ecosystem. You can read the slide deck on FigJam, access the full recorded presentation, or find related SAP research findings on explainability in the user research library.
Venn diagram showing the anatomy of an explanation
The Two Main Goals of Explainability
AI explanations address two main goals that form the foundation of explainable AI: understandability and intervenability:
<div> <div>Key Goal</div> <div>What It Means</div> <div>Key Question Answered</div> <div>Purpose</div> <div>Examples</div> </div> <div> <div>Understandability</div> <div>Helps users <em>make sense of what the AI is doing</em></div> <div><em>“What happened and why?”</em></div> <div>Builds confidence and trust in AI outputs, and is especially important in high-risk scenarios</div> <div>Present the factors that influenced a recommendation in plain language</div> </div> <div> <div>Intervenability</div> <div>Helps users <em>take action when needed</em></div> <div><em>“What can I do about it?”</em></div> <div>Empowers users to monitor, correct, contest, or troubleshoot AI decisions, supporting user autonomy</div> <div>Provide an option to override an AI decision or request a human review, enabling human agency</div> </div>
Why These Goals Matter
<div> <div>Goal</div> <div>What It Means</div> <div>Why It Matters</div> </div> <div> <div>Understandability</div> <div>Users understand AI outputs</div> <div>Builds adequate trust and confidence. See <a href="https%3A%2F%2Fwww.sap.com%2Fdesign-system%2Ffiori-design-web%2Ffoundations%2Fai-and-joule-design%2Fguidelines%2Fexplainable-ai-principles%23designing-for-trust">Designing for Trust</a>.</div> </div> <div> <div>Intervenability</div> <div>Users can act on AI decisions</div> <div>Ensures user control and agency</div> </div>
The EU GDPR and EU AI Act on EUR-Lex set key obligations for transparency, interpretability, and human oversight, but neither outlines a detailed taxonomy of explanation types. However, the UK GDPR highlights six main explanation types that cover both local and global levels:
<div> <div>Explanation Type</div> <div>Description</div> <div>Purpose</div> </div> <div> <div><strong>Rationale</strong> explanation</div> <div>Reasons behind decisions, non-technical</div> <div>Challenge decisions, change behavior</div> </div> <div> <div><strong>Responsibility</strong> explanation</div> <div>Accountability and contact information</div> <div>Challenge decisions, traceable accountability, provide information</div> </div> <div> <div><strong>Data</strong> explanation</div> <div>Data used in decisions</div> <div>Challenge decisions, reassurance</div> </div> <div> <div><strong>Fairness</strong> explanation</div> <div>Efforts to mitigate bias</div> <div>Challenge decisions, reassurance</div> </div> <div> <div><strong>Safety</strong> and performance explanation</div> <div>Reliability and robustness</div> <div>Challenge decisions, provide information, reassurance, help users gain understanding or confidence</div> </div> <div> <div><strong>Impact</strong> explanation</div> <div>Effects on users and society</div> <div>Help users understand impact or consequences, human in control, reassurance</div> </div>
Explanation Content
An AI explanation isn’t a fixed attribute. Instead, it’s a multifaceted concept shaped by multiple interrelated components.
<div> <div>Explanation Content</div> <div>Description</div> </div> <div> <div>Source</div> <div>Data used to inform an AI insight and the origin of the explanation requirement.</div> </div> <div> <div>Timing</div> <div>When the explanation is generated <strong>and</strong> when it’s presented. Timing may cover proactive or reactive explanations.</div> </div> <div> <div>Trigger</div> <div>What causes the explanation to appear.</div> </div> <div> <div>Autonomy</div> <div>Distinguishes between reactive explanations (requested by the user) and proactive explanations (initiated automatically or by other events)</div> </div> <div> <div>Content</div> <div> <p>What the explanation communicates. Consider the following subcategories:</p> <p>1) Sensitivity<br>2) Personal data</p> <p>3) Confidentiality<br>4) Minimum content</p> </div> </div> <div> <div>Level (scope)</div> <div>The focus of the explanation: local, global, or both</div> </div> <div> <div>Criticality</div> <div>The importance of the explanation: mandatory, recommended, or undetermined</div> </div> <div> <div>Goal(s)</div> <div>The purpose for seeking the explanation: understandability or intervenability. These goals and their subgoals are covered in the <a href="https%3A%2F%2Fvideo.sap.com%2Fmedia%2Ft%2F1_wdain60t">full recorded presentation</a>. For more details, see the <a href="https%3A%2F%2Fsap.sharepoint.com%2Fsites%2F206415%2FLists%2FUser%2520Studies%2FDispForm.aspx%3FID%3D674%26e%3Ddby1Iz">SAP explainability research findings</a> in the User Research Library.</div> </div> <div> <div>Intended recipient</div> <div>The target audience for the explanation: An opportunity for SAP to define various intended recipients across the organization</div> </div>
Progressive Disclosure
Progressive disclosure in explainable AI means presenting information in layers, starting with simple, high-level explanations. Users can then access more detailed or technical information if they want to. This method helps avoid overwhelming users with complexity and aligns explanations with the user’s needs and expertise.
Traditional UX often aims for seamlessness: removing friction, hiding complexity, and smoothing transitions. However, in AI-driven systems, seamlessness can lead to opacity, false trust, or user disempowerment. Seamful design takes a different approach by acknowledging imperfection and supporting users in building a realistic mental model of how the system works.
Seamful design is useful when a system shows expected or unexpected limitations, and there’s a risk of over- or under-reliance. This approach plays a key role in developing calibrated trust and understanding what a tool can and can’t do.
Rather than hiding seams, seamful design turns these moments into design patterns that are integral to the solution. This empowers users to decide when and how to engage with limitations. However, it’s important to reveal system limitations at a level of detail that’s appropriate for the context.
<div> <div>Scenario</div> <div>Seam</div> <div>Design Choice</div> <div>Example</div> </div> <div> <div>Deliberate slow-downs (speed vs. reflection)</div> <div>Fast automation can bypass human judgment</div> <div>Introduce pauses that encourage user reflection</div> <div>In healthcare decision-support, the AI may require doctors to acknowledge critical risks before moving forward, rather than auto-filling recommendations</div> </div> <div> <div>Require user input before critical actions</div> <div>The AI can’t infer intent perfectly</div> <div>Add friction by asking for clarification or confirmation before high-stakes actions</div> <div>A financial AI assistant might say, “I’ve prepared a wire transfer to X. Please review and confirm.”</div> </div>
guideline
If your use case includes known limitations, strategically disclose them and pair the message with SAP’s ethical principles to build trust and support user agency.
Sources
<div> <div>Title</div> <div>Year</div> <div>Author(s)</div> </div> <div> <div><a href="https%3A%2F%2Fdoi.org%2F10.1145%2F3708504">A typology of explanations to support Explainability-by-Design</a></div> <div>2025</div> <div>Niko Tsakalakis, Sophie Stalla-Bourdillon, Dong Huynh, and Luc Moreau</div> </div> <div> <div><a href="https%3A%2F%2Fico.org.uk%2Ffor-organisations%2Fuk-gdpr-guidance-and-resources%2Fartificial-intelligence%2Fexplaining-decisions-made-with-artificial-intelligence%2Fpart-1-the-basics-of-explaining-ai%2Fwhat-goes-into-an-explanation%2F">What goes into an explanation?</a></div> <div>2024</div> <div>Information Commissioner’s Office (ICO)</div> </div> <div> <div><a href="https%3A%2F%2Fwww.dcs.gla.ac.uk%2F~matthew%2Fpapers%2Fubicomp2003HCISystems.pdf">Seamful Design and Ubicomp Infrastructure</a></div> <div>2003</div> <div>Matthew Chalmers</div> </div> <div> <div><a href="https%3A%2F%2Fdoi.org%2F10.1145%2F3637396">Seamful XAI: Operationalizing Seamful Design in Explainable AI</a></div> <div>2024</div> <div>Upol Ehsan, Q. Vera Liao, Samir Passi, Mark O. Riedl, and Hal Daumé</div> </div> <div> <div><a href="https%3A%2F%2Farxiv.org%2Fpdf%2F2408.05345v2">Explainable AI Reloaded: Challenging the XAI Status Quo in the Era of Large Language Models</a></div> <div>2024</div> <div>Upol Ehsan and Mark O. Riedl</div> </div> <div> <div><a href="https%3A%2F%2Fdoi.org%2F10.1145%2F3491101.3503727">Human-Centered Explainable AI (HCXAI): Beyond Opening the Black-Box of AI</a></div> <div>2022</div> <div>Upol Ehsan, Philipp Wintersberger, Q. Vera Liao, Elizabeth Anne Watkins, Carina Manger, Hal Daumé Iii, Andreas Riener, and Mark O Riedl</div> </div> <div> <div><a href="https%3A%2F%2Fdoi.org%2F10.48550%2FarXiv.2207.00007">“Explanation” is Not a Technical Term: The Problem of Ambiguity in XAI</a></div> <div>2022</div> <div>Leilani H. Gilpin, Andrew R. Paley, Mohammed A. Alam, Sarah Spurlock, and Kristian J. Hammond</div> </div> <div> <div><a href="https%3A%2F%2Fdoi.org%2F10.1016%2Fj.inffus.2023.101805">Explainable Artificial Intelligence (XAI): What we know and what is left to attain Trustworthy Artificial Intelligence</a></div> <div>2023</div> <div>Sajid Ali, Tamer Abuhmed, Shaker El-Sappagh, Khan Muhammad, Jose M. Alonso-Moral, Roberto Confalonieri, Riccardo Guidotti, Javier Del Ser, Natalia Díaz-Rodríguez, and Francisco Herrera</div> </div> <div> <div><a href="https%3A%2F%2Farxiv.org%2Fpdf%2F2202.11748">The Need for Interpretable Features: Motivation and Taxonomy</a></div> <div>2022</div> <div>Alexandra Zytek, Ignacio Arnaldo, Dongyu Liu, Laure Berti-Equille, Kalyan Veeramachaneni</div> </div> <div> <div><a href="https%3A%2F%2Fwww.bbc.com%2Fnews%2Farticles%2Fc2l799gxjjpo">What is AI, how does it work and why are some people concerned about it?</a></div> <div>2025</div> <div>BBC News</div> </div>
