Intro

As our world grows increasingly interconnected and our technology becomes increasingly intelligent, the design of agentic systems that are capable of autonomous execution and decision-making presents consequential challenges and opportunities. We all bear a significant responsibility to place humans and ethics at the forefront of our technology processes to prioritize the needs, values, experiences of users and the greater society. By doing this, we can create agentic systems and experiences that not only enhance user experiences, but also foster user empowerment and principles of responsible AI.

The guidelines below aim to provide clarity in design and development of agentic systems in ways that are valuable, enhances transparency, empowers humans to remain in control, and mitigates potential risk associated with human-AI interactions. We hope that these guidelines will provide clear direction for creating ethical, user-friendly enterprise agentic systems that align with diverse needs and rights of all users.

Designing Human & Agent Interactions

With the design of multi-agent systems (MASs), it is important to create a user experience that minimizes complexity and enhances usability. This section outlines key principles for structuring agent interactions to ensure users engage with a single orchestrating agent, without the need for direct communication between multiple sub-agents.

Guidelines:

1. Structure the agent ecosystem around one primary orchestrating agent (Joule) that serves as the interface for user interactions.
2. Place the responsibility on the system, not the user, for performance and generation of outcomes.

Why: Our research indicates that across various use-cases, most people expect to and prefer to manage a single AI agent over engaging with multiple agents within an ecosystem, as this reduces cognitive load. Understanding and selecting agents to direct their prompts is expected to introduce additional cognitive burden. People expect Joule to function as their seamless AI assistant, integrating with their existing systems to support their goals and enhance their efficiency.

Exposing Agents to End-Users

Guideline:

1. Limit exposure to the communication happening between agents

Why: The research clearly shows that people prefer and expect to engage with a single AI Assistant. This was consistent in both the primary research and secondary research - where we learned that revealing interaction between agents has shown to be confusing, unnecessary, and sometimes uncomfortable and problematic to users.

What we saw was that people expected Joule to have access to everything and to be able to be their one AI assistant.

Interacting with more than one agent was confusing and unnecessary to most participants. A few participants also mentioned just wanting the output without the back-and-forth interaction with different agents.

Expectations are being shaped by “general” AI’s like ChatGPT - and people are mentioning that they expect the AI to be able to respond with “no limits”.

Avoiding Antropomorphism of AI Agents

Antropomorphism is the attribution of intrinsic human qualities or human-like capabilities onto non-humans, like systems in our case.

Guideline:

1. Use language and terminology that supports:

   1. transparency in the capabilities and limitations in technology without implying human or human-like capabilities
   2. trust calibration by helping users to manage realistic expectations, not based on false assurances
   3. help users understand how the system supports their goals and tasks

Why: Internal and external research studies have shown that that people currently have a threshold for comfort towards anthropomorphism. Interacting with a system via natural language for non-embodied agents in particular, people expect a degree of humanness with conversations in a way that’s natural, emotive to a certain degree, and encourages engagement.

Our research shows people perceive minimal value in the trade-off made for anthropomorphizing agents. While this takeaway may differ depending on the use-case or at a future point in time when people are more familiar with the technological concept of agents, for now, people prefer efficient interactions with systems that prioritize clarity and efficiency.

Guideline:

1. Apply a thoughtful approach to messaging and visuals as context clues to further signal and provide understanding on agentic processes and systems.

   1. Leverage non-anthropomorphic language,
      e.g. using modifiers like activate, launch, and AI
   2. Be careful when using anthropomorphic language
      e.g., thoughts, engage
   3. Avoid using icons or visuals that resemble humans for AI systems or processes

Why: Through research we have learned that by anthropomorphizing AI, people using these systems can develop unrealistic expectations, have mismatched trust calibration, and consequently having too much trust in AI systems which could result in decreased confidence, loss of trust, and decreased usage and adoption.

Ensuring Explainability and Interpretability

When it comes to multi-agent systems, explainability is key for helping enterprise users understand what’s happening all along the way. It’s essential to transparency and will provide people with the clarity they require to have confidence in the system.

Guideline:

1. Explainability must be interpretable and support both user understanding and outcomes. At the same time, there is a need to balance explainability with cognitive load and efficiency.
2. Ensure that explainability is not just a checkbox but supports clarity and understanding of the process and outcomes for users in relation to their goals.

Why: Our secondary research shows that there are benefits to leveraging non-anthropomorphic design to increase transparency, user-agency, and trust calibration. The observations we made in this primary research further supported some use of design patterns like progress indicators, status indicators, and progressive disclosure patterns. These were widely recognized by participants AND supported their understanding of their task.

Studies show that we must ensure

interpretabilityGuideline:

1. Give transparency to business users all along the way. Examples include real time processing indicators, system actions and statuses.
   1. Apply flexible strategies (e.g. progressive disclosure) to make data sources, agent reasoning, and human-ai alignment visible, traceable, understandable, and valuable, allowing humans to be in control (this recommendation is consistent with both the Primary and Secondary Research we conducted).

Why: Our secondary research shows that it’s important to reuse and leverage non-anthropomorphic design patterns (processing indicators, status indicators, and progressive disclosure) for explainability and interpretability of AI systems and flexibility for balancing information with cognitive load. We also saw these patterns used in the market during our competitive analysis.

The Terminology We Use Matters

Guideline:

1. When designing for and referring to multi-agent systems in the user interface, use language and terminology that supports

   1. transparency in the capabilities and limitations in technology without implying human or human-like capabilities
   2. trust calibration by helping users to manage realistic expectations, not based on false assurances
   3. help users understand how the system supports their goals and tasks

Recommendation: When using an agentic system or process, instead of using 'agent' as a term, leverage specific names of data sources or describe explicit AI processes. It’s also important to make it clear by using “AI Agent” or “Agentic AI”.

1. Be thoughtful when using messaging and visuals as context clues to further signal and provide understanding on agentic processes and systems

   1. leverage non - anthropomorphic, language
      eg., activate
   2. caution on using anthropomorphic language
      e.g., thoughts, engage
   3. avoid using icons or visuals the resemble humans for AI systems or processes

Why: Both our primary and secondary research consistently showed that, “almost ALL participants associate the term “Agent” with a human”. Participants were generally unsure about what an “agent” is. Some participants expected HR agent to be a human or a human HR team member.

While most participants associated the term “agent” with a person, when used in conjunction with a modifier like “launch” or “activate” – they associated “agent” with AI. The opposite was true when it came to words like “reasoning”, “thoughts”, or “engaged”. In those cases – they still associated “agent” with a human. We also saw that using a person icon next to HR Agent created confusion and plays a role in how they perceived the agent as human.

Without having an understanding of agentic terms, participants value understanding the data sources and algorithmic process over agentic labels to understand AI outcomes.

Designing for Human-in-Control in Agentic Systems

Guideline:

1. Leverage principles of explainability, interpretability, transparency contributes to human learning and being + feeling in control.
2. Expectations for human-in-control may vary depending on use-cases and context. Continue to understand end-user expectations and attitudes towards being in-control for different tasks to effectively calibrate degree of human-in-control with AI.
3. Provide easy ways for users to provide AI feedback in order to influence AI learning and quality of AI outcomes.

Why: Our research shows that people are aware that AI requires human input for improved outcomes, but to influence AI effectively, they also understand their role and acknowledge the need to enable user controls for human feedback for increased effective AI collaboration and outcomes. This highlights how explainability, interpretability, empowers reciprocal learning with AI and human-in-control.

References

Multi-Agent Systems

• Secondary Research + Competitive Analysis
• Primary Research

Definitions

AI Agents: Intelligent systems that show reasoning, planning, and memory and can leverage and interact with diverse tools and learn to help achieve objectives or goals. May or may not be an LLM powered agent.

LLM powered Agents: AI systems that utilize LLMs to interact (with users and/or other agents), answer queries, or perform tasks through conversations.

Autonomous agents: Systems that can make decisions and take actions in pursuit of a goal (Goyal et al. 2024).

Agentic workflows: Multistep processes that use one or more AI agents in iterative and dynamic ways to pursue relatively more complex goals and workflows.

Antropomorphism: Anthropomorphism is the attribution of intrinsic human qualities onto non-humans (Springer, 2024).

Multi-agent System: System made up of multiple agents within a shared environment.

Non-Embodied Multi-Agent Systems (Joule): Non-embodied multi-agent systems do not have a physical or virtual body that interacts directly with the environment. These agents typically interact through communication channels or by modifying shared resources or databases. Examples include software agents coordinating airline schedules, trading algorithms interacting in financial markets, or agents in a simulation where they communicate but do not physically move.

Trust Calibration: The process of humans adjusting their level of trust in a system based on perceived expectations, capabilities, and performance.