Agentic AI is becoming a potential “killer app” for CIOs looking to increase the return on their growing AI investments. KPMG has already dubbed 2025 as the “year of agentic AI”. The advisory firm’s AI Pulse Survey for Q4 2024 found that 51% of organizations are exploring the use of AI agents and another 37% are piloting the technology, which comprises intelligent, often autonomous agents that can understand natural language, bridge information gaps, integrate across systems, and even take action.

While near-term targets for automation include administrative duties and call center tasks, what gets people most excited is how agents can integrate into existing applications and end-to-end business processes because it provides a much clearer path to ROI than early iterations of generative AI (GenAI) technologies.

“The first round of tools or platforms based on large language models were, from a user experience perspective, completely disconnected from business workflows,” says Isaac Sacolick, president and founder of StarCIO, a digital transformation consultancy. “Agentic AI is bringing this experience directly into the tools that people use every day. When you start to see these types of agents in action, you can quickly see the promise they hold.”

Timo Elliott, VP of Marketing and Global Innovation Evangelist with SAP, puts a finer point on agentic AI’s potential: “Ultimately, agents are going to make AI a lot more useful.”

Agentic AI raises the stakes for CIOs, who are increasingly tabbed to lead AI projects. The KPMG survey found that 71% of respondents say CIOs are leading AI initiatives, followed by CEOs (17%) and chief innovation officers (10%). That’s a significant shift from the Q2 2024 iteration of the survey, when 49% of respondents cited CEOs as the primary AI leaders.

As CIOs map out their strategies, it’s becoming clear that AI in general, and agentic AI specifically, will change how they manage their organization’s IT environment and how they deliver services to the rest of the business. With the ability of agents to automate a broad swath of end-to-end business processes—learning and changing as they go—business users will be interacting with them closely as part of their daily activities. As a result, CIOs will oversee significant shifts in software development, IT operating models, staffing, and IT governance.

“Every aspect of the agentic AI stack is constantly shifting,” says Elliott. As agents allow business users to do more without involving IT, the need for “adult supervision”—in the form of compliance, security, governance, and privacy guidelines—will become more urgent, he says. “It’s going to be chaos until people get the hang of it.”

Here are six questions CIOs should be asking to mitigate the chaos and successfully deploy and manage agentic AI across the business.

1. How do AI agents differ from chatbots and other “assistant” technology or automation tools?

Software-driven automation is not a new concept to CIOs. Robotic process automation (RPA) emerged in the early 2000s as a rules-based method for automating simple, repetitive tasks, such as data extraction or order processing. The launch of the generative AI application ChatGPT in November 2022 added a layer of cognitive capabilities to process automation, allowing developers to create a new generation of automated “assistants” that could interact with customers and help employees with a variety of activities, from software development to content creation.

Agentic AI raises the intelligence bar even higher, adding new levels of autonomy and reasoning that allow AI agents to work together, make decisions based on context, and trigger actions across different business functions.

“Every organization has key end-to-end business processes, which are mostly already automated,” Elliott explains. “AI agents offer a new way of patching problems, either by automating steps or by handling exceptions more flexibly.” In this manner, agents can begin to make or suggest process improvements as they go along.

In a blog post for Infoworld, Sacolick notes that several types of AI agents exist, “classified by how they make decisions and perform actions. Model-based agents replace rules with AI models and supporting data, while goal and utility-based agents compare different scenarios before selecting a course of action. The more sophisticated AI learning agents use feedback loops to improve results, while hierarchical agents work in a group to deconstruct complex tasks.”

Developers at Milvus, which makes an open-source vector database for AI applications, provide an example of hierarchical agents in the context of a warehouse robotics system: "A top-level 'orchestrator' agent might oversee inventory management, deciding which products need restocking. It could delegate subtasks to mid-level 'zone manager' agents, each responsible for a section of the warehouse. These mid-level agents might then assign specific pick-and-place tasks to low-level 'robot controllor' agents operating individual machines."

While these technical distinctions are important for CIOs as they map out a deployment strategy, Sacolick recommends that IT leaders focus on a different angle when selling the idea to their C-suite colleagues: “The CIO’s role is to unpack the jargon and explain it to the C-suite in terms of the value, what problem they’re solving, or where the long-term impact will be—using whatever language [business leaders] are familiar with.”

2. How do we prioritize use cases for agentic AI?

CIOs wary of disrupting business with AI pilots that target  vital operational areas such as ERP, human resources, sales or supply chain are getting an assist from software vendors that have been integrating AI agents into their enterprise applications. It’s a good place to start testing agentic AI’s capabilities.

“Look at your core applications first, because they hold the data and the processes where agents can provide the most value,” says Sacolick. “Ultimately it comes down to where the user is.”

This focus on where “work gets done” is the key to an organization’s ability to shift from capturing incremental productivity gains, where much of the early GenAI experiments have focused, to improving business performance by rethinking and transforming end-to-end processes, says Francesco Brenna, VP & Senior Partner for AI Integration Services with IBM Consulting. “Many businesses are still far from seeing value at scale from AI implementations,” he says. “The first generation of AI assistants are good at helping you find the information you need to do your job, but they’re not really focused on helping you actually get the job done. This is where agentic AI can really have an impact.”

Agents that are embedded into existing applications will make it easier to implement the technology across the business, says Elliott. “Building AI agent proofs of concept is surprisingly easy,” he says. “Making them ready for real-world production environments is surprisingly hard.”

From application vendors to hyperscale cloud providers, the tech industry is working hard to help ease that process. For example, agents embedded in SAP’s financial management products can help an accounts receivable clerk quickly settle a disputed invoice from a customer. An agent tasked with monitoring an inbox that houses customer inquiries would identify the dispute and alert the accounts receivable team, generating a case number and a summary of the issue. Next, the agent interacts with a help desk agent to review previous cases and the customer knowledge base to identify possible solutions. It then proposes a few suggestions to resolve the dispute—all in a manner of minutes.

A member of the accounts receivable team is then prompted to review the case, including the original customer e-mail and the proposed solutions. The clerk also can review the agent's reasoning and actions before making an informed decision on which solution to offer the customer. Based on the solution, the agent can then draft an e-mail for the clerk to approve and send.

“Application vendors know where to get the right data and understand the goals and context of the action being taken,” says Elliott. “And they have the economies of scale to figure out how to set up these agents for each business decision or automation opportunity across all their customers.”

3. How do we ensure that agents running across business functions play nicely with each other?

The above example underscores the importance of maintaining “human-in-the-loop” checks and balances as agents take on more autonomy—and begin interacting with each other across different parts of the business. “Make sure you can actually track the behavior of the agents and react to their behavior to make sure they are following your rules,” says Brenna. CIOs will need to make sure these types of governance policies and processes are in place to protect sensitive data and to ensure, Sacolick notes, “that agents don’t break things as they go.”

“Most businesses will seek fully autonomous options for low-risk areas where there is trust in decision accuracy, reliability around automation, and the need to respond quickly at scale,” Sacolick says. “Human-in-the-middle allows people to bring added context to decision-making when AI agents may not have sufficient visibility and history to make complex decisions.”

For example, Sacolick explains, healthcare AI agents may provide recommendations to doctors when evaluating patient symptoms, but the doctors still need to review and trigger prescriptions. An even higher level of human oversight, known as “human-at-the-helm,” is an option for “large-scale and speedy decisions, where AI agents are autonomous, but knowledgeable and accountable experts can adjust decision authorities, or fully unplug an AI agent when necessary,” says Sacolick.

Human-based checks and balances are vital for validating agent-based outputs and recommendations and, if needed, manually change course should unintended consequences—including hallucinations or other errors—arise.

“Agents being wrong is not the same thing as humans being wrong,” says Elliott. “Agents can be really wrong in ways that would get a human fired if they made the same mistake. We need safeguards so that if an agent calls the wrong API, it’s obvious to the person overseeing that task that the response or outcome is unreasonable or doesn’t make sense.”

These orchestration and observability layers will be increasingly important as agents are implemented across the business.

“As different parts of the organization [automate] manual processes, you can quickly end up with a patchwork-quilt architecture that becomes almost impossible to upgrade or rethink,” says Elliott. The good news, he adds, is that whereas early GenAI tools relied on a single long “chain of thought” prompt, agents can more easily be broken down into component parts, or subagents, and each one can be worked on until it is reliable.

From there, CIOs will need to develop a multi-agentic framework with an orchestration layer that tracks and governs behaviors as agents interact with each other and with more traditional rules-based processes, says Brenna. “Agentic AI will not solve all your problems,” he says. “Some processes will still require a deterministic, event-driven approach.”

4. How will agentic AI improve how IT delivers technology services to the business?

IT teams won’t just be supporting other business functions in the deployment of AI agents. They’re also finding opportunities to improve their own operations and service delivery models. AI coding agents are already taking on programming tasks, reducing software development time and costs. Soon, coding agents not only will write the code, but separate agents will review code for errors; Sheldon Monteiro, executive vice president and chief product officer at Publicis Sapient, tells CIO.

“With DevOps toolchains already automating workflows, adding AI agents is a natural evolution,” Monteiro says. “These agents can autonomously reverse engineer specifications from code, forward engineer test cases and code from specifications, and approve artifacts that meet certain threshold criteria, improving the overall level of automation.”

More broadly, agents will help improve visibility across enterprise architectures and help IT teams identify pain points, troubleshoot underperforming areas, and make changes and upgrades that can increase uptime and improve IT project success rates.

“Enterprise technology architectures in most companies are massively complicated—every part is linked to every other part,” says Elliott. “AI is really good at synthesizing massively complicated datasets and finding connections that humans don't know exist. [AI agents] will help us move and change things more confidently.”

AI agents can also improve the IT organization’s ability to perform comprehensive and ongoing risk assessments to increase protection and decrease the frequency and effect of cybersecurity incidents.

Elliott shares an example from an enterprise architect who was overseeing a migration to S/4HANA. He tasked an AI agent with reviewing the existing system and all its interconnections, then assess any potential trouble spots that could delay or derail the project. The agent identified a crucial incompatibility between applications that the architect’s team had missed during their initial evaluation. “Agents can help undo those Gordian knots more easily and more quickly than humans can to make migration projects go more smoothly,” Elliott says. “Anything that helps a CIO de-risk their world would be one of the biggest possible changes to their lives.”

Longer term, we’re likely to see IT organizations shift from supporting applications to supporting agents. “Today, IT is very much focused on applications for finance, procurement, et cetera,” says Brenna. “As agents are integrated across all these applications, the application landscape will blur. Agents will become the key component of IT architecture.”

This shift will require a new discovery and operating model to ensure people can find the right agents to help them do their jobs. “As agents evolve, they may need to be managed like humans, with IT serving as an HR-like function for agents,” Elliott says. Andreas Welsch, chief AI strategist at AI consultancy Intelligence Briefing, advocates for an “Agent Resources” department that draws on traditional HR principles to define the roles of agents and measure their performance.

5. How does agentic AI change collaboration between CIOs and their C-suite colleagues?

The effects of agentic AI will also trickle up to the C-suite, with CIOs likely to champion even closer alignment and integration with business leaders across finance, HR, procurement, supply chain, sales and marketing, and other functional areas. Given the embedded nature of AI agents in business processes, IT teams are likely to become more focused on these functions.

With agents taking on many programming and data science activities, it will be important for technologists to gain a deeper understanding of business processes so they can tailor agentic products accordingly, says Brenna.

As agents gather more and more feedback, they will add more value through their ability to analyze existing processes and workflows and suggest changes to improve performance. That’s when the concept of “push-button innovation” becomes very plausible, says Elliott. “Innovation becomes less like a staircase where you manually have to go up each step, and more like an escalator where the system takes you automatically to the next levels without you having to intervene, with each process constantly learning and improving and upgrading itself,” he says.

For example, a CIO, working with the CFO or the head of supply chain operations, could deploy a set of agents to scour industry news for hints on what could be done better or how other organizations or industries respond to sudden disruptions such as an unexpected trade war among several nations. “These agents work with your organization’s ‘innovation agent’ to suggest changes to ERP workflows, work with the ‘strategic SI’ agent to implement the changes, which would in turn work with the ‘human organization and change management’ teams of agents,” to ensure adoption and compliance, Elliott explains.

The CIO’s role in change management also will escalate as these scenarios play out. “We always talk about the importance of change management,” says Brenna. “But in this case, it's even more critical to ensure adoption of any of the solutions you're implementing.”

While emphasizing adoption and the ability to capture short-term business value, it’s also important for CIOs to work with other business leaders to tee up what Elliot calls “second-order thinking” to prepare for the longer-term business impact of AI agents. For example, he says, “if we extrapolate from current trends, which areas of the business might be able to be completely rethought, and what do we need to do to prepare for that?”

6. How do we manage the costs of large-scale agentic AI?

Ah, yes, the inevitable money discussion. As AI and GenAI deployments grow, so do concerns about costs. In IDC’s 2024 Future Enterprise Resiliency and Spending Survey, three of the top eight barriers IT executives cited regarding broader GenAI adoption are cost-related: high adoption costs (No. 1 concern), ineffective cost management (No. 4), and excessive infrastructure costs (No. 6). Agentic AI adds more uncertainty to the mix.

“The cost aspect is critical,” says Brenna. “Make sure you have transparency into resource consumption and the cost of deploying and maintaining proprietary LLMs [large language models].”

Some interesting pricing models for using AI agents are developing. Box CEO Aaron Levie raises the possibility of pricing agents like traditional labor, based on how long it takes an agent to complete a task. Another option is pricing agents on a per-outcome basis, which Levie says “allows for a simple relationship between what the customer needs and what they’re paying to get accomplished.” A third model involves pay-as-you-go “conversation-based” pricing. Regardless of which model they buy into, CIOs will need to map out a clear path to how agentic AI investments will create value for the business.

“Experimenting is good,” says Sacolick, “but ultimately you have to say, ‘Here’s what we’ve invested, here’s what we put in production, and here’s the value we got out of it.’”

New value metrics are likely to include time-to-decision improvements and decision accuracy, says Sacolick, offering an example: “A marketer can use agents to analyze campaigns with hundreds of experiments and variations and make daily optimizations based on performance.”

Because agents will be built into essential business processes and workflows, experts believe it will be easier to assess ROI than CIOs have experienced with other AI investments. “There’s a much more pragmatic sentiment right now because CIOs know there’s ROI out there,” says Sacolick. “They just need to capture it.”

Read more