Picture this: Q2 is wrapping up, and you’ve just had a eureka moment – a brand new product idea with blockbuster potential. Now you need the budget to build and launch ASAP.

Depending on how your company handles forecasting, this scenario could present a head-smacking problem or a golden opportunity:

• Problem: If you work at Company A, where the budget for this year’s innovation fund was set last year, the executive committee may be loath to unleash additional cash.

• Opportunity: If you work at Company B, which adjusts revenue and cash flow forecasts throughout the year based on actual events, the executive committee will be happy to invest. In fact, they’ll be able to tell you exactly how much money is available and when.

If these two companies are racing each other to market with similar ideas, Company B is your likely winner. Furthermore, the sales team for Company A is likely burdened with hitting revenue forecasts that have been rendered outdated by more recent events.

This more flexible approach, it turns out, is the future of forecasting. In fact, for leading companies, it’s the present. Against today’s backdrop of shorter product lifecycles, faster consumer changes, and relentless advances in technology, the traditional approach of creating a five-quarters-out forecast is less reliable than ever. It’s also a potential competitive disadvantage, as advances in data quality, real-time collection and analysis, and AI make near-real-time forecasts possible.

The better, more responsive approach goes by different names, – some call it dynamic planning, others call it agile forecasting. There’s a whole movement called Beyond Budgeting dedicated to these concepts. Whatever the name, the idea isn’t new, but more forward-thinking companies are finally moving in this direction.

Its value shows in bottom-line results. In a 2023 study by McKinsey & Company, respondents from top-performing organizations were 1.6 times more likely to use advanced analytics for finance tasks such as cost analysis and budgeting and for business operations tasks such as predictive modeling and pricing.

Making this leap isn’t easy. The entire financial system is oriented around past performance, as enshrined in the hallowed quarterly earnings report. Most businesses have managed budgeting the same way for decades: one year’s spreadsheet provides the foundation for the next. Nevertheless, the time for change is now. It’s crucial to reorient budgeting and planning around more flexible and comprehensive responses.

This article will show you why it’s worth the effort, and how to get started.

How the shopworn budgeting process dings your results

Now that businesses have so much data about everything, it seems sensible to use that data differently. The challenge, of course, is that not all businesses have changed the way they think about planning.

If you’re in a traditional finance organization, you likely spend about 80% of your time processing and analyzing events that happened already. This remains standard operating procedure at many businesses for many reasons: an institutional resistance to change, a lack of innovative tools, investments already committed to antiquated IT tools, and structural challenges stemming from systems that aren’t configured for real-time budgeting.

Another problem is that many companies still manage budgeting based on a fiscal year. The long time frame can result in familiar (but unfortunate) processes and behaviors such as managers sandbagging forecasts to reduce expectations, arbitrary end-of-quarter sales deadline pressures, and accounting hijinks.

Many people think of financial markets as real-time, reactive entities, but the traditional earnings reports and quarterly check-ins on financial performance are rigid and not real time at all. Stock prices react to analyst expectations and performance against them, and straying too far from projected annual or quarterly earnings can hurt a company’s share price.

Put simply, the system resists evasive action because everything is linked to that long-winded, year-in-advance process.

Dynamic planning addresses this head-on. For example, the approach administered by the Beyond Budgeting Institute exists in part to free organizations of command-and-control cultures. Leaders there spend the bulk of their time ruminating on how to make processes such as budgeting and forecasting more responsive and dynamic.

According to Bjarte Bogsnes, chair of the Beyond Budgeting Roundtable, a global network that promotes the Beyond Budgeting principles, the current mainstream approach to budgeting fails for several reasons:

• The process is time-consuming, even among the most agile traditional organizations.
• The link between allocations and strategy is often weak.
• Spending decisions are made too early and at a level that is too senior.
• Assumptions built into budget decisions are outdated.
• Once budget decisions are set, they can prevent pivots that can add value.

Hyoun Park, CEO of industry analysis firm Amalgam Insights, says COVID-19 demonstrated these drawbacks and also presented a golden opportunity for companies with traditional approaches to budgeting to rethink how they operate. “It was a time when 90% of all business assumptions went out the window,” he says. “Companies learned the hard way that they needed more flexibility. They realized they needed to be more dynamic. They woke up to how much more agile they had to be.”

What agile forecasting really means now

If we’re being literal, agile means “nimble” and “able to move quickly and easily.” In the world of forecasting, agile also means “operating with an eye on the future.”

In the budgeting context, agile means making a conscious decision to continually adapt a business based on what’s happening today as well as what you believe will happen in the months and years ahead.

One conversation that got subject matter experts thinking differently about forecasting was a chat that a group of finance nerds had with leaders at a Fortune 500 company years ago – a conversation about forecasts based on actual performance. The nerds harped on how expectations must be driven by actuals, i.e. past performance. The Fortune 500 leaders responded that they didn’t care about actuals precisely because those were in the past; instead they cared about where everyone at the company thought the budgets needed to grow. “We care about the forecast because the forecast will tell us where we need to invest, where we need to grow, what products we need to push, and what products we need to get rid of,” the leaders said. “If you really want to manage your business, you want to manage it based on what you believe is going to happen” – not what happened last year.

It’s true that today, the idea of the entire forecasting chain working in literal real time remains a work in progress. Financial systems can immediately update dashboards when they get new data, but most related forecasting paradigms don’t incorporate that data at the very instant things happen in the real world. What’s more, some partner systems aren’t yet passing real-time data up and down the supply chain, either because they don’t process the data that fast themselves or because of lags and batch processes in transmission. And external factors (including changing weather patterns, population shifts, emerging geographic risks, and technology developments) can take time to be reflected in third-party data that’s also part of the forecasting process.

Given these factors, agile forecasting hasn’t yet reached its full promise. But when compared with the old formal process that happens quarterly, semiannually, or annually from a budgeting perspective, agile forecasting is dynamic, organic, and fundamentally reflective of the changing needs of businesses today.

Why dynamic planning works better

The dream of forecasting more dynamically goes way back.

In the early days, a lot of the thinking on the subject was just that – thinking. It became more feasible as data analytics matured and the possibilities for forecasting evolved from compiling historical data toward sense and respond. This occurred as other new and innovative approaches, as outlined below, were unfolding and leaders began rethinking ways to manage budgeting:

• One of the most popular concepts was “zero-based budgeting,” developed in the 1970s by former corporate accounting manager Peter Pyhrr, in which all annual budgets had to be built and justified from scratch rather than simply tweaking last year’s allocations.
• Two Harvard professors defined “activity-based costing” in 1987, aspiring to provide a more accurate basis for forecasting.
• This book on dynamic planning was published in 1994.
• The Beyond Budgeting movement began in 1998.
• “Rolling forecasts” were hyped in the early 2000s.

While the details, successes, and drawbacks of these new ideas differed, the thinking behind them was the same: there had to be a better way.

Most of these frameworks require a shakeup that goes beyond spreadsheets. Bogsnes says the Beyond Budgeting version frees institutions from an obsession with command-and-control management and other platitudes about what’s best for maximizing performance. The movement now espouses 12 management and leadership principles – which Bogsnes says apply perfectly to a more agile approach to managing an organization, including forecasting.

Dynamic planning also requires heightened collaboration and a data-focused, analytical mindset.

The key is for finance to work proactively with other areas, such as supply chain and procurement, empowering leaders to make the best and most profitable financial decisions. Incorporating near-real-time information enables businesses to be lean and efficient. They can simplify the forecasting process by focusing only on what they need to reach realistic targets and on the here and now rather than distant fiscal quarters. This leads to an ability to adapt quickly; eventually the setup becomes so fine-tuned that the business can immediately respond to new developments.

Since data is at the center of everything, companies that want to use agile forecasting must embrace an analytics mindset.

This concept plays out as a progression. First, companies must accept that agile forecasting means taking a wider and broader approach. They can’t just rely on one or two data points (say, revenue and expenses); instead they need to look at system-wide data before deciding how to prepare for what’s next.

This, in turn, spotlights the need for data standardization and harmonization. A business that’s going to look deeply into its data to expedite and streamline forecasting must make sure the data exists in the same language and forms. As data is standardized, quality can improve, as it must in order to deliver improved forecasts that inform better decisions.

Road map to more agile forecasting

Once business leaders have wrapped their organizational heads around a new way of thinking about the future, it’s time to follow the path to true business agility. This approach is a total evolution of thinking on the subject; instead of approaching the process at a fixed point in time, it constantly considers the changing landscape.

Bogsnes says business units should update their forecasts every time something happens that they believe justifies an update instead of updating at fixed points in time.

“With this approach, you do not have a predefined frequency or predefined time horizon for updating the forecasts,” he explains. “That update is not done for corporate, or what some might call, ‘The Guys Upstairs.’ Managers do it for themselves in order to run their own lines of business, then the information goes into a common database so everyone can see it.”

Based on conversations with Bogsnes and Park, here are four steps to make forecasting more agile. Fittingly, most of these concepts echo agile software development principles, which means leaders may already be familiar with them.

Step 1: Decentralize decisions and set priorities.

Instead of top-loading an organization with forecasters who are not directly connected to business units, empower lower-level managers to manipulate data they see in real time and to make decisions based on this data.

As Park puts it, smart people will make smart decisions.

“If you have hundreds or thousands of cost centers and you don't have enough managers to be able to actually manage what's going on, you're never going to get to dynamic forecasting,” he says. “It can be a big mistake having one person trying to track 300 different cost centers or departments and never being able to get into enough detail to understand what's going on.”

It’s also a good idea to incorporate technology (in some cases, AI) to automate some of these processes based on priorities at the lower levels. If you think of a business as a container ship, the concept is that it’s easier to turn more quickly if you have people working to turn earlier in the process.

Step 2: Separate the processes for forecasting, budgeting, and investing in innovation.

Agile forecasting can be introduced only after the traditional budget purposes have been separated into different processes, Bogsnes says.

“Companies make budgets as a forecast of what next year could look like,” he says. “The same budget, however, also sets financial targets, and they also use the budget as a resource allocation mechanism, handing out bags of money to different parts of the organization. Our recommendation is not to do these things in one process because these are very much conflicting purposes. A target, for instance, is an aspiration, while a forecast is an expectation. You can’t mix the two.”

“With separate targets, you can design each one in much more intelligent ways,” Bogsnes says.

In a related vein, SAP Insights has previously written about innovation metrics; when combined with agile forecasting concepts, these metrics can unlock more effective future investments in new products and ideas.

Step 3: Build capabilities to sense and respond.

Companies need to move quickly when data suggests action, and it’s important that they have the protocols in place to do so. In some organizations, this might mean giving certain departments authority to make decisions about forecasting on their own. In others, it might mean implementing technology to automatically engage in certain actions if data satisfies specific conditions – the capability often called prescriptive analytics.

Park adds that companies also should build in the capability to about-face.

“It’s critically important to be able to pull up at a stop sign at any moment, immediately step on the brakes, and have an all-hands or all-manager meeting whenever there is a truly significant issue,” he says. “There always needs to be an emergency brake on the forecasting process – and it can’t take a month to do it.”

Step 4: Cut costs that are not related to strategy.

As more organizations investigate how to become leaner, a growing number are embracing the idea that nearly everything can be obtained as a service. Fundamentally, this is a more strategic way of looking at business and at forecasting.

By trimming budgets in all other areas, companies effectively can trim risk associated with those budgets, streamlining the forecasting process and paving the way to becoming nimbler.

Park says that this approach also can provide for the unknown when applied with the right algorithms.

“With more algorithmic forecasting capabilities, we have a better idea of the what-ifs that may occur on a multiyear basis as well as the type of investments that we may need to make if we’re trying to grow the company toward a three- or five-year goal,” he says. Considering the potential effects of layoffs, Park adds that a more tactical forecasting approach “might show that there's a certain level of investment to fully onboard somebody into a specific position,” which would help companies get a more realistic understanding of the costs of personnel decisions.

From forecasts to real-world results

In a Forbes magazine article, Bogsnes describes how the move toward agility affected Equinor, a Norwegian multinational energy company that suspended its annual budgeting process years ago. This included abolishing the traditional investment budget, for which all spending decisions were made in the fall.

“The principle is [that] the bank is always open,” Bogsnes says. “You can always forward a project for approval. ‘Yes’ or ‘no’ depends on the quality of the project and our capacity to fund it and implement it as things stand. Capacity information comes from our dynamic forecasting, where we constantly monitor future cash flows.”

Any company can embrace agile forecasting, but some are better prepared than others to succeed.

According to Park, companies that have recently restructured tend to have an easier time with this more dynamic approach because they’ve already looked long and hard at how they can improve efficiencies.

AI also can play a big role in agile forecasting, providing executives with technology to interpret more data in real time. In theory, as AI continues to improve, so too will the ability for businesses incorporating it to respond quickly and seamlessly to changes in data over time. Park envisions getting to the point where AI is telling executives where to look, providing forecasts, and recommending different approaches – all automatically.

In the meantime, as these processes mature, agile forecasting at its best becomes akin to continuous planning, a constantly evolving approach to the future based on signals coming in from across the organization and the outside world.