A story as old as time. A person, group, company, governmental entity – doesn’t matter which – undertakes a project, any project. The team jumps out to a quick start, flush with high hopes and a firm belief that their project is special and unique. Come in on time and below budget? No sweat.

Then the unforeseen happens, the delays begin, the scope creeps, and before anyone knows it, the budget and timeline are history. Everyone just hopes to get away from the fiasco before the stench of failure spreads. With real disasters, careers can be ruined. Most of the time, the project is just a trash heap of failed expectations.

Think this sounds too harsh? Bent Flyvbjerg, executive chairman and co-founder of Oxford Global Projects, has studied and collected data on projects – particularly, government “megaprojects” such as infrastructure projects and hosting the Olympics – for decades. His data shows that projects perform as forecast much less than 1% of the time. The good news? You don’t have to keep banging your head on the wall.

You can learn from Flyvbjerg and Dan Gardner, who have coauthored a practical book called How Big Things Get Done: The Surprising Factors That Determine the Fate of Every Project, from Home Renovations to Space Exploration and Everything in Between, which distills the common themes of project failure and occasional success. We spoke with Gardner about the many ways projects can fail and how to break the cycle. Luckily for us mortals, he advises best practices that are memorable and easy to grasp. Now the trick is to keep these in mind when the next big initiative comes around.

Dan Gardner, author of How Big Things Get Done: The Surprising Factors That Determine the Fate of Every Project, from Home Renovations to Space Exploration and Everything in Between.

SAP Insights: Project management is one of the perennial topics of humankind, is it not?

Dan Gardner: Oh, absolutely. Projects are major undertakings. They are the points at which we try to turn visions into reality. And getting a project right is so central to not just business; it's so central to human life. We should be better at it by now. But most projects fail. The large ones – what we call “megaprojects” in the book – fail almost every time.

The essential elements of a project, which will determine its success or failure, are human perception and prioritization. And when we look at the patterns of how projects fail, we see patterns in human decision-making.

Q: Is it likely that business and technology executives will approach the ideas in your book thinking that IT projects are very different from construction projects and other types of projects? You know, we're unique, and this guy doesn't know what we're about?

Gardner: Not only is it likely, I would bet my house that is what people will think, because that's what people think in every domain. The people who work in construction think construction projects are unlike every other type of project. The people who are in defense think the same about defense projects. People in IT think the same about IT projects.

There definitely are elements that are unique to each of those project types. But human nature cuts across project types and geographies. If you don't understand human nature, and you don't understand how we get tripped up by universal thought processes, you will get tripped up by those processes. The only way to catch and correct them is to understand them fully before you begin.

Q: The central thesis of the book is that in order to increase your odds of having a successful project, you need to “think slow and act fast” as opposed to “think fast and act slow.” Tell us more.

Gardner: I am fully aware that “slow down; think carefully before you act” is just about as far removed from a new insight as possible. We have these old expressions as proof – “measure twice; cut once.” Or “make haste, slowly,” which is actually the personal motto of a Roman emperor. So this is really old advice. But when you look at how projects fail, what you see time and time again is that people are thinking fast – engaging in quick, superficial planning – and then acting slow as a consequence.

They launch. And they're excited because things are moving along fast, and everybody wants progress. But of course, because you engage in quick, superficial planning, problems will inevitably arise; subsequently, the project will start to bog down. And what starts as a sprint ends up becoming a slow, painful slog. So you have to reverse it: think slow; act fast.

The critical question then is why don't people do it if it's so bloody obvious? There are a few reasons. The first factor at work is politics – who does what, who gets what. There are all sorts of incentives for people to engage in quick, superficial planning. Imagine you are a contractor bidding on a contract. If you overlook anticipated problems and challenges, you can reduce your estimated cost and put that into your bid. When the inevitable happens and problems arise, you can then demand that the contract be revised accordingly because of “unforeseen circumstances.” Or, as we explore in our book, politicians have deliberately lowballed the cost, time needed, and degree of difficulty of public projects just to get them rolling.

The second factor is psychology – how people perceive, think, and make decisions. One great insight of modern psychology is that our quick, intuitive information processing is dominant. Most people will look at a problem and very quickly come to feel that, yeah, I've got this nailed down. We're good. Let's go. But that often leads to problems because it weighs against methodical planning.

There are also cases where psychology mixes with political interest. If we think back to that example of the politicians, they're pushing public projects ahead with minimal planning because they have the sunk cost fallacy on their side. When whoever is in charge faces inevitable issues, they’re going to continue with the project even if it’s over budget or off schedule because people hate pulling the plug when time and money have already been spent.

There’s even a third factor in the business world – a bias for action. You've got to get going. Planning is one of those things you feel like you have to do, but you don't really want to do it. What you really want is to have shovels in the ground and to see that hole getting bigger, because that's what real progress is. When you combine these elements, they push in the direction of think fast, and therefore act slow. That's why you see over and over again people engaging in quick and superficial planning, even though we should all know by now that is a terrible mistake.

Q: This bias for quick action seems even more pronounced in the technology sector, does it not? The volume of available data is growing, and projects that used to take five years now need to be done in five months or five weeks. And with the world changing so quickly, the perception is that you can't afford to take time to be deliberate. How does that come into play?

Gardner: That's absolutely correct. The sheer volume of information and the velocity of that information has been accelerating for at least 30 years. That's one more pressure point saying, “Move faster; make a decision.” So it’s further accelerating and pushing us in a direction that cultivates quick and superficial planning, which is death to projects.

If you go to Silicon Valley and you say the word “plan,” people roll their eyes. They hate planning. It makes them cringe. They’ll instead focus on product development, saying, “Let’s get a product that’s good enough. Let’s put it up for some customers, have them try it, and collect feedback. Then we’ll revise the product, and we’ll put it back in front of customers and so on. We’ll iterate, and the product will get better.” That’s the conventional wisdom in Silicon Valley. But the essential reason it works is because successful planning is an active process of experimenting, testing, and then iterating, iterating, iterating.

If you look at Frank Gehry, the great Canadian-born American architect, a lot of people know him as a wildly imaginative architect. What they don’t know is that Frank Gehry has a long-standing reputation for delivering his buildings on budget and on time, which is basically unheard of in his world. How does he do it? He spends an enormous amount of time planning as a process of exploring – that is, experimenting, testing, and iterating.

He’ll start with sketches, then he’ll move to physical models. And eventually he develops highly detailed, digital simulations. And all the while, he’s saying, “Well, what if I tried this? Would that work? What if I tried that, would that work?” He does these iterations, quite literally thousands and thousands. And only after he has done this incredibly exhaustive planning process, to the point where he is extremely confident that he has an airtight plan that can be delivered on budget on time, does construction begin.

Q: How do they not get completely exhausted? Clearly, they have the discipline and culture to be able to do this. But is there not a danger of people just getting very burned out and sick of the whole thing?

Gardner: One of the frustrations of this approach is that people want to see the project moving ahead. But if you’re engaged in this iterative process, you will see progress because you’re going to know that iteration number two, boy, when you look back on it, that was weak. And iteration number six is so much better. And iteration number seven is going to be better still.

That is a pretty inspiring process. And I can tell you as a writer that it’s exhausting. You think, “I don’t want to do another draft.” And yet when you finish it, you’ll look at it and go, “Yeah, that is a whole lot better.”

Q: One classic pitfall is that people fail to prioritize experience when it comes to staffing their projects, feeling that theirs is unique and that “no one’s ever done it before.” Can you talk about that?

Gardner: People talk about building the tallest, the largest, the widest whatever. When you start using those superlatives, there is a very good chance your process will minimize, if not entirely eliminate, experience.

Experience can apply to technology, too. If your project involves brand-new, unproven technology, you have minimized the importance of experience. You need to correct for that. Beware being the first mover on a technology. There may be an advantage. But it’s more likely that the lasting advantage will go to the organizations that are second or third in line to adopt the technology, provided they’ve learned from the first company’s experience. And it’s hard to remember that when everyone is excited about a new technology like generative AI.

Q: How do you personally avoid the classic pitfalls when you undertake a book? Didn’t you feel your book was unique?

Gardner: Oh, yeah, every book is your own special little child. [laughs] It's like every big project is its own special, unique blessing. As we’ve touched on before, that’s how people are naturally going to feel about any big project they tackle. What they're not going to do, because our psychology does not incline us toward this, is think of their project as being part of a category.

That is actually a very dangerous mistake. If I think of my project as being unique, that means I can't learn from other projects. So there are blind spots when I go to create a forecast for how long a project will take, what it’ll cost, and what challenges could crop up.

Whereas if you think of your project as being part of a category of hundreds if not thousands of similar projects that came before it, you can look at actual project outcomes. The timeline, budget, and unexpected events are all built into the data, and I can use all that information from that category of projects as a foundation for my forecast.

This is called “reference class forecasting.” And there is a ton of research to show that in fact it succeeds. It produces far more accurate forecasts than looking at your project as its own special little unique creature. But, of course, one of the biggest challenges to reference class forecasting is getting lots of good data.

Q: We’ve discussed the importance of up-front planning and all the reasons that projects can fail. But how often does it actually happen?

My co-author Bent Flyvbjerg has been collecting project data of all types and sizes for a decade or two now. His data is absolutely clear that the number of projects that come in on budget, on time, and on benefit is incredibly low, like 0.5%. And yet, everywhere you go, all you see is people claiming, “Your project was on budget, on time, and on benefits.” It's kind of like Las Vegas; we are all winners. That seems improbable.

A lot of people are massaging the data – they don't want the true results of their projects to be collected or used in reference class forecasting. But if businesses want their projects to have a better chance at success, they must start with better data collection.

Psychology tells us that being social was our killer app. Executives can use that. Gather information from internal projects and those being run from others in your field. Whether they fail spectacularly or succeed, you can use all of that data to guide your own project. If you aren't learning from experience, you won't get better. Period.