Published on the 13/09/2024 | Written by Heather Wright

Up to 1000% cost blowout…

AI costs are as big a risk now as security or hallucinations with 500 to 1000 percent errors in cost estimates for generative AI possible, Australian and New Zealand organisations have been warned.

Attendees at the Gartner IT Symposium/Xpo 2024 on the Gold Coast this week heard repeated warning that cost is one of the greatest near-term threats to AI success.

“You want AI adoption? You’re going to pay for it.”

Mary Mesaglio, Gartner distinguished VP analyst, says AI investments are not like conventional IT investments, where a capex investment is made and then the ongoing costs are relatively known.

“With GenAI it really easy to waste money,” she says.

Volatility in operating costs based on usage means some companies can quickly become victims of their success.

“You want AI adoption? You’re going to pay for it. You wanted sophisticated queries? Every time you make a more sophisticated prompt you are consuming more tokens.”

On average in 2023, organisations already deploying GenAI spent between US$300,000 and $2.9 million, just in the proof-of-concept side.

And costs are already going up.

“Vendors are raising prices by up to 30 percent as they incorporate GenAI.”

She notes similarities to the early days of cloud, when those first cloud computing bills often prompted shock – and confusion around what was being charged for. The same is likely to happen with AI bills, she warns.

“No matter what pace you’re running at you need to understand the bill. That means understanding the cost components and pricing model options and how to reduce them.”

Kristian Steenstrup, Gartner distinguished VP analyst warned that if you don’t understand how your GenAI costs scale you can make a 500 to 1000 percent error in your cost estimates.

“This cost derailment can come from all sorts of places: From interference cost, data prep and grounding cost. And it can be a big number,” Steenstrup says.

At a recent CFO conference in Australia there was plenty of talk about concerns about AI cost overruns. While those spoken to by iStart at the CIO Symposium weren’t about to admit to such large cost overruns, many said they weren’t surprised by the numbers cited. One CIO told iStart a recent modification to their GenAI model had cost AU$50,000.

Gartner figures show 92 percent of Australian and New Zealand CIOs report that managing costs limits their ability to get value from AI.

Mesaglio says one way of avoiding some of the costs used by some clients is using an API with their own web front end, instead of buying a packaged AI product.
“When you do a proof of concept, don’t just test whether the technology works and if employees like it, use the PoC to also understand how your costs will scale. When you do this your proof of concept becomes a proof of value.”

But she also cautions against not paying enough attention to AI use, and the mis-use of GenAI, saying too many are using GenAI foundation models to answer simple questions they should be using search for.

“You should not be using a GenAI foundation model for questions like ‘what’s the capital of Costa Rica?’ You just cut down 10 times more trees by asking a foundational model that, than asking Google Search.”

Despite all the cost risk, CEOs apparently think the hype is justified. While just 21 percent believed AI would significantly impact their industry back in 2022, that figure soared to 59 percent in 2023 and 74 percent this year.

But if CEOs are enthusiastic, CIOs are wary: 47 percent of CIOs say their enterprise AI/genAI initiatives haven’t met ROI expectations.

Steenstrup urged organisations to identify what the key benefits they’re seeking from AI are, and ‘set their pace’ – AI steady, or AI accelerated – around that.

For 43 percent of A/NZ organisations, the key benefits sought are around employee productivity, which can be managed through an AI-steady approach – an approach for those whose industries aren’t being disrupted, who have modest AI ambition and 10 or fewer AI initiatives.

But for the 37 percent of A/NZ organisations focused on process improvement or the 20 percent focused in business model innovation, Steenstrup says it’s a more expensive AI-accelerated approach that will need to be taken.

On the productivity front, Mesaglio warned of the dangers of ‘productivity leakage’, noting research showing that A/NZ companies which have successfully integrated AI into their daily work say they are getting back a little over three hours a week in saved time.

She questioned whether that would be used for other work, or to enjoy a coffee.

Gartner expects companies to experience 10-30 percent leakage, depending on the use cases.

“The bottom line is that time saved doesn’t necessarily mean increased business benefit,” she says.

Productivity gains also won’t be equally distributed, Steenstrup says, varying by employee depending on complexity of the job and employee experience. In low-complexity roles such as call centres, less experienced workers can gain big productivity gains with AI providing access to information. The more experience call centre worker already knows the information – in fact they probably helped train the AI model.

For roles like lawyers, it is the more experienced workers, who can tell good AI output from bad, who see the biggest gains. A less experienced lawyer, on the other hand, won’t have the experience to know whether AI output is good or bad.

“What makes achieving AI productivity easier is matching the job complexity and job experience. The rule of thumb is you get more AI productivity when you match low complexity and low experience, or when you match high complexity with high experience,” he says.

“Get this right and you unlock AI productivity across the enterprise.”