Calculating the cost of AI

Krishna Grenville-Goble: Hi, I'm Krishna Grenville-Goble and this is The Insight in 15.

Today we're going to be exploring how to balance AI innovation and cost. To help us with that question I'm joined by Su Crighton, Partner in our Technology and Data team here at KPMG UK. As usual, we’ve only got 15 minutes, so we're going to jump straight in.

Su, great to have you with us.

Su Crighton: Thank you. It's really good to be here.

Krishna: In a previous episode we were talking about cybersecurity and tokens were mentioned. Can you explain what they are and how they make calculating the cost of AI different from other technology?

Su: Of course. So tokens are essentially the sort of the cost units that AI is priced in. So when you get a price list from your AI vendor, as long as you're not on a monthly subscription. So individual users might be on a monthly subscription where they just get sort of a certain limit.

But for most business purposes, you are priced per token and you'll get a price per million tokens and effectively a token, sort of, roughly speaking, a word or a special character or a little fragment of code. And when you put a prompt into AI, all the words or the code script or the input files that forms your input token.

So that's what you're putting into the model. And what comes out of the model is its response, which is your output tokens. And so input and output tokens get priced differently. And the output token price is generally priced based on the complexity of the model you're using, how much reasoning needs to be done and where you get to.

So your number of tokens, depending on the complexity of what you're asking for, can increase significantly. And that's how your pricing will get done. So why this differs to an awful lot of other technology costs is because it's very consumption driven.

And so you could actually easily, if you think about a developer or a researcher asking a lot of questions which is going off and pulling in a lot of data, or setting up agents that in themselves go off and call other pieces, you can actually end up with an almost exponential number of tokens being used, which leads to a very unpredictable AI cost.

And so it's why you have to track it very real time.

Krishna: So does this model make it more difficult to track costs?

Su: It can do. What's very hard is that when you're using the tokens, there's lots of different people in the organisation who might be using them. And it's an immediate cost hit.

If you're only looking at your financials on a traditional sort of monthly basis, you could be looking back and find that you've blown your budget really badly, and you weren't tracking that. You weren't managing it.

Krishna: So as AI scales across an enterprise, how do I maintain visibility of cost?

Su: So this is where, I mean, our recommendation is very much that right from the start when you're using AI, you need to build in some kind of token monitoring. And that might be fairly basic. You might have more complex dashboards, that sort of thing.

The criticality is that the people who are consuming understand how many tokens are being used, because ultimately, what you want to be able to do is connect your AI cost to the value it's delivering and make sure those are proportionate. And so understanding, at all times, how many tokens am I using? What's that costing me?

Then allows the people who are using it to sort of review it and say, actually, do I need to use that expensive a model? Are there ways in which I could maybe run my query later at night, batch it up so it's not real time? Do I want to reduce the amount of information that is being pulled into it? All of which can sort of moderate the total cost.

And then you've got that balance between cost and output, and making sure you've got the right quality of output for the right balance of cost.

Krishna: So there's a lot of factors to consider there. What sort of governance and guardrails do I have to have in place?

Su: So I think the guardrails and governance you need really depend on your organisation, what you're trying to do.

Our recommendation is generally that you, whether it's an AI centre of excellence or somewhere that you've got the right policies. So those policies might be about security. When it comes to cost, we would look at saying potentially you have model caps. So the really expensive high-end models, the frontier models that cost more, might not have a blanket cap.

But for example, you might need to go through a different approval process for using those. So you could have intelligent routing policies that just say, actually, for the type of work you're doing, this is the most sensible model because it's the most cost efficient. And then have the governance process that allows you to review that if you need to.

We see some organisations who put in just token caps. So for example, say developers, engineers, they have this much budget to spend. And once it's gone, it's gone. That can be quite a crude metric. And I think, as I was saying before, you've got to balance what you're getting as the outcome with what you're doing as the income.

So personally, I don't think having a blanket token cap is the right thing to do. But just constantly monitoring and measuring that, you might have a cap by role or by use case. But that's where I think having a central place where you've got some established guardrails, you're making sure that you've got clear visibility.

And so linked to that visibility is wherever someone is using AI, you've got to have that ownership. You've got to have that accountability so that you can link the spend and be able to come back and challenge.

Krishna: So AI touches every part of the business. So where does the responsibility for the controls lie?

Su: So this is something we've been debating at length. And I think the simple answer is it rather depends on the organisation.

So generally speaking, we would tend to be talking to tech and finance teams and data teams, depending on how the business is set up, about where the actual controls and guardrails and governance sit.

But it's something that everyone in the business, because everyone's a consumer and everyone's getting excited about it, and where the benefits live across different parts of the business. It is actually a board level discussion.

Krishna: Given what we've said, how do I ensure that the focus on cost doesn't slow down my innovation on AI?

Su: And this is, I think, the real challenge. And where I was saying earlier that personally, I don't think that just putting flat spending caps is the right thing to do, because there are some phenomenal use cases for AI where it will drive absolutely brilliant value for the business, and you need to be able to enable that. You need to be able to see it through.

It's why I very strongly believe that the right thing to do is get this clear visibility, so that all the people who are using AI have got the ability to see, right, what is the stuff I've developed using? What is the cost of this query? What is the cost of these agents? And make the decision as to whether it's worth it.

Now, that might then need to go through the governance process of spending review if it's a particularly high spend. And whatever your business case review process is, to be able to look at and say, look, I'm generating this much because of the way I'm innovating. It's worth it. Can I do it?

So I think this is where you've got to get the right controls in to say, actually, how do we make it sufficiently available but be able to watch?

Krishna: So at what point do I need to get the governance in place and sort of what happens if I'm already quite far down the journey?

Su: So I think the easy answer to that is ideally you get the governance in place before you start. And that doesn't have to be super hefty governance. You can do token tracking, monitoring, use spreadsheets and Power BI dashboards or that sort of thing to keep it quite simple.

But equally, a lot of organisations have already started. They're sort of halfway down the line. It's never too late. But I think the sooner you put in, at the very least, the basic visualisation and tracking, so you really understand your AI spend.

And increasingly, I mean, we talk to CFOs and CEOs and they are increasingly asking questions about, well, actually, how much is this costing? What do I get back for it? And that's what you've got to be able to answer.

So putting in the basic, can I monitor the costs, track it? Can I link it to value? Is, at the very basic, the governance you need. And then you've got wider, ideally model routing, different things that you can do to continually optimise that cost, but you can refine that as you go.

Krishna: When organisations are looking at cost, apart from tokens, what else should they be looking at?

Su: So with AI you've got two areas to think about. You've got the initial development and then you've got the ongoing running costs.

So with the initial development, depending on what you're doing, you've got the costs of the people who are developing the capability. You've got the costs of potentially training a model, setting it up. The token cost will usually cover things like what infrastructure you're running it on and various different choices you've made.

And then when you get into running it and something that's running ongoing, you've got to think about those ongoing costs.

So again, you've got the infrastructure it's running on. You've got the data that's being consumed that you've got to think about being pulled in.

And the ongoing token costs, potentially not just the immediately visible ones, but the ones that the agents are then running off, calling other things, bringing things in.

So you've got to look at that as a whole.

Krishna: So when organisations are looking at these costs, is there anything that they're missing?

Su: So I think one of the hardest things to get your head around is we've talked about tokens. We've talked about token being that unit cost.

The real difficult thing is that not all tokens are equal. And so if you look at the price lists that you'll often get from AI vendors, you've got lots and lots and lots of variables depending on the infrastructure you're using, the type of model you're using, whether you've got caching, whether you've got prompt optimisation, all these different things.

And so your price per million tokens from the same vendor can vary up to a factor of 100 times depending on what you're doing. So when I talk about the visualisation of the costing and the prices, it's not simply a number of tokens because 100 tokens used in one way could cost significantly higher than 100 tokens used in another.

Krishna: In previous episodes we've talked about second-tier benefits of AI. So time saved means you can deliver value elsewhere. So how easy is it really to put a figure on AI ROI?

Su: So I think really calculating AI ROI depends very much on your business. So for example, in a contact centre environment, if AI agents are helping the call centre agents to provide answers, or even avoiding the customer having to come to a call centre agent because they've been able to solve the problem on our web chat, then you're directly reducing the number of calls, you're improving customer satisfaction and potentially reducing the number of people you need to facilitate that.

But that's quite a specific use case. In other examples, for example, where engineers are using AI to develop code and to develop solutions, your real value might actually be time to market. So it's that you might get new features out a couple of months ahead of where you would previously have got them out. And so that's slightly less tangible in that maybe you're just pulling forward revenue.

So there's lots of different ways. It really depends what value looks like to your organisation. And when we're talking to our clients we're always recommending: look at your business outcomes, look at your business strategy and make sure that what you're doing with AI is directly attributable and really, really think about how it goes on.

If you've got productivity improvements, if you've got new product generation, if you're enabling different things that directly bring in money, that's where you've got to think about it and then look at the complete costs associated with AI. Not just the immediate costs you see, but look at the complete picture, the data, the infrastructure, everything, and then see if it's really driving value.

Krishna: So Su, we're nearly at the end of time. We are going to ask the question we ask of all of our guests. What is the one thing that clients should take away from this conversation?

Su: I think if you do nothing else, you've got to get visibility of your AI spend. You don't want to have surprises. Again, we've seen the headlines of the organisations that have had surprises and have had to sort of go, oh, we've got to pull back.

You can't manage what you can't see. And that needs to be made visible to the people who are consuming it. So whether that's developers who are using AI, whether it's data engineers, whether it's people in other parts of the business who are choosing to use AI to drive productivity, if they understand what that AI use is costing, they are then able to make decisions around whether it's sufficient to drive value, whether it's going in the right place and build the ROI case.

But if they don't understand what it costs, they can't do that. So yeah, the visibility of your AI spend is the absolute one thing I would say. If you do nothing else, please do that.

Krishna: Well that's our 15 minutes up. Su, thanks very much for joining us. You'll find this and all of our episodes on Apple Podcasts, Spotify and YouTube.

Join us next time for The Insight in 15.