Cheaper tokens, bigger bill

AI is getting cheaper isn't it? Apparently not...

Recently I have been finding myself talking with clients and colleagues about making AI more efficient. We see new models making it cheaper and increased competition reducing costs but Gartner are saying the opposite is happening.

According to them, routing a single task to a reasoning agent costs at least 5x what the same task costs as a basic chatbot answer, and often a good deal more as the task gets more complex. (Inference cost is what you pay every time the model actually runs, as opposed to what it cost to build in the first place. It is the per-use bill, and with agents this determines if you can afford them!)

Why? Well, tokens are getting cheaper, and that is what's creating the problem. Cheap tokens make an ambitious workflow affordable, so teams are building the ambitious workflow, and the ambitious workflow burns far more tokens than the chatbot it replaced. A chatbot reads a customer query and returns an answer, once. An agent reasons, negotiates and questions itself, in loops, and every loop costs money. Gartner calls this the Inference Paradox, and on the back of it forecasts inference cost per agentic workflow rising more than 5x through 2028.

So what? First, this is a forecast, and forecasts in this area have not always been reliable. Second, Gartner sells advice of exactly this kind, so take that into account when you read it.

The action I'd take? If you are implementing agentic AI, think the financial implications through properly before you scale anything. Set expectations because if the outcomes take time to arrive, the cost will climb first and come down later, if it ever comes down at all.