The conversation about AI in enterprises has shifted from experimentation and innovation to cost and sustainability. In recent weeks, many organizations, including Microsoft, Uber, Amazon and Starbucks, have had to recalibrate their use of AI because of its high and unpredictable costs. Enterprises are discovering that what begins as an inexpensive pilot can quickly become a significant operating expense when AI is used across thousands of employees and workflows—particularly with agentic systems that reason, plan, invoke tools, spawn sub-agents and iteratively refine their own outputs.
And it is not just costs that scale with AI usage, but also energy consumption, carbon emissions, water usage and hardware requirements. Therefore, what saves AI costs for enterprises also reduces AI’s environmental impact. Call it enlightened self-interest!
We explored what causes these budget busts and what can be done about them at a recent APAC AI and Sustainability session with Navveen Balani, Executive Director, Green Software Foundation (GSF). He argued that this is not a temporary challenge but a structural shift: AI is priced less like software that is purchased once and more like a utility whose meter keeps running. And the meter is running hot in many organizations.
What causes budget busts?
There are broadly two reasons for runaway costs: using a more powerful model than a workload requires, and using that model inefficiently. In many cases, there may be no need for generative AI at all, such as tasks requiring simple data classification. Even where GenAI capabilities are needed, organizations often use more expensive models than necessary. It is a bit like sending a Ferrari to deliver every pizza order. Sure, you can do it, but the delivery will cost more than the pizza, and it all adds up. Pairing each workload with the right model is therefore far more important than using the most powerful model for every task. Horses for courses, if you like.
Second, the number of tokens (the basic unit of compute and pricing) consumed also depends on how efficiently models are used. An AI agent performs many tasks—ingesting goals and breaking them into smaller steps, reasoning through those steps, selecting tools and models, taking action, reflecting on its actions and retrying to achieve the desired result. Each of these steps consumes tokens, and they can multiply rapidly if not designed well. Vague goals, verbose prompts, unbounded retries and unnecessary context sharing can each increase token usage, and together they compound into runaway costs. A common reason these issues go unchecked is that organizations prioritize speed of launch, assuming they can optimize later—which, of course, rarely happens.
Enter Lean Agentic AI
The solution, according to Navveen, is designing agents that are efficient by design—an approach he calls Lean Agentic AI (and the subject of his recent book). Essentially, it involves:
- measuring the cost and carbon emissions associated with each agent task—you cannot manage what you do not measure;
- right-sizing everything: using the smallest model that clears the bar, the leanest prompt, and only the context that is actually needed;
- putting a cap on retry loops; and
- caching frequently repeated parts of prompts that do not need to be processed every time. Research has shown that roughly three in four agent-and-tool requests share more than half of their prompt with prior requests, and nearly half of conversational requests do the same. Cached tokens typically cost only a fraction of the normal input price on subsequent calls.
A measure of cost and impact
The Green Software Foundation has developed the Software Carbon Intensity standard, now adopted by ISO, which turns the impact of an agent task into a single comparable metric using telemetry already collected for billing. It brings together data on energy use, carbon emissions, costs and token consumption into a single pane for each agent task. GSF is also working on water intensity standards, which will provide a more complete picture of AI’s environmental impact.
While these standards do not, by themselves, reduce costs or emissions, they provide visibility into where an organization stands and help set measurable improvement targets. However, none of this will happen unless someone is accountable for AI efficiency within the organization. Time for an AI Efficiency Czar?
Using AI is increasingly no longer just about writing prompts. It is becoming an engineering discipline that balances output quality with cost, sustainability and latency. GSF is about two months away from launching an academy to train professionals in Green AI. It already offers Green Software courses and learning material that have been used by hundreds of thousands of developers.
Cost as part of the AI sovereignty conversation?
AI sovereignty is a major topic today, with enterprises concerned about their lack of control over model weights and the risk of corporate IP leaking into public LLMs. In a widely shared interview with CNBC, Alex Carp, CEO of Palantir, argued that frontier LLM providers are overcharging enterprises for tokens while simultaneously absorbing valuable corporate knowledge and competitive advantage. What, he asks, will stop these model providers from eventually competing with their enterprise customers?
While Palantir’s position is understandable, given it offers an on-premise AI solution with Nvidia, it does reflect a broader concern among enterprises and governments about not only strategic control but also the long-term economics of relying on hosted frontier models. Expect cost and AI sovereignty to be discussed increasingly in the same breath.
