GPT-4o output tokens cost $10 per million. Claude Sonnet charges $15 per million output tokens. These numbers appear on every pricing page, and they are almost completely useless for budgeting.
The FinOps Foundation published its GenAI token pricing analysis in early 2026, and the core finding was blunt: “the advertised per-token price for GenAI is misleading; the real costs are in the details.” Two independent analyses, from IndexNine and Zenskar, arrived at the same figure: hidden costs from retry logic, retrieval augmentation, context window management, and embedding generation inflate actual AI spending by 40 to 60 percent beyond what most teams track.
For FinOps practitioners who have spent years making cloud spend predictable, token pricing introduces a fundamentally different cost surface. An EC2 instance running for 720 hours produces the same bill every month. An AI agent researching a customer question, drafting a response, validating it against company policy, and revising based on feedback can burn through 50,000 to 500,000 tokens before returning a single answer. That range, a full order of magnitude, makes token cost forecasting look more like weather prediction than financial planning.
Context Windows: The Compounding Cost Nobody Budgets For
The single most expensive hidden cost in production AI systems is context window creep.
Large language models are stateless. Every API call must include the full conversation history, system instructions, and any retrieved context the model needs. The FinOps Foundation’s working group documented the math: in a three-turn customer service interaction, Turn 3 contains only 6 new tokens, but the system transmits all 61 accumulated tokens. The model processes everything, and you pay for everything.
This compounds fast. System prompts alone add 500 to 3,000 tokens to every request. Retrieval-augmented generation (RAG) pipelines inject retrieved documents that can run 2,000 to 8,000 tokens per call. Multimedia inputs compound further because images sent in an initial turn incur vision processing fees again on every subsequent turn.
The result: input tokens dominate enterprise AI spend despite output tokens costing 3 to 5 times more per unit, simply because input volume compounds with every interaction.
Output Tokens Are the Silent Budget Killer
While input tokens accumulate through context, output tokens carry the higher unit price. Across major providers, output tokens cost 4 to 8 times more than input tokens for the same model.
In output-heavy workflows (content generation, code writing, report drafting), output can account for 80 to 90 percent of the total bill. Most internal dashboards track total tokens without distinguishing input from output, which means the most expensive component of every call goes unexamined.
Model selection matters here more than anywhere else. A reasoning model that “thinks” before answering may produce better output, but the thinking tokens count toward your bill. Frontier reasoning models can generate thousands of internal reasoning tokens that never appear in the user-visible response. The FinOps Foundation notes that hyperscaler platforms show pricing variance of approximately 30 percent for identical open-source models, influenced by inference speed, quantization, context window length, and thinking-step allowances.
The False Economy of Cheap Models
The instinct to select the lowest-cost model for every task creates its own cost spiral. The FinOps Foundation’s analysis is direct: selecting the cheapest model creates hidden expenses through longer prompts, retry attempts, and verbose outputs requiring refinement.
A smaller model that needs three attempts to produce acceptable output costs more than a capable model that gets it right the first time. A model that generates 2,000 tokens of rambling output when 400 would suffice costs five times more in output tokens, plus the downstream engineering time to parse the result.
This is where intelligent model routing becomes a FinOps discipline, not just an engineering optimization. Route simple classification tasks to a small model. Route complex reasoning to a capable one. Route latency-insensitive batch work through asynchronous endpoints that offer 50 percent or greater discounts. The routing decision is a cost decision, and FinOps teams should own the policy.
Three Consumption Models, Three Cost Profiles
Deloitte’s 2026 analysis of enterprise AI spending identifies three distinct consumption models with different economics:
API consumption offers transparent, metered costs but introduces forecasting volatility. Token usage varies by prompt design, model behavior, and user interaction patterns. At moderate scale, API costs are competitive, but the unpredictability makes budget variance a recurring problem for finance teams accustomed to fixed infrastructure costs.
Packaged AI software (Microsoft Copilot, Salesforce Einstein, GitHub Copilot) bundles token costs into per-seat licensing. Predictable for budgeting, but token visibility disappears entirely. You pay the same whether your team uses 100 tokens or 100,000 tokens per day. Without visibility, you cannot optimize.
Owned infrastructure (on-premise GPU clusters, dedicated inference endpoints) internalizes token economics entirely. Deloitte’s three-year simulations show more than 50 percent cost savings versus API pricing once token production reaches critical mass, though approximately 50 percent of the total cost comes from non-GPU factors: networking, power and cooling, facilities, and the software stack.
The FinOps decision is not which model is cheapest per token. It is which consumption model matches your organization’s scale, predictability requirements, and optimization maturity.
Developer AI Tools: The Emerging Blind Spot
Enterprise AI cost discussions focus on production inference, but developer tooling is becoming a significant, largely untracked cost center. Vantage’s 2026 analysis highlights the problem: a developer using Cursor or Claude Code for the same workload can generate a token bill that varies by an order of magnitude depending on model choice, context window depth, and session length.
The attribution gap compounds the visibility problem. Anthropic and Cursor offer developer-level spend attribution. OpenAI requires additional instrumentation. AWS Bedrock provides the least granularity because marketplace proxying obscures per-user consumption.
Most companies remain in the measurement phase, with CTOs building infrastructure to connect token costs to engineering output. Vantage recommends tracking tokens per feature shipped, cost per PR, and cost per release as the early metrics that connect AI spend to engineering value.
From running fractional COO engagements across IT organizations, I have seen this pattern before. It mirrors the early days of cloud computing, when teams provisioned instances without cost awareness and the bill arrived 30 days later as a surprise. The companies that built cost attribution into developer workflows early gained a structural advantage they never lost.
Five Practices for Token Cost Management
Token economics requires a different FinOps muscle than cloud cost management. VMs and storage follow deterministic pricing; tokens follow probabilistic distributions. The same prompt can produce variable output token counts across different models, making per-transaction costs a range, not a number.
Set token budgets by team and use case. Not spending limits that kill productivity, but guardrails generous enough to prevent five-figure overnight bills while building baseline usage data. Adjust limits over time as you gather real consumption patterns.
Separate input and output token tracking. Aggregate token metrics hide the real cost drivers. Output tokens at 4 to 8 times the input price deserve their own line in the dashboard. This single change reveals optimization opportunities that aggregate tracking obscures.
Implement model routing policies. Define which models are authorized for which task types. A classification task does not need a frontier reasoning model. A customer-facing response might. The FinOps tools landscape is evolving fast here, with Amnic, Vantage, and Finout leading in native LLM spend ingestion.
Use prompt caching and batch processing. Explicit prompt caching reduces costs for repetitive workloads. Batch processing offers 50 percent or greater discounts for non-urgent tasks delivered within 24 hours. Both require deliberate FinOps coordination between engineering and finance.
Build the 1.7x multiplier into every AI budget. Zenskar’s analysis recommends a realistic total budget of approximately 1.7 times your base token calculation: 25 percent for usage growth, 30 percent for infrastructure overhead, and 15 percent for experimentation. This is not padding; it is the actual cost of running AI in production.
The advertised cost of an AI token is a starting point, not a forecast. FinOps teams that treat it as a forecast will spend the next year explaining budget overruns to the CFO.