LLM token pricing is hard to predict because your bill depends on three things that change with every request: how many tokens you send in, how many the model writes back, and which model does the work. Add conversation history, retries and automated agents, and the same feature can cost very different amounts from one week to the next.
This post explains how per-token pricing works, why AI costs creep up, what a realistic conversation costs, and which levers help you reduce LLM costs.
How does LLM token pricing work?
LLM token pricing charges you for the tokens a model processes, usually quoted as a price per million tokens, with separate rates for input and output. Input tokens are everything you send, including instructions, the user’s question, any documents, and the earlier conversation. Output tokens are what the model writes back. The two are priced separately, and output is often the more expensive of the pair.
The cost per token is tiny, which is exactly why the totals surprise people. A fraction of a cent per request becomes a large number across thousands of users and automated calls. Rates differ by provider and change over time, so check each provider’s current pricing page before you build a budget around any figure.
Three things drive what you pay:
- Tokens in. Longer prompts and bigger documents cost more. The context window, the amount of text a model can read at once, sets the ceiling, and everything inside it is billed as input on each call. This is context window cost.
- Tokens out. Longer answers cost more, and they are often priced higher per token than the input.
- Model tier. Larger, more capable models cost more per token than smaller ones. Different providers also count tokens differently, so the same text can produce different totals on different platforms.
A taxi meter can be a useful analogy. The fare depends on distance, waiting time and car class at once. Tokens are the distance. Re-sent history and retries are the waiting time; you pay while going nowhere. The model tier is the car class. Two rides across the same town can cost very different amounts, and the meter gives you no warning in advance.
Additional Read: What is AI tokenomics (and why it isn’t about crypto)
Why do AI costs creep up over time?
AI costs creep up because each request tends to carry more tokens than the one before it, and software can make more requests than people expect. Three habits of real applications cause most of it:
- History grows the context. A model does not remember earlier calls. Most applications resend the earlier conversation with every new message, so the input grows each turn even when the new message is short.
- Retries repeat work. A retry sends the same input again, and the new attempt is billed. An application that retries automatically can raise spend without anyone noticing.
- Tool calls and agents add tokens. Each tool call adds the request, the result, and often another model turn. Agents repeat this in loops, with no person approving each step.
None of these appears as its own line on an invoice. They appear as a larger total, which is why the bill feels unpredictable even when usage looks steady.
What does a 10-message conversation really cost?
A 10-message conversation can cost about twice as much when the application resends its history, even though the answers stay the same length. Here is a hypothetical example. The numbers are invented to keep the arithmetic simple and are not any provider’s real rates.
Assume a support assistant with a 400-token system prompt. Each turn adds a 100-token user message and a 300-token answer. The hypothetical rates are $1 per million input tokens and $4 per million output tokens, so a single input token costs a millionth of a dollar.
| Each turn stands alone | History resent each turn | |
|---|---|---|
| Input tokens, 10 turns | 5,000 | 23,000 |
| Output tokens, 10 turns | 3,000 | 3,000 |
| Cost per conversation (hypothetical) | $0.017 | $0.035 |
| Cost at 10,000 conversations a month | $170 | $350 |
Input tokens grew 4.6 times, from 5,000 to 23,000, while output stayed the same, and the cost roughly doubled. Nothing about the user changed. Only the way the application builds each request did.
How can you reduce LLM costs?
You can reduce LLM costs by sending fewer tokens, asking for shorter answers, reusing repeated prompts and using the smallest model that does the job well. Five levers cover most cases:
- Shorten the prompt. Trim the system prompt, remove duplicated instructions, and summarize older history instead of resending it in full.
- Cap the output. Ask for concise answers and set a maximum length where the use case allows it.
- Use prompt caching where it is offered. Some providers charge less for a repeated prompt prefix, such as a long system prompt that is identical across calls. Terms differ, so check how each provider prices cached tokens.
- Match model size to the task. Use a smaller, cheaper model for classification, extraction and routing, and keep the larger model for hard reasoning.
- Limit retries and loops. Set clear caps on automatic retries and on how many steps an agent may take.
These levers work, but they raise a practical question: which one is worth pulling in your organization? The answer depends on where the tokens are actually going.
Additional Read: Understanding token licensing: Pros, cons, and use cases
Why can’t you optimize AI spend without attribution?
You cannot optimize AI spend well without attribution, because a provider invoice shows totals, not causes. It may break spend down by model or by API key, but it rarely says which user, project or team drove the growth.
Without that, every lever above is a guess. A proxy that sits between your applications and the model providers can record tokens by user, project and model as requests pass through, for example OpenLM’s AI Proxy Monitoring. Otherwise, you would not know whether to trim the support assistant’s prompt, move the research tool to a smaller model, or cap an agent that loops. With attribution by user, project and model, the same numbers become decisions: you can see which lever is worth pulling and check afterwards whether it worked.
You can only optimize what you can attribute. Turning token counts into a cost per user, project and department is the subject of AI unit economics.
Frequently asked questions
How is LLM token pricing calculated?
Providers multiply the number of input tokens by an input rate and the number of output tokens by an output rate, then add the two. Rates are usually quoted per million tokens and vary by model.
What is the difference between input and output tokens?
Input tokens are what you send to the model, including instructions, documents and earlier conversation. Output tokens are what the model writes back. They are typically priced separately, and output is often priced higher.
Why is my AI bill higher than expected?
The usual causes are growing conversation history, automatic retries, tool calls and agent loops, all of which add tokens without a person noticing. A larger model than the task needs is the other common cause.
What is prompt caching?
Prompt caching lets a provider reuse a repeated part of a prompt, such as a long system prompt, instead of processing it from scratch each time. Some providers charge less for cached tokens, but terms differ, so check each provider’s pricing page.
How can I reduce LLM costs without hurting quality?
Start with the changes that do not affect answers: shorter prompts, summarized history, caps on retries and caching for repeated prefixes. Then test whether a smaller model handles simpler tasks as well as a larger one.
