How do you estimate the token cost of using a large language model?

Token consumption by scenario

The table deliberately omits unit prices: vendors revise rate cards every few months, so any figure written here would go stale and mislead. What it gives instead is the order-of-magnitude relationship and the cost drivers, which are comparatively stable — a plain Q&A call and a long-document extraction call can differ by two orders of magnitude in token count, and their cost levers are entirely different. Find your scenario, identify the dominant variable, then pull the current unit price from the official rate card and put it into the formula.

Getting to a budget number you can actually use

Step one: do not guess token counts, measure them. Take 30–50 real requests, run them through the target vendor's tokeniser, and record actual input and output tokens per request, keeping both the mean and the P90 — P90 matters more, because long-tail requests routinely contribute most of the bill. Step two: estimate volume from existing business data — how many cases are handled today, what share will route through AI, and how many model calls one case triggers on average. Step three: apply the formula — monthly cost = (mean input tokens × input price + mean output tokens × output price) × calls per day × 30. Step four: multiply by 1.3 to 1.5 to cover retries, evaluation runs and the waste of a trial period, all of which always appear on a real invoice. Step five: rerun it with P90 instead of the mean and treat that as the budget ceiling. After a month of real traffic, come back and correct the numbers; everything before that is an order-of-magnitude estimate.

Cost levers, ordered by return on effort

One: cut the input. In a RAG setup, dropping top-k from ten to three while adding a reranker usually saves tokens and improves accuracy at the same time — a rare win on both sides. Two: enable context caching. Put the long system prompt and fixed tool definitions at the very front of the prompt and keep them byte-identical; once cached, that portion costs far less, and note that a single changed character invalidates it. Three: route by difficulty. Send simple classification, intent detection and format conversion to a small or local model, and reserve the flagship for requests that genuinely need reasoning — usually the largest saving available and the one most often overlooked. Four: constrain output — require JSON, cap the length, forbid restating the question. Five: trim multi-turn history with a sliding window or periodic summarisation. Six: use batch endpoints for offline work, where many vendors discount non-realtime processing substantially. After the first three, most workloads land at an acceptable bill.

Where this applies

When this answer does not hold

• This answer deliberately quotes no unit prices. Vendors revise rate cards every few months, so a hard-coded figure would be stale and misleading. Use the current official rate card when you model costs.
• The Chinese-to-token ratio varies noticeably by tokeniser and no universal coefficient exists. Any "one character equals N tokens" claim holds only for a specific model, so measure with the target vendor's tokeniser.
• This assumes per-token billing on a hosted API. Self-hosted cost structure is entirely different — hardware depreciation, power and operations staff, with a marginal cost approaching electricity — and this formula does not apply.
• Caching, batch endpoints and discount schemes differ widely between vendors, and some carry constraints such as a minimum prefix length or a limited cache lifetime. Confirm against your actual call pattern before budgeting on a generic description.

Related questions

Related applications