AI tokens and context windows explained: what they mean for cost and quality

If you look at the pricing page of any AI provider, you’ll see prices “per million tokens” and models described by their “context window”. These two ideas decide how much an AI feature costs, how fast it answers, and how much it can take into account. Here is what they mean in plain words.

What is a token?#

AI models don’t read letters or whole words. They read tokens: small pieces of text. A common word is often one token; a longer or rarer word is split into several; spaces and punctuation count too.

As a rough guide for English, one token is about three-quarters of a word, so 1,000 tokens is roughly 750 words. Other languages, code and unusual text often use more tokens for the same length. The exact split depends on the model, and providers offer tools to count tokens precisely.

How tokens decide the cost#

AI APIs charge for two things, priced separately:

• Input tokens: everything you send. That’s the visitor’s question, plus your instructions, your FAQs, the chat history and any documents.
• Output tokens: everything the model writes back. These usually cost several times more per token than input.

The important part for a business is that input is usually the bigger number. A visitor asks a 20-word question, but your feature sends that plus 2,000 words of instructions and FAQs, every single time. A chat also resends the whole conversation with each new message, so long chats cost more per message as they go.

What is a context window?#

The context window is the maximum number of tokens the model can consider at once: input and output together. Think of it as the model’s desk. Everything it needs for this answer has to fit on the desk: instructions, documents, conversation and its reply.

Modern models have large context windows, often enough for hundreds of pages. But a bigger desk isn’t a reason to fill it:

• Cost grows with every token you send.
• Speed: more input takes longer to process.
• Quality: a model given only the relevant information usually answers better than one given everything and left to find the needle.

The model also has no memory between requests. Anything it should know, including earlier messages in a chat, must be sent again each time, inside the context window.

How to keep AI features affordable#

1. Send what’s relevant, not everything. For larger content, use RAG to include only the parts that match the question.
2. Keep instructions tight. A clear, focused system prompt is cheaper and often works better than a long one; see how to write prompts that get better AI answers.
3. Limit the output. Ask for short answers when short answers are what you need, and set a maximum output length in the API call.
4. Trim long chats. Keep the last few messages, or replace older ones with a short summary.
5. Use prompt caching. Many providers, including Anthropic, can cache a long, unchanging start of a prompt (your instructions and FAQs), so repeated requests cost less and respond faster.
6. Pick the right model size. Smaller, faster models are much cheaper and handle many tasks, such as sorting messages or short answers, perfectly well. Save the largest model for tasks that need it.
7. Set limits. Cap the number of messages per visitor and set a monthly spending limit in the provider’s dashboard.

A quick example#

A support assistant has 1,500 words of instructions and FAQs (about 2,000 tokens). Each visitor question adds about 50 tokens, and each answer is about 150 tokens.

Per question, that’s roughly 2,050 input tokens and 150 output tokens. Multiply by the number of questions per month and the provider’s current prices, and you have a realistic budget. Notice that the instructions are almost all of the input: that’s why trimming them, or caching them, makes the biggest difference.

In short#

• Tokens are pieces of text; you pay for what you send and what comes back.
• The context window is how much the model can consider at once, and it has no memory beyond it.
• The cheapest and most accurate setup sends the model only what it needs for each answer.

If you want an AI feature on your site that’s useful and stays within budget, I do AI integration.