A token is a sub-word unit of text -- a chunk that a language model reads as a single piece. Tokens are not words, not characters, and not bytes. They are the result of running text through a tokenizer, which splits text according to a vocabulary learned during model training. The tokenizer maps each token to a unique integer ID, and the model operates exclusively on those IDs.

The key insight is that token boundaries do not align with word boundaries. Common words like "the" or "cat" might be single tokens, while rare or complex words like "tokenization" might split into multiple tokens: "token" + "ization". This means the number of tokens in a text is not the same as the number of words -- it is usually less for common English, but can be dramatically more for non-Latin scripts, code, or unusual text.

The Rough Rule of Thumb

For English text processed by modern tokenizers (GPT-4's cl100k_base, Llama's tokenizer, etc.), the approximate conversion ratios are:

• 1 token ≈ 4 characters of English text
• 1 token ≈ 0.75 words (so 100 words ≈ 133 tokens)
• 1 word ≈ 1.3 tokens on average
• 1 token ≈ 1 syllable in English (very rough)
• 1 line of code ≈ 10-20 tokens depending on length
• 1 emoji ≈ 1-4 tokens (varies by tokenizer)

Tokens Are Not Characters

Consider the word "unbelievable". A character-based approach would see 11 characters. A word-based approach would see 1 word. A sub-word tokenizer might see: "un" + "believe" + "able" = 3 tokens. The advantage of sub-word tokenization is that it handles rare words by decomposing them into meaningful parts the model has seen before, while keeping common words as single tokens for efficiency.

Visualizing Token Splits

Here is how different texts tokenize with a typical BPE tokenizer:

The Vocabulary

Every model has a fixed vocabulary -- a table mapping token strings to integer IDs. The vocabulary size varies by model:

Tokenization is the process of converting raw text into a sequence of token IDs. It happens in three stages:

1. Normalization -- Text is cleaned (whitespace handling, Unicode normalization, lowercasing for some models).
2. Pre-tokenization -- Text is split into rough chunks (words, punctuation, whitespace) using regex rules.
3. Sub-word splitting -- Each pre-token is further split using the model's vocabulary and merge rules (BPE), or vocabulary lookup (WordPiece), until all pieces are in the vocabulary.

The four major sub-word tokenization algorithms used in modern LLMs are:

• BPE (Byte-Pair Encoding) -- GPT-4, Llama 3, Mistral, Claude
• WordPiece -- BERT, DistilBERT, Electra
• SentencePiece -- Llama 2, T5, ALBERT, Gemini (wraps BPE or Unigram)
• Tiktoken -- OpenAI's optimized BPE implementation (GPT-3.5, GPT-4, GPT-4o)

The Pre-tokenization Regex

Before sub-word splitting, text is divided into chunks using a regex pattern. GPT-4's cl100k_base uses this pattern (simplified):

Byte-Pair Encoding (BPE) is the dominant tokenization algorithm for modern LLMs. It was originally developed in 1994 as a data compression technique and adapted for neural language models by Sennrich et al. in 2015. GPT-2 popularized it, and today it powers GPT-4, Llama 3, Mistral, Claude, and most other major models.

How BPE Works

BPE training builds a vocabulary by iteratively merging the most frequent adjacent byte/character pairs:

1. Start with a vocabulary of all individual characters (or bytes).
2. Count all adjacent pairs in the training corpus.
3. Merge the most frequent pair into a new token, add it to the vocabulary.
4. Repeat steps 2-3 until the vocabulary reaches the target size.

At inference time, the tokenizer applies the same merge rules in order. Text is first split into characters/bytes, then merge rules are applied greedily in the order they were learned:

Byte-Level BPE

GPT-2 and later models use byte-level BPE, which operates on raw bytes (256 possible values) rather than Unicode characters. This means the base vocabulary is always 256 bytes, and every possible text input can be tokenized -- there are no "unknown" tokens. This solves the out-of-vocabulary problem that plagued earlier word-level and character-level tokenizers. The trade-off is that non-ASCII characters (which require multiple bytes in UTF-8) consume more tokens.

WordPiece

WordPiece is the tokenization algorithm used by BERT, DistilBERT, and Electra. It is similar to BPE but uses a different merge criterion. Instead of merging the most frequent pair, WordPiece merges the pair that maximizes the likelihood of the training corpus when added to the vocabulary. The result is a vocabulary that tends to favor sub-words that appear in more diverse contexts.

WordPiece uses a "##" prefix to denote continuation sub-words. For example, "playing" might tokenize as ["play", "##ing"], where "##ing" indicates that "ing" is a continuation of the previous piece, not a standalone word. This helps the model distinguish word-internal sub-words from standalone words.

SentencePiece

SentencePiece is not a single algorithm -- it is a tokenization library and framework developed by Google that can implement either BPE or Unigram language model tokenization. Its key innovation is that it treats the input as a raw stream of Unicode characters without assuming word boundaries (no whitespace pre-splitting). This makes it language-agnostic and particularly well-suited for languages that do not use spaces between words (Chinese, Japanese, Thai).

• Llama 2 uses SentencePiece with BPE, 32K vocab
• Llama 3 uses SentencePiece with BPE, 128K vocab (tiktoken-compatible)
• T5 uses SentencePiece with Unigram model, 32K vocab
• ALBERT uses SentencePiece with a 30K Unigram vocab
• Gemini uses SentencePiece, ~256K vocab

SentencePiece uses a special unicode character ░ (U+2581, "LIGHT SHADE") internally to represent spaces, so that whitespace is part of the tokenization rather than a delimiter. This means "Hello world" and "Helloworld" tokenize differently even though there are no word boundaries -- the space is encoded into the tokens themselves.

Unigram Language Model Tokenization

The Unigram approach (used by T5 via SentencePiece) works differently from BPE. Instead of building up by merging, it starts with a large vocabulary and prunes tokens that least reduce the likelihood of the corpus. The final vocabulary is the set of sub-words that best explains the training data. At inference time, multiple segmentations are possible and the tokenizer picks the one with the highest probability (using a Viterbi-like search). This makes Unigram tokenization probabilistic -- the same text always tokenizes the same way given the vocabulary, but the algorithm considers multiple possible segmentations.

Tiktoken is OpenAI's open-source, high-performance BPE tokenizer library, written in Rust with Python bindings. It is the tokenizer used for all OpenAI API models (GPT-3.5-turbo, GPT-4, GPT-4o, o1, etc.). Tiktoken is significantly faster than pure-Python BPE implementations -- typically 3-6x faster than HuggingFace's tokenizers library for the same vocabulary.

Tiktoken Encodings

Tiktoken uses different encoding files (called "encodings") for different model generations:

Tiktoken Is Just BPE

Despite the name, Tiktoken does not introduce a new tokenization algorithm -- it is a byte-level BPE implementation with a specific set of merge rules and regex pre-tokenization. The "tik" refers to its speed (tick-tock, fast). The real innovation is in the vocabulary design (which merges were learned) and the engineering (Rust core, regex-based pre-tokenization, cached merge lookups).

Most LLM APIs charge per token, not per request, per word, or per character. This means understanding token counts is essential for cost estimation. Pricing is typically split into input tokens (the prompt you send) and output tokens (the response the model generates). Output tokens are usually 3-5x more expensive than input tokens because generation requires autoregressive decoding (one forward pass per token) while input processing is a single forward pass.

Per-Token Pricing Model

API providers price per 1,000 or per 1,000,000 tokens. Here is a comparison of major API pricing (as of mid-2026, approximate):

Calculating Real Costs

To estimate the cost of an API call, you need to count both input and output tokens:

Hidden Token Costs

Several factors inflate token costs beyond the obvious prompt + response:

• System prompts -- Every call includes a system prompt (often 200-500 tokens). If you send it on every call, it adds up. Use prompt caching to reduce this.
• Conversation history -- In a chat application, you must resend the entire conversation on each turn. A 10-turn conversation might have 3,000-10,000 input tokens by the last turn.
• Few-shot examples -- Including 3-5 examples in your prompt adds 500-2,000 tokens. Cache these if possible.
• Output token limits -- You pay for all tokens the model generates, including any it generates before the visible response (reasoning tokens in o1/o3 models).
• Tool/function definitions -- JSON schemas for tools consume tokens. A complex function-calling setup might add 300-1,000 input tokens.
• Structural overhead -- JSON, XML, or markdown formatting in your prompt adds tokens. Minimize unnecessary formatting.

The context window is the maximum number of tokens a model can process in a single inference call. This includes all tokens: the system prompt, the user's input, any conversation history, tool definitions, few-shot examples, retrieved documents (in RAG), and the space reserved for the model's response. If the total exceeds the context window, something must be cut.

What Fits in a Context Window?

Practical estimates of what different context window sizes can hold (assuming ~1.3 tokens per English word, ~500 words per page):

What Does NOT Fit

Even with a 128K context window, you cannot fit:

• An entire enterprise codebase (millions of lines, tens of millions of tokens)
• A full legal database or case law collection
• More than ~2-3 average-length books
• Large CSV/JSON datasets (10K+ rows become millions of tokens)
• Years of conversation logs or chat histories
• Complete video/audio transcripts beyond a few hours

For these use cases, retrieval-augmented generation (RAG) is the standard approach: store the full corpus in a vector database, retrieve only the relevant chunks, and include just those in the context window. This is both more cost-effective and often higher quality than stuffing everything into context.

Context windows have grown dramatically. The original GPT-3 (2020) had a 2K context window. By 2024, Gemini 1.5 Pro reached 2M tokens. This 1,000x increase has transformed what LLMs can do -- from reading a single paragraph to reading an entire codebase. Here is a comparison of popular models and their context windows:

When the total token count (input + requested output) exceeds the model's context window, something must give. Different APIs and inference engines handle this differently:

Strategy 1: Hard Error (API Rejection)

Most commercial APIs (OpenAI, Anthropic, Google) return an HTTP 400 error if you exceed the context window. The request never reaches the model. You get a message like:

Strategy 2: Automatic Truncation

Some inference engines (llama.cpp, Ollama, certain API wrappers) silently truncate the input. They drop tokens from the beginning of the conversation (the oldest messages) until the input fits within the context window. This is common in chat applications built on local models. The user sees a response, but the model has "forgotten" the earliest parts of the conversation.

Strategy 3: Sliding Window

A sliding window approach keeps only the most recent N tokens of context. As new messages arrive, old messages fall off the end. This is how many chat UIs maintain a continuous conversation with a fixed context model. The model always sees a window of recent context, with older context discarded:

Strategy 4: Summarization / Compression

Instead of dropping old context, you summarize it. When the conversation approaches the context limit, the system takes the oldest messages, asks the model to summarize them, and replaces the originals with the summary. This preserves the key information while reducing token count:

Strategy 5: RAG (Retrieval-Augmented Generation)

For large knowledge bases, RAG is the standard solution. Instead of putting all documents in context, you store them in a vector database, retrieve only the most relevant chunks for the current query, and include just those in the prompt. This scales to unlimited knowledge while keeping context window usage bounded:

The KV cache (Key-Value cache) is the primary memory mechanism that makes autoregressive LLM inference feasible. Without it, generating a 1,000-token response would require re-processing all preceding tokens for every single new token -- a quadratic operation. The KV cache stores the intermediate attention computations (keys and values) for all processed tokens so they do not need to be recomputed.

How the KV Cache Works

In a transformer's self-attention layer, each token produces a query, key, and value vector. During generation, the model needs the keys and values of all previous tokens to compute attention for the current token. The KV cache stores these keys and values so only the current token's Q, K, V need to be computed:

• Without KV cache: Generating token N requires re-computing attention for all N-1 previous tokens. Cost per token: O(N). Total cost for N tokens: O(N²).
• With KV cache: Generating token N requires computing Q, K, V for only token N, then attending against the cached K, V of all previous tokens. Cost per token: O(1) compute + O(N) memory. Total cost for N tokens: O(N) compute, O(N) memory.

KV Cache Memory Calculation

The KV cache size grows linearly with context length. The formula is:

Techniques to Reduce KV Cache Memory

A larger context window does not guarantee better retrieval or understanding. The "lost in the middle" phenomenon, documented by Liu et al. (2023), shows that models perform best at retrieving information from the beginning and end of the context, with significantly degraded performance for information placed in the middle.

The U-Shaped Performance Curve

In the original study, researchers placed a relevant fact at different positions within a long context and measured the model's ability to retrieve it. The results showed a U-shaped curve:

• Position 0-10% (beginning): ~75-80% retrieval accuracy
• Position 45-55% (middle): ~50-55% retrieval accuracy
• Position 90-100% (end): ~75-80% retrieval accuracy

This means that even with a 128K context window, placing a critical document in the middle of the context can result in a 25-30% drop in retrieval accuracy compared to placing it at the beginning or end.

Why It Happens

Several factors contribute to lost-in-the-middle:

• Attention dilution -- With more tokens, attention scores are distributed across more candidates. The softmax over thousands of tokens makes it harder for any single middle token to stand out.
• Positional bias in training -- Training data often has important information at the beginning (instructions) or end (questions, conclusions). Models learn this distributional bias.
• Recency and primacy effects -- Similar to human memory, models exhibit stronger attention to the first and last items in a sequence.
• Attention sink -- The first few tokens (especially the BOS token) act as "attention sinks" that absorb disproportionate attention, creating a beginning bias.

Mitigating Lost in the Middle

1. Put critical information first or last -- Place the most important context at the beginning (after system prompt) or at the end (right before the user's question).
2. Use RAG instead of stuffing -- Retrieve only relevant chunks rather than dumping 100K tokens. Fewer, more relevant tokens mean better attention.
3. Reorder retrieved documents -- If using RAG, place the most relevant documents at the beginning and end of the context, less relevant ones in the middle.
4. Repeat key instructions -- If a critical instruction must be followed, state it at the beginning and end of the context.
5. Use structured prompts -- XML/JSON tags, headers, and delimiters help the model locate information regardless of position.
6. Chunk and map-reduce -- Process long documents in chunks and combine results, rather than sending the entire document at once.

1. Know Your Token Budget

Before sending any request, calculate: system_prompt_tokens + user_input_tokens + conversation_history_tokens + max_output_tokens. This must be ≤ the model's context window. Use a token counter (see the CLI and Python sections below) to measure each component.

2. Use Prompt Caching

Both OpenAI and Anthropic now offer prompt caching. If you send the same prompt prefix (system prompt, documents, examples) across multiple calls, the provider caches the KV cache for that prefix. Subsequent calls with the same prefix are charged at 50-90% less and are significantly faster. This is essential for applications with large static contexts.

3. Implement Conversation Memory Management

For chat applications, implement a memory manager that:

• Always preserves the system prompt (never truncate it)
• Keeps the most recent N turns verbatim
• Summarizes older turns into a compact summary block
• Drops the oldest summarized content if still over budget
• Uses function calling to offload facts to a database (long-term memory)

4. Use RAG for Knowledge Bases

For any knowledge base larger than the context window, use RAG. The standard pipeline:

1. Chunk documents into 500-1000 token pieces with overlap
2. Embed each chunk using an embedding model (text-embedding-3-small, etc.)
3. Store embeddings in a vector database (Pinecone, Weaviate, Qdrant, Chroma)
4. Retrieve top-K chunks for each query (K = 3-10 typically)
5. Construct a prompt with only the retrieved chunks
6. Generate the answer grounded in the retrieved context

5. Choose the Right Context Length

Do not always use the maximum context. Longer context costs more (both compute and money), is slower, and may suffer quality degradation. Match context to the task:

6. Minimize Token Waste

• Avoid redundant instructions -- Do not repeat the same guidance in the system prompt and the user message.
• Use concise formatting -- Markdown adds fewer tokens than verbose JSON. Avoid unnecessary whitespace.
• Trim documents -- Remove boilerplate, headers, footers, and navigation text from documents before including them.
• Pre-process code -- Remove comments and blank lines if the task does not need them. Minify where appropriate.
• Use references -- Instead of including full documents, include section references and retrieve on demand.
• Batch similar queries -- If you have multiple questions about the same document, ask them in one call rather than re-sending the document each time.

Tiktoken CLI

The most common tool for counting OpenAI-model tokens is tiktoken. Install it and use it from the command line:

HuggingFace Transformers Tokenizer

For non-OpenAI models (Llama, Mistral, BERT, etc.), use the HuggingFace transformers library:

Ollama Token Counting

If you run models locally with Ollama, you can count tokens using the Ollama API:

Quick Shell One-Liners

Basic Tiktoken Usage (OpenAI Models)

Counting Tokens for API Calls

HuggingFace Tokenizer (Non-OpenAI Models)

Context Budget Manager

Comparing Tokenizers Across Models