Tokens and Context

 

1. Tokens and Context Windows

Now we reach one of the most important concepts in LLM application architecture:

Context

An LLM can only process a finite amount of context in a single request.

Think about an AI assistant answering a question about a large enterprise knowledge base.

You might have:

User question
        +
Conversation history
        +
System instructions
        +
Retrieved documents
        +
Tool results
        +
Other context

All of this consumes tokens.

So you can't simply say:

"Send the entire company knowledge base to the LLM."

Instead, you need to decide:

What information is actually relevant to this request?

That is an architecture problem.

---

2. This Becomes Extremely Important in RAG

Consider a Retrieval-Augmented Generation system.

A user asks:

"What is our policy for refreshing ISV authentication tokens?"

Your knowledge base may contain thousands of documents.

You don't want to send all of them to the LLM.

Instead:

                User Question
                     │
                     ↓
                 Retrieval
                     │
                     ↓
             Relevant Documents
                     │
                     ↓
              Selected Chunks
                     │
                     ↓
                   LLM
                     │
                     ↓
                  Answer

The retrieval system needs to select the most useful information.

And suddenly, tokenization becomes an architectural consideration.

You have to ask:

How much retrieved information should we send to the model?

---

3. The RAG Trade-off

Imagine your retrieval system finds 20 potentially relevant chunks.

You could send all 20.

But that may mean:

20 chunks
   ↓
15,000 tokens
   ↓
Higher cost
Higher processing
More context
Potentially more irrelevant information

Or you could select the best 5:

5 chunks
   ↓
4,000 tokens
   ↓
Lower cost
Less context
Potentially faster
Potentially more focused

But there's a trade-off.

If you retrieve too little:

You may omit information needed to answer the question.

If you retrieve too much:

You may increase cost, latency, and noise.

Therefore:

Retrieval quality isn't just about finding relevant documents. It's also about selecting the right amount of information to place into the model's context.

---

4. Tokens Are Also Important for Prompt Design

Consider these two prompts.

Prompt A

Answer the user's question using the provided documents.

Prompt B

You are an enterprise security assistant.

Your task is to answer the user's question using only
the information contained in the retrieved documents.

Do not invent information.

If the documents don't contain enough information,
say that you don't have enough information.

Cite the relevant source...

Prompt B contains more instructions.

More instructions mean more input tokens.

That doesn't automatically mean Prompt A is better.

The goal isn't:

Use as few tokens as possible.

The goal is:

Use the right amount of tokens to produce the required behavior.

This is an important distinction in production AI systems.

---

5. Token Optimization

As AI applications scale, engineers often look for ways to reduce unnecessary tokens.

Some approaches include:

Reduce unnecessary prompt instructions

Don't repeat information that the model already has.

Trim conversation history

You may not need the entire conversation in every request.

Summarize older conversations

Instead of passing hundreds of messages:

100 messages
      ↓
Conversation summary
      ↓
Relevant recent messages

Improve RAG retrieval

Retrieve fewer but more relevant chunks.

Remove duplicate context

Don't send the same information multiple times.

Compress structured information

Represent information efficiently when appropriate.

These techniques can improve:

• Cost

• Latency

• Context utilization

• Sometimes even answer quality

---

6. A Subtle but Important Point: More Context ≠ Better Answers

It is tempting to think:

"If more context is good, then maximum context must be better."

Not necessarily.

Imagine asking:

"What is the refund policy?"

and giving the model:

50 highly relevant paragraphs
+
500 unrelated paragraphs

The model has more information, but the signal-to-noise ratio may be worse.

A better system might provide:

5 highly relevant paragraphs

This is why modern AI architecture often focuses on:

Context quality, not simply context quantity.

---

7. From Words to Tokens to Intelligence

The complete journey looks something like this:

Human language
      ↓
Text
      ↓
Tokenizer
      ↓
Tokens
      ↓
Token IDs
      ↓
Embeddings
      ↓
Transformer
      ↓
Self-Attention
      ↓
Multi-Head Attention
      ↓
Multiple Transformer Layers
      ↓
Next-token probabilities
      ↓
Generated tokens
      ↓
Text

This is the foundation of how an LLM turns language into something a neural network can process.

---

The Big Takeaway

The most important thing to remember isn't:

"A token is approximately four characters."

That's only a rough approximation.

The important lesson is:

Words are not the fundamental unit that an LLM processes. Tokens are.

And once you understand that, several important engineering questions become much easier to reason about:

Cost

How many tokens am I sending and generating?

Latency

How much context am I asking the model to process?

Context

How much information can I provide in one request?

RAG

Which retrieved information deserves a place in the model's context?

Prompt engineering

How much instruction does the model actually need?

AI architecture

How do I design the system so that the model receives the right information without unnecessary context?

That is why something as simple as tokenization becomes an important concept when building production-grade LLM applications.

If words are how humans communicate, tokens are one of the fundamental units through which LLMs process that communication.