Tokens: The Fundamental Unit of LLMs

When we interact with an LLM, we naturally think in terms of words.

We type:

"Forward Deployed Engineering is becoming an important model for enterprise AI deployment."

It feels like the model receives that sentence as a collection of words.

It doesn't.

Before an LLM can process the text, it first converts the text into tokens.

And understanding tokens is one of the first important steps toward understanding how LLMs actually work.

---

1. See Tokenization for Yourself

One of the easiest ways to understand tokens is to actually see them.

Open the official OpenAI Tokenizer:

OpenAI Tokenizer

Enter something like:

"Forward Deployed Engineering is becoming an important model for enterprise AI deployment."

The tokenizer will show you how the sentence is broken down.

Don't focus too much on the exact number of tokens.

Instead, notice how the text is divided.

You may find that a token corresponds to:

• A complete word

• Part of a word

• Punctuation

• A space combined with a word

• A common sequence of characters

This leads to the first important principle:

Words ≠ Tokens

---

2. So What Exactly Is a Token?

A token is essentially a piece of text that the model processes as a unit.

For example, a word might remain a single token:

"engineering"

But another word might be split:

"unbelievable"
       ↓
"un" + "believ" + "able"

The exact splitting depends on the tokenizer and the vocabulary it was trained with.

Tokens aren't necessarily words.

They can be:

Token
 ├── complete word
 ├── part of a word
 ├── punctuation
 ├── whitespace + word
 └── common character sequence

This is why counting words isn't enough when you're working with LLMs.

---

3. Why Don't LLMs Just Use Words?

At first, using words seems like the obvious choice.

Why break:

"engineering"

into smaller pieces?

There are several reasons.

Imagine an LLM had a separate token for every possible word.

The vocabulary would become enormous.

English alone contains a huge number of words, and there are countless:

• Names

• Technical terms

• Product names

• URLs

• Code

• Abbreviations

• Misspellings

• New words

Instead, tokenization allows the model to reuse smaller pieces.

For example:

engine
engineering
engineer
engineered

may share some underlying token pieces.

This makes it possible for the model to represent a much larger range of text using a manageable vocabulary.

---

4. A Useful Rule of Thumb

For common English text, a rough approximation is:

1 token ≈ 4 characters

But this is only a rule of thumb.

Actual tokenization varies significantly depending on the text.

For example:

Normal English

"Hello, how are you?"

may tokenize relatively efficiently.

Technical text

"MultiHeadAttention"

may be split into several pieces.

Code

def calculate_attention_scores():

can tokenize differently from normal prose.

Numbers

928374928374

may be broken into multiple tokens.

Other languages

Token efficiency can also vary considerably across languages.

Therefore:

Never assume that one word equals one token.

---

5. Tokens Are What the LLM Actually Processes

This is a key conceptual shift.

We start with:

Human
  ↓
Text

The model doesn't directly process that text as humans understand it.

Instead:

Text
  ↓
Tokenizer
  ↓
Tokens
  ↓
Token IDs
  ↓
Embeddings
  ↓
Transformer

For example, conceptually:

"Hello world!"
       ↓
   Tokenizer
       ↓
["Hello", " world", "!"]
       ↓
[15496, 995, 0]
       ↓
Token embeddings
       ↓
Transformer

The actual token IDs depend on the tokenizer/model.

The important point is that the LLM ultimately operates on numerical representations of tokens, not directly on words.

---

6. Tokens and LLM Costs

This is where tokenization becomes an engineering concern.

LLM APIs typically charge based on tokens.

There can be separate pricing for:

• Input tokens

• Output tokens

• Sometimes cached tokens or other categories

So consider an application where you send:

User question
+
System instructions
+
Conversation history
+
Retrieved documents
+
Tool results

All of that can contribute to the input token count.

If your application sends unnecessarily large prompts, the cost can increase.

For example:

Small prompt
     ↓
1,000 tokens
     ↓
Lower cost


Large prompt
     ↓
20,000 tokens
     ↓
Higher cost

This becomes particularly important at enterprise scale.

A few thousand unnecessary tokens might not matter much for one request.

But multiply that by:

10,000 users
×
100 requests/day
×
large prompts

and token efficiency becomes an architecture and cost-management issue.

---

7. Tokens and Latency

Tokens can also affect performance.

The model needs to process the input context and generate the output.

A simplified picture is:

More input tokens
       +
More output tokens
       ↓
More computation
       ↓
Potentially more latency

There are many other factors that affect latency — model architecture, hardware, serving infrastructure, batching, caching, network time, and so on.

So token count isn't the only factor.

But it is an important one.

This is why an AI application shouldn't blindly send everything it knows to the model.

---