Nearby lessons
16 of 30Why Tokenization is Needed
Concept in One Line
Tokenization is needed because computers work with numbers, and breaking text into small pieces makes language easier for AI to process.
Why This Concept Is Important
- Explains how GPT reads text.
- Helps understand token limits and costs.
- Important for interviews and prompt design.
Key Points to Remember
- Computers work with numbers, not raw text.
- Tokenization breaks text into small units.
- Tokens can be words, parts of words, or symbols.
- Tokenization makes text machine-readable.
- Smaller pieces improve prediction accuracy.
Why Tokenization Is Needed
Text by itself is hard for a machine to process directly.
Tokens give the model small, manageable pieces.
Those pieces are easier to convert into numbers.
That is why tokenization is the first practical step before the model can think.
Simple Tokenization Example
Sentence: ChatGPT is powerful
Possible tokens:
- Chat
- GPT
- is
- powerful
Tokens depend on the tokenizer.
Simple Flow
Raw Text -> Tokenization -> Tokens -> Numbers -> Model
Why Smaller Pieces Help
- New words appear every day.
- Splitting words avoids confusion.
- Smaller pieces improve prediction.
- Tokenization helps handle unknown words better.
Interview / Exam Points
Q1: Why is tokenization needed?
Tokenization is needed because computers understand numbers, not raw text, so text must be broken into tokens first.
Q2: What does tokenization make text?
Tokenization makes text machine-readable.
Common Confusions to Avoid
- Tokenization is not the same as token limits.
- Tokenization is not word meaning.
- Tokenization is a preprocessing step.
- Tokenization happens before encoding and model processing.
One Line to Remember
Tokenization makes human text ready for machine processing.
Quick Self-Check
For Students
Computers understand raw text directly - Yes / No?
Tokenization makes text machine-readable - Yes / No?
(Correct answers: No, Yes)