Nearby lessons
15 of 30What is a Token?
Concept in One Line
A token is a small piece of text, such as a word, part of a word, or a symbol, that GPT uses instead of reading full sentences directly.
Why This Concept Is Important
- GPT does not understand full words like humans.
- Token count decides cost, speed, and limits.
- This concept is very important for prompts and interviews.
Key Points to Remember
- GPT reads tokens, not full sentences.
- One word can be one token or many tokens.
- Spaces and symbols also count as tokens.
- More tokens means more processing.
- Context window is measured in tokens.
How Tokens Work
Simple Explanation
Example sentence: I am learning AI
Possible tokens:
- I
- am
- learning
- AI
GPT breaks text into small chunks.
Why Tokens Are Needed
- Computers understand numbers, not text.
- Tokens are easy to convert into numbers.
- Smaller units help the model make better predictions.
Tokens make language machine-friendly.
Simple Flow
Text -> Tokens -> Numbers -> Processing -> Output
Interview / Exam Points
Q1: What is a token in GPT?
A token is a small unit of text used by GPT for processing language.
Q2: Does one word always equal one token?
No. One word can be split into multiple tokens.
Common Confusions to Avoid
- A token is not always equal to a word.
- Token count is not the same as character count.
- Token count affects cost and limits.
- GPT processes tokens internally.
One Line to Remember
GPT understands text only through tokens.
Quick Self-Check
For Students
GPT reads full sentences directly - Yes / No?
Token count affects context window - Yes / No?
(Correct answers: No, Yes)
🧠 Test Your Knowledge
3 QuestionsProgress: 0 / 3
Keep Going!Why Tokenization Is Needed