Nearby lessons
17 of 30Types of Tokens
Concept in One Line
Tokens are of different types because GPT breaks text into small meaningful parts like words, parts of words, spaces, and symbols.
Why This Concept Is Important
- Helps understand why token count increases.
- Explains cost and context window clearly.
- Important for prompt writing and interviews.
Key Points to Remember
- Tokens are not only full words.
- A single word can become multiple tokens.
- Spaces and symbols are also tokens.
- Different languages create different token counts.
- Token type depends on tokenizer design.
Main Types of Tokens
Word Tokens
Full words as tokens.
Example:
AI -> AI
Hello -> Hello
Sub-Word Tokens
Part of a word becomes a token.
Very common in GPT.
Example:
learning -> learn + ing
unhappiness -> un + happiness
Helps the model understand new words.
Space Tokens
Spaces are also counted as tokens.
Example: Hello AI
Tokens may look like:
Hello
AI
Space before a word matters.
Punctuation Tokens
Symbols are separate tokens.
Examples: . , ? !
Sentence: Hello, AI!
Tokens include comma and exclamation.
Special Tokens
Used internally by the model.
Examples:
Start of sentence
End of sentence
Padding tokens
Mostly hidden from users.
Simple Example
Sentence: I love AI!
Possible tokens:
- I
- love
- AI
- !
Not word by word, but meaningful chunks.
Simple Flow
Text -> Split into token types -> Convert into numbers -> Processing
Interview / Exam Points
Q1: What are the types of tokens in GPT?
Word tokens, sub-word tokens, space tokens, punctuation tokens, and special tokens.
Q2: Why are sub-word tokens used?
To handle new words and reduce vocabulary size.
Common Confusions to Avoid
- Token is not always a word.
- Token is not a character.
- Space can be a token.
- One word can have many tokens.
One Line to Remember
Tokens are small meaningful pieces, not full words always.
Quick Self-Check
For Students
Spaces can be tokens - Yes / No?
One word can be split into multiple tokens - Yes / No?
(Correct answers: Yes, Yes)