LLM chevron_right Introduction to Tokenization and Text Representation in NLP

Introduction to Tokenization and Text Representation in NLP

calendar_today schedule 7 min read

Learning Objectives: By the end of this lesson, you will be able to:

  • Explain the importance of tokenization and text representation in NLP
  • Implement basic tokenization and TF-IDF from scratch and with libraries
  • Identify and avoid common mistakes in text preprocessing

1. Welcome and Introduction (5 minutes)

Icebreaker Activity

Have you ever wondered how your phone’s autocorrect or a search engine understands what you mean, even if you make a typo? Or why some chatbots seem to “get” your intent, while others don’t?

Lesson Overview

  • Duration: Approximately 2.5 hours including all exercises (or 60–90 minutes when selected activities are assigned)
  • Prerequisites: Basic Python programming, familiarity with text data
  • Materials Needed: Python (Jupyter Notebook or IDE), NLTK or spaCy, sample text files

2. Learning Objectives and Session Outline (5 minutes)

What We’ll Cover Today:

  1. Tokenization: What it is and why it matters
  2. Text normalization: Stemming, lemmatization, and cleaning
  3. Text representation: Bag-of-Words, n-grams, and TF-IDF
  4. Hands-on practice and exercises
  5. Common pitfalls and how to avoid them

Success Criteria:

By the end of this session, you should be able to preprocess text for NLP tasks and explain the rationale behind each step.

3. Foundation Knowledge (15-20 minutes)

Core Concepts

Tokenization

Tokenization is the process of splitting text into smaller units called tokens. These tokens can be words, subwords, or even characters, depending on the application.

Real-World Analogy: Think of tokenization like breaking a sentence into Lego blocks. Each block (token) can be analyzed or rearranged independently.

Why This Matters: Computers can’t understand raw text. Tokenization is the first step in making text data usable for algorithms.

Text Normalization

Depending on the task, normalization may include stemming or lemmatization. Lowercasing, punctuation removal, and stop-word removal are optional and can discard useful meaning.

Text Representation

Once text is tokenized and normalized, it needs to be represented numerically. Common methods include Bag-of-Words, n-grams, and TF-IDF.

Key Terminology

  • Token: A single unit of text, such as a word or subword (e.g., “cat”, “##ing”).
  • Stemming: Reducing words to their root form (e.g., “running” → “run”).
  • Lemmatization: Reducing words to their dictionary form (e.g., “better” → “good”).
  • TF-IDF: Term Frequency-Inverse Document Frequency, a statistic that reflects how important a word is to a document in a collection.
  • n-gram: A sequence of n tokens (e.g., bigram = 2 tokens).

4. Detailed Explanation and Examples (25-30 minutes)

Understanding Tokenization and Text Representation

Tokenization is foundational in NLP. Without it, computers can’t process or analyze text. There are different types of tokenization:

  • Word Tokenization: Splitting text into words.
  • Subword Tokenization: Splitting words into smaller units (useful for rare words).
  • Character Tokenization: Splitting text into individual characters.

Practical Applications

  • Search engines use tokenization to match queries to documents.
  • Chatbots use tokenization to understand user input.
  • Sentiment analysis relies on tokenized text to determine sentiment.

Common Misconceptions

  1. Misconception: Tokenization is always straightforward. Reality: Tokenization can be complex, especially for languages without spaces or with compound words.
  2. Misconception: Stemming and lemmatization are interchangeable. Reality: Stemming is rule-based and can be crude; lemmatization uses vocabulary and is more accurate.

5. Live Coding Demonstration (20-25 minutes)

Example 1: Basic Tokenization in Python

# Basic word tokenization
text = "Natural Language Processing is fun!"
tokens = text.split()
print(tokens)

Explanation:

  • Line 1: Defines a sample text.
  • Line 2: Splits the text into words using spaces.
  • Line 3: Prints the list of tokens.

Example 2: Using NLTK for Tokenization and Stemming

import nltk
from nltk.tokenize import word_tokenize
from nltk.stem import PorterStemmer
nltk.download('punkt')

text = "Cats are running faster than dogs."
tokens = word_tokenize(text)
stemmer = PorterStemmer()
stemmed = [stemmer.stem(token) for token in tokens]
print("Tokens:", tokens)
print("Stemmed:", stemmed)

Key Points:

  • NLTK provides robust tokenization and stemming tools.
  • Stemming may produce non-dictionary words (e.g., “running” → “run”).
  • Always check the output for correctness.

Example 3: TF-IDF Vectorization with scikit-learn

from sklearn.feature_extraction.text import TfidfVectorizer
corpus = [
    "I love machine learning",
    "Machine learning is powerful",
    "I love coding in Python"
]
vectorizer = TfidfVectorizer()
X = vectorizer.fit_transform(corpus)
print(vectorizer.get_feature_names_out())
print(X.toarray())

Debugging Demonstration

Show common errors and how to fix them:

# Compare task-dependent normalization rather than assuming lowercase is correct
text = "Cats are Running."
tokens = text.split()
print(tokens)  # ['Cats', 'are', 'Running.']

# Corrected version
text = text.lower()
tokens = text.split()
print(tokens)  # ['cats', 'are', 'running.']

6. Hands-On Practice (30-40 minutes)

Exercise 1: Basic Practice (10 minutes)

Objective: Tokenize and normalize a short paragraph.

Instructions:

  1. Write a function with optional lowercase and remove_punctuation switches, and compare the outputs.
  2. Test it on the sentence: “Hello, world! NLP is amazing.”

Starter Code:

import string

def simple_tokenize(text, lowercase=False, remove_punctuation=False):
    # Implement both optional transformations, then split the text.
    pass

print(simple_tokenize("Hello, world! NLP is amazing."))

Solution:

import string

def simple_tokenize(text, lowercase=False, remove_punctuation=False):
    if lowercase:
        text = text.lower()
    if remove_punctuation:
        text = text.translate(str.maketrans('', '', string.punctuation))
    return text.split()

print(simple_tokenize("Hello, world! NLP is amazing.", True, True))
# Output: ['hello', 'world', 'nlp', 'is', 'amazing']

Exercise 2: Intermediate Challenge (15 minutes)

Objective: Implement a function to compute the TF-IDF score for a word in a document.

Requirements:

  • Write a function that takes a word, a document, and a list of documents (corpus).
  • Compute term frequency (TF) and inverse document frequency (IDF).
  • Return the TF-IDF score.

Hints:

  • TF = (number of times word appears in document) / (total words in document)
  • IDF = log(N / (1 + number of documents containing the word)), where N is total number of documents

Exercise 3: Creative Application (15 minutes)

Objective: Build a simple search function over a small set of documents using TF-IDF.

Challenge:

  • Given a list of documents, allow the user to enter a query.
  • Return the document most relevant to the query using cosine similarity of TF-IDF vectors.

Extension Ideas:

  • Highlight the matching words in the result.
  • Allow for multi-word queries.

7. Common Pitfalls and Troubleshooting (10 minutes)

Frequent Mistakes

  1. Problem: Applying normalization without testing its effect Cause: Treating lowercasing and punctuation removal as universal rules Solution: Preserve raw text and validate each transformation for the task

  2. Problem: Using stemming when lemmatization is needed Cause: Not understanding the difference Solution: Use lemmatization for more accurate normalization

Debugging Tips

  • Print intermediate results to check each step
  • Use library functions for complex tasks
  • Read error messages carefully; they often point to the exact issue

8. Best Practices and Professional Tips (5 minutes)

Industry Standards

  • Always preprocess text before analysis
  • Use established libraries (NLTK, spaCy, scikit-learn) for reliability
  • Document your preprocessing pipeline for reproducibility

Next Steps in Your Learning Journey

  • Explore subword tokenization (BPE, WordPiece)
  • Learn about word embeddings (Word2Vec, GloVe)
  • Try more advanced NLP tasks (sentiment analysis, text classification)

Accuracy and Task-Design Notes

Lowercasing, punctuation removal, stop-word removal, stemming, and lemmatization are optional, task-dependent transformations, not a universal pipeline. For example, case distinguishes US from us, punctuation carries sentiment and sentence boundaries, and pretrained tokenizers normally expect text close to their training distribution. Preserve the original text and validate each normalization choice on the downstream metric.

For a hand-calculated TF-IDF exercise, use one formula consistently. A convenient smoothed definition is idf(t) = log((1 + N) / (1 + df(t))) + 1, matching scikit-learn’s default convention. It remains positive even when a term appears in every document.

9. Summary and Review (5 minutes)

Key Takeaways

  1. Tokenization and normalization are essential first steps in NLP
  2. TF-IDF is a powerful method for representing text numerically
  3. Always check your preprocessing for errors and inconsistencies

Quick Review Questions

  1. What is tokenization and why is it important?
  2. How does TF-IDF differ from simple word counts?
  3. What are common pitfalls in text preprocessing?

For Instructors

Teaching Tips

  • Encourage students to experiment with different tokenization methods
  • Discuss real-world challenges (e.g., tokenizing non-English text)
  • Adjust timing based on student familiarity with Python

Assessment Ideas

  • Short quiz on terminology and concepts
  • Coding assignment: implement a mini search engine
  • Peer review: students check each other’s preprocessing code

Adaptation Notes

  • For beginners: focus on word tokenization and basic TF-IDF
  • For advanced students: introduce subword tokenization and embeddings
  • For online delivery: use collaborative coding platforms (e.g., Google Colab)