NLP (2026): Complete Guide with Python Code

Type "running shoes" into a search box and you expect results that mention "run", "runner" and "runs" too. Behind that small convenience sits one of the oldest and most practical tricks in Natural Language Processing: stemming. In this guide you will learn exactly what stemming is, how the famous algorithms work step by step, how to use them in Python, where they fail, and whether stemming still matters in an age of large language models.

This is a hands-on guide. Every Python snippet below has been run, and the outputs shown are real. By the end you will be able to choose the right stemmer for your project, plug it into a scikit-learn pipeline, and explain the difference between stemming and lemmatization confidently in an interview, a viva, or a code review.

Table of Contents

1. What Is Stemming in NLP?
2. Why Stemming Matters in Text Preprocessing
3. Stemming vs Lemmatization
4. How Stemming Works Under the Hood
5. Over-Stemming and Under-Stemming
6. Stemming Algorithms Explained
7. Porter vs Snowball vs Lancaster: Real Output Comparison
8. Stemming in Python with NLTK
9. A Complete Text Preprocessing Pipeline
10. Stemming with scikit-learn and TF-IDF
11. Build a Simple Stemmer from Scratch
12. Multilingual Stemming (Including Arabic and Urdu)
13. Real-World Applications
14. Is Stemming Still Relevant in 2026?
15. How to Evaluate a Stemmer
16. Best Practices and Common Mistakes
17. Frequently Asked Questions
18. Conclusion

1. What Is Stemming in NLP?

Stemming is the process of reducing a word to its base or root form — called the stem — by chopping off prefixes and, much more commonly, suffixes. The goal is simple: different surface forms of the same word should be treated as one item by a computer.

Consider these words:

• connect, connected, connecting, connection, connections
• argue, argued, argues, arguing
• fish, fishing, fished

To a human, each line obviously belongs to one idea. To a computer, they are completely different strings. A search engine, a spam filter or a sentiment classifier that treats "connected" and "connection" as unrelated words loses useful information. A stemmer fixes that by mapping all of them to a shared stem:

connectedconnectingconnection→connect

arguearguedarguing→argu

Notice the second example. The stem argu is not a real English word. That is perfectly normal and is the single most important thing to understand about stemming: a stem does not need to be a dictionary word. It only needs to be consistent, so that related words end up identical. Stemming is a heuristic, rule-based shortcut, not a linguistic analysis.

A short history

Stemming is older than the web. Julie Beth Lovins published the first widely cited stemming algorithm in 1968. In 1980, Martin Porter published the algorithm that became the de facto standard for English, and it is still the default in many libraries more than four decades later. Porter later designed Snowball, a small language for writing stemmers, and used it to create an improved English stemmer (often called Porter2) plus stemmers for many other languages. Around 1990, Chris Paice and Gareth Husk at Lancaster University released the aggressive Lancaster stemmer. These three — Porter, Snowball and Lancaster — are the ones you will meet most often in practice.

Key definition to remember: Stemming = fast, rule-based suffix stripping that maps related word forms to a common (possibly non-word) stem, to reduce vocabulary size and improve matching.

2. Why Stemming Matters in Text Preprocessing

Raw text is messy. Before a classical machine learning model can learn from it, text usually passes through a preprocessing pipeline: lowercasing, tokenization, stop-word removal, and some form of normalization. Stemming is one of the main normalization tools. Here is why it earns its place.

2.1 It shrinks the vocabulary

Models such as Bag-of-Words and TF-IDF create one feature (column) per unique word. A corpus of news articles might contain "run", "runs", "running", "runner" and "runners" as five separate columns, each with fewer examples to learn from. Stemming collapses many of these into one column. A smaller vocabulary means smaller matrices, faster training, less memory, and — crucially — less sparsity. Each remaining feature has more evidence behind it, which often helps models trained on small or medium datasets.

2.2 It improves recall in search

In information retrieval, recall measures how many of the relevant documents you actually found. If a user searches "connecting databases" and a great article only uses "database connection", an exact-match system misses it. When both the query and the documents are stemmed, both become connect databas, and the article is found. This is why search engines such as Elasticsearch, OpenSearch, Apache Solr and PostgreSQL full-text search ship with stemmers built into their text analyzers.

2.3 It is extremely fast

A stemmer looks only at the characters of a single word and applies a handful of rules. It needs no dictionary lookups, no part-of-speech tagging and no neural network. That makes it ideal for indexing millions of documents, processing streaming data, or running on low-power hardware.

2.4 It reduces noise for classical models

For tasks like topic modeling (LDA), document clustering, keyword extraction and simple text classification, the exact tense or plural form of a word rarely carries meaning. "Election", "elections" and "elected" all point to the same topic. Merging them lets the model focus on what a document is about instead of how each word happened to be inflected.

Rule of thumb: Stemming helps most when you care about topic and relevance (search, clustering, classification on small data). It helps least — and can hurt — when you care about exact meaning and grammar (translation, question answering, text generation).

3. Stemming vs Lemmatization: What's the Difference?

This is the most frequently asked question about stemming, and it shows up in almost every NLP exam and interview. Both techniques reduce words to a base form, but they do it in completely different ways.

Stemming chops letters off the end of a word using rules, without knowing what the word means or which part of speech it is. Lemmatization uses a vocabulary (such as WordNet) and morphological analysis to return the lemma — the real dictionary form of the word — and usually needs to know whether the word is a noun, verb or adjective.

Here is a real comparison using NLTK's PorterStemmer and WordNetLemmatizer (with the correct part-of-speech supplied to the lemmatizer):

Word	Porter Stem	WordNet Lemma	What happened
running	run	run	Both agree
studies	studi	study	Stem is not a real word; lemma is
ran	ran	run	Irregular verb: only the lemmatizer knows it
better	better	good	Lemmatizer uses its dictionary (as adjective)
was	wa	be	Stemmer blindly strips the "s"

The pattern is clear. Lemmatization is more accurate and produces readable output. Stemming is faster and simpler, but makes mistakes with irregular forms.

Aspect	Stemming	Lemmatization
Approach	Rule-based suffix stripping	Dictionary + morphological analysis
Output	Stem (may not be a real word)	Lemma (always a valid word)
Needs POS tags?	No	Usually yes, for best results
Speed	Very fast	Slower
Handles irregular forms (ran, was, better)?	No	Yes
Resources needed	Just the algorithm	Lexical database or language model
Best for	Search indexing, large-scale IR, quick baselines	Chatbots, QA, analysis where meaning matters

Which one should you choose?

Use stemming when speed and recall matter more than readability: search engines, indexing huge corpora, quick classification baselines, or when you have no lemmatizer for your language. Use lemmatization when you need human-readable output, when precision matters, or when downstream steps depend on real words (for example, looking words up in a sentiment lexicon). When in doubt, try both and measure — the "best" choice is whichever gives better results on your validation data.

4. How Stemming Works Under the Hood

Almost every popular stemmer is an affix-stripping stemmer. It holds an ordered list of rules of the form:

IF word ends with SUFFIX AND some CONDITION is true
THEN replace SUFFIX with REPLACEMENT

For example: "if the word ends in -ational, replace it with -ate" (relational → relate). The interesting part is the condition. Without conditions, a rule like "remove -ing" would turn "sing" into "s" and "king" into "k". Good stemmers protect short words with length or structure checks.

4.1 The "measure" idea in the Porter stemmer

Porter's clever trick is to describe every word as a sequence of consonant groups (C) and vowel groups (V):

[C](VC)^m[V]

The number m is called the measure of the word. It roughly counts how many vowel-consonant "syllable chunks" a word has:

• tree, by → m = 0
• trouble, oats → m = 1
• troubles, private → m = 2

Rules can then say things like "remove -ement only if the remaining stem has m > 1". That's why "replacement" becomes "replac" but "cement" stays "cement" — after removing -ement, "c" has a measure of 0, so the rule does not fire.

4.2 Rules are applied in steps

Instead of one giant list, stemmers group rules into ordered steps. Each step usually removes one "layer" of suffix. A long word may pass through several steps. Here is a classic real example, verified with NLTK's Porter stemmer:

generalizations→generalization→generalize→general→gener

Plural "s" is removed first, then -ization becomes -ize, then -ize is dropped, and finally -al is removed. Peeling suffixes one layer at a time is what lets a small set of rules handle thousands of words.

4.3 Other families of stemmers

Affix stripping is the most common approach, but not the only one:

• Dictionary / lookup stemmers store a table mapping each word form to its stem. Very accurate for known words, useless for new ones, and the table can be huge.
• Hybrid stemmers such as the Krovetz stemmer combine light rules with a dictionary check, so they only strip a suffix if the result is a real word. Output is more readable, recall is lower.
• Statistical / corpus-based stemmers learn which suffixes to strip from a large text collection, for example by grouping words that share long prefixes and appear in similar contexts. Useful for languages with no hand-written stemmer.
• N-gram approaches skip stemming entirely and break words into overlapping character chunks (run, unn, nni …). Related words naturally share many n-grams.

5. Over-Stemming and Under-Stemming

No stemmer is perfect. Its mistakes fall into two categories, and understanding them is the key to picking the right algorithm.

5.1 Over-stemming (false positives)

Over-stemming happens when the stemmer cuts too much and merges words that have different meanings. Real outputs from NLTK:

• university and universe → both become univers (Porter, Snowball and Lancaster all do this)
• organization → organ (Porter) — now it collides with the body part
• generously → gen (Lancaster) — almost all meaning lost
• news → new (Lancaster) — a different word entirely

Over-stemming hurts precision: a search for "university" might return documents about the universe.

5.2 Under-stemming (false negatives)

Under-stemming happens when the stemmer cuts too little and fails to merge words that should be together:

• ran stays ran, while running becomes run — they never meet
• better stays better; it never connects to "good"
• children stays children; it never connects to "child"

Under-stemming hurts recall: relevant documents are missed because their word forms were not merged.

The fundamental trade-off: Aggressive stemmers (like Lancaster) reduce under-stemming but cause more over-stemming. Light stemmers (like Snowball) cause less over-stemming but leave more words un-merged. There is no setting that eliminates both — you choose the balance that suits your task.

6. Stemming Algorithms Explained

6.1 Lovins Stemmer (1968)

The Lovins stemmer was the first influential stemming algorithm. It contains a list of roughly 290 endings and removes the longest matching ending in a single pass, guided by around 29 context conditions. After stripping, a set of "recoding" rules fixes up spelling at the boundary (for example, handling double letters). It is fast because it works in one step, but its huge suffix list was difficult to maintain, and it is rarely used today except for historical interest and research comparisons.

6.2 Porter Stemmer (1980)

The Porter stemmer is the most famous stemming algorithm in history. It uses the measure concept described above and applies around 60 rules in five main steps:

1. Step 1a — Plurals: sses → ss, ies → i, s → (removed). So caresses → caress, ponies → poni, cats → cat.
2. Step 1b — Past tense and progressive: removes -ed and -ing when the stem contains a vowel, then tidies up (adding back an "e" or removing a doubled consonant). So hopping → hop, caring → care.
3. Step 1c — Turns a final y into i when there is another vowel in the stem: happy → happi.
4. Steps 2 and 3 — Map double and single derivational suffixes to simpler ones: -ational → -ate, -ization → -ize, -fulness → -ful, -icate → -ic.
5. Steps 4 and 5 — Remove remaining suffixes like -al, -ance, -ment, -ive when the measure is large enough, and clean up a final "e" or double "l".

Strengths: well-understood, moderate aggressiveness, available everywhere, reproducible. Weaknesses: English-only, produces many non-words (studi, happi, easili), and its original rules contain a few quirks that later versions fixed.

Note for NLTK users: NLTK's PorterStemmer runs in an "NLTK extensions" mode by default, which includes a few small fixes over the 1980 paper. If you need results that exactly match the original paper (for example, to reproduce a research result), use PorterStemmer(mode=PorterStemmer.ORIGINAL_ALGORITHM).

6.3 Snowball Stemmer / Porter2 (2001+)

Martin Porter created Snowball, a small string-processing language designed specifically for writing stemmers, and used it to write a revised English stemmer usually called Porter2. In NLTK it is available as SnowballStemmer("english"). Its improvements include better handling of words ending in -ly, special handling for a list of exception words, and a smarter definition of where the "stem region" begins.

Real differences you can observe:

• fairly → Porter: fairli, Snowball: fair
• generously → Porter: gener, Snowball: generous

Snowball also includes stemmers for many other languages, including Arabic, Danish, Dutch, Finnish, French, German, Hungarian, Italian, Norwegian, Portuguese, Romanian, Russian, Spanish and Swedish. For English text in 2026, Snowball is generally the recommended default stemmer.

6.4 Lancaster (Paice/Husk) Stemmer (1990)

The Lancaster stemmer is iterative: it keeps applying rules from a table until no rule matches. Each rule says which ending to remove, what to add, whether the word must be "intact" (unchanged so far), and whether to continue or stop. Because it keeps going, it is much more aggressive than Porter.

Sometimes that aggression produces nicer output — studies → study, happiness → happy, ponies → pony. More often it over-stems heavily: organization → org, generously → gen, national → nat, maximum → maxim, caring → car. Use Lancaster when you need maximum vocabulary reduction and can tolerate more false matches.

6.5 Regexp Stemmer (custom rules)

NLTK's RegexpStemmer lets you define your own suffix patterns with a regular expression and a minimum word length. It is useful for domain-specific text, quick experiments, or teaching — but it has no intelligence beyond your pattern. With the pattern ing$|s$|e$|able$ and a minimum length of 4, "running" becomes runn (it doesn't know about doubled consonants) and "caring" becomes car.

6.6 Krovetz Stemmer (1993)

Robert Krovetz designed a light, dictionary-checked stemmer. It handles inflections (plurals, past tense, -ing) and some derivations, but only accepts a result if it appears in its dictionary. Output is almost always a real word, and over-stemming is rare. It is popular in academic information-retrieval toolkits and available in Apache Lucene as KStemFilter. The trade-off is lower recall than Porter or Snowball.

6.7 Other notable stemmers

• Dawson stemmer — an extension of Lovins with a larger suffix list, processed efficiently.
• Lucene / Elasticsearch "minimal" and "light" stemmers — conservative stemmers for many languages (for example minimal_english mostly just handles plurals), used when precision matters.
• ISRI stemmer — a root-extraction stemmer for Arabic, available in NLTK.
• Statistical stemmers (e.g., YASS, GRAS) — learn suffix rules from a corpus, helpful for low-resource languages.

7. Porter vs Snowball vs Lancaster vs Regexp: Real Output Comparison

Theory is helpful, but nothing beats seeing real outputs side by side. Every value in the table below was produced by NLTK (Regexp stemmer configured with ing$|s$|e$|able$, minimum length 4).

Word	Porter	Snowball	Lancaster	Regexp
running	run	run	run	runn
connection	connect	connect	connect	connection
studies	studi	studi	study	studie
happiness	happi	happi	happy	happines
easily	easili	easili	easy	easily
fairly	fairli	fair	fair	fairly
generously	gener	generous	gen	generously
organization	organ	organ	org	organization
university	univers	univers	univers	university
caring	care	care	car	car
relational	relat	relat	rel	relational
maximum	maximum	maximum	maxim	maximum
was	wa	was	was	was
news	news	news	new	new
fisher	fisher	fisher	fish	fisher

What the table tells us

• Snowball is the most balanced. It fixes Porter's odd outputs like fairli and wa, and avoids Lancaster's extreme cuts.
• Porter is close to Snowball and is a safe, well-known choice — ideal when you need to match older research or systems.
• Lancaster sometimes produces nicer stems (study, happy) but over-stems badly (org, gen, car, new).
• Regexp is only as good as your rules. Great for domain-specific suffixes, poor as a general stemmer.

Stemmer	Aggressiveness	Speed	Languages	Recommended for
Porter	Medium	Fast	English	Classic baselines, reproducing research
Snowball (Porter2)	Medium-light	Fast	English + many others	Default choice for most projects
Lancaster	High	Fast	English	Maximum vocabulary reduction
Krovetz	Light	Medium	English	Search where readable stems matter
Regexp	You decide	Very fast	Any	Custom domain rules, teaching

8. Stemming in Python with NLTK

The Natural Language Toolkit (NLTK) is the easiest way to experiment with stemming in Python. Let's go from installation to multilingual stemming.

8.1 Installation

pip install nltk

The stemmers themselves need no extra downloads. Tokenizers and stop-word lists do, so download them once:

import nltk
nltk.download("punkt_tab")   # tokenizer models
nltk.download("stopwords")   # stop-word lists
nltk.download("wordnet")     # only needed for lemmatization comparisons

8.2 Porter Stemmer

from nltk.stem import PorterStemmer

porter = PorterStemmer()
words = ["connect", "connected", "connecting", "connection", "connections"]

for w in words:
    print(f"{w:12} -> {porter.stem(w)}")

connect      -> connect
connected    -> connect
connecting   -> connect
connection   -> connect
connections  -> connect

Five different strings become one. That is stemming doing exactly what it was designed for.

8.3 Stemming a whole sentence

Stemmers work on single words, so tokenize first:

from nltk.tokenize import word_tokenize

sentence = "The children are running faster than the runners who ran yesterday."
tokens = word_tokenize(sentence)
print([porter.stem(t) for t in tokens])

['the', 'children', 'are', 'run', 'faster', 'than', 'the', 'runner', 'who', 'ran', 'yesterday', '.']

Notice the limitations in action: children and ran are untouched because they are irregular forms. Note also that Porter lowercases its output by default (The became the).

8.4 Snowball Stemmer

from nltk.stem import SnowballStemmer

snowball = SnowballStemmer("english")
for w in ["fairly", "generously", "running", "was"]:
    print(f"{w:11} -> {snowball.stem(w)}")

fairly      -> fair
generously  -> generous
running     -> run
was         -> was

Snowball has a handy option, ignore_stopwords=True, which leaves common stop words untouched:

print(SnowballStemmer("english").stem("having"))                          # have
print(SnowballStemmer("english", ignore_stopwords=True).stem("having"))   # having

8.5 Lancaster Stemmer

from nltk.stem import LancasterStemmer

lancaster = LancasterStemmer()
for w in ["studies", "happiness", "organization", "generously"]:
    print(f"{w:13} -> {lancaster.stem(w)}")

studies       -> study
happiness     -> happy
organization  -> org
generously    -> gen

8.6 Regexp Stemmer

from nltk.stem import RegexpStemmer

regexp = RegexpStemmer(r"ing$|s$|e$|able$", min=4)
for w in ["cats", "running", "caring", "comfortable"]:
    print(f"{w:12} -> {regexp.stem(w)}")

cats         -> cat
running      -> runn
caring       -> car
comfortable  -> comfort

The min=4 argument means words shorter than 4 characters are never stemmed, which protects words like "is" and "as".

8.7 Compare all stemmers at once

A small helper makes it easy to test stemmers on your own vocabulary before committing to one:

from nltk.stem import PorterStemmer, SnowballStemmer, LancasterStemmer

stemmers = {
    "Porter": PorterStemmer(),
    "Snowball": SnowballStemmer("english"),
    "Lancaster": LancasterStemmer(),
}

def compare(words):
    print(f"{'word':14}" + "".join(f"{n:12}" for n in stemmers))
    for w in words:
        print(f"{w:14}" + "".join(f"{s.stem(w):12}" for s in stemmers.values()))

compare(["argued", "fishing", "presumably", "university", "universe"])

word          Porter      Snowball    Lancaster
argued        argu        argu        argu
fishing       fish        fish        fish
presumably    presum      presum      presum
university    univers     univers     univers
universe      univers     univers     univers

8.8 Faster stemming for big data: PyStemmer

NLTK is written in pure Python, which is fine for thousands of documents but slow for tens of millions. PyStemmer wraps the official C implementation of the Snowball stemmers and is dramatically faster, while producing Snowball-compatible output:

pip install PyStemmer

import Stemmer

stemmer = Stemmer.Stemmer("english")
print(stemmer.stemWords(["running", "connection", "generously"]))
# ['run', 'connect', 'generous']

Use stemWords() on a whole list at once rather than calling a function per word — it avoids Python overhead and is the fastest way to stem large token lists. Libraries such as the BM25 package bm25s use PyStemmer for exactly this reason.

What about spaCy? spaCy deliberately does not ship a stemmer — it provides lemmatization instead (token.lemma_). If your project already uses spaCy, you can still run an NLTK or PyStemmer stemmer on token.text, but consider whether spaCy's lemmas already give you what you need.

9. A Complete Text Preprocessing Pipeline with Stemming

In real projects, stemming is one stage in a pipeline. The order of stages matters. A common and sensible order is:

Raw text→Lowercase→Remove noise (URLs, punctuation, numbers)→Tokenize→Remove stop words→Stem→Features / model

Why remove stop words before stemming? Because stop-word lists contain real words ("having", "was", "does"). If you stem first, "was" may become "wa" and no longer matches the list. Here is a clean, reusable implementation:

import re
from nltk.tokenize import word_tokenize
from nltk.corpus import stopwords
from nltk.stem import SnowballStemmer

STOP_WORDS = set(stopwords.words("english"))
stemmer = SnowballStemmer("english")

def preprocess(text: str) -> list[str]:
    # 1. Lowercase
    text = text.lower()
    # 2. Remove URLs, then anything that isn't a letter or space
    text = re.sub(r"https?://\S+|www\.\S+", " ", text)
    text = re.sub(r"[^a-z\s]", " ", text)
    # 3. Tokenize
    tokens = word_tokenize(text)
    # 4. Remove stop words and very short tokens
    tokens = [t for t in tokens if t not in STOP_WORDS and len(t) > 2]
    # 5. Stem
    return [stemmer.stem(t) for t in tokens]

print(preprocess(
    "Stemming algorithms are reducing inflected words to their root forms, "
    "making searches more effective!"
))

['stem', 'algorithm', 'reduc', 'inflect', 'word', 'root', 'form', 'make', 'search', 'effect']

Fifteen words became ten meaningful stems. A model now sees "search" and "effect" as features — which would also match "searching", "searched", "effective" and "effectively" in other documents.

9.1 Applying the pipeline to a pandas DataFrame

import pandas as pd

df = pd.DataFrame({
    "review": [
        "The delivery was amazingly fast and the packaging looked great",
        "Terrible packaging, items arrived damaged and broken",
        "Fast delivering, great prices, will be ordering again",
    ]
})

df["tokens"] = df["review"].apply(preprocess)
df["clean_text"] = df["tokens"].apply(" ".join)
print(df[["clean_text"]])

Store both the token list and a joined string: the list is handy for analysis, and the string is what most vectorizers expect.

9.2 Caching for speed

Real text repeats the same words constantly. Caching stems avoids recomputing them and can speed up large jobs considerably:

from functools import lru_cache

@lru_cache(maxsize=200_000)
def cached_stem(word: str) -> str:
    return stemmer.stem(word)

Swap stemmer.stem(t) for cached_stem(t) in the pipeline and you get the same results, faster.

10. Stemming with scikit-learn and TF-IDF

scikit-learn's CountVectorizer and TfidfVectorizer don't include stemming, but they accept a custom tokenizer. That's all you need:

import re
from nltk.corpus import stopwords
from nltk.stem import PorterStemmer
from sklearn.feature_extraction.text import TfidfVectorizer

stop_words = set(stopwords.words("english"))
porter = PorterStemmer()

def stem_tokenizer(text):
    tokens = re.findall(r"[a-z]+", text.lower())
    return [porter.stem(t) for t in tokens if t not in stop_words]

docs = [
    "The runner was running in the race",
    "Runners run every morning",
    "She loves connecting with connected people",
]

# Without stemming
plain = TfidfVectorizer(stop_words="english").fit(docs)
print(plain.get_feature_names_out())

# With stemming
stemmed = TfidfVectorizer(tokenizer=stem_tokenizer, token_pattern=None).fit(docs)
print(stemmed.get_feature_names_out())

['connected' 'connecting' 'loves' 'morning' 'people' 'race' 'run' 'runner'
 'runners' 'running']
['connect' 'everi' 'love' 'morn' 'peopl' 'race' 'run' 'runner']

The stemmed vocabulary has fewer, denser features: "connected" and "connecting" merged into connect, and "running" merged with "run". (You can also see that NLTK's and scikit-learn's stop-word lists differ — scikit-learn removed "every", NLTK did not. Always be deliberate about which list you use.) Setting token_pattern=None silences a warning, because the default pattern is ignored once you pass a custom tokenizer.

10.1 Putting it in a full classification pipeline

from sklearn.pipeline import Pipeline
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import cross_val_score

pipe = Pipeline([
    ("tfidf", TfidfVectorizer(tokenizer=stem_tokenizer, token_pattern=None,
                              ngram_range=(1, 2), min_df=2)),
    ("clf", LogisticRegression(max_iter=1000)),
])

# X = list of texts, y = labels
# scores = cross_val_score(pipe, X, y, cv=5, scoring="f1_macro")
# print(scores.mean())

Experiment properly: Train the same pipeline three times — no normalization, stemming, and lemmatization — and compare cross-validated scores. On small datasets stemming often gives a small boost; on large datasets the difference frequently disappears. Let the numbers decide, not habit.

One practical warning: the stemming function must be identical at training time and prediction time. If you train with Snowball and later deploy with Porter, many features will silently stop matching. Keeping the stemmer inside the scikit-learn Pipeline (as above) and saving the whole pipeline with joblib prevents this mistake.

11. Build a Simple Stemmer from Scratch

Writing a tiny stemmer yourself is the best way to truly understand the ideas of rules, conditions and ordering. This toy version is nowhere near Porter quality, but it shows the core mechanics in about 30 lines:

VOWELS = set("aeiou")

def has_vowel(stem: str) -> bool:
    return any(ch in VOWELS for ch in stem)

def simple_stem(word: str) -> str:
    w = word.lower()
    if len(w) <= 3:                      # protect short words
        return w

    # Step 1: plurals
    if w.endswith("sses"):
        w = w[:-2]                       # caresses -> caress
    elif w.endswith("ies"):
        w = w[:-3] + "y"                 # ponies -> pony
    elif w.endswith("s") and not w.endswith("ss"):
        w = w[:-1]                       # cats -> cat

    # Step 2: -ing / -ed, only if a vowel remains
    for suffix in ("ing", "ed"):
        if w.endswith(suffix) and has_vowel(w[:-len(suffix)]):
            w = w[:-len(suffix)]
            # undo doubled consonant: hopp -> hop
            if len(w) > 2 and w[-1] == w[-2] and w[-1] not in "lsz":
                w = w[:-1]
            break

    # Step 3: common derivational endings (longest first)
    for suffix, repl in (("ational", "ate"), ("ization", "ize"),
                         ("fulness", "ful"), ("ness", ""), ("ment", "")):
        if w.endswith(suffix) and len(w) - len(suffix) >= 3:
            w = w[:-len(suffix)] + repl
            break
    return w

for word in ["caresses", "ponies", "hopping", "relational",
             "happiness", "sing", "agreement"]:
    print(f"{word:11} -> {simple_stem(word)}")

caresses    -> caress
ponies      -> pony
hopping     -> hop
relational  -> relate
happiness   -> happi
sing        -> sing
agreement   -> agree

Look at how each design decision matters. The vowel check stops "sing" from becoming "s". Rule order matters: plurals go first so "connections" can lose its "s" before other rules run. The minimum-length check in Step 3 stops tiny words from being destroyed. Every real stemmer is essentially a carefully tuned, much larger version of this idea. A great exercise is to extend this function and compare its output against NLTK's Snowball stemmer on a list of 500 common words.

12. Multilingual Stemming (Including Arabic and Urdu)

English has relatively simple morphology. Many languages are far richer, and stemming becomes both more useful and more difficult.

12.1 European languages with Snowball

from nltk.stem import SnowballStemmer

print(SnowballStemmer.languages)
print(SnowballStemmer("spanish").stem("corriendo"))        # corr
print(SnowballStemmer("french").stem("continuellement"))   # continuel
print(SnowballStemmer("german").stem("katzen"))            # katz

NLTK's Snowball supports Arabic, Danish, Dutch, English, Finnish, French, German, Hungarian, Italian, Norwegian, Portuguese, Romanian, Russian, Spanish and Swedish (plus "porter" for the original English algorithm). For languages like Finnish and Hungarian, where a single word can carry many suffixes, stemming has an even bigger effect on vocabulary size than in English.

12.2 Arabic

Arabic words are built from (usually three-letter) roots placed into patterns, with prefixes, suffixes and sometimes infixes. Simple suffix stripping is not enough, so specialized stemmers exist. NLTK includes the ISRI root-based stemmer:

from nltk.stem.isri import ISRIStemmer

print(ISRIStemmer().stem("المكتبات"))   # "the libraries" -> root كتب (k-t-b, "writing")

Researchers distinguish root-based stemming (reducing to the core root, very aggressive) from light stemming (only removing common prefixes and suffixes). For Arabic search, light stemming often gives better precision, because many unrelated words share a root.

12.3 Urdu and other South Asian languages

For readers working with Urdu, Pashto, Hindi or Punjabi data, there is an important practical point: mainstream Python libraries do not ship a widely adopted Urdu stemmer. Urdu combines Persian, Arabic and native morphology, with plural endings (such as ـوں and ـیں), case markers and a mix of borrowed suffixes. Several research stemmers have been proposed in academic papers, but none has become a standard the way Porter has for English.

Your realistic options are:

1. Write a light rule-based stemmer with a curated list of common suffixes (using the from-scratch pattern in Section 11), and evaluate it on a hand-labeled word list.
2. Skip stemming and use subword tokenization (SentencePiece or a BPE tokenizer), which learns useful word pieces from data and handles any script.
3. Use a multilingual transformer model (for example, a multilingual BERT-family encoder) whose tokenizer already handles morphology implicitly.

For a final-year project or thesis on Urdu NLP, building and evaluating a light stemmer is a genuinely useful and publishable contribution.

Never use an English stemmer on non-English text. Porter will happily chop "s" and "ing" off Romanized Urdu or Spanish words and produce garbage. Detect the language first, then pick a matching stemmer — or none at all.

13. Real-World Applications of Stemming

13.1 Search engines and site search

This is stemming's home ground. Elasticsearch and OpenSearch let you add stemming to an analyzer with a few lines of configuration (for example the porter_stem, snowball, kstem or language-specific stemmer token filters). Both documents and queries pass through the same analyzer, so "connecting databases" matches "database connection". PostgreSQL full-text search uses Snowball stemmers inside to_tsvector('english', ...), and SQLite's FTS5 offers a Porter tokenizer.

13.2 Hybrid search for RAG systems

Modern retrieval-augmented generation (RAG) systems commonly combine dense vector search with classic keyword search using BM25. The BM25 side almost always benefits from stemming, because it relies on exact token matches. If you are building a chatbot over your company's documents, a stemmed BM25 index alongside embeddings often catches exact terms — product codes, names, error messages — that vectors miss.

13.3 Text classification and spam filtering

Naive Bayes spam filters, support-ticket routers, and news categorizers built with TF-IDF features use stemming to merge variants like "winner", "winning" and "wins", giving each feature more training examples.

13.4 Sentiment analysis

Stemming can help classical sentiment models by merging "loved", "loving" and "loves". Be careful, though: lexicon-based tools such as VADER expect real words, so stemmed input will break their dictionary lookups.

13.5 Topic modeling and clustering

LDA and k-means clustering on TF-IDF vectors are very sensitive to vocabulary size. Stemming reduces fragmentation, often producing cleaner, more coherent topics. Many practitioners keep a mapping from each stem to its most frequent original word, so topic keywords can be displayed as "election, campaign, vote" instead of "elect, campaign, vote".

13.6 Keyword extraction and SEO tools

SEO and content tools stem keywords to group variants ("optimize", "optimizing", "optimization") when measuring keyword density or comparing a page against competitors. This gives a truer picture of how often a topic is covered.

13.7 Plagiarism and duplicate detection

Simple paraphrasing often changes word forms — "they analyzed the results" vs "analyzing the result". Stemming before computing similarity (for example, with shingles or cosine similarity) helps detect these lightly rewritten copies.

14. Is Stemming Still Relevant in 2026?

With powerful large language models everywhere, it's fair to ask whether a 1980 algorithm still deserves a place in your toolkit. The honest answer is: stemming is no longer used inside modern neural models, but it is still very much alive around them.

Where you should NOT stem

Transformer models like BERT, GPT-style LLMs and modern embedding models use subword tokenizers (BPE, WordPiece, SentencePiece). These tokenizers already break rare words into meaningful pieces, and the models learn from the full, unstemmed context. Feeding them stemmed text such as "the runner wa run in the race" actually removes information they rely on — tense, plurality and grammar — and generally makes results worse. So:

• Do not stem text before fine-tuning or prompting a transformer.
• Do not stem text before creating dense embeddings for semantic search.
• Do not stem text you plan to display to users or generate from.

Where stemming is still the right tool

• Keyword and BM25 search — including the keyword half of hybrid RAG systems.
• Database full-text search — PostgreSQL, SQLite FTS5, Elasticsearch and OpenSearch analyzers.
• Classical ML baselines — TF-IDF + Logistic Regression or Naive Bayes is still a strong, cheap, explainable baseline that you should benchmark before reaching for an LLM.
• Edge and low-resource settings — mobile apps, embedded devices and offline tools where running a large model is impractical.
• Data analysis and exploration — quickly counting topic frequencies across millions of documents.
• Teaching and interviews — stemming remains core NLP knowledge and a very common exam and interview topic.

The 2026 mindset: Treat stemming as a precision tool for lexical systems (token matching, counting, sparse vectors), not as a universal preprocessing step. Neural systems want raw text; lexical systems love stemmed text. Many production systems use both side by side.

15. How to Evaluate a Stemmer

Choosing a stemmer by gut feeling is a common mistake. Here are practical ways to evaluate one.

15.1 Task-based (extrinsic) evaluation — the most important

Plug each stemmer into your real system and measure what you actually care about:

• For search: precision, recall, Mean Average Precision (MAP), nDCG or MRR on a set of test queries with known relevant documents.
• For classification: accuracy, macro-F1, or AUC using cross-validation.
• For clustering/topic modeling: topic coherence scores and human inspection of topics.

15.2 Intrinsic evaluation

You can also evaluate a stemmer directly using hand-made groups of words that should share a stem (e.g. connect / connected / connection) and words that should not (e.g. university / universe). Then measure:

• Under-stemming index — how often words in the same group get different stems.
• Over-stemming index — how often words from different groups get the same stem.
• Index compression factor — how much the vocabulary shrinks: (unique words − unique stems) ÷ unique words.

Paice's evaluation method formalizes these ideas and is widely used in stemming research — useful if you are writing an academic paper or a thesis.

15.3 A quick vocabulary-reduction check in Python

def compression(tokens, stemmer):
    unique_words = set(tokens)
    unique_stems = {stemmer.stem(t) for t in unique_words}
    return 1 - len(unique_stems) / len(unique_words)

# tokens = a big list of lowercase words from your corpus
# for name, s in stemmers.items():
#     print(name, round(compression(tokens, s), 3))

Expect Lancaster to give the highest compression and Snowball the lowest of the three. Higher compression is not automatically better — it just tells you how aggressive each stemmer is on your data.

16. Best Practices and Common Mistakes

Best practices

1. Default to Snowball for English. It's balanced, maintained and fixes several Porter quirks.
2. Lowercase before stemming. Most stemmers expect lowercase input, and inconsistent casing splits your vocabulary.
3. Remove stop words before stemming, so stop-word lists still match their real-word entries.
4. Use the same stemmer everywhere — training, prediction, indexing and querying. Wrap it in a pipeline and save the pipeline.
5. Protect special tokens. Product codes, usernames, hashtags, chemical names and acronyms should often skip stemming. Keep a "do not stem" list for your domain.
6. Keep a stem-to-word map for display. Show users "election", not "elect".
7. Cache stems for large corpora, or use PyStemmer for raw speed.
8. Measure, don't assume. Compare no stemming, stemming and lemmatization on your validation set.

Common mistakes

• Stemming text before sending it to an LLM or embedding model. This removes useful information and usually lowers quality.
• Expecting stems to be real words. "studi" and "happi" are correct outputs, not bugs.
• Using Lancaster by default. Its aggressive cuts surprise people (caring → car, news → new). Choose it deliberately.
• Stemming only the documents, not the queries (or vice versa) in a search system. Both sides must be normalized identically.
• Applying an English stemmer to other languages or to mixed-language text without detection.
• Stemming before named-entity recognition. Run NER on the original text — "Wells Fargo" should not become "well fargo".
• Feeding stemmed text into lexicon-based tools such as VADER sentiment or spell-checkers that expect dictionary words.

17. Frequently Asked Questions

What is stemming in NLP in simple words?

Stemming is cutting the ends off words so that related forms look the same to a computer. "Playing", "played" and "plays" all become "play". It's a fast, rule-based way to normalize text.

What is the difference between stemming and lemmatization?

Stemming uses rules to chop suffixes and may produce non-words (studies → studi). Lemmatization uses a dictionary and grammar to return the real base word (studies → study, ran → run). Stemming is faster; lemmatization is more accurate.

Which stemmer is best for English?

For most projects, the Snowball (Porter2) English stemmer is the best default. Use Porter to match older systems or research, Lancaster when you need maximum vocabulary reduction, and Krovetz when readable output matters in search.

Why does the Porter stemmer produce words that aren't real?

Because it was designed to produce consistent stems, not dictionary words. As long as "happy", "happiness" and "happily" map to the same stem, the goal is achieved for search and classification — even if that stem is "happi".

Does stemming improve machine learning accuracy?

Sometimes. It often helps classical models (TF-IDF, Naive Bayes, Logistic Regression) on small or medium datasets by reducing sparsity. On large datasets the gain is often small, and with transformer models it usually hurts. Always test on your own data.

Should I use stemming with BERT, GPT or other LLMs?

No. Modern transformer models use subword tokenizers and learn from raw text. Stemming removes grammatical information they depend on. Use stemming for keyword/BM25 search and classical models, not for neural model inputs.

What are over-stemming and under-stemming?

Over-stemming merges words with different meanings (university and universe → univers). Under-stemming fails to merge words that belong together (ran and running). Aggressive stemmers over-stem more; light stemmers under-stem more.

Is stemming case-sensitive?

Most stemmers are designed for lowercase input. NLTK's Porter and Snowball stemmers lowercase their output by default, but you should still lowercase text explicitly in your pipeline for consistency.

Can I use stemming for Urdu or Arabic?

For Arabic, yes — NLTK provides the ISRI stemmer and an Arabic Snowball stemmer. For Urdu there is no widely adopted standard library stemmer, so you would typically write a light rule-based stemmer or use subword tokenization and multilingual models instead.

Does Google use stemming?

Search engines have long matched word variants — a search for "running shoes" also finds "run" and "runner". Modern web search relies heavily on neural language understanding, so word-variant matching is now one part of a much larger system rather than a standalone stemmer.

How fast is stemming?

Very fast. Pure-Python NLTK stemmers handle many thousands of words per second, and C-based libraries like PyStemmer are much faster still — easily enough to stem millions of documents.

18. Conclusion

Stemming is one of those rare techniques that is both decades old and still genuinely useful. At its heart it is a simple idea — strip suffixes so that related words look the same — but the details reveal real engineering trade-offs between speed, recall, precision and readability.

Here's what to take away from this guide:

• Stemming reduces words to stems using rules; stems don't have to be real words.
• Lemmatization is slower but returns real dictionary words and handles irregular forms.
• Porter is the classic, Snowball is the modern default, Lancaster is the aggressive option, and Regexp is for custom rules.
• Watch out for over-stemming (merging unrelated words) and under-stemming (missing related ones).
• In Python, NLTK makes stemming a one-liner, scikit-learn accepts a custom stemming tokenizer, and PyStemmer gives you speed at scale.
• In 2026, use stemming for lexical systems — BM25, full-text search, TF-IDF models, hybrid RAG — and skip it for neural inputs.

The best next step is to open a notebook, take a dataset you care about, and train a simple TF-IDF classifier three ways: no normalization, Snowball stemming, and lemmatization. Compare the scores. In fifteen minutes you'll have real evidence about what works for your data — and a much deeper understanding of text preprocessing than any definition can give you.

Found this guide useful? Bookmark it, share it with a classmate preparing for an NLP exam, and leave a comment with the stemmer you ended up choosing for your project — and why.