Parts of Speech (POS) Tagging in NLP: Complete Guide with Python Examples (2026)

Parts of Speech (POS) Tagging in NLP: Complete Guide with Python Examples (2026)

Read the sentence "Time flies like an arrow." You instantly understand that "time" is the subject, "flies" is the verb, and "like an arrow" describes how it moves. Now read "Fruit flies like a banana." Suddenly "flies" is a noun (the insects), and "like" is a verb meaning "enjoy." The words look almost identical, yet their grammatical roles are completely different. Humans resolve this ambiguity without effort. Computers need a dedicated technique to do it: Part-of-Speech (POS) tagging.

POS tagging is one of the oldest and most important tasks in Natural Language Processing. It labels every word in a sentence with its grammatical category — noun, verb, adjective, adverb, and so on — based on both the word itself and its context. Those labels power lemmatization, parsing, named entity recognition, information extraction, grammar checkers, text-to-speech systems, search engines, and countless analytics pipelines. In this complete 2026 guide you will learn what POS tagging is, which tagsets are used in practice, how the classic algorithms (rule-based, HMM with the Viterbi algorithm, perceptron, CRF) and modern neural and transformer taggers work, and how to use them in Python with NLTK, spaCy, Stanza, Flair, and Hugging Face. You will also build an HMM tagger from scratch, train and evaluate your own tagger, and finish with practical mini-projects.

Quick Summary (TL;DR)

• POS tagging assigns a grammatical label (noun, verb, adjective…) to every token in a sentence, using context to resolve ambiguity.
• The two most common tagsets are Penn Treebank (36 fine-grained English tags such as NN, VBD, JJR) and Universal POS (UPOS) (17 cross-lingual tags such as NOUN, VERB, ADJ).
• Classic algorithms include rule-based taggers, Hidden Markov Models with Viterbi decoding, averaged perceptrons, and CRFs; modern taggers use BiLSTMs and transformers.
• In Python: NLTK for learning, spaCy for fast production, Stanza for multilingual accuracy, Flair and Hugging Face for transformer-level performance.
• Modern English taggers reach roughly 97–98% token accuracy on news text, but accuracy drops on social media, domain-specific, and code-mixed text — so always evaluate on your own data.

Table of Contents

1. What Is POS Tagging in NLP?
2. The Parts of Speech in English
3. POS Tagsets: Penn Treebank vs Universal Dependencies
4. Why POS Tagging Matters: Applications
5. Why POS Tagging Is Hard
6. POS Tagging Approaches and Algorithms
7. HMM and the Viterbi Algorithm Explained
8. Build an HMM POS Tagger from Scratch in Python
9. POS Tagging with NLTK
10. POS Tagging with spaCy
11. POS Tagging with Stanza
12. POS Tagging with Flair and Hugging Face Transformers
13. Train and Evaluate Your Own POS Tagger
14. Evaluation Metrics and Error Analysis
15. Practical Projects Using POS Tags
16. Library Comparison Table
17. Common Mistakes and Best Practices
18. Frequently Asked Questions
19. Conclusion

1. What Is POS Tagging in NLP?

Part-of-Speech tagging (also called grammatical tagging or word-category disambiguation) is the task of assigning each word in a text the grammatical category it belongs to in that specific context. The input is a sequence of tokens; the output is a sequence of tags of the same length. Because the output is aligned one-to-one with the input, POS tagging is the textbook example of a sequence labeling problem — the same family of problems that includes named entity recognition and chunking.

Input : The   quick  brown  fox   jumps  over  the   lazy  dog   .
Penn  : DT    JJ     JJ     NN    VBZ    IN    DT    JJ    NN    .
UPOS  : DET   ADJ    ADJ    NOUN  VERB   ADP   DET   ADJ   NOUN  PUNCT

The key phrase is in context. A dictionary can tell you that "book" can be a noun or a verb, but only the sentence tells you which one applies: "I read a book" (noun) versus "Please book a table" (verb). A POS tagger is essentially a disambiguation machine. It knows the possible tags for each word and uses the surrounding words to choose the most likely one.

A short history

POS tagging has a long research history that mirrors the history of NLP itself. Early systems in the 1960s and 1970s, such as the tagger built for the Brown Corpus, used hand-written rules. In the 1980s and 1990s, statistical methods took over: Hidden Markov Models achieved strong accuracy by learning tag sequences from annotated corpora, and Eric Brill's transformation-based tagger showed that rules could be learned automatically. The 2000s brought discriminative models such as Maximum Entropy taggers, Conditional Random Fields, and the averaged perceptron, pushing English accuracy close to 97%. In the 2010s, neural networks (BiLSTMs with character embeddings) and then pre-trained transformers like BERT pushed accuracy a little higher and, more importantly, made taggers more robust to unknown words and new domains. Today, POS tagging is considered largely "solved" for well-edited English, but it remains an active, practical problem for low-resource languages, noisy text, and specialized domains.

2. The Parts of Speech in English

Traditional English grammar describes eight or nine parts of speech. Computational tagsets refine these into finer categories, but the classic categories are still the best place to start.

Part of speech	Function	Examples
Noun	Names a person, place, thing, or idea	teacher, Lahore, laptop, freedom
Pronoun	Replaces a noun	she, they, it, himself, who
Verb	Expresses an action or state	run, write, is, became, seems
Adjective	Describes or modifies a noun	fast, blue, larger, happiest
Adverb	Modifies a verb, adjective, or other adverb	quickly, very, never, here
Preposition (adposition)	Shows a relationship between words	in, on, under, between, of
Conjunction	Connects words, phrases, or clauses	and, but, or, because, although
Interjection	Expresses emotion	wow, oh, ouch, hey
Determiner / Article	Introduces and specifies a noun	the, a, an, this, every, some

Linguists also distinguish between open classes and closed classes. Open classes — nouns, verbs, adjectives, and adverbs — constantly gain new members ("to google," "a selfie," "an LLM"). Closed classes — pronouns, prepositions, conjunctions, determiners — rarely change. This matters for taggers: closed-class words are easy because the tagger has seen all of them many times, while open-class words include every new name, product, and slang term the tagger has never seen. Handling these unknown words well is one of the main things that separates good taggers from mediocre ones.

3. POS Tagsets: Penn Treebank vs Universal Dependencies

A tagset is the inventory of labels a tagger can assign. Different corpora and libraries use different tagsets, and confusing them is one of the most common sources of bugs. The two you will meet most often are the Penn Treebank tagset and the Universal POS tagset.

3.1 The Penn Treebank tagset (36 word tags)

The Penn Treebank (PTB) tagset was created for the Penn Treebank project in the early 1990s and remains the default for English in NLTK and in spaCy's fine-grained token.tag_ attribute. It encodes extra grammatical detail such as number for nouns, tense and form for verbs, and degree for adjectives and adverbs. It defines 36 word-level tags, plus several tags for punctuation.

Tag	Description	Example	Tag	Description	Example
CC	Coordinating conjunction	and, but	PRP$	Possessive pronoun	my, their
CD	Cardinal number	three, 2026	RB	Adverb	quickly
DT	Determiner	the, a	RBR	Adverb, comparative	faster
EX	Existential there	there (is)	RBS	Adverb, superlative	fastest
FW	Foreign word	déjà vu	RP	Particle	(give) up
IN	Preposition / subordinating conj.	in, because	SYM	Symbol	%, +
JJ	Adjective	big	TO	"to"	to
JJR	Adjective, comparative	bigger	UH	Interjection	oh, wow
JJS	Adjective, superlative	biggest	VB	Verb, base form	run
LS	List item marker	1), a)	VBD	Verb, past tense	ran
MD	Modal	can, should	VBG	Verb, gerund / present participle	running
NN	Noun, singular or mass	dog, water	VBN	Verb, past participle	eaten
NNS	Noun, plural	dogs	VBP	Verb, non-3rd person singular present	run (I run)
NNP	Proper noun, singular	Peshawar	VBZ	Verb, 3rd person singular present	runs
NNPS	Proper noun, plural	Americans	WDT	Wh-determiner	which, that
PDT	Predeterminer	all (the), both	WP	Wh-pronoun	who, what
POS	Possessive ending	's	WP$	Possessive wh-pronoun	whose
PRP	Personal pronoun	I, she, they	WRB	Wh-adverb	where, when

Memory trick: The first letter usually tells you the coarse class — N* = nouns, V* = verbs, J* = adjectives, R* = adverbs, W* = wh-words. That is exactly why the Penn-to-WordNet mapping used for lemmatization only checks the first character of the tag.

3.2 The Universal POS tagset (UPOS, 17 tags)

Fine-grained tagsets differ from language to language, which made cross-lingual NLP painful. The Universal Dependencies (UD) project introduced a single set of 17 coarse tags that apply to every language. UPOS is used by Stanza, by spaCy's token.pos_ attribute, and by the UD treebanks covering well over 100 languages.

UPOS	Meaning	Examples
ADJ	Adjective	big, old, green
ADP	Adposition (preposition / postposition)	in, to, during; Urdu "میں", "سے"
ADV	Adverb	very, tomorrow, well
AUX	Auxiliary verb	is, has (done), will, should
CCONJ	Coordinating conjunction	and, or, but
DET	Determiner	a, the, this, every
INTJ	Interjection	psst, ouch, hello
NOUN	Noun	girl, tree, idea
NUM	Numeral	1, 2026, seven
PART	Particle	's, not, to (infinitive)
PRON	Pronoun	I, you, herself, who
PROPN	Proper noun	Mary, Karachi, Google
PUNCT	Punctuation	. , ( ) ?
SCONJ	Subordinating conjunction	if, while, that (complementizer)
SYM	Symbol	$, %, §, ©, :)
VERB	Verb	run, eat, think
X	Other	foreign words, typos, unanalyzable tokens

3.3 Which tagset should you use?

Use UPOS when you work with multiple languages, need simple coarse categories for filtering (for example, "keep only nouns and adjectives"), or want consistency across tools. Use Penn Treebank when you need finer English distinctions such as tense (VBD vs VBZ), number (NN vs NNS), or degree (JJ vs JJR vs JJS). spaCy gives you both at once: token.pos_ holds the UPOS tag and token.tag_ holds the fine-grained Penn tag. UD treebanks also include rich morphological features (such as Tense=Past, Number=Plur), which give you the detail of Penn tags in a cross-lingual format.

4. Why POS Tagging Matters: Applications

POS tags are rarely the final product. They are a building block that makes almost every higher-level NLP task easier and more accurate.

• Lemmatization: Knowing that "saw" is a verb lets the lemmatizer return "see" instead of "saw." This is why every serious lemmatizer depends on a tagger. (See our Lemmatization in NLP guide for details.)
• Syntactic parsing: Dependency and constituency parsers use POS tags as core features to decide which words attach to which.
• Named Entity Recognition: Proper-noun tags (NNP, PROPN) are strong hints for entity boundaries. Our NLP Text Tagging: POS and NER guide covers this connection in depth.
• Information extraction: Patterns such as "adjective + noun" or "noun + verb + noun" extract key phrases and simple relations from unstructured text.
• Keyword and keyphrase extraction: Algorithms like RAKE, TextRank, and YAKE-style systems often restrict candidates to noun phrases identified with POS tags.
• Sentiment and opinion mining: Adjectives and adverbs carry much of the sentiment in reviews; aspect-based sentiment links adjectives to the nouns they describe ("battery is weak," "screen is bright").
• Text-to-speech: Pronunciation depends on POS: "reCORD" (verb) vs "RECord" (noun), "preSENT" vs "PRESent," "read" (present) vs "read" (past).
• Grammar and style checking: Tools detect errors like "a apple," passive voice overuse, or subject-verb disagreement using tag patterns.
• Machine translation and word sense disambiguation: Knowing a word's category narrows its possible meanings and translations.
• Search and query understanding: Search engines use tags to identify the head noun of a query ("cheap flights to Dubai") and to boost the most important terms.

5. Why POS Tagging Is Hard

If most words have only one possible tag, why is POS tagging a research problem at all? Because the hard cases are extremely common. In English, many of the most frequent words are ambiguous, so a large share of the tokens in running text could take more than one tag.

5.1 Lexical ambiguity

Words such as "back," "set," "run," "like," "that," "round," and "down" can belong to many categories. "Back" alone can be an adjective ("the back door"), a noun ("my back hurts"), an adverb ("come back"), or a verb ("back the project").

5.2 Unknown and rare words

New words appear constantly: product names, usernames, hashtags, technical jargon, borrowings from other languages. A tagger must guess their category from clues such as capitalization, suffixes ("-tion" suggests a noun, "-ly" an adverb, "-ize" a verb), digits, hyphens, and context.

5.3 Domain shift

Most English taggers were trained largely on news and web text. Biomedical abstracts, legal contracts, source-code comments, and chat messages have different vocabulary and grammar, and accuracy can drop noticeably when you move to them.

5.4 Noisy and informal text

Social media posts break capitalization rules, drop words, use emojis and abbreviations ("u r gr8"), and mix languages. Code-mixed text such as Roman Urdu with English ("yaar this movie was bohat boring") is especially challenging because standard taggers are trained on single-language, standard-script text.

5.5 Tokenization errors

A tagger can only be as good as its tokens. If "can't" is split into "ca" and "n't" by one tool and kept whole by another, or if "New York-based" is split unexpectedly, tags will be inconsistent. Always tag with the same tokenizer that the tagger was trained with.

5.6 Annotation conventions

Even human annotators disagree on some tags. Is "that" in "I know that she left" a determiner, pronoun, or subordinating conjunction? Different tagsets answer differently. This sets a practical ceiling on achievable accuracy; inter-annotator agreement on English is high but not perfect.

6. POS Tagging Approaches and Algorithms

Over six decades, researchers have developed several families of POS tagging algorithms. Each new family improved accuracy and robustness, but the older methods are still worth knowing: they are fast, interpretable, and often good enough, and they explain the ideas that modern neural taggers build on.

6.1 Rule-based tagging

Rule-based taggers start with a lexicon listing the possible tags for each word, then apply hand-written disambiguation rules. A typical rule might be: "If an ambiguous word follows a determiner, choose the noun reading" — so in "the book," "book" becomes a noun. Rule-based systems are transparent and need no training data, but writing and maintaining thousands of rules is slow, and the rules break on informal text.

6.2 Transformation-based learning (Brill tagger)

Eric Brill's transformation-based tagger combined the best of rules and statistics. It starts by giving every word its most frequent tag, then learns an ordered list of correction rules from annotated data, such as "change NN to VB when the previous tag is TO." The learned rules are human-readable, which made the approach popular for teaching and debugging. NLTK includes an implementation in nltk.tag.brill.

6.3 Stochastic (probabilistic) tagging: n-grams and HMMs

Statistical taggers learn probabilities from a tagged corpus. The simplest is a unigram tagger, which always picks a word's most frequent tag — surprisingly, this alone gets around 90% accuracy on English news text, because so many words are unambiguous or strongly biased. Bigram and trigram taggers also look at the previous one or two tags. The Hidden Markov Model (HMM) formalizes this idea and finds the best tag sequence for an entire sentence at once, using the Viterbi algorithm. HMM-based taggers such as TnT ("Trigrams'n'Tags") reached accuracy in the mid-to-high 90s on English and remained competitive for years.

6.4 Discriminative models: MaxEnt, perceptron, CRF

HMMs are generative: they model how words are produced from tags, which makes it awkward to add rich features. Discriminative models instead directly model the probability (or score) of tags given the words, and can use arbitrary overlapping features: the word itself, its prefix and suffix, whether it is capitalized, whether it contains a digit or hyphen, the neighboring words, and previous tags.

• Maximum Entropy (MaxEnt) taggers use logistic regression over these features, one decision per token.
• The averaged perceptron learns feature weights with a simple, fast online algorithm and greedy or beam-search decoding. NLTK's default English tagger, PerceptronTagger, is based on this approach, and it is fast and accurate.
• Conditional Random Fields (CRFs) model the whole tag sequence jointly, combining rich features with global decoding. CRFs were the state of the art for many sequence-labeling tasks before deep learning.

6.5 Neural taggers: BiLSTM and BiLSTM-CRF

Neural taggers replace hand-crafted features with learned representations. A typical architecture embeds each word (often combining a word embedding with a character-level CNN or LSTM to handle unknown words), runs a bidirectional LSTM over the sentence so each position sees both left and right context, and predicts a tag with a softmax or CRF layer. These models are robust to unseen words and need little feature engineering.

6.6 Transformer-based taggers

Today's most accurate taggers fine-tune pre-trained transformers such as BERT, RoBERTa, XLM-RoBERTa, or DeBERTa as token classification models. The pre-trained model already understands syntax remarkably well, so a small linear layer on top of its contextual embeddings achieves top accuracy with modest training data. The trade-offs are size and speed: transformer taggers are much heavier than a perceptron or a small spaCy model, so they are best when accuracy matters more than throughput, or when a GPU is available.

6.7 LLM-based tagging

You can also prompt a general-purpose large language model to tag a sentence and return JSON. This is handy for quick experiments, unusual languages, or creating seed annotations, but it is slower, costlier, and less consistent than a dedicated tagger, and LLMs sometimes change tokenization or invent tags. For production pipelines, a trained tagger is still the right tool.

Approach	Typical English accuracy*	Speed	Needs training data?	Strength
Most-frequent-tag baseline	~90%	Fastest	Yes (counts)	Trivial baseline
Rule-based / Brill	~93–96%	Fast	Rules or data	Interpretable
HMM (bigram/trigram)	~95–96.5%	Fast	Yes	Simple, global decoding
Averaged perceptron / MaxEnt / CRF	~96.5–97.3%	Fast	Yes	Rich features
BiLSTM-CRF	~97.3–97.6%	Medium	Yes	Robust to unknown words
Fine-tuned transformer	~97.5–98%	Slower	Yes (less)	Best accuracy, transfer learning

*Approximate ranges reported on standard English news benchmarks (Penn Treebank Wall Street Journal sections). Exact numbers depend on data splits, tagsets, and implementation. Expect lower numbers on out-of-domain text.

7. HMM and the Viterbi Algorithm Explained

The Hidden Markov Model is the most famous classical POS tagging algorithm, and understanding it gives you intuition for every sequence-labeling model that followed. Let us build that intuition step by step.

7.1 The model

An HMM assumes that a hidden sequence of states (the POS tags) generates an observed sequence of emissions (the words). It has two kinds of probabilities, both estimated by counting in a tagged corpus:

• Transition probabilities P(tag_i | tag_{i-1}): how likely one tag is to follow another. For example, P(NOUN | DET) is high, because nouns often follow determiners, while P(DET | DET) is low.
• Emission probabilities P(word | tag): how likely a tag is to produce a given word. For example, P("book" | NOUN) is much larger than P("book" | VERB), simply because "book" is used as a noun far more often.

Under the HMM's simplifying assumptions (each tag depends only on the previous tag, and each word depends only on its own tag), the probability of a tag sequence T for a sentence W is the product, over all positions, of the transition probability times the emission probability. Tagging means finding the sequence T that makes this product as large as possible.

7.2 Why we need Viterbi

A 20-word sentence with 17 possible tags per word has 17²⁰ possible tag sequences — far too many to enumerate. The Viterbi algorithm solves this with dynamic programming. It moves left to right through the sentence, and for every position and every tag, it keeps only the best-scoring path that ends in that tag, plus a backpointer to where that path came from. Because the HMM only looks one tag back, the best path to tag t at position i must extend one of the best paths at position i−1. This reduces the work from exponential to O(n × k²), where n is the sentence length and k the number of tags.

7.3 A small worked example

Consider the two-word sentence "Fish swim" with just two tags, NOUN and VERB. Suppose our counts gave these probabilities:

Probability	Value	Probability	Value
P(NOUN | start)	0.7	P(VERB | start)	0.3
P(VERB | NOUN)	0.6	P(NOUN | NOUN)	0.4
P(NOUN | VERB)	0.5	P(VERB | VERB)	0.5
P("fish" | NOUN)	0.02	P("fish" | VERB)	0.01
P("swim" | NOUN)	0.001	P("swim" | VERB)	0.02

Step 1 — first word "fish": score(NOUN) = 0.7 × 0.02 = 0.014; score(VERB) = 0.3 × 0.01 = 0.003.

Step 2 — second word "swim": For VERB at position 2, the best previous tag is NOUN: 0.014 × 0.6 × 0.02 = 0.000168 (coming from VERB would give 0.003 × 0.5 × 0.02 = 0.00003). For NOUN at position 2, the best is again from NOUN: 0.014 × 0.4 × 0.001 = 0.0000056.

Step 3 — backtrack: The highest final score is VERB (0.000168), whose backpointer is NOUN. The best sequence is therefore NOUN VERB — "Fish (noun) swim (verb)." Real taggers do exactly this, just with 17 or 45 tags, tens of thousands of words, and log-probabilities instead of raw products to avoid numerical underflow.

7.4 Smoothing and unknown words

If a word never appeared with a tag in training, its emission probability is zero, and a single zero wipes out the whole path. Practical HMMs therefore use smoothing (for example, add-alpha or Laplace smoothing) and special handling for unknown words, such as estimating emission probabilities from suffixes, capitalization, and digits. Trigram HMMs like TnT also interpolate trigram, bigram, and unigram transition estimates to balance context and data sparsity.

8. Build an HMM POS Tagger from Scratch in Python

Nothing beats implementing the algorithm yourself. The following tagger trains on the Penn Treebank sample that ships with NLTK (mapped to Universal tags), uses add-alpha smoothing, and decodes with Viterbi in log space. It is around 70 lines and runs in seconds.

import math
from collections import Counter, defaultdict
import nltk
from nltk.corpus import treebank

nltk.download("treebank")
nltk.download("universal_tagset")

class HMMTagger:
    START, END = "<S>", "</S>"

    def __init__(self, alpha: float = 0.01):
        self.alpha = alpha

    def fit(self, tagged_sents):
        self.trans = defaultdict(Counter)   # prev_tag -> Counter(next_tag)
        self.emit = defaultdict(Counter)    # tag -> Counter(word)
        for sent in tagged_sents:
            prev = self.START
            for word, tag in sent:
                self.trans[prev][tag] += 1
                self.emit[tag][word.lower()] += 1
                prev = tag
            self.trans[prev][self.END] += 1
        self.tags = sorted(self.emit)
        self.vocab = {w for t in self.emit for w in self.emit[t]}
        self.trans_tot = {p: sum(c.values()) for p, c in self.trans.items()}
        self.emit_tot = {t: sum(c.values()) for t, c in self.emit.items()}
        return self

    def log_t(self, prev, tag):
        n_next = len(self.tags) + 1
        num = self.trans[prev][tag] + self.alpha
        den = self.trans_tot.get(prev, 0) + self.alpha * n_next
        return math.log(num / den)

    def log_e(self, tag, word):
        num = self.emit[tag][word.lower()] + self.alpha
        den = self.emit_tot[tag] + self.alpha * (len(self.vocab) + 1)
        return math.log(num / den)

    def tag(self, words):
        if not words:
            return []
        V = [{t: self.log_t(self.START, t) + self.log_e(t, words[0]) for t in self.tags}]
        back = [{t: None for t in self.tags}]
        for i in range(1, len(words)):
            V.append({})
            back.append({})
            for t in self.tags:
                prev, score = max(((p, V[i - 1][p] + self.log_t(p, t)) for p in self.tags),
                                  key=lambda x: x[1])
                V[i][t] = score + self.log_e(t, words[i])
                back[i][t] = prev
        last = max(self.tags, key=lambda t: V[-1][t] + self.log_t(t, self.END))
        path = [last]
        for i in range(len(words) - 1, 0, -1):
            path.append(back[i][path[-1]])
        return list(zip(words, reversed(path)))

# Train / test split
sents = list(treebank.tagged_sents(tagset="universal"))
cut = int(len(sents) * 0.9)
train, test = sents[:cut], sents[cut:]
hmm = HMMTagger(alpha=0.01).fit(train)

print(hmm.tag("The old man the boats".split()))

# Evaluate token accuracy
correct = total = 0
for sent in test:
    words = [w for w, _ in sent]
    for (_, gold), (_, pred) in zip(sent, hmm.tag(words)):
        correct += gold == pred
        total += 1
print(f"HMM accuracy: {correct / total:.2%}")

On this small training set, the tagger typically scores in the low 90s. Most of its errors come from unknown words: because we smooth emissions uniformly, an unseen word gives the tagger almost no lexical evidence, so it relies purely on transitions. Try these improvements as exercises — each one teaches an important lesson:

1. Add a suffix-based emission model for unknown words (e.g., words ending in "-ly" → ADV, "-ing" → VERB, capitalized → PROPN).
2. Upgrade from bigram to trigram transitions with interpolation.
3. Train on the full Penn Treebank or a large UD English treebank instead of the 10% sample.
4. Replace the uniform alpha with tuned smoothing values chosen on a development set.

Garden-path fun: "The old man the boats" is a famous garden-path sentence where "man" is a verb ("the old [people] staff the boats"). Most taggers — including ours — tag "man" as a noun, which shows how strongly statistics favor the common reading. Our toy HMM even tags the unseen word "boats" as punctuation, because with no lexical evidence it simply picks the tag that most often ends a sentence — a vivid demonstration of why unknown-word handling matters so much.

9. POS Tagging with NLTK

NLTK provides a ready-to-use English tagger through nltk.pos_tag(). Under the hood, it is an averaged perceptron trained on Penn Treebank-style data, so it is far more accurate than our toy HMM.

9.1 Basic tagging

import nltk

for pkg in ["punkt_tab", "averaged_perceptron_tagger_eng", "universal_tagset"]:
    nltk.download(pkg)

sentence = "Fruit flies like a banana, but time flies like an arrow."
tokens = nltk.word_tokenize(sentence)
print(nltk.pos_tag(tokens))

[('Fruit', 'NNP'), ('flies', 'VBZ'), ('like', 'IN'), ('a', 'DT'),
 ('banana', 'NN'), (',', ','), ('but', 'CC'), ('time', 'NN'),
 ('flies', 'NNS'), ('like', 'IN'), ('an', 'DT'), ('arrow', 'NN'), ('.', '.')]

Exact tags can vary slightly between NLTK versions. Look closely and you will see the tagger got both "flies" backwards: it calls the insects in "Fruit flies" a verb (VBZ) and the verb in "time flies" a plural noun (NNS), and it tags the first "like" as a preposition rather than a verb. A per-token accuracy of 97% still leaves room for exactly these errors on tricky sentences — which is why this sentence is a classic, and why you should always inspect tagger output on your own text rather than trusting a headline accuracy number.

9.2 Universal tags and tag explanations

print(nltk.pos_tag(tokens, tagset="universal"))

# Look up what a Penn tag means (needs: nltk.download("tagsets_json"))
nltk.download("tagsets_json")
nltk.help.upenn_tagset("VBZ")
nltk.help.upenn_tagset("JJ.*")   # regular expressions work too

9.3 Tagging many sentences efficiently

from nltk.tokenize import sent_tokenize, word_tokenize

text = """Pakistan's tech sector is growing quickly. Many startups in Peshawar
and Lahore are building AI products. Investors are watching closely."""

sentences = [word_tokenize(s) for s in sent_tokenize(text)]
for tagged in nltk.pos_tag_sents(sentences):
    print(tagged)

pos_tag_sents() loads the model once and tags a batch of sentences, which is faster than calling pos_tag() repeatedly inside a loop.

9.4 Counting parts of speech in a document

from collections import Counter

tags = [tag for sent in nltk.pos_tag_sents(sentences, tagset="universal")
        for _, tag in sent]
print(Counter(tags).most_common())

A POS distribution is a surprisingly useful stylistic fingerprint. Academic writing tends to have a high proportion of nouns and adjectives, conversational text more pronouns and verbs, and marketing copy more adjectives and adverbs. Researchers use these distributions for authorship attribution, readability estimation, and genre classification.

10. POS Tagging with spaCy

spaCy is the go-to library for production NLP in Python. Its pipelines tag, parse, lemmatize, and recognize entities in one pass, and they are optimized for speed. For POS tagging, spaCy gives you three layers of information on every token:

• token.pos_ — the coarse Universal POS tag (e.g., VERB).
• token.tag_ — the fine-grained Penn Treebank tag for English (e.g., VBG).
• token.morph — morphological features such as tense, number, person, and verb form.

10.1 Basic usage

import spacy

nlp = spacy.load("en_core_web_sm")
doc = nlp("Apple is looking at buying a U.K. startup for $1 billion.")

for token in doc:
    print(f"{token.text:8} {token.pos_:6} {token.tag_:5} "
          f"{spacy.explain(token.tag_):42} {token.morph}")

Apple    PROPN  NNP   noun, proper singular                      Number=Sing
is       AUX    VBZ   verb, 3rd person singular present          Mood=Ind|Number=Sing|Person=3|Tense=Pres|VerbForm=Fin
looking  VERB   VBG   verb, gerund or present participle         Aspect=Prog|Tense=Pres|VerbForm=Part
at       ADP    IN    conjunction, subordinating or preposition
buying   VERB   VBG   verb, gerund or present participle         Aspect=Prog|Tense=Pres|VerbForm=Part
a        DET    DT    determiner                                 Definite=Ind|PronType=Art
U.K.     PROPN  NNP   noun, proper singular                      Number=Sing
startup  NOUN   NN    noun, singular or mass                     Number=Sing
for      ADP    IN    conjunction, subordinating or preposition
$        SYM    $     symbol, currency
1        NUM    CD    cardinal number                            NumType=Card
billion  NUM    CD    cardinal number                            NumType=Card
.        PUNCT  .     punctuation mark, sentence closer          PunctType=Peri

Notice how spaCy separates "is" as an auxiliary (AUX) from main verbs like "looking" (VERB). Penn Treebank does not make this distinction (both are VB*), but UPOS does, which is very handy when you want to extract only meaningful action words. The helper spacy.explain() turns any tag or label into a human-readable description, which is great when you meet an unfamiliar tag.

10.2 Working with morphological features

doc = nlp("She has written three books and is writing another.")
for token in doc:
    if token.pos_ in {"VERB", "AUX"}:
        print(token.text, token.morph.get("Tense"), token.morph.get("VerbForm"))

Morphological features let you answer questions such as "Is this sentence in the past tense?" or "Which nouns are plural?" without memorizing dozens of Penn tags. This is especially useful for multilingual pipelines, because the same feature names are used across all UD languages.

10.3 Visualizing tags and dependencies

from spacy import displacy

doc = nlp("The quick brown fox jumps over the lazy dog.")
# In Jupyter:
displacy.render(doc, style="dep", options={"compact": True})
# In a script, start a local server instead:
# displacy.serve(doc, style="dep")

displaCy draws each word with its POS tag underneath and arrows for syntactic dependencies. It is an excellent teaching tool and a fast way to debug why your extraction rules are failing on a particular sentence.

10.4 Fast batch tagging

texts = ["First document text ...", "Second document text ..."] * 5000

nlp = spacy.load("en_core_web_sm", disable=["parser", "ner", "lemmatizer"])
for doc in nlp.pipe(texts, batch_size=1000):
    tags = [(t.text, t.pos_) for t in doc]

If you only need POS tags, disable the parser, NER, and lemmatizer. Keep tok2vec, tagger, and attribute_ruler — in the English pipelines, the attribute ruler maps fine-grained tags to UPOS, so removing it leaves pos_ empty.

10.5 Choosing a spaCy model

Model	Size	Speed	Tagging accuracy	Use when
en_core_web_sm	Small	Fastest	Good	Prototyping, large volumes, CPU-only servers
en_core_web_md	Medium	Fast	Better	You also want word vectors
en_core_web_lg	Large	Fast	Better	Best non-transformer accuracy
en_core_web_trf	Large (transformer)	Slow on CPU	Best	Accuracy-critical tasks, GPU available

11. POS Tagging with Stanza

Stanza, from Stanford NLP, offers neural pipelines for over 60 languages, all trained on Universal Dependencies treebanks. Each word receives a UPOS tag, a language-specific tag (XPOS), and morphological features (FEATS). Stanza is slower than spaCy on CPU but is often more accurate, especially outside English.

import stanza

stanza.download("en")
nlp = stanza.Pipeline("en", processors="tokenize,mwt,pos")

doc = nlp("Barack Obama was born in Hawaii. He was elected president in 2008.")
for sent in doc.sentences:
    for word in sent.words:
        print(f"{word.text:10} {word.upos:6} {word.xpos:5} {word.feats or ''}")

11.1 Tagging Urdu and other languages

stanza.download("ur")
nlp_ur = stanza.Pipeline("ur", processors="tokenize,pos")

doc = nlp_ur("پشاور ایک تاریخی شہر ہے")
for word in doc.sentences[0].words:
    print(word.text, word.upos)

Urdu is a good example of why Universal tags matter. Urdu uses postpositions ("میں," "سے," "کو") after nouns instead of prepositions before them; UPOS handles this naturally with the single ADP (adposition) tag. The same code works for Hindi, Arabic, Persian, Turkish, and dozens of other languages by changing only the language code.

12. POS Tagging with Flair and Hugging Face Transformers

12.1 Flair

Flair is a PyTorch-based NLP library from Humboldt University of Berlin, known for its contextual string embeddings and strong sequence-labeling models. Its pre-trained English POS model is accurate and very easy to use.

# pip install flair
from flair.data import Sentence
from flair.models import SequenceTagger

tagger = SequenceTagger.load("flair/pos-english")   # Penn Treebank tags
sentence = Sentence("I love Berlin and I will book a table tomorrow.")
tagger.predict(sentence)

for token in sentence:
    print(token.text, token.get_label("pos").value,
          round(token.get_label("pos").score, 3))

Flair also reports a confidence score for every tag. Low-confidence predictions are perfect candidates for human review or active learning: if you are building a custom dataset, sending only the uncertain tokens to annotators saves a lot of labeling effort.

12.2 Hugging Face Transformers

The Hugging Face Hub hosts many POS models fine-tuned from BERT, RoBERTa, and multilingual encoders. You can run any of them with the token-classification pipeline:

# pip install transformers torch
from transformers import pipeline

pos = pipeline("token-classification",
               model="vblagoje/bert-english-uncased-finetuned-pos",
               aggregation_strategy="simple")

for item in pos("The startup released its first AI model last week."):
    print(f"{item['word']:10} {item['entity_group']:6} {item['score']:.3f}")

Setting aggregation_strategy="simple" merges subword pieces back into whole words, so "startup" is reported once even if the tokenizer split it into "start" + "##up." Model names and availability on the Hub change over time, so search the Hub for "pos" and your language to find current, well-maintained options, and check each model card for the tagset it uses.

Subword alignment: Transformer tokenizers split words into pieces. Without aggregation, you get one prediction per piece. When you fine-tune your own model, the usual convention is to label only the first subword of each word and ignore the rest (label -100) in the loss. Our NER guide shows this alignment code step by step.

13. Train and Evaluate Your Own POS Tagger

Pre-trained taggers are great, but sometimes you need your own: a new language, an unusual domain (clinical notes, legal text, code comments), a custom tagset, or a lightweight model you fully control. NLTK makes it easy to train and compare several classic taggers on the Penn Treebank sample.

13.1 N-gram taggers with backoff

import nltk
from nltk.corpus import treebank
from nltk.tag import DefaultTagger, UnigramTagger, BigramTagger, TrigramTagger

sents = list(treebank.tagged_sents())
cut = int(len(sents) * 0.9)
train, test = sents[:cut], sents[cut:]

t0 = DefaultTagger("NN")                    # everything is a noun
t1 = UnigramTagger(train, backoff=t0)       # most frequent tag per word
t2 = BigramTagger(train, backoff=t1)        # + previous tag
t3 = TrigramTagger(train, backoff=t2)       # + previous two tags

for name, tagger in [("Default", t0), ("Unigram", t1),
                     ("Bigram", t2), ("Trigram", t3)]:
    print(f"{name:8} accuracy: {tagger.accuracy(test):.2%}")

Here is what this produced when we ran it (90/10 split of the 3,914-sentence NLTK sample):

Default  accuracy: 14.70%
Unigram  accuracy: 88.12%
Bigram   accuracy: 89.06%
Trigram  accuracy: 88.99%

The backoff chain is the key idea: when the trigram tagger has never seen a context, it asks the bigram tagger; if that fails, the unigram tagger; and finally the default tagger. Accuracy climbs steeply from the default baseline (just under 15%, the share of tokens that really are NN) to the unigram tagger, then improves only a little with context. The trigram tagger is even marginally worse than the bigram one: with only about 3,500 training sentences, most trigram contexts are seen too rarely to estimate reliably. That is data sparsity in action, and it is exactly why smoothing, interpolation and, later, discriminative models were invented. Older tutorials use tagger.evaluate(test); that method was deprecated in favor of accuracy().

13.2 A regex tagger for unknown words

from nltk.tag import RegexpTagger

patterns = [
    (r"^-?\d+([.,]\d+)?$", "CD"),     # numbers
    (r".*ing$", "VBG"),               # gerunds
    (r".*ed$", "VBD"),                # past tense
    (r".*es$", "VBZ"),                # 3rd person present
    (r".*ly$", "RB"),                 # adverbs
    (r".*s$", "NNS"),                 # plural nouns
    (r"^[A-Z].*$", "NNP"),            # capitalized words
    (r".*", "NN"),                    # default
]
regex = RegexpTagger(patterns)
t1 = UnigramTagger(train, backoff=regex)
t2 = BigramTagger(train, backoff=t1)
print(f"Unigram+Bigram with regex backoff: {t2.accuracy(test):.2%}")

Unigram+Bigram with regex backoff: 93.33%

Replacing the "everything is a noun" default with a small set of suffix rules usually gives a noticeable boost, because unknown words now get a sensible guess. This is exactly the unknown-word problem we saw in the HMM section, solved with a few lines of linguistic knowledge.

13.3 Training NLTK's averaged perceptron

from nltk.tag.perceptron import PerceptronTagger

perceptron = PerceptronTagger(load=False)
perceptron.train(train, nr_iter=5)
print(f"Perceptron accuracy: {perceptron.accuracy(test):.2%}")

Perceptron accuracy: 96.37%

Tagger (same training data)	Test accuracy
Default (always NN)	14.70%
Unigram + default backoff	88.12%
Bigram + backoff	89.06%
Trigram + backoff	88.99%
Our HMM from scratch (Universal tags)	91.77%
Unigram + Bigram + regex backoff	93.33%
Averaged perceptron	96.37%

Note: the NLTK sample contains -NONE- placeholder tokens for empty syntactic elements, which are trivially easy to tag and inflate every score slightly. Real-world accuracy on clean text without them is a little lower.

The perceptron uses rich features — the word, its suffix and prefix, capitalization, neighboring words, and previous tags — and typically beats every n-gram tagger trained on the same data. It is the same algorithm behind NLTK's default pos_tag(), and a good example of how far a simple linear model goes with well-designed features.

13.4 Training a POS tagger with spaCy

For production-grade custom taggers, spaCy's training system is the most practical route. Convert your data (for example, a CoNLL-U file from a UD treebank) into spaCy's binary format, generate a config, and train:

# Convert CoNLL-U files to spaCy's .spacy format
python -m spacy convert train.conllu ./corpus --converter conllu
python -m spacy convert dev.conllu   ./corpus --converter conllu

# Create a training config for a tagger + morphologizer
python -m spacy init config config.cfg --lang en --pipeline tagger,morphologizer --optimize accuracy

# Train
python -m spacy train config.cfg --output ./pos_model \
    --paths.train ./corpus/train.spacy --paths.dev ./corpus/dev.spacy

spaCy's tagger component learns fine-grained tags (stored in token.tag_), while the morphologizer learns UPOS tags and morphological features together. The training output prints tag accuracy on the development set after each evaluation step, and saves the best model to ./pos_model/model-best, which you can load with spacy.load().

14. Evaluation Metrics and Error Analysis

The standard metric for POS tagging is token-level accuracy: the number of correctly tagged tokens divided by the total number of tokens. Because some tags are rare, accuracy alone can hide problems, so also look at per-tag precision, recall, and F1, and at the confusion matrix.

from collections import Counter
from sklearn.metrics import classification_report

gold, pred = [], []
for sent in test:
    words = [w for w, _ in sent]
    gold += [t for _, t in sent]
    pred += [t for _, t in perceptron.tag(words)]

print(classification_report(gold, pred, zero_division=0))

confusions = Counter((g, p) for g, p in zip(gold, pred) if g != p)
for (g, p), n in confusions.most_common(10):
    print(f"gold={g:5} predicted={p:5} count={n}")

Running this on the perceptron above, the ten most frequent confusions were:

gold=NN    predicted=JJ    count=34
gold=JJ    predicted=NN    count=27
gold=NN    predicted=NNP   count=23
gold=JJ    predicted=NNP   count=17
gold=NNPS  predicted=NNP   count=14
gold=JJ    predicted=RB    count=12
gold=VBG   predicted=NN    count=11
gold=VBN   predicted=JJ    count=10
gold=IN    predicted=RP    count=10
gold=VBN   predicted=VBD   count=10

The per-tag report also showed rare tags such as PDT, RBS and NNPS with very low recall — they simply have too few training examples. On English, the most common confusions are usually predictable: NN vs NNP (common vs proper nouns, especially for capitalized words at the start of sentences), JJ vs NN (noun modifiers like "computer science"), VBD vs VBN (past tense vs past participle: "He raised money" vs "money raised last year"), and IN vs RP (preposition vs particle: "look up the word" vs "walk up the hill"). Knowing your top confusions tells you exactly where to add training data or rules.

14.1 Accuracy on known vs unknown words

Always split your accuracy into known words (seen in training) and unknown words (not seen). A tagger can show high overall accuracy while still performing much worse on unknown words, and unknown words are exactly what you meet when moving to a new domain. If unknown-word accuracy is low, improve your suffix features, add character-level models, or use pre-trained embeddings.

14.2 Evaluate on your own domain

Benchmarks such as the Penn Treebank Wall Street Journal sections or the English Web Treebank are useful for comparing algorithms, but they say little about your tweets, contracts, or chat logs. Hand-tag 200–500 sentences from your real data — it takes a few hours — and you will have the single most valuable evaluation set in your project.

15. Practical Projects Using POS Tags

POS tags become truly valuable when you use them to build something. Here are four small, practical projects you can adapt for your own work, blog analytics, or final-year project.

15.1 Keyphrase extraction with noun-phrase chunking

Chunking (shallow parsing) groups tagged words into phrases using patterns over POS tags. A classic pattern for keyphrases is "zero or more adjectives followed by one or more nouns." NLTK's RegexpParser applies such grammars directly:

import nltk
from collections import Counter

grammar = r"KP: {<JJ.*>*<NN.*>+}"     # adjectives* + nouns+
chunker = nltk.RegexpParser(grammar)

def keyphrases(text: str) -> Counter:
    phrases = Counter()
    for sent in nltk.sent_tokenize(text):
        tree = chunker.parse(nltk.pos_tag(nltk.word_tokenize(sent)))
        for subtree in tree.subtrees(filter=lambda t: t.label() == "KP"):
            phrases[" ".join(w.lower() for w, _ in subtree.leaves())] += 1
    return phrases

article = """Natural language processing helps computers understand human language.
Modern language models use deep neural networks. Text preprocessing and
POS tagging remain important steps in many natural language processing pipelines."""
print(keyphrases(article).most_common(5))

[('natural language processing', 1), ('computers', 1), ('human language', 1),
 ('modern language models', 1), ('deep neural networks', 1)]

Look at the full list and you will also find 'many natural language processing pipelines' and 'pos'. The tagger labelled "many" as an adjective (JJ), so it was absorbed into the phrase, and "POS tagging" split because "tagging" was tagged as a verb. Real keyphrase extractors fix this by removing determiners and quantifiers such as "many" from the start of phrases, keeping a short stop list, and counting sub-phrases ("natural language processing" inside the longer phrase) as matches. Small refinements like these are where most of the quality of a keyword tool comes from.

This tiny function is the core of many SEO tools and content analyzers: run it over your blog posts, and you get the main topics of each article ranked by frequency. Combine it with TF-IDF across your whole blog to find phrases that are distinctive for each post.

15.2 Aspect-based opinion mining with spaCy

Product reviews often follow the pattern "the [aspect] is [opinion]." With POS tags and dependency relations, we can link adjectives to the nouns they describe:

import spacy

nlp = spacy.load("en_core_web_sm")

def aspect_opinions(text: str):
    pairs = []
    for token in nlp(text):
        if token.pos_ != "ADJ":
            continue
        if token.dep_ == "amod":                     # "great camera"
            pairs.append((token.head.text, token.text))
        elif token.dep_ == "acomp":                  # "camera is great"
            subjects = [c for c in token.head.children if c.dep_ == "nsubj"]
            for subj in subjects:
                pairs.append((subj.text, token.text))
    return pairs

review = ("The new battery lasts long but the camera is disappointing "
          "and the screen looks very bright.")
print(aspect_opinions(review))

[('battery', 'new'), ('camera', 'disappointing'), ('screen', 'bright')]

From here you can attach a sentiment score to each opinion word and aggregate by aspect across thousands of reviews: "battery: mostly positive; camera: mostly negative." That is real business insight produced from nothing more than POS tags and a few dependency rules.

15.3 A simple passive-voice detector for writers

def passive_sentences(text: str):
    doc = nlp(text)
    found = []
    for sent in doc.sents:
        if any(t.dep_ in {"nsubjpass", "auxpass"} for t in sent):
            found.append(sent.text)
    return found

text = ("The report was written by the intern. We published it yesterday. "
        "Mistakes were made.")
print(passive_sentences(text))

['The report was written by the intern.', 'Mistakes were made.']

Grammar tools like this rely on the tag VBN (past participle) plus a form of "be," which the parser labels as a passive auxiliary. You could extend this into a full writing assistant that flags weak adverbs (RB next to verbs), long noun stacks, or overuse of the verb "be."

15.4 POS features for text classification

POS distributions are useful features for authorship attribution, spam detection, readability scoring, and AI-generated-text detection. Here is a simple feature extractor that turns a document into normalized UPOS frequencies you can feed to any scikit-learn classifier alongside TF-IDF features:

UPOS = ["ADJ", "ADP", "ADV", "AUX", "CCONJ", "DET", "INTJ", "NOUN", "NUM",
        "PART", "PRON", "PROPN", "PUNCT", "SCONJ", "SYM", "VERB", "X"]

def pos_profile(text: str) -> list[float]:
    doc = nlp(text)
    counts = {tag: 0 for tag in UPOS}
    for token in doc:
        counts[token.pos_] = counts.get(token.pos_, 0) + 1
    total = max(len(doc), 1)
    return [counts[tag] / total for tag in UPOS]

print([round(x, 2) for x in pos_profile("I really love this phone, it is so fast!")])

16. POS Tagging Library Comparison (2026)

Library	Algorithm	Tagset(s)	Languages	Speed	Best for
NLTK	Averaged perceptron (plus trainable n-gram, HMM, Brill, CRF)	Penn, Universal	English by default (trainable for others)	Medium	Learning, experiments, custom classic taggers
spaCy	Neural (CNN or transformer)	UPOS + Penn + morph features	25+ trained pipelines	Fast	Production pipelines
Stanza	BiLSTM-based neural	UPOS, XPOS, FEATS	60+	Medium (fast on GPU)	Multilingual accuracy, research
Flair	Contextual string embeddings + BiLSTM-CRF	Penn, UPOS (by model)	Several	Medium	High accuracy with confidence scores
Hugging Face	Fine-tuned transformers	Depends on model	Many	Slower on CPU	State-of-the-art accuracy, custom fine-tuning
TextBlob	Wraps NLTK/Pattern	Penn	English	Medium	Beginners, quick scripts

Our recommendation: Learn the concepts with NLTK, build real applications with spaCy, switch to Stanza for non-English languages (including Urdu and Hindi), and reach for a fine-tuned transformer when you need the last percent of accuracy or must adapt to a specialized domain.

17. Common Mistakes and Best Practices

17.1 Common mistakes

1. Tagging preprocessed text. Lowercasing, removing stop words, or stripping punctuation before tagging destroys the context the tagger relies on. Always tag the original, natural sentence first.
2. Mixing tagsets. Comparing NLTK's Penn tags with spaCy's pos_ UPOS tags (or with a model that uses a third tagset) leads to silent bugs. Normalize to one tagset before comparing or combining outputs.
3. Tagging single words without context. pos_tag(["book"]) can only guess. Tag full sentences whenever possible.
4. Ignoring tokenization differences. Different tools split contractions, hyphens, and URLs differently. Use the tokenizer that matches your tagger.
5. Trusting benchmark accuracy on new domains. A tagger that scores 97% on news may do far worse on tweets or medical notes. Evaluate on your own data.
6. Calling the model in a loop. Use pos_tag_sents() in NLTK and nlp.pipe() in spaCy for batch processing.
7. Removing the attribute ruler in spaCy. In English pipelines, it fills pos_; disabling it leaves UPOS tags empty.
8. Forgetting the subword problem with transformers. Use aggregation_strategy at inference and proper label alignment when training.

17.2 Best practices checklist

• ✅ Pick a tagset intentionally: UPOS for simplicity and multilingual work, Penn for fine English detail.
• ✅ Tag natural, untouched sentences, then filter or normalize afterwards.
• ✅ Start with spaCy en_core_web_sm; move to larger or transformer models only if evaluation shows you need them.
• ✅ Build a small hand-tagged evaluation set from your real data.
• ✅ Report accuracy separately for known and unknown words, and study the confusion matrix.
• ✅ Use confidence scores (Flair, transformers) to route uncertain cases to human review.
• ✅ Pin library and model versions; re-run evaluation after upgrades.
• ✅ For code-mixed or Roman Urdu text, plan for custom training data — off-the-shelf taggers are not built for it.

17.3 POS tagging interview questions

1. What is the difference between a generative and a discriminative tagger? A generative model (HMM) models the joint probability of words and tags; a discriminative model (MaxEnt, CRF, perceptron) directly models tags given words and can use arbitrary overlapping features.
2. What is the time complexity of the Viterbi algorithm? O(n × k²) for a bigram HMM, where n is the sentence length and k the number of tags.
3. How do you handle unknown words? Use suffix, prefix, capitalization, and digit features; character-level neural models; subword embeddings; or pre-trained transformers.
4. Why is UPOS useful? It provides a consistent, cross-lingual tagset, making multilingual models and comparisons possible.
5. Why can the trigram tagger be worse than the bigram tagger? Data sparsity: with limited training data, most trigram contexts are rare, so estimates are unreliable without smoothing and interpolation.

18. Frequently Asked Questions (FAQ)

What is POS tagging in NLP with an example?

POS tagging assigns a grammatical category to each word in context. For "She reads books," a tagger outputs She/PRON, reads/VERB, books/NOUN (or PRP, VBZ, NNS in Penn Treebank tags).

Which is the best POS tagger in Python?

spaCy is the best general choice for production because it is fast, accurate, and easy to use. Stanza is excellent for non-English languages, NLTK is ideal for learning, and fine-tuned transformer models give the highest accuracy.

What is the difference between Penn Treebank and Universal POS tags?

Penn Treebank has 36 fine-grained English word tags that encode details like tense and number. Universal POS has 17 coarse tags designed to work consistently across all languages.

How accurate is POS tagging?

Modern English taggers achieve roughly 97–98% token accuracy on edited news text. Accuracy is lower on social media, specialized domains, and low-resource or code-mixed languages.

Is POS tagging still useful with large language models?

Yes. LLMs do not need explicit tags internally, but POS tagging remains a fast, cheap, and deterministic tool for lemmatization, keyword extraction, grammar checking, information extraction, linguistic research, and data preprocessing at scale.

What is the Viterbi algorithm in POS tagging?

Viterbi is a dynamic programming algorithm that finds the most probable sequence of tags for a sentence under a Hidden Markov Model, keeping only the best path into each tag at every position instead of enumerating every possible sequence.

Can I do POS tagging for Urdu in Python?

Yes. Stanza provides Urdu (and Hindi) models trained on Universal Dependencies treebanks. Roman Urdu and code-mixed text usually require custom training data.

What is the difference between POS tagging and NER?

POS tagging labels the grammatical role of every word (noun, verb…). Named Entity Recognition finds and classifies real-world names such as people, organizations, and locations, often spanning multiple words. Both are sequence labeling tasks, and POS tags are useful features for NER.

19. Conclusion

Part-of-Speech tagging is one of the foundations of NLP. It turns a flat string of words into structured grammatical information, resolving ambiguity using context so that "book" in "book a flight" is a verb and "book" in "read a book" is a noun. That information powers lemmatization, parsing, entity recognition, keyphrase extraction, grammar checking, opinion mining, and text-to-speech.

In this guide, you learned the major tagsets (Penn Treebank and Universal POS), why tagging is hard, and how the algorithms evolved from hand-written rules to HMMs with Viterbi decoding, discriminative perceptrons and CRFs, and finally neural and transformer models. You built an HMM tagger from scratch, compared n-gram, regex, HMM and perceptron taggers with real measured results, used NLTK, spaCy, Stanza, Flair and Hugging Face in Python, learned how to evaluate and debug taggers, and built four practical projects on top of POS tags.

The best next step is hands-on: take a few hundred sentences from a domain you care about — your blog posts, product reviews, or project data — tag them with spaCy, and inspect the errors. Then continue with our companion guides: Lemmatization in NLP (which depends directly on POS tags) and NLP Text Tagging in Python: POS Tagging & Named Entity Recognition, where you will learn to extract people, organizations, places and more from text.

Have a question about POS tagging, or a language you want to tag? Leave a comment below — and share this guide with someone starting their NLP journey.