NLP Text Tagging in Python: Complete Guide to POS Tagging & Named Entity Recognition (NER)

NLP Text Tagging in Python: Complete Guide to POS Tagging & Named Entity Recognition (NER)

Every day, the world produces an enormous amount of unstructured text: news articles, emails, support tickets, resumes, contracts, medical notes, social media posts, and chat messages. Hidden inside that text are the facts businesses actually care about — who did something, what company was involved, where it happened, when it happened, and how much money changed hands. Turning that messy text into clean, structured data starts with one family of techniques: text tagging.

In this complete guide, you will learn the two most important text tagging tasks in Natural Language Processing: Part-of-Speech (POS) tagging, which labels the grammatical role of every word, and Named Entity Recognition (NER), which finds and classifies real-world names such as people, organizations, locations, dates, and amounts. You will see how both are solved as sequence labeling problems, understand tagging schemes like BIO and BILOU, and implement everything in Python with NLTK, spaCy, Stanza, and Hugging Face Transformers. Then you will go further: add custom rule-based entities, combine POS and NER into an information extraction pipeline, train your own NER model with spaCy, fine-tune a BERT model, evaluate properly with entity-level metrics, and build real projects such as a resume skill extractor and a PII redaction tool.

Quick Summary (TL;DR)

• Text tagging (sequence labeling) assigns a label to every token in a text. POS tagging and NER are its two most common forms.
• POS tagging labels grammar (NOUN, VERB, ADJ…); NER labels real-world entities (PERSON, ORG, GPE, DATE, MONEY…), often spanning several words.
• Multi-word entities are encoded with BIO / IOB2 or BILOU tags such as B-PER I-PER O.
• In Python, spaCy gives fast, production-ready POS + NER in one call; Hugging Face transformers give the highest accuracy; EntityRuler adds custom entities with zero training data.
• Evaluate NER with entity-level precision, recall and F1 (e.g., with seqeval), not token accuracy.

Table of Contents

1. What Is Text Tagging in NLP?
2. Types of Text Tagging Tasks
3. Tagging Schemes: IO, BIO, IOB2 and BILOU
4. POS Tagging: A Practical Recap
5. What Is Named Entity Recognition?
6. How NER Works: Rules, CRFs, Neural Nets and Transformers
7. Setting Up Your Environment
8. NER with NLTK
9. POS + NER with spaCy
10. Custom Entities with spaCy's EntityRuler
11. NER with Hugging Face Transformers
12. NER with Stanza (Multilingual)
13. Combining POS and NER: An Information Extraction Pipeline
14. Train a Custom NER Model with spaCy
15. Fine-Tune a BERT Model for NER
16. Evaluating Taggers: Token vs Entity-Level Metrics
17. NER with Large Language Models
18. Real-World Projects
19. Performance, Scaling and Privacy
20. Common Mistakes and Best Practices
21. Frequently Asked Questions
22. Conclusion

1. What Is Text Tagging in NLP?

Text tagging is the process of attaching labels to pieces of text. In NLP, the most important form is token-level tagging, also called sequence labeling: the text is split into tokens (words, numbers, punctuation), and the model assigns one label to each token. The output sequence has exactly the same length as the input sequence, which makes sequence labeling different from text classification (one label per document) and from text generation (an output of arbitrary length).

Tokens : Ayesha  Khan    works  at  Microsoft  in  Karachi  .
POS    : PROPN   PROPN   VERB   ADP PROPN      ADP PROPN    PUNCT
NER    : B-PER   I-PER   O      O   B-ORG      O   B-LOC    O

The same sentence carries two layers of information. The POS layer describes grammar: "works" is a verb, "Karachi" is a proper noun. The NER layer describes meaning in the real world: "Ayesha Khan" is a person, "Microsoft" an organization, "Karachi" a location. Neither layer alone tells the full story, but together they let a program understand that a person is employed by a company in a city — which is exactly the kind of structured fact you want to extract.

Text tagging is sometimes also used more loosely to mean document tagging — attaching topic labels or keywords to whole articles, like the labels on a blog post. That is a classification task and is covered by different techniques. In this guide, "text tagging" means token-level sequence labeling, with POS and NER as the stars.

2. Types of Text Tagging Tasks

POS tagging and NER are the most common sequence labeling tasks, but they are part of a larger family. Knowing the family helps you recognize that the same tools and models can solve many different problems.

Task	What it labels	Example labels	Typical use
POS tagging	Grammatical category of each word	NOUN, VERB, ADJ / NN, VBD, JJ	Lemmatization, parsing, keyword extraction
Named Entity Recognition	Spans that name real-world entities	PERSON, ORG, GPE, DATE, MONEY	Information extraction, search, knowledge graphs
Chunking (shallow parsing)	Phrases such as noun or verb phrases	B-NP, I-NP, B-VP	Keyphrase extraction, simple relation patterns
Morphological tagging	Grammatical features of each word	Tense=Past, Number=Plur	Rich-morphology languages, grammar tools
Slot filling	Pieces of a user request	B-from_city, B-date, B-dish	Chatbots and voice assistants
Semantic role labeling	Who did what to whom	ARG0, ARG1, ARGM-TMP	Question answering, event extraction
Domain entity tagging	Domain-specific concepts	DRUG, DISEASE, GENE, SKILL, CLAUSE	Biomedical, legal, HR and finance NLP

All of these can be modeled with the same architectures — CRFs, BiLSTM-CRFs, and transformer token classifiers — and evaluated with the same metrics. Once you understand POS and NER, you can build any of the others by changing the training data and label set.

3. Tagging Schemes: IO, BIO, IOB2 and BILOU

POS tags apply to single tokens, so they need no special encoding. Entities, however, often span multiple tokens: "University of Oxford," "15 March 2026," "New York Stock Exchange." To represent spans with one label per token, we use a tagging scheme that marks where each entity begins and continues.

• IO: Every token is either I-TYPE (inside an entity) or O (outside). Simple, but it cannot separate two adjacent entities of the same type.
• BIO / IOB2: B-TYPE marks the beginning of an entity, I-TYPE marks tokens inside it, and O marks tokens outside any entity. This is the most widely used scheme today (CoNLL-2003 datasets on the Hugging Face Hub, most transformer NER models).
• IOB1: An older variant where B- is used only when an entity immediately follows another entity of the same type. It appears in some legacy datasets; convert it to IOB2 before training.
• BILOU / BIOES: Adds L- (last token) and U- (unit, a single-token entity); BIOES uses E- and S- for the same ideas. The extra detail can help models learn entity boundaries; spaCy uses BILOU internally.

Token	IO	BIO (IOB2)	BILOU
Ayesha	I-PER	B-PER	B-PER
Khan	I-PER	I-PER	L-PER
joined	O	O	O
the	O	O	O
University	I-ORG	B-ORG	B-ORG
of	I-ORG	I-ORG	I-ORG
Oxford	I-ORG	I-ORG	L-ORG
in	O	O	O
2026	I-DATE	B-DATE	U-DATE

Converting BIO tags back into entity spans is a common utility you will need when working with raw model outputs:

def bio_to_spans(tokens, tags):
    """Convert BIO tags into (entity_text, label) spans."""
    spans, current, label = [], [], None
    for token, tag in zip(tokens, tags):
        if tag.startswith("B-") or (tag.startswith("I-") and label != tag[2:]):
            if current:
                spans.append((" ".join(current), label))
            current, label = [token], tag[2:]
        elif tag.startswith("I-"):
            current.append(token)
        else:  # "O"
            if current:
                spans.append((" ".join(current), label))
            current, label = [], None
    if current:
        spans.append((" ".join(current), label))
    return spans

tokens = ["Ayesha", "Khan", "joined", "the", "University", "of", "Oxford", "in", "2026"]
tags   = ["B-PER", "I-PER", "O", "O", "B-ORG", "I-ORG", "I-ORG", "O", "B-DATE"]
print(bio_to_spans(tokens, tags))

[('Ayesha Khan', 'PER'), ('University of Oxford', 'ORG'), ('2026', 'DATE')]

Notice the defensive check: if an I- tag appears without a matching B- (a common model error), the function starts a new entity instead of crashing or silently merging it with an unrelated one.

4. POS Tagging: A Practical Recap

Part-of-Speech tagging labels every token with its grammatical category in context. We cover it in depth — tagsets, HMMs, the Viterbi algorithm, and training your own tagger — in our dedicated guide, Parts of Speech (POS) Tagging in NLP: Complete Guide with Python Examples (2026). Here is the essential recap you need for this article.

• Two tagsets dominate: Universal POS (17 coarse, cross-lingual tags such as NOUN, VERB, PROPN) and Penn Treebank (36 fine-grained English tags such as NN, VBD, NNP).
• Modern English taggers reach roughly 97–98% token accuracy on edited news text, less on informal or specialized text.
• POS tags matter for NER because proper nouns (PROPN/NNP) are strong hints for entities, and because POS patterns help extract the relations between entities.

import spacy

nlp = spacy.load("en_core_web_sm")
doc = nlp("Ayesha Khan, a senior engineer at Microsoft, flew from Karachi to Berlin.")

print([(t.text, t.pos_, t.tag_) for t in doc])

In the rest of this guide, we will use POS tags as a supporting signal: to filter entity candidates, to find the verbs that connect entities, and to build richer extraction rules.

5. What Is Named Entity Recognition?

Named Entity Recognition (NER) is the task of locating spans of text that refer to real-world entities and classifying them into predefined categories. The term "named entity" originally meant proper names — people, organizations, locations — but modern NER systems also detect dates, times, money, percentages, quantities, products, events, laws, and domain-specific types.

NER is really two problems solved together: detection (where does an entity start and end?) and classification (what type is it?). Both can be hard. "Washington" might be a person, a city, a state, or a government; "Apple" might be a fruit or a company; "Jordan" might be a country, a river, or a basketball player. As with POS tagging, context decides.

5.1 Common entity label sets

The CoNLL-2003 benchmark uses just four types: PER (person), ORG (organization), LOC (location), and MISC (miscellaneous names such as nationalities and events). The richer OntoNotes 5 scheme, used by spaCy's English pipelines, has 18 types:

Label	Description	Example
PERSON	People, including fictional characters	Ayesha Khan, Sherlock Holmes
NORP	Nationalities, religious or political groups	Pakistani, European
FAC	Facilities: buildings, airports, highways, bridges	Jinnah International Airport
ORG	Companies, agencies, institutions	Microsoft, University of Oxford
GPE	Geopolitical entities: countries, cities, states	Pakistan, Peshawar, Texas
LOC	Non-GPE locations: mountains, rivers, regions	K2, the Indus River
PRODUCT	Objects, vehicles, foods, software (not services)	iPhone 17, Toyota Corolla
EVENT	Named hurricanes, battles, wars, sports events	Cricket World Cup
WORK_OF_ART	Titles of books, songs, films	The Alchemist
LAW	Named documents made into laws	GDPR
LANGUAGE	Any named language	Urdu, Pashto
DATE	Absolute or relative dates or periods	15 March 2026, last week
TIME	Times smaller than a day	9:30 am, two hours
PERCENT	Percentages	45%
MONEY	Monetary values	$2 million, Rs. 5,000
QUANTITY	Measurements of weight or distance	5 kilometers
ORDINAL	First, second, etc.	third
CARDINAL	Numerals not covered by another type	three, 120

Choose the label set based on your application. A news analytics product might need all 18; a resume parser needs PERSON, ORG, DATE, plus custom labels like SKILL, DEGREE, and JOB_TITLE; a clinical system needs DRUG, DOSAGE, SYMPTOM, and DISEASE.

5.2 Why NER matters

• Information extraction: Turn news, reports, and filings into databases of companies, people, deals, and dates.
• Search and recommendation: Index documents by entities so users can filter "articles about Pakistan mentioning solar energy companies."
• Customer support: Detect product names, order numbers, and locations in tickets for automatic routing.
• Knowledge graphs: Entities become nodes; extracted relations become edges.
• Privacy and compliance: Find and redact names, phone numbers, and addresses (PII) before sharing data.
• RAG and LLM applications: Use entities as metadata filters for retrieval, or to verify that a generated answer mentions the right people and organizations.
• HR tech: Extract skills, employers, degrees, and dates from resumes and job posts.

6. How NER Works: Rules, CRFs, Neural Nets and Transformers

6.1 Rule-based and dictionary (gazetteer) NER

The simplest systems use lists of known names (gazetteers) and regular expressions: a list of cities, a list of companies, a regex for phone numbers or email addresses. Rules are precise, transparent, and need no training data, which makes them ideal for well-defined patterns (invoice numbers, CNIC-style ID formats, product codes) and closed lists (your company's product catalog). Their weakness is recall: they cannot recognize names that are not in the list, and they ignore context.

6.2 Feature-based machine learning: CRFs

Before deep learning, the best NER systems were Conditional Random Fields with hand-crafted features: the word, lowercased word, prefixes and suffixes, capitalization patterns ("Xxxxx," "XXX," "xxxx"), whether the word contains digits, gazetteer membership, the POS tag, and the same features for neighboring words. The CRF learns how these features predict BIO tags, and decodes the most likely tag sequence for the whole sentence, which prevents invalid sequences like O I-PER.

6.3 Neural NER: BiLSTM-CRF

Neural models replaced feature engineering with learned representations: word embeddings plus character-level encoders feed a bidirectional LSTM, and a CRF layer on top produces consistent tag sequences. BiLSTM-CRF models were the state of the art for several years and are still a strong, lightweight option.

6.4 Transformer-based NER

Today, the most accurate NER systems fine-tune pre-trained transformers (BERT, RoBERTa, DeBERTa, XLM-RoBERTa) as token classifiers. Because the pre-trained model already understands language deeply, it needs relatively few labeled examples and generalizes well to new names. On CoNLL-2003 English, strong systems report entity-level F1 scores in the low-to-mid 90s.

6.5 Span-based and generative NER

Some newer approaches classify candidate spans directly instead of tagging tokens, which handles nested entities ("Bank of [Pakistan]" containing a location inside an organization). Others use generative models, including LLMs, to output entities as text or JSON. Zero-shot models such as GLiNER let you specify entity types in plain language at inference time, without retraining — a useful middle ground between rigid label sets and expensive LLM calls.

Approach	Training data	Accuracy	Speed	Best for
Rules / gazetteers	None	High precision, low recall	Very fast	IDs, codes, closed lists
CRF with features	Moderate	Good	Fast	Small, CPU-only systems
BiLSTM-CRF / spaCy CNN	Moderate	Very good	Fast	Production at scale
Fine-tuned transformer	Less (transfer learning)	Excellent	Slower	Accuracy-critical tasks
Zero-shot / LLM	None or few examples	Variable to excellent	Slow	New entity types, prototyping, labeling help

6.6 What a feature-based tagger actually "sees"

It helps to look at NER through the eyes of a classic feature-based model, because modern neural models learn the same kinds of clues automatically. Take the token "Karachi" in the sentence "She moved to Karachi last year." A CRF would describe it with features such as: the word is "karachi" in lowercase; its shape is "Xxxxx" (capital letter followed by lowercase letters); it is not the first word of the sentence, so the capital letter is meaningful; its suffix is "chi"; its POS tag is NNP (proper noun); the previous word is "to" and the word before that is "moved"; the next word is "last"; and, if a gazetteer is available, the word appears in a list of world cities. Each feature carries a learned weight for each label. "Proper noun after 'to' following a movement verb" pushes strongly toward a location; "appears in a city list" pushes further; and the transition weights make sure that a B-LOC can be followed by I-LOC but an O cannot be followed directly by an I-LOC.

A transformer does not need these hand-written features. Its self-attention layers look at every word in the sentence at once and build a contextual vector for "Karachi" that already encodes "proper name, used as a destination of movement." The final classification layer turns that vector into a BIO label. The underlying intuition, however, is the same: the word's own form, its grammatical role, and its context together decide its entity type. This is also why POS information remains valuable — it summarizes the grammatical role in a single, cheap signal that rules and lightweight models can use directly.

6.7 Beyond flat NER: nested entities, linking and relations

Standard NER assumes entities are flat and non-overlapping, but real text is richer. In "the University of Peshawar," the whole phrase is an organization while "Peshawar" inside it is a city — a nested entity. BIO tagging can represent only one layer, so systems that need nesting use span-based models or multiple tagging layers. Most business applications can ignore nesting, but biomedical and legal NLP often cannot.

Recognizing that "Microsoft" is an organization is also only the first step. Entity linking (or entity disambiguation) maps each mention to a unique entry in a knowledge base, so that "Microsoft," "Microsoft Corp.," and "MSFT" all point to the same record, and "Jordan" the country is kept separate from "Jordan" the person. Relation extraction then identifies how entities are connected — works_for, located_in, acquired. Together, NER, linking, and relation extraction form the classic pipeline for building knowledge graphs, and you will build a simple version of the relation step in Section 13.

7. Setting Up Your Environment

# Create a virtual environment (recommended)
python -m venv tagging-env
source tagging-env/bin/activate        # Windows: tagging-env\Scripts\activate

# Core libraries
pip install nltk spacy stanza pandas seqeval

# Transformers (add a CUDA build of torch if you have an NVIDIA GPU)
pip install transformers datasets evaluate accelerate torch

# spaCy English model
python -m spacy download en_core_web_sm

import nltk
for pkg in ["punkt_tab", "averaged_perceptron_tagger_eng",
            "maxent_ne_chunker_tab", "words"]:
    nltk.download(pkg)

Recent NLTK versions renamed several data packages (for example maxent_ne_chunker_tab instead of maxent_ne_chunker, and averaged_perceptron_tagger_eng instead of averaged_perceptron_tagger). If code from an older tutorial raises a LookupError, download the new names above.

8. NER with NLTK

NLTK's built-in NER uses ne_chunk(), a maximum-entropy chunker that runs on top of POS-tagged tokens. This makes the POS → NER dependency very visible: the chunker literally takes POS tags as input. It recognizes PERSON, ORGANIZATION, GPE, and a few other types.

import nltk
from nltk.chunk import tree2conlltags

text = ("Ayesha Khan, a senior engineer at Microsoft, flew from Karachi to Berlin "
        "on 15 March 2026 to present a $2 million research project with the "
        "University of Oxford.")

tokens = nltk.word_tokenize(text)
tagged = nltk.pos_tag(tokens)          # POS tagging first
tree = nltk.ne_chunk(tagged)           # then NER on top of POS tags

for subtree in tree:
    if hasattr(subtree, "label"):
        print(subtree.label(), " ".join(word for word, _ in subtree.leaves()))

PERSON Ayesha
GPE Khan
ORGANIZATION Microsoft
GPE Karachi
PERSON Berlin
ORGANIZATION University
GPE Oxford

This is the real output from a current NLTK release, and it is a useful lesson in itself. NLTK found most capitalized names, but it split "Ayesha Khan" into two entities, called "Berlin" a person, and broke "University of Oxford" into two pieces. It also ignored the date and the money amount entirely, because the model does not support those types. NLTK's NER is fine for learning how chunk-based NER works, but it is not a good choice for production.

8.1 Converting NLTK trees to IOB tags

for word, pos, ner in tree2conlltags(tree)[:8]:
    print(f"{word:10} {pos:5} {ner}")

tree2conlltags() converts the nested tree into a flat list of (word, POS, IOB-tag) triples — the same three-column format used by CoNLL datasets. This is handy when you want to compare NLTK with other taggers or feed its output into a different tool.

9. POS + NER with spaCy

spaCy runs tokenization, POS tagging, dependency parsing, lemmatization, and NER in a single call, so you get both tagging layers at once. Its English pipelines are trained on OntoNotes and recognize all 18 entity types listed earlier.

9.1 Extracting entities

import spacy

nlp = spacy.load("en_core_web_sm")
doc = nlp(text)

for ent in doc.ents:
    print(f"{ent.text:28} {ent.label_:8} {spacy.explain(ent.label_)}")

Ayesha Khan                  PERSON   People, including fictional
Microsoft                    ORG      Companies, agencies, institutions, etc.
Karachi                      GPE      Countries, cities, states
Berlin                       GPE      Countries, cities, states
15 March 2026                DATE     Absolute or relative dates or periods
$2 million                   MONEY    Monetary values, including unit
the University of Oxford     ORG      Companies, agencies, institutions, etc.

Compare this with NLTK's output on the same sentence: spaCy keeps "Ayesha Khan" and "the University of Oxford" as complete spans, classifies "Berlin" correctly, and adds the DATE and MONEY entities. That difference is why spaCy is a common default for production NER.

9.2 Token-level view: POS and BIO tags together

for token in doc[:8]:
    bio = token.ent_iob_ + ("-" + token.ent_type_ if token.ent_type_ else "")
    print(f"{token.text:10} {token.pos_:6} {bio}")

Ayesha     PROPN  B-PERSON
Khan       PROPN  I-PERSON
,          PUNCT  O
a          DET    O
senior     ADJ    O
engineer   NOUN   O
at         ADP    O
Microsoft  PROPN  B-ORG

Every token exposes ent_iob_ (B, I, or O) and ent_type_, so you can export spaCy's predictions in BIO format for evaluation or for training other models.

9.3 Useful entity attributes

for ent in doc.ents:
    print(ent.text, ent.label_, ent.start_char, ent.end_char,   # character offsets
          ent.start, ent.end,                                    # token offsets
          ent.root.text, ent.root.dep_, ent.sent.text[:30])      # syntactic head, sentence

Character offsets let you highlight entities in the original text or store them in a database; token offsets let you slice the Doc; and ent.root connects the entity to the dependency tree, which we will use for relation extraction.

9.4 Visualizing entities

from spacy import displacy

displacy.render(doc, style="ent", jupyter=True)        # in Jupyter / Colab
html = displacy.render(doc, style="ent", page=True)    # or save as an HTML page
open("entities.html", "w", encoding="utf-8").write(html)

displaCy highlights each entity with a colored label. Saving it as HTML is a quick way to share examples with non-technical teammates or to embed annotated examples in a blog post.

9.5 Knowing the limits of small models

Small models make mistakes, especially on names that are rare in their training data. In our tests, en_core_web_sm handled the sentence above perfectly, but on another sentence it labeled the city "Lahore" as a PERSON. Larger models (en_core_web_lg, en_core_web_trf) are noticeably better on such cases. If your data contains many South Asian, African, or other non-Western names and places, test carefully: these are often under-represented in English training corpora, and a few rules or a little fine-tuning can fix most errors.

10. Custom Entities with spaCy's EntityRuler

Statistical models only know the labels they were trained on. When you need new entity types (SKILL, PRODUCT_CODE, COURSE) or want to guarantee that certain names are always recognized, use spaCy's EntityRuler. It adds rule-based entities using exact phrases or token patterns, with no training data required.

import spacy

nlp = spacy.load("en_core_web_sm")
ruler = nlp.add_pipe("entity_ruler", before="ner")   # rules take priority over the model

patterns = [
    {"label": "ORG", "pattern": "Sarhad University"},
    {"label": "PRODUCT", "pattern": "Claude"},
    {"label": "SKILL", "pattern": [{"LOWER": "machine"}, {"LOWER": "learning"}]},
    {"label": "SKILL", "pattern": [{"LOWER": {"IN": ["python", "laravel", "react"]}}]},
]
ruler.add_patterns(patterns)

doc = nlp("Ali studied machine learning at Sarhad University and builds apps "
          "with Python, Laravel and React using Claude.")
print([(ent.text, ent.label_) for ent in doc.ents])

[('Ali', 'PERSON'), ('machine learning', 'SKILL'), ('Sarhad University', 'ORG'),
 ('Python', 'SKILL'), ('Laravel', 'SKILL'), ('React', 'SKILL'), ('Claude', 'PRODUCT')]

Rule-based and statistical entities now work together: the model still found "Ali" as a PERSON, while the ruler added skills, a university, and a product. Placing the ruler before="ner" means rule matches take priority and the model fills in the rest; placing it after="ner" means rules only add entities where the model found none.

10.1 Patterns using POS tags

Token patterns can match on POS tags, lemmas, shapes, and more, which lets you combine grammar and meaning:

patterns = [
    # "Rs. 5,000" or "PKR 12000" style amounts
    # spaCy splits "Rs." into "Rs" + ".", so the dot is an optional token
    {"label": "MONEY", "pattern": [{"TEXT": {"IN": ["Rs", "PKR"]}},
                                   {"TEXT": ".", "OP": "?"}, {"LIKE_NUM": True}]},
    # Job titles: optional adjective + "engineer|developer|manager"
    {"label": "JOB_TITLE", "pattern": [{"POS": "ADJ", "OP": "?"},
                                       {"LEMMA": {"IN": ["engineer", "developer", "manager"]}}]},
]
nlp = spacy.load("en_core_web_sm")
nlp.add_pipe("entity_ruler", before="ner").add_patterns(patterns)

for s in ["The senior developer earns Rs. 250,000 per month.",
          "She paid PKR 12000 to the manager."]:
    print([(e.text, e.label_) for e in nlp(s).ents])

[('senior developer', 'JOB_TITLE'), ('Rs. 250,000', 'MONEY')]
[('PKR 12000', 'MONEY'), ('manager', 'JOB_TITLE')]

Without that optional dot token, the first sentence would only produce 250,000 as a plain CARDINAL — a good reminder to always check how spaCy tokenizes your text before writing token patterns. A quick print([t.text for t in nlp(s)]) saves a lot of debugging.

Scale tip: For thousands of exact phrases (a full product catalog or a skills taxonomy), load them as phrase patterns from a JSONL file with ruler.from_disk("patterns.jsonl"). Phrase matching is very fast, even with large lists.

11. NER with Hugging Face Transformers

For the best accuracy, use a transformer model fine-tuned for NER. The Hugging Face pipeline API makes this a few lines of code:

from transformers import pipeline

ner = pipeline("ner", model="dslim/bert-base-NER", aggregation_strategy="simple")

for entity in ner("Ayesha Khan joined the University of Oxford after working at Microsoft in Karachi."):
    print(f"{entity['word']:24} {entity['entity_group']:5} {entity['score']:.3f} "
          f"[{entity['start']}:{entity['end']}]")

dslim/bert-base-NER is a widely used BERT model fine-tuned on CoNLL-2003, so it predicts the four CoNLL types (PER, ORG, LOC, MISC). Each result includes a confidence score and character offsets. The aggregation_strategy parameter controls how subword predictions are merged into words and entities:

• "none" — one prediction per subword token (raw output).
• "simple" — groups adjacent tokens with the same entity type; the most common choice.
• "first", "average", "max" — word-level strategies that pick a single label per word using the first subword, the average score, or the maximum score.

11.1 Multilingual NER

For non-English or mixed-language text, use a multilingual model fine-tuned for NER (for example, XLM-RoBERTa-based models). Search the Hub for "ner" plus your language, check the model card for its label set, training data, and license, and test on a sample of your own text before committing. Models and their availability change over time, so prefer actively maintained ones with clear documentation.

11.2 Running transformer NER efficiently

import torch
from transformers import pipeline

device = 0 if torch.cuda.is_available() else -1
ner = pipeline("ner", model="dslim/bert-base-NER",
               aggregation_strategy="simple", device=device)

texts = ["First news article ...", "Second news article ..."] * 500
results = ner(texts, batch_size=32)      # batched inference

Batching and a GPU make a large difference for transformers. Also note that BERT-style models have a maximum input length (commonly 512 subword tokens), so split long documents into sentences or chunks before tagging, and map entity offsets back to the original document.

12. NER with Stanza (Multilingual)

import stanza

stanza.download("en")
nlp = stanza.Pipeline("en", processors="tokenize,ner")

doc = nlp("Ayesha Khan flew from Karachi to Berlin on Monday.")
for ent in doc.ents:
    print(ent.text, ent.type)

# Token-level BIOES tags
for token in doc.sentences[0].tokens:
    print(token.text, token.ner)

Stanza's English NER uses the OntoNotes label set, and it provides NER models for a range of other languages too. Its token-level output uses the BIOES scheme (B-, I-, E-, S-), which you can convert to BIO by mapping E- to I- and S- to B-. Check Stanza's model page for the current list of languages with NER support, since it covers fewer languages for NER than for POS tagging.

13. Combining POS and NER: An Information Extraction Pipeline

The real power of text tagging appears when you combine layers. NER tells you which entities a sentence mentions; POS tags and dependency relations tell you how they are connected. Together they let you extract simple facts, or triples, of the form (subject, relation, object).

import spacy
import pandas as pd

nlp = spacy.load("en_core_web_sm")

def expand_to_entity(token, doc):
    """Return the full entity span covering a token, or the token's noun chunk."""
    for ent in doc.ents:
        if ent.start <= token.i < ent.end:
            return ent.text, ent.label_
    for chunk in doc.noun_chunks:
        if chunk.start <= token.i < chunk.end:
            return chunk.text, "NOUN_PHRASE"
    return token.text, token.pos_

def extract_triples(text: str):
    doc = nlp(text)
    rows = []
    for token in doc:
        if token.pos_ != "VERB":
            continue
        subjects = [c for c in token.children if c.dep_ in ("nsubj", "nsubjpass")]
        objects = [c for c in token.children if c.dep_ in ("dobj", "attr")]
        # objects of prepositions: "invested IN startups"
        for prep in (c for c in token.children if c.dep_ == "prep"):
            objects += [g for g in prep.children if g.dep_ == "pobj"]
        for s in subjects:
            for o in objects:
                s_text, s_type = expand_to_entity(s, doc)
                o_text, o_type = expand_to_entity(o, doc)
                rows.append({"subject": s_text, "subject_type": s_type,
                             "relation": token.lemma_,
                             "object": o_text, "object_type": o_type,
                             "sentence": token.sent.text})
    return pd.DataFrame(rows)

news = ("Microsoft acquired GitHub in 2018. "
        "Ayesha Khan leads the AI team in Karachi. "
        "The company invested $2 million in local startups.")
print(extract_triples(news)[["subject", "subject_type", "relation", "object", "object_type"]])

      subject subject_type relation          object  object_type
0    Microsoft          ORG  acquire       GitHub          ORG
1    Microsoft          ORG  acquire         2018         DATE
2  Ayesha Khan       PERSON     lead  the AI team  NOUN_PHRASE
3  The company  NOUN_PHRASE   invest   $2 million        MONEY

In about 40 lines, we turned free text into a table of facts: Microsoft (ORG) acquired GitHub (ORG); Ayesha Khan (PERSON) leads the AI team; the company invested $2 million (MONEY). Each layer contributes something different:

• POS tags find the candidate relation words (VERB) and help define noun phrases.
• Dependency labels (nsubj, dobj, pobj) identify who acts and what is acted upon.
• NER types the arguments, so you can filter for exactly the facts you need, such as ORG–acquire–ORG for an acquisitions database.
• Lemmas normalize the relation ("acquired," "acquires," "acquiring" → "acquire"), which makes aggregation easy.

Notice the imperfections too. "2018" was attached as an object because it hangs off the preposition "in," so it looks just like a real object to our simple rule. "Karachi" and "local startups" are missing: the parser attached "in Karachi" to "team" and "in local startups" to "$2 million" rather than to the verbs, so our verb-centred rule never saw them. And "The company" was not resolved to Microsoft. This is typical of rule-based extraction — every rule is only as good as the parse it reads. Production systems add preposition filtering (treat "in + DATE" as a time attribute, not an object), coreference resolution (linking "the company" and "it" back to "Microsoft"), and entity linking (mapping "Microsoft" to a unique ID in a knowledge base like Wikidata). This simple pipeline is still an excellent starting point — and a great final-year-project foundation.

13.1 Using POS tags to clean NER output

POS tags are also useful to sanity-check entities. For example, a PERSON entity that contains no proper nouns is suspicious, and leading determiners like "the" are often unwanted in a database:

def clean_entities(doc):
    cleaned = []
    for ent in doc.ents:
        span = ent
        while len(span) > 1 and span[0].pos_ == "DET":     # drop leading "the"
            span = span[1:]
        if ent.label_ in {"PERSON", "ORG", "GPE"} and not any(t.pos_ == "PROPN" for t in span):
            continue                                          # likely a false positive
        cleaned.append((span.text, ent.label_))
    return cleaned

print(clean_entities(nlp("She joined the University of Oxford and the team in Berlin.")))

14. Train a Custom NER Model with spaCy

When rules are not enough and pre-trained labels do not match your needs, train your own NER model. spaCy v3's config-driven training system makes this reproducible and fast. The workflow has four steps: label data → convert to DocBin → configure → train.

14.1 Prepare labeled data

Training examples are texts plus character offsets for each entity. In practice you would annotate a few hundred to a few thousand examples with a tool such as Label Studio, Doccano, or Prodigy, and export them. Here is the format with a tiny example dataset:

TRAIN_DATA = [
    ("Hiring a Laravel developer with React experience in Peshawar.",
     [(9, 16, "SKILL"), (32, 37, "SKILL"), (52, 60, "LOCATION")]),
    ("We need a data scientist who knows Python and PyTorch.",
     [(10, 24, "JOB_TITLE"), (35, 41, "SKILL"), (46, 53, "SKILL")]),
    ("Senior Next.js engineer required in Lahore, remote possible.",
     [(7, 14, "SKILL"), (36, 42, "LOCATION")]),
]

Offset errors are the #1 training bug. Off-by-one character offsets silently produce entities that do not align with token boundaries, and spaCy skips them. Always validate your data with the conversion code below and inspect any warnings.

14.2 Convert to spaCy's binary format

import spacy
from spacy.tokens import DocBin

nlp = spacy.blank("en")

def to_docbin(data, path):
    db = DocBin()
    for text, annotations in data:
        doc = nlp.make_doc(text)
        ents = []
        for start, end, label in annotations:
            span = doc.char_span(start, end, label=label, alignment_mode="contract")
            if span is None:
                print(f"Skipping misaligned entity: {text[start:end]!r} in {text!r}")
            else:
                ents.append(span)
        doc.ents = ents
        db.add(doc)
    db.to_disk(path)

to_docbin(TRAIN_DATA, "train.spacy")
to_docbin(TRAIN_DATA, "dev.spacy")   # use a separate, held-out set in real projects!

14.3 Create a config and train

# Generate a sensible default config for an English NER pipeline
python -m spacy init config config.cfg --lang en --pipeline ner --optimize efficiency

# Train (add --gpu-id 0 if you have a GPU)
python -m spacy train config.cfg --output ./ner_model \
    --paths.train ./train.spacy --paths.dev ./dev.spacy

During training, spaCy prints the loss and the entity-level precision (ENTS_P), recall (ENTS_R), and F-score (ENTS_F) on the dev set. When it finishes, the best model is saved to ./ner_model/model-best:

import spacy

nlp_custom = spacy.load("./ner_model/model-best")
doc = nlp_custom("Looking for a React developer in Islamabad who also knows Python.")
print([(ent.text, ent.label_) for ent in doc.ents])

14.4 Tips for better custom NER

• Data volume: A few hundred well-labeled examples per entity type is a practical minimum; more varied examples matter more than many similar ones.
• Consistent guidelines: Write a one-page annotation guide ("Do we include 'Senior' in JOB_TITLE?") before labeling. Inconsistent labels cap your accuracy.
• Hold out real test data: Never evaluate on training data. Keep a test set the model never sees until the end.
• Start from a pre-trained pipeline or use a transformer-based config (--optimize accuracy with a GPU) when you need higher accuracy.
• Combine with rules: An EntityRuler for closed lists plus a trained model for open-ended entities is often the best of both worlds.
• Use model-assisted labeling: Let your first model pre-annotate new data; humans only correct mistakes, which is much faster than labeling from scratch.

15. Fine-Tune a BERT Model for NER

For state-of-the-art accuracy, fine-tune a pre-trained transformer as a token classifier. The example below fine-tunes bert-base-cased on the CoNLL-2003 dataset with the Hugging Face Trainer. The most important (and most commonly broken) step is label alignment: transformers split words into subword tokens, but labels belong to words.

import numpy as np
import evaluate
from datasets import load_dataset
from transformers import (AutoTokenizer, AutoModelForTokenClassification,
                          DataCollatorForTokenClassification, TrainingArguments, Trainer)

# If your datasets version rejects script-based datasets, use a parquet
# mirror of CoNLL-2003 from the Hub instead (search "conll2003").
raw = load_dataset("conll2003")
label_names = raw["train"].features["ner_tags"].feature.names   # ['O', 'B-PER', 'I-PER', ...]
id2label = dict(enumerate(label_names))
label2id = {v: k for k, v in id2label.items()}

checkpoint = "bert-base-cased"            # cased models work better for NER
tokenizer = AutoTokenizer.from_pretrained(checkpoint)

def tokenize_and_align(batch):
    enc = tokenizer(batch["tokens"], truncation=True, is_split_into_words=True)
    all_labels = []
    for i, word_labels in enumerate(batch["ner_tags"]):
        word_ids = enc.word_ids(batch_index=i)
        labels, previous = [], None
        for wid in word_ids:
            if wid is None:
                labels.append(-100)              # special tokens: ignored by the loss
            elif wid != previous:
                labels.append(word_labels[wid])  # first subword gets the word's label
            else:
                labels.append(-100)              # later subwords are ignored
            previous = wid
        all_labels.append(labels)
    enc["labels"] = all_labels
    return enc

tokenized = raw.map(tokenize_and_align, batched=True,
                    remove_columns=raw["train"].column_names)

model = AutoModelForTokenClassification.from_pretrained(
    checkpoint, num_labels=len(label_names), id2label=id2label, label2id=label2id)

seqeval = evaluate.load("seqeval")

def compute_metrics(eval_pred):
    logits, labels = eval_pred
    preds = np.argmax(logits, axis=-1)
    true_preds = [[label_names[p] for p, l in zip(pr, la) if l != -100]
                  for pr, la in zip(preds, labels)]
    true_labels = [[label_names[l] for p, l in zip(pr, la) if l != -100]
                   for pr, la in zip(preds, labels)]
    r = seqeval.compute(predictions=true_preds, references=true_labels)
    return {"precision": r["overall_precision"], "recall": r["overall_recall"],
            "f1": r["overall_f1"], "accuracy": r["overall_accuracy"]}

args = TrainingArguments(
    output_dir="bert-ner-conll",
    eval_strategy="epoch",
    save_strategy="epoch",
    learning_rate=2e-5,
    per_device_train_batch_size=16,
    num_train_epochs=3,
    weight_decay=0.01,
    load_best_model_at_end=True,
    metric_for_best_model="f1",
)

trainer = Trainer(model=model, args=args,
                  train_dataset=tokenized["train"],
                  eval_dataset=tokenized["validation"],
                  data_collator=DataCollatorForTokenClassification(tokenizer),
                  compute_metrics=compute_metrics)
trainer.train()
print(trainer.evaluate(tokenized["test"]))

A few important details:

• Label -100 tells PyTorch's loss function to ignore a position. We use it for special tokens ([CLS], [SEP]) and for every subword after the first in a word.
• Cased models (bert-base-cased, roberta-base) usually beat uncased ones for NER, because capitalization is a strong entity signal.
• On a single modern GPU, three epochs on CoNLL-2003 take minutes rather than hours. On CPU, use a smaller model such as DistilBERT, or train on a subset first to test your code.
• Older tutorials use evaluation_strategy; recent versions of Transformers renamed it to eval_strategy.
• To use your own data, convert it to the same structure — a tokens list and an ner_tags list of label IDs per sentence — and everything else stays the same.

After training, load the model in a pipeline exactly as you did with dslim/bert-base-NER: pipeline("ner", model="bert-ner-conll/checkpoint-XXXX", aggregation_strategy="simple"), or use trainer.save_model() to save a clean copy first.

16. Evaluating Taggers: Token vs Entity-Level Metrics

How you measure a tagger depends on the task. For POS tagging, token accuracy is standard because every token has exactly one correct label. For NER, token accuracy is misleading: most tokens are O, so a model that predicts O everywhere can score very high accuracy while finding zero entities. NER is therefore evaluated at the entity level, usually with the strict CoNLL criterion: a predicted entity counts as correct only if both its boundaries and its type exactly match a gold entity.

• Precision = correct predicted entities ÷ all predicted entities (how many of the model's entities are right).
• Recall = correct predicted entities ÷ all gold entities (how many of the real entities the model found).
• F1 = harmonic mean of precision and recall — the standard headline number.

from seqeval.metrics import classification_report, f1_score, accuracy_score

y_true = [["B-PER", "I-PER", "O", "O", "B-ORG", "I-ORG", "I-ORG", "O", "B-LOC"],
          ["O", "B-ORG", "O", "B-LOC", "O"]]
y_pred = [["B-PER", "I-PER", "O", "O", "B-ORG", "I-ORG", "O",     "O", "B-LOC"],
          ["O", "B-ORG", "O", "B-PER", "O"]]

print(f"Token accuracy: {accuracy_score(y_true, y_pred):.2%}")
print(f"Entity F1     : {f1_score(y_true, y_pred):.2%}")
print(classification_report(y_true, y_pred))

Token accuracy: 85.71%
Entity F1     : 60.00%
              precision    recall  f1-score   support

         LOC       1.00      0.50      0.67         2
         ORG       0.50      0.50      0.50         2
         PER       0.50      1.00      0.67         1

   micro avg       0.60      0.60      0.60         5
   macro avg       0.67      0.67      0.61         5
weighted avg       0.70      0.60      0.60         5

This small example shows why entity-level metrics matter. Token accuracy looks respectable at about 86%, but only three of the five gold entities were found exactly. Cutting one token off "University of Oxford" makes the whole ORG entity wrong, and labeling a location as a person counts as both a false positive (PER) and a false negative (LOC). Entity-level F1 (60%) reflects what a user of the system would actually experience.

16.1 Error analysis for NER

Group errors into categories to decide what to fix:

• Boundary errors: right type, wrong span ("University" instead of "University of Oxford"). Often fixed with more training examples of long entities or BILOU encoding.
• Type errors: right span, wrong type ("Berlin" as PERSON). Often fixed with more context-rich examples or a gazetteer feature.
• Missed entities (false negatives): rare names, lowercase names, or unusual formats. Fixed with more data, data augmentation, or rules.
• Spurious entities (false positives): capitalized common words at sentence start, headings in ALL CAPS. Fixed with negative examples and POS-based filters.

16.2 Strict vs partial matching

The strict CoNLL criterion is the standard for research papers, but it can be harsher than your application needs. If your goal is to highlight names in a document for a human reviewer, finding "University" when the gold entity is "University of Oxford" is still partly useful. Evaluation frameworks from shared tasks such as SemEval define additional modes — exact (boundaries only), partial (overlapping boundaries earn half credit) and type (correct type with any overlap) — and libraries such as nervaluate implement them. Report strict F1 so your numbers are comparable with published results, and add a partial-match score when it better reflects what your users experience. Whatever you choose, use the same scorer for every model you compare, because small differences in matching rules can move F1 by several points.

17. NER with Large Language Models

Large language models can perform NER from a plain-language instruction, with no training at all. You describe the entity types you want, ask for JSON output, and parse the result. This is genuinely useful in 2026, but it has clear trade-offs you should understand before using it in production.

import json

PROMPT = """Extract named entities from the text below.
Return ONLY a JSON list of objects with keys "text" and "label".
Allowed labels: PERSON, ORG, LOCATION, DATE, MONEY, SKILL.
Copy entity text exactly as it appears in the input.

Text: {text}"""

def llm_ner(text: str, call_llm) -> list[dict]:
    """call_llm is any function that sends a prompt to your LLM and returns its text reply."""
    reply = call_llm(PROMPT.format(text=text))
    try:
        entities = json.loads(reply)
    except json.JSONDecodeError:
        return []
    # Keep only entities that really occur in the text, and recover their offsets
    results = []
    for ent in entities:
        start = text.find(ent.get("text", ""))
        if start != -1 and ent.get("label"):
            results.append({**ent, "start": start, "end": start + len(ent["text"])})
    return results

The validation step at the end is essential. LLMs can paraphrase entity text, invent entities that are not in the input, or return malformed JSON. Checking that each entity really appears in the text — and recovering its character offsets — turns a free-form reply into reliable, highlightable spans.

Aspect	Trained NER model (spaCy / BERT)	LLM prompting
Setup effort	Needs labeled data (or a pre-trained model)	Just a prompt
New entity types	Requires retraining	Change the prompt
Speed and cost	Fast and cheap, runs locally	Slower, per-token cost or heavy local hardware
Consistency	Deterministic	Can vary between calls; needs validation
Offsets	Exact character offsets	Must be recovered and checked
Privacy	Data stays on your machine	Depends on provider and deployment
Best use	High-volume production	Prototyping, rare entity types, generating training labels

A powerful pattern combines both: use an LLM to pre-label a few thousand examples, have humans review and correct them, then train a small, fast spaCy or DistilBERT model on the cleaned data. You get the flexibility of the LLM during development and the speed, privacy, and low cost of a dedicated model in production.

18. Real-World Projects

18.1 Resume and job-post skill extractor

Recruitment platforms need to match candidates and jobs by skills, locations, and experience. Combining the pre-trained model (for people, organizations, places, dates) with an EntityRuler (for skills) gives a useful extractor on day one:

import spacy
from collections import defaultdict

SKILLS = ["python", "django", "laravel", "php", "react", "next.js", "node.js",
          "machine learning", "deep learning", "nlp", "sql", "docker", "aws"]

nlp = spacy.load("en_core_web_sm")
ruler = nlp.add_pipe("entity_ruler", before="ner",
                     config={"phrase_matcher_attr": "LOWER"})   # case-insensitive phrases
ruler.add_patterns([{"label": "SKILL", "pattern": s} for s in SKILLS])

def parse_profile(text: str) -> dict:
    doc = nlp(text)
    profile = defaultdict(set)
    for ent in doc.ents:
        if ent.label_ in {"SKILL", "ORG", "GPE", "DATE", "PERSON"}:
            profile[ent.label_].add(ent.text)
    return {k: sorted(v) for k, v in profile.items()}

resume = """Ali Raza is a full-stack developer based in Peshawar. From 2022 to 2025 he
worked at Systems Limited building Laravel and React applications, and he recently
completed projects in Machine Learning and NLP using Python and Docker."""
print(parse_profile(resume))

{'PERSON': ['Ali Raza'], 'GPE': ['Peshawar'], 'DATE': ['2022 to 2025'],
 'ORG': ['Systems Limited building'], 'SKILL': ['Docker', 'Laravel', 'Machine Learning',
 'NLP', 'Python', 'React']}

The skills, person, place and date are all right, but look at the organization: the small model swallowed the next word and returned "Systems Limited building." Boundary errors like this are the most common NER mistake in real text. You can fix this one with the POS-based cleanup from Section 13.1 (trim trailing tokens that are not proper nouns), with an EntityRuler entry for known employers, or by using a larger model. From here, you can normalize skills to a taxonomy (e.g., "ML" → "Machine Learning"), compute a match score between a resume and a job post using the overlap of their skill sets, and rank candidates. A trained NER model for JOB_TITLE, DEGREE, and EXPERIENCE_YEARS is a natural next step — and an excellent final-year project idea.

18.2 PII redaction tool

Before sharing support transcripts, logs, or documents with analysts or external AI services, many organizations must remove personally identifiable information. NER finds names and places; regular expressions catch structured identifiers such as emails and phone numbers:

import re
import spacy

nlp = spacy.load("en_core_web_sm")

EMAIL = re.compile(r"[\w.+-]+@[\w-]+\.[\w.]+")
PHONE = re.compile(r"(?:\+?\d{1,3}[\s-]?)?(?:\(?\d{2,4}\)?[\s-]?)\d{3,4}[\s-]?\d{3,4}")
REDACT_LABELS = {"PERSON", "GPE", "LOC", "FAC", "ORG"}

def redact(text: str) -> str:
    text = EMAIL.sub("[EMAIL]", text)
    text = PHONE.sub("[PHONE]", text)
    doc = nlp(text)
    out, last = [], 0
    for ent in doc.ents:
        if ent.label_ in REDACT_LABELS:
            out.append(text[last:ent.start_char])
            out.append(f"[{ent.label_}]")
            last = ent.end_char
    out.append(text[last:])
    return "".join(out)

msg = ("Hi, this is Sara Ahmed from Islamabad. Please email me at sara.ahmed@example.com "
       "or call +92 300 1234567 about my order.")
print(redact(msg))

Hi, this is [PERSON] from [GPE]. Please email me at [EMAIL] or call [PHONE] about my order.

Important: NER-based redaction reduces risk but is never perfect — models miss unusual names and mis-type others. For regulated data (health, finance, legal), combine NER with strict regex rules, allow-lists and deny-lists, human review of samples, and measure recall on a labeled test set. Specialized open-source tools such as Microsoft Presidio package these techniques together.

18.3 News entity analytics dashboard

Run NER over a stream of news articles, count entity mentions per day, and plot the trends: which companies, people, and cities are rising in coverage? Add a sentiment score per sentence and you get an "entity sentiment tracker" — a popular portfolio project. Combine it with the triple extraction from Section 13 to capture events such as acquisitions, investments, and partnerships.

18.4 Smarter search for your blog or website

Tag every post with its entities and noun phrases, store them as metadata, and let readers filter by organization, technology, or location. The same metadata improves internal linking — automatically suggesting related posts that mention the same entities — which is also good for SEO.

19. Performance, Scaling and Privacy

19.1 Speed tips

• Batch everything: nlp.pipe(texts, batch_size=...) in spaCy, pos_tag_sents() in NLTK, and batch_size in Hugging Face pipelines.
• Disable unused components: If you only need entities, spacy.load("en_core_web_sm", disable=["parser", "lemmatizer"]). Keep the parser if you use noun_chunks or dependency-based relations.
• Use multiprocessing with n_process in spaCy for CPU-bound workloads.
• Use a GPU for transformers. CPU inference with BERT-sized models is often too slow for large volumes; consider DistilBERT or quantized models when you must stay on CPU.
• Split long documents into sentences or paragraphs before transformer tagging, respecting the model's maximum sequence length.
• Cache results in a database or Parquet files; never tag the same text twice.

19.2 Choosing the right tool

Need	Recommended approach
Learn the concepts	NLTK (pos_tag + ne_chunk), then spaCy
Fast English POS + NER in production	spaCy en_core_web_sm / lg
Highest accuracy on standard entity types	Fine-tuned transformer (Hugging Face) or spaCy en_core_web_trf
Custom entity types, little data	EntityRuler + a small trained spaCy model
Custom entity types, lots of data	Fine-tune BERT / RoBERTa / DeBERTa
Non-English (Urdu, Hindi, Arabic, etc.)	Stanza or a multilingual transformer (XLM-RoBERTa-based)
Brand-new entity types for a prototype	LLM prompting or zero-shot NER models, then distill into a small model

19.3 Privacy and responsibility

NER systems often process personal data by definition — names, locations, contact details. Keep processing local when possible, log as little raw text as you can, apply data retention limits, and follow the regulations that apply to your users. Also be aware of bias: models trained mostly on Western news may recognize some names and places far better than others. Evaluate on data that represents your real users, and fix gaps with targeted training data or rules.

20. Common Mistakes and Best Practices

20.1 Common mistakes

1. Evaluating NER with token accuracy. Use entity-level precision, recall, and F1 with seqeval or spaCy's scorer.
2. Lowercasing text before NER. Capitalization is one of the strongest entity signals. Keep the original casing.
3. Misaligned character offsets in training data, which spaCy silently drops. Validate with doc.char_span().
4. Labeling every subword when fine-tuning transformers. Label the first subword, set the rest to -100.
5. Mixing label schemes (IOB1 vs IOB2, CoNLL vs OntoNotes) between datasets, models, and evaluation scripts.
6. Ignoring domain shift. A model trained on news will struggle with tweets, contracts, and medical notes. Measure on your own data.
7. Trusting LLM output blindly. Validate that extracted entities actually appear in the text.
8. Feeding very long documents into transformer models without chunking, causing silent truncation.

20.2 Best practices checklist

• ✅ Define your label set and annotation guidelines before collecting data.
• ✅ Start with a pre-trained pipeline plus EntityRuler; train only when needed.
• ✅ Keep a held-out, hand-labeled test set from your real domain.
• ✅ Report entity-level F1 per type, and study boundary vs type errors.
• ✅ Use POS tags and dependencies to connect entities into relations.
• ✅ Batch, cache, and disable unused components for speed.
• ✅ Pin library and model versions; re-evaluate after upgrades.
• ✅ Treat extracted personal data responsibly and redact where required.

20.3 Interview questions on POS tagging and NER

1. Why is NER evaluated at the entity level? Because most tokens are O; token accuracy rewards predicting nothing. Users care about complete, correctly typed entities.
2. What is the difference between BIO and BILOU? BILOU adds explicit "last" and "unit" tags, giving the model more boundary information.
3. How do you handle subword tokenization in transformer NER? Align labels to the first subword of each word, ignore other subwords with -100, and aggregate predictions back to words at inference.
4. How can POS tags help NER? Proper-noun tags signal entity candidates, and POS patterns help filter false positives and extract relations between entities.
5. When would you use rules instead of a model? For closed lists and well-defined formats (product codes, emails, IDs), where rules are precise, fast, and need no training data.

21. Frequently Asked Questions (FAQ)

What is text tagging in NLP?

Text tagging (sequence labeling) assigns a label to each token in a text. The most common types are POS tagging, which labels grammatical roles, and Named Entity Recognition, which labels real-world entities such as people, organizations, and locations.

What is the difference between POS tagging and NER?

POS tagging labels the grammatical category of every word (noun, verb, adjective). NER finds spans that name real-world things and classifies them (PERSON, ORG, GPE, DATE). POS tags apply to every token; entities cover only some tokens and often span multiple words.

Which Python library is best for NER?

spaCy is the best general-purpose choice for fast, production-ready NER. Hugging Face transformer models give the highest accuracy. NLTK is useful for learning, and Stanza is a good option for several non-English languages.

What is BIO tagging?

BIO tagging marks the Beginning of an entity with B-TYPE, tokens Inside it with I-TYPE, and tokens Outside any entity with O. It lets a token-level model represent multi-word entities such as "University of Oxford."

How do I create a custom NER model?

Label example texts with character offsets for your entity types, convert them to spaCy's DocBin format, generate a config with spacy init config, and train with spacy train. For higher accuracy, fine-tune a transformer such as BERT on BIO-tagged data.

How much data do I need to train a custom NER model?

A few hundred varied, consistently labeled examples per entity type is a practical starting point. Transformers need less data than models trained from scratch, and combining rules with a model reduces the amount further.

Can LLMs replace NER models?

LLMs are excellent for prototyping and for generating training labels, but dedicated NER models are faster, cheaper, more consistent, and easier to run privately. Many teams use LLMs to label data and then train a small model for production.

Is NER possible for Urdu or Roman Urdu?

Urdu NER is possible with multilingual transformer models fine-tuned on Urdu data and with research datasets; coverage is thinner than for English. Roman Urdu and code-mixed text usually require collecting and labeling your own data, often bootstrapped with rules and LLM pre-labeling.

22. Conclusion

Text tagging is how NLP turns unstructured language into structured information. POS tagging reveals the grammar of a sentence; Named Entity Recognition reveals the people, organizations, places, dates, and amounts it talks about. Both are solved as sequence labeling, encoded with schemes like BIO, and powered by the same evolution of models — from rules and CRFs to BiLSTMs, transformers, and LLMs.

In this guide, you used NLTK, spaCy, Stanza, and Hugging Face to tag real text, compared their outputs on the same sentence, added custom entities with the EntityRuler, combined POS tags, dependencies, and entities into an information extraction pipeline, trained a custom NER model with spaCy, fine-tuned BERT with correct label alignment, evaluated with entity-level metrics, and built practical tools for resume parsing and PII redaction.

Your best next step is to pick one project from Section 18 and build it end to end with your own data. Start with spaCy and an EntityRuler, label a small test set, measure entity-level F1, and improve from there. If you want to strengthen your foundations, read our companion guides: Parts of Speech (POS) Tagging in NLP and Lemmatization in NLP.

Which entities do you want to extract from your data? Tell us in the comments — and share this guide with anyone building their first NLP project.