Named Entity Recognition (NER) in NLP: The Complete Python Guide for 2026

Named Entity Recognition (NER) in NLP: The Complete Python Guide for 2026

Learn NER from scratch with spaCy, Hugging Face Transformers, GLiNER and LLMs, with working code, custom training, evaluation and a real project

Table of Contents

Introduction: Why Every Developer Should Understand NER
What Is Named Entity Recognition (NER)?
Why NER Matters: Real-World Use Cases
Common Entity Types
How NER Works Under the Hood
The Evolution of NER: From Rules to LLMs
Setting Up Your Python Environment
Method 1: NER with NLTK (The Classic Approach)
Method 2: NER with spaCy (The Production Workhorse)
Method 3: NER with Hugging Face Transformers
Method 4: Zero-Shot NER with GLiNER
Method 5: NER with Large Language Models
Training a Custom NER Model with spaCy
Fine-Tuning BERT for NER with Hugging Face
How to Evaluate NER Models Correctly
Comparing NER Approaches in 2026
Hands-On Project: Build an Entity Extraction API with FastAPI
Common Challenges in NER (and How to Solve Them)
Best Practices for Production NER
Frequently Asked Questions (FAQ)
Conclusion: Your Next Steps with NER

Introduction: Why Every Developer Should Understand NER

Every day, the world produces an almost unimaginable amount of unstructured text: news articles, customer support tickets, medical notes, legal contracts, resumes, tweets, product reviews and chat logs. Hidden inside all of that text are the facts businesses actually care about. Who is the customer? Which product failed? What company signed the deal, in which city, on what date, for how much money?

Humans pick out these details instantly. Computers do not. To a computer, a sentence is just a string of characters. Named Entity Recognition (NER) is the Natural Language Processing (NLP) technique that bridges this gap. It reads raw text and automatically finds and labels the important "things" in it, such as people, organizations, locations, dates and amounts.

NER is one of the oldest tasks in NLP, yet in 2026 it is more relevant than ever. It powers search engines, chatbots, fraud detection, resume screening, healthcare analytics and the retrieval pipelines behind modern AI assistants. And the tooling has never been better: you can now go from zero to a working entity extractor in five lines of Python, or build a zero-shot model that finds entity types it was never trained on.

In this complete guide you will learn:

What Named Entity Recognition is and how it works under the hood
The common entity types and the BIO tagging scheme
How NER evolved from rule-based systems to transformers, GLiNER and LLMs
Five practical ways to do NER in Python: NLTK, spaCy, Hugging Face, GLiNER and LLMs
How to train a custom spaCy NER model and fine-tune BERT on your own data
How to evaluate NER models properly with precision, recall and F1
How to build and deploy a real entity extraction API with FastAPI
Best practices, common pitfalls and production tips

Whether you are a student working on a final year project, a developer adding smart features to a web app, or a data scientist building an information extraction pipeline, this guide will give you both the theory and the code you need.

What Is Named Entity Recognition (NER)?

Named Entity Recognition is a subtask of information extraction that identifies spans of text referring to real-world objects and classifies them into predefined categories. In simple words, NER answers two questions about any piece of text:

Where are the important names and values? (entity detection, also called span detection)
What type is each one? (entity classification)

Consider this sentence:

Sundar Pichai announced on Tuesday that Google will invest $2 billion in a new data center in Lahore, Pakistan.

A good NER system would produce:

Entity	Label	Meaning
Sundar Pichai	PERSON	A person's name
Tuesday	DATE	A date or day
Google	ORG	An organization
$2 billion	MONEY	A monetary value
Lahore	GPE	City (geo-political entity)
Pakistan	GPE	Country (geo-political entity)

Notice that NER works on spans, not just single words. "Sundar Pichai" is one entity made of two tokens, and "$2 billion" is one entity made of three. Detecting where an entity starts and ends is just as important as getting its label right.

Also notice that NER depends on context. The word "Apple" is an organization in "Apple released a new iPhone", but a fruit (and not an entity at all) in "I ate an apple." "Jordan" can be a person, a country or a brand of shoes. Good NER models learn to use the surrounding words to resolve this ambiguity.

Why NER Matters: Real-World Use Cases

NER is rarely the final product. Instead, it is a building block that turns messy text into structured data that other systems can use. Here are some of the most common applications in 2026:

Search and recommendation: Search engines and e-commerce sites extract brands, products and locations from queries so that "cheap Samsung phones in Karachi" is understood as a brand plus a city, not just a bag of words.
Customer support automation: Ticketing systems pull out order numbers, product names and account IDs so tickets can be routed automatically and agents see key details instantly.
Resume parsing and HR tech: Applicant tracking systems extract names, skills, universities, job titles and companies from CVs to rank candidates.
Healthcare and life sciences: Clinical NER finds diseases, drugs, dosages and symptoms in doctor notes and research papers, supporting drug discovery and patient analytics.
Finance and compliance: Banks extract company names, tickers, amounts and dates from filings and news to feed trading signals, risk models and anti-money-laundering checks.
Legal tech: Contract analysis tools identify parties, effective dates, jurisdictions and monetary obligations across thousands of documents.
Privacy and PII redaction: NER detects personal data such as names, phone numbers, emails and ID numbers so it can be masked before data is stored, shared or sent to an AI model.
Knowledge graphs: Entities become the nodes of a knowledge graph, and relation extraction connects them ("Satya Nadella" CEO-of "Microsoft").
Retrieval-Augmented Generation (RAG): Modern AI assistants use NER to tag document chunks with entities, enabling metadata filtering and more accurate retrieval than vector search alone.

If your application handles text and needs to "understand" what that text is about, there is a good chance NER can help.

Common Entity Types

Different datasets and libraries use different label sets. The two you will see most often are the CoNLL-2003 scheme (four classic types) and the OntoNotes 5.0 scheme (eighteen types) used by spaCy's English models.

Label	Description	Example
PERSON	People, including fictional	Ada Lovelace, Imran Khan
ORG	Companies, agencies, institutions	OpenAI, State Bank of Pakistan
GPE	Countries, cities, states	Pakistan, Peshawar, Texas
LOC	Non-GPE locations, mountains, rivers	K2, the Indus River
NORP	Nationalities, religious or political groups	Pakistani, European
PRODUCT	Objects, vehicles, foods, software	iPhone 17, Toyota Corolla
EVENT	Named events, wars, sports events	Cricket World Cup
DATE	Absolute or relative dates	10 October 2026, last week
TIME	Times smaller than a day	3:30 PM, two hours
MONEY	Monetary values	$500, Rs. 2,000
PERCENT	Percentages	45%, ten percent
CARDINAL	Numerals not covered by other types	three, 1,024

CoNLL-2003 keeps it simpler with PER (person), ORG (organization), LOC (location) and MISC (miscellaneous). Domain-specific projects usually define their own labels, such as DRUG, DOSAGE and SYMPTOM for medicine, or SKILL, DEGREE and JOB_TITLE for resumes. Choosing a clear, well-documented label set is one of the most important decisions in any NER project.

How NER Works Under the Hood

Step 1: Tokenization

Before any model can find entities, the text must be split into tokens, which are usually words and punctuation marks. "Google's office in Lahore." becomes something like ["Google", "'s", "office", "in", "Lahore", "."]. Transformer models go further and split rare words into subword pieces (for example, "Peshawar" might become "Pesh", "##awar"), which matters later when we align labels.

Step 2: Sequence labeling with the BIO scheme

Most NER models treat the problem as sequence labeling: they assign one tag to every token. To represent multi-word entities, we use the BIO (also called IOB2) scheme:

B-TYPE marks the beginning of an entity
I-TYPE marks a token inside an entity (a continuation)
O marks a token outside any entity

Token	BIO Tag
Sundar	B-PER
Pichai	I-PER
visited	O
New	B-LOC
York	I-LOC
with	O
Google	B-ORG
engineers	O

The B tag is what lets a model separate two entities of the same type that sit next to each other. Some systems use the richer BILOU scheme (Begin, Inside, Last, Outside, Unit), which spaCy uses internally, but BIO is the most common format in datasets.

Step 3: Classification with context

The model then predicts a tag for each token using features of the token and its context. How it does that is where the different generations of NER differ.

The Evolution of NER: From Rules to LLMs

Understanding the history helps you pick the right tool, because every generation is still useful somewhere.

1. Rule-based and dictionary systems

The earliest systems used hand-written rules, regular expressions and lists of known names (called gazetteers). A rule might say "a capitalized word followed by 'Inc.' or 'Ltd.' is an ORG". Rules are fast, transparent and need no training data, which makes them perfect for highly structured entities like emails, phone numbers, invoice IDs and product codes. Their weakness is that they are brittle: they break on new spellings, typos and ambiguous words, and they are expensive to maintain at scale.

2. Statistical machine learning (HMM, MaxEnt, CRF)

In the 2000s, models such as Hidden Markov Models and especially Conditional Random Fields (CRFs) learned patterns from labeled data. Engineers designed features by hand: is the word capitalized, what is its suffix, what are the previous and next words, is it in a gazetteer? CRFs were state of the art for years and are still a good lightweight baseline.

3. Deep learning (BiLSTM-CRF)

Around 2015, neural networks removed most manual feature engineering. The BiLSTM-CRF architecture read the sentence in both directions, combined word embeddings with character-level features, and used a CRF layer on top to produce valid tag sequences. Accuracy jumped significantly.

4. Transformers (BERT, RoBERTa, DeBERTa)

Since 2018, pre-trained transformers have dominated. Models like BERT, RoBERTa, DeBERTa-v3 and ModernBERT are pre-trained on huge text corpora and then fine-tuned for token classification. Because they see the full context of every word through self-attention, they handle ambiguity far better than earlier models. Fine-tuned transformers are still the accuracy leaders when you have good labeled data for a fixed label set.

5. Zero-shot NER with GLiNER

Traditional models can only find the labels they were trained on. In 2023, GLiNER (Generalist and Lightweight model for NER) changed this. Instead of predicting from a fixed list, GLiNER treats NER as a matching problem: you give it the text and a list of entity type names in plain English (for example "programming language" or "medical condition"), and it scores which text spans match which labels. The original GLiNER paper reported that its largest variant beat ChatGPT and several fine-tuned LLMs in zero-shot evaluation on out-of-domain NER benchmarks, while being small enough to run on a CPU. In 2026 the family has grown further: a GLiNER bi-encoder architecture published in February 2026 pre-computes label embeddings, reaching 61.5% micro-F1 on the CrossNER zero-shot benchmark and up to 130 times higher throughput at 1,024 labels.

6. Large Language Models (LLMs)

Generative LLMs can extract entities from a simple instruction, even for complex, descriptive entity types and nested structures, with no training at all. They are excellent for prototyping, low-volume tasks and messy domains. The trade-offs are cost, latency, occasional hallucinations and less precise character offsets, which is why many production systems use LLMs to label data and then train a smaller, faster model.

Setting Up Your Python Environment

All examples in this guide use Python 3.10 or newer. Create a virtual environment and install the libraries:

python -m venv ner-env
source ner-env/bin/activate      # Windows: ner-env\Scripts\activate

pip install -U spacy nltk transformers datasets evaluate seqeval
pip install -U torch gliner fastapi uvicorn pydantic

# Download spaCy English pipelines
python -m spacy download en_core_web_sm
python -m spacy download en_core_web_trf   # optional, transformer-based

Tip: If you are on a laptop without a GPU, everything except BERT fine-tuning will run comfortably on the CPU. For fine-tuning, a free Google Colab or Kaggle GPU is more than enough.

Method 1: NER with NLTK (The Classic Approach)

NLTK (Natural Language Toolkit) is a teaching-friendly library and a good way to see the classic pipeline: tokenize, tag parts of speech, then chunk named entities.

import nltk

# One-time downloads (newer NLTK versions use the *_tab / *_eng resources)
for pkg in ["punkt_tab", "averaged_perceptron_tagger_eng",
            "maxent_ne_chunker_tab", "words"]:
    nltk.download(pkg, quiet=True)

text = "Barack Obama met Angela Merkel in Berlin to discuss NATO."

tokens = nltk.word_tokenize(text)
pos_tags = nltk.pos_tag(tokens)
tree = nltk.ne_chunk(pos_tags)

for subtree in tree:
    if hasattr(subtree, "label"):
        entity = " ".join(word for word, tag in subtree.leaves())
        print(f"{entity:20} -> {subtree.label()}")

Output:

Barack Obama         -> PERSON
Angela Merkel        -> PERSON
Berlin               -> GPE
NATO                 -> ORGANIZATION

NLTK's chunker is fast and dependency-light, but its accuracy is well below modern models, it has a small label set, and it struggles with lowercase or informal text. Use it for learning and quick experiments, not production.

Method 2: NER with spaCy (The Production Workhorse)

spaCy is the most popular industrial NLP library in Python. It is fast, well documented, and ships with pre-trained pipelines for many languages. For English you can choose between small CPU-friendly models (en_core_web_sm, md, lg) and a transformer-based model (en_core_web_trf) that is slower but more accurate.

Basic entity extraction

import spacy

nlp = spacy.load("en_core_web_sm")

text = ("Sundar Pichai announced on Tuesday that Google will invest "
        "$2 billion in a new data center in Lahore, Pakistan.")

doc = nlp(text)

for ent in doc.ents:
    print(f"{ent.text:15} {ent.label_:8} {ent.start_char:>4}-{ent.end_char:<4} "
          f"{spacy.explain(ent.label_)}")

Output (may vary slightly by model version):

Sundar Pichai   PERSON      0-13   People, including fictional
Tuesday         DATE       27-34   Absolute or relative dates or periods
Google          ORG        40-46   Companies, agencies, institutions, etc.
$2 billion      MONEY      59-69   Monetary values, including unit
Lahore          GPE        94-100  Countries, cities, states
Pakistan        GPE       102-110  Countries, cities, states

Each entity is a Span object with useful attributes: the text, the label, character offsets (start_char, end_char) and token offsets (start, end). Character offsets are essential when you want to highlight entities in a web UI or redact them.

Visualizing entities with displaCy

spaCy includes displaCy, a visualizer that highlights entities in color. In a Jupyter notebook, run:

from spacy import displacy

displacy.render(doc, style="ent", jupyter=True)

# Outside a notebook, export to HTML instead:
html = displacy.render(doc, style="ent", page=True)
with open("entities.html", "w", encoding="utf-8") as f:
    f.write(html)

This is a great way to debug a model and to demo results to non-technical stakeholders.

Processing many documents efficiently

Calling nlp() in a loop is slow for large datasets. Use nlp.pipe(), which batches documents, and disable components you do not need:

texts = load_my_documents()  # a list or generator of strings

with nlp.select_pipes(enable=["tok2vec", "ner"]):
    for doc in nlp.pipe(texts, batch_size=256, n_process=2):
        entities = [(e.text, e.label_) for e in doc.ents]
        save_to_database(doc.text, entities)

On a typical laptop, the small English model can process thousands of short documents per second this way.

Adding rules with the EntityRuler

Pre-trained models do not know your company's products, internal codes or local names. spaCy's EntityRuler lets you add pattern-based entities that work alongside the statistical model. This hybrid approach is one of the most practical techniques in real projects.

import spacy

nlp = spacy.load("en_core_web_sm")
ruler = nlp.add_pipe("entity_ruler", before="ner")

patterns = [
    {"label": "ORG", "pattern": "Sarhad University"},
    {"label": "PRODUCT", "pattern": [{"LOWER": "laravel"}]},
    {"label": "PRODUCT", "pattern": [{"LOWER": "next"}, {"TEXT": "."}, {"LOWER": "js"}]},
    # Regex-like token pattern for ticket IDs such as TCK-48213
    {"label": "TICKET_ID", "pattern": [{"TEXT": {"REGEX": r"^TCK-\d{5}$"}}]},
]
ruler.add_patterns(patterns)

doc = nlp("Ticket TCK-48213: a Sarhad University student reports a Laravel bug.")
print([(e.text, e.label_) for e in doc.ents])

Placing the ruler before the "ner" component means its matches take priority and the statistical model fills in the rest. Place it after if you only want rules to add entities the model missed.

Method 3: NER with Hugging Face Transformers

The Hugging Face Hub hosts thousands of pre-trained NER models for different languages and domains. The pipeline API makes them easy to use:

from transformers import pipeline

ner = pipeline(
    "token-classification",
    model="dslim/bert-base-NER",
    aggregation_strategy="simple",   # merge subword pieces into whole entities
)

text = "Elon Musk said Tesla will open a factory in Berlin next year."

for ent in ner(text):
    print(f"{ent['word']:12} {ent['entity_group']:5} "
          f"score={ent['score']:.3f} span=({ent['start']}, {ent['end']})")

Output:

Elon Musk    PER   score=0.999 span=(0, 9)
Tesla        ORG   score=0.998 span=(15, 20)
Berlin       LOC   score=0.999 span=(44, 50)

The aggregation_strategy parameter is important. Without it, you get one prediction per subword token (like "Ber" and "##lin") with raw B- and I- tags. With "simple", "first", "average" or "max", the pipeline groups tokens into complete entities. "simple" is a good default; "first" or "max" can give cleaner results when words are split into many pieces.

Choosing a model from the Hub

Search the Hub with the "token-classification" filter and look at:

Label set: Does it predict the entity types you need?
Language: Multilingual models such as XLM-RoBERTa-based NER models can handle Urdu, Arabic, Hindi and dozens of other languages.
Domain: Biomedical, legal, financial and social media models usually beat general models on their own domain.
Size and license: A distilled model may be fast enough for real-time APIs; always check the license before commercial use.

Running on a GPU is as simple as adding device=0 to the pipeline call, and you can pass a list of texts with batch_size for higher throughput.

Method 4: Zero-Shot NER with GLiNER

What if you need entity types that no pre-trained model supports, like "programming language", "university", "job title" or "medical symptom", and you have no labeled data? This is exactly the problem GLiNER solves.

from gliner import GLiNER

model = GLiNER.from_pretrained("urchade/gliner_medium-v2.1")

text = """Syed is a full-stack developer from Peshawar who studied at
Sarhad University. He builds web apps with Laravel, React and Next.js
and recently joined Systems Limited as a Software Engineer."""

labels = ["person", "city", "university", "programming framework",
          "company", "job title"]

entities = model.predict_entities(text, labels, threshold=0.4)

for e in entities:
    print(f"{e['text']:22} -> {e['label']:22} ({e['score']:.2f})")

Typical output:

Syed                   -> person                 (0.93)
Peshawar               -> city                   (0.95)
Sarhad University      -> university             (0.91)
Laravel                -> programming framework  (0.88)
React                  -> programming framework  (0.86)
Next.js                -> programming framework  (0.84)
Systems Limited        -> company                (0.89)
Software Engineer      -> job title              (0.90)

No training, no annotation, just a list of labels in plain English. A few practical tips for GLiNER:

Write clear, specific labels. "programming framework" works better than "tech". Lowercase natural-language labels usually work well.
Tune the threshold. Lower values (0.3) increase recall but add noise; higher values (0.5 to 0.6) favor precision. Test on a small sample of real data.
Chunk long documents. GLiNER models have a maximum input length (often around 384 to 512 tokens). Split long texts into sentences or paragraphs and adjust offsets when merging results.
Consider the bi-encoder variants for many labels. The newer bi-encoder models encode labels separately, which lets you cache label embeddings and is reported to be 2 to 3 times faster than the original architecture with dozens of entity types.
Fine-tune when needed. GLiNER can also be fine-tuned on a few hundred examples of your own data, often giving a large accuracy boost.

GLiNER has become a favorite for rapid prototyping, PII detection and domains where label sets change often.

Method 5: NER with Large Language Models

LLMs can perform NER from a well-written prompt. The key to doing this reliably is to ask for structured JSON output and validate it in code. Here is an example using the Anthropic Python SDK and Pydantic; the same pattern works with any LLM provider or a local open-weight model.

import json
from anthropic import Anthropic
from pydantic import BaseModel, ValidationError

class Entity(BaseModel):
    text: str
    label: str

class EntityList(BaseModel):
    entities: list[Entity]

client = Anthropic()  # reads ANTHROPIC_API_KEY from the environment

LABELS = ["PERSON", "ORG", "LOCATION", "DATE", "MONEY", "PRODUCT"]

def llm_ner(text: str) -> list[Entity]:
    prompt = f"""Extract named entities from the text below.
Allowed labels: {", ".join(LABELS)}.
Copy each entity exactly as it appears in the text. Do not invent entities.
Return only JSON in this format:
{{"entities": [{{"text": "...", "label": "..."}}]}}

Text: {text}"""

    response = client.messages.create(
        model="claude-haiku-5-5",
        max_tokens=1024,
        messages=[{"role": "user", "content": prompt}],
    )
    raw = response.content[0].text
    try:
        data = EntityList.model_validate(json.loads(raw))
    except (json.JSONDecodeError, ValidationError):
        return []
    # Keep only entities that really occur in the text (guards against hallucination)
    return [e for e in data.entities if e.text in text and e.label in LABELS]

print(llm_ner("Ali paid Rs. 15,000 to Daraz for a Samsung Galaxy on 3 March."))

Notice the three safety nets: a strict output format, schema validation with Pydantic, and a check that every extracted entity actually appears in the source text. Since LLMs return text rather than offsets, you can recover character positions with text.find() or a regular expression.

When should you use an LLM for NER?

Great for: prototyping, small volumes, complex or descriptive entity types, multilingual and messy text, and generating training labels (often called "LLM-assisted annotation" or "distillation").
Less ideal for: high-volume, low-latency pipelines, strict privacy requirements where text cannot leave your servers, and cases where you need exact, reproducible offsets.

A very effective 2026 workflow is: use an LLM (or GLiNER) to pre-label a few thousand examples, have a human review and correct them, then train a small spaCy or BERT model that runs cheaply and quickly in production.

Training a Custom NER Model with spaCy

Pre-trained models are general-purpose. When your domain has its own vocabulary, such as skills on resumes or drug names in clinical notes, training a custom model usually gives the best balance of speed and accuracy.

Step 1: Prepare annotated data

You need examples of text with the character offsets and labels of each entity. Annotation tools such as Label Studio, Doccano or Prodigy export this format. A tiny example:

TRAIN_DATA = [
    ("I have 3 years of experience with Python and Django.",
     {"entities": [(34, 40, "SKILL"), (45, 51, "SKILL")]}),
    ("Graduated from NUST with a BS in Computer Science.",
     {"entities": [(15, 19, "UNIVERSITY"), (27, 49, "DEGREE")]}),
    ("Worked as a Backend Engineer at Careem for two years.",
     {"entities": [(12, 28, "JOB_TITLE"), (32, 38, "ORG")]}),
    # ... hundreds more examples
]

Watch your offsets. A single off-by-one error causes spaCy to skip the entity. Always validate them, as shown next.

Step 2: Convert to spaCy's binary format

import spacy
from spacy.tokens import DocBin

def to_docbin(data, output_path):
    nlp = spacy.blank("en")
    db = DocBin()
    for text, annot in data:
        doc = nlp.make_doc(text)
        spans = []
        for start, end, label in annot["entities"]:
            span = doc.char_span(start, end, label=label,
                                 alignment_mode="contract")
            if span is None:
                print(f"Skipping misaligned entity: {text[start:end]!r}")
            else:
                spans.append(span)
        doc.ents = spans
        db.add(doc)
    db.to_disk(output_path)

# Split your data (e.g., 80/20) before converting
to_docbin(train_examples, "train.spacy")
to_docbin(dev_examples, "dev.spacy")

Step 3: Create a config and train

spaCy v3 uses a configuration file that captures every training setting, which makes experiments reproducible.

# Generate a config optimized for accuracy (use --optimize efficiency for speed)
python -m spacy init config config.cfg --lang en --pipeline ner --optimize accuracy

# Train
python -m spacy train config.cfg --output ./resume_ner \
    --paths.train ./train.spacy --paths.dev ./dev.spacy

# Optional: check your data for problems before training
python -m spacy debug data config.cfg \
    --paths.train ./train.spacy --paths.dev ./dev.spacy

Training prints a table with loss and the entity metrics ENTS_P, ENTS_R and ENTS_F (precision, recall and F1) on the dev set after each evaluation step. The best model is saved to resume_ner/model-best.

Step 4: Use your model

import spacy

nlp = spacy.load("./resume_ner/model-best")
doc = nlp("Senior Laravel Developer with an MS from FAST, skilled in Vue and Docker.")
print([(e.text, e.label_) for e in doc.ents])

How much data do you need? As a rough guide, a few hundred well-annotated examples per label give a usable model, and one to two thousand give a strong one. Quality and consistency matter more than raw quantity. If you add the GPU-based transformer config (by passing --gpu-id 0 when generating it), accuracy improves further at the cost of speed.

Fine-Tuning BERT for NER with Hugging Face

If you want maximum accuracy, fine-tuning a transformer is the standard approach. Here is a complete, minimal training script on the classic CoNLL-2003 dataset. You can swap in your own dataset as long as it has a "tokens" column and a "ner_tags" column.

Load the data

from datasets import load_dataset

# If this name stops resolving in your datasets version, search the Hub
# for a Parquet copy of CoNLL-2003 and use that dataset ID instead.
dataset = load_dataset("eriktks/conll2003")

label_names = dataset["train"].features["ner_tags"].feature.names
id2label = {i: l for i, l in enumerate(label_names)}
label2id = {l: i for i, l in enumerate(label_names)}
print(label_names)
# ['O', 'B-PER', 'I-PER', 'B-ORG', 'I-ORG', 'B-LOC', 'I-LOC', 'B-MISC', 'I-MISC']

Tokenize and align labels

This is the step most beginners get wrong. The dataset has one label per word, but the tokenizer may split a word into several subword tokens. We give the label to the first subword and mark the rest, plus special tokens like [CLS] and [SEP], with -100 so the loss function ignores them.

from transformers import AutoTokenizer

checkpoint = "bert-base-cased"   # cased models are better for NER
tokenizer = AutoTokenizer.from_pretrained(checkpoint)

def tokenize_and_align(batch):
    tokenized = tokenizer(batch["tokens"], truncation=True,
                          is_split_into_words=True)
    all_labels = []
    for i, word_labels in enumerate(batch["ner_tags"]):
        word_ids = tokenized.word_ids(batch_index=i)
        prev_word = None
        labels = []
        for word_id in word_ids:
            if word_id is None:
                labels.append(-100)              # special token
            elif word_id != prev_word:
                labels.append(word_labels[word_id])  # first subword
            else:
                labels.append(-100)              # other subwords
            prev_word = word_id
        all_labels.append(labels)
    tokenized["labels"] = all_labels
    return tokenized

tokenized_ds = dataset.map(tokenize_and_align, batched=True,
                           remove_columns=dataset["train"].column_names)

Define metrics with seqeval

import numpy as np
import evaluate

seqeval = evaluate.load("seqeval")

def compute_metrics(eval_pred):
    logits, labels = eval_pred
    preds = np.argmax(logits, axis=-1)
    true_preds = [[label_names[p] for p, l in zip(pr, lb) if l != -100]
                  for pr, lb in zip(preds, labels)]
    true_labels = [[label_names[l] for p, l in zip(pr, lb) if l != -100]
                   for pr, lb in zip(preds, labels)]
    r = seqeval.compute(predictions=true_preds, references=true_labels)
    return {"precision": r["overall_precision"], "recall": r["overall_recall"],
            "f1": r["overall_f1"], "accuracy": r["overall_accuracy"]}

Train the model

from transformers import (AutoModelForTokenClassification, TrainingArguments,
                          Trainer, DataCollatorForTokenClassification)

model = AutoModelForTokenClassification.from_pretrained(
    checkpoint, num_labels=len(label_names),
    id2label=id2label, label2id=label2id)

args = TrainingArguments(
    output_dir="bert-ner-conll",
    eval_strategy="epoch",
    save_strategy="epoch",
    learning_rate=2e-5,
    per_device_train_batch_size=16,
    per_device_eval_batch_size=32,
    num_train_epochs=3,
    weight_decay=0.01,
    load_best_model_at_end=True,
    metric_for_best_model="f1",
)

trainer = Trainer(
    model=model,
    args=args,
    train_dataset=tokenized_ds["train"],
    eval_dataset=tokenized_ds["validation"],
    data_collator=DataCollatorForTokenClassification(tokenizer),
    compute_metrics=compute_metrics,
)

trainer.train()
trainer.save_model("bert-ner-conll/final")
tokenizer.save_pretrained("bert-ner-conll/final")

On a single modern GPU this takes roughly 15 to 30 minutes, and a fine-tuned bert-base-cased model typically reaches an entity-level F1 of about 0.90 to 0.92 on the CoNLL-2003 test set. Larger models such as RoBERTa-large or DeBERTa-v3 push that higher. Load your fine-tuned model with the same pipeline() call from Method 3 by pointing model to "bert-ner-conll/final".

How to Evaluate NER Models Correctly

A model that "looks good" on three example sentences can still fail badly in production. Proper evaluation needs a held-out test set and the right metrics.

Entity-level precision, recall and F1

Precision: Of all entities the model predicted, how many were correct? Low precision means many false alarms.
Recall: Of all real entities in the text, how many did the model find? Low recall means many misses.
F1 score: The harmonic mean of precision and recall, giving a single balanced number.

The standard (CoNLL-style) evaluation is strict: a prediction counts as correct only if both the span boundaries and the label exactly match the gold annotation. Predicting "New" as LOC when the gold entity is "New York" is wrong. Token-level accuracy is misleading for NER because most tokens are "O", so a model that predicts nothing at all can still score over 80% accuracy.

from seqeval.metrics import classification_report

y_true = [["B-PER", "I-PER", "O", "B-ORG", "O", "B-LOC", "I-LOC"]]
y_pred = [["B-PER", "I-PER", "O", "B-ORG", "O", "B-LOC", "O"]]

print(classification_report(y_true, y_pred, digits=3))

The per-label report shows that PER and ORG are perfect while LOC fails, because "New York" was cut short. Per-label breakdowns like this tell you exactly where to focus your next round of data collection.

Error analysis checklist

Boundary errors: The right type but wrong span ("Bank" instead of "State Bank of Pakistan").
Type confusion: The right span but wrong label (ORG vs GPE for "Pakistan Cricket Board" vs "Pakistan").
Missed entities: Usually rare names, lowercase text or new domain terms.
False positives: Capitalized words at the start of a sentence or in headlines.

Always look at real errors, not just the numbers. Twenty minutes of reading mistakes will teach you more than any single metric.

Comparing NER Approaches in 2026

Approach	Training data	Speed	Accuracy	Best for
Rules / regex	None	Very fast	High on fixed patterns	IDs, emails, codes
NLTK	None	Fast	Low to moderate	Learning, quick tests
spaCy (sm/lg)	Optional	Very fast	Good	Production pipelines on CPU
spaCy (trf) / BERT fine-tuned	Hundreds to thousands	Moderate	Very high	Fixed labels, max accuracy
GLiNER	None (optional fine-tune)	Fast	Good to very good	Custom labels, zero-shot
LLM prompting	None	Slow, paid per call	Good to very good	Prototypes, complex types, labeling

There is no single best method. A very common production design combines several: rules for structured patterns, a trained model for the core entity types, and GLiNER or an LLM for long-tail or newly requested types.

Hands-On Project: Build an Entity Extraction API with FastAPI

Let us turn what we have learned into a small but realistic service. This API combines spaCy's pre-trained model, custom rules and GLiNER zero-shot labels, and returns clean JSON that a React, Next.js or Laravel front end can consume.

# app.py
from fastapi import FastAPI
from pydantic import BaseModel
import spacy
from gliner import GLiNER

app = FastAPI(title="Entity Extraction API")

# Load models once at startup, not per request
nlp = spacy.load("en_core_web_sm")
ruler = nlp.add_pipe("entity_ruler", before="ner")
ruler.add_patterns([
    {"label": "EMAIL", "pattern": [{"LIKE_EMAIL": True}]},
    {"label": "URL", "pattern": [{"LIKE_URL": True}]},
])
gliner = GLiNER.from_pretrained("urchade/gliner_medium-v2.1")

class ExtractRequest(BaseModel):
    text: str
    custom_labels: list[str] = []
    threshold: float = 0.45

class EntityOut(BaseModel):
    text: str
    label: str
    start: int
    end: int
    source: str

@app.post("/extract", response_model=list[EntityOut])
def extract(req: ExtractRequest):
    results = []
    doc = nlp(req.text)
    for ent in doc.ents:
        results.append(EntityOut(text=ent.text, label=ent.label_,
                                 start=ent.start_char, end=ent.end_char,
                                 source="spacy"))
    if req.custom_labels:
        for e in gliner.predict_entities(req.text, req.custom_labels,
                                         threshold=req.threshold):
            results.append(EntityOut(text=e["text"], label=e["label"].upper(),
                                     start=e["start"], end=e["end"],
                                     source="gliner"))
    # Remove spans already covered by an earlier result
    seen, unique = set(), []
    for r in sorted(results, key=lambda r: (r.start, -(r.end - r.start))):
        if not any(r.start < e and r.end > s for s, e in seen):
            unique.append(r)
            seen.add((r.start, r.end))
    return unique

Run it locally and test it:

uvicorn app:app --reload

curl -X POST http://127.0.0.1:8000/extract \
  -H "Content-Type: application/json" \
  -d '{"text": "Contact Hina at hina@example.com. She knows Python and Flutter.",
       "custom_labels": ["programming language", "framework"]}'

FastAPI automatically generates interactive documentation at /docs, so you can test the endpoint in the browser. From here you can add authentication, request size limits, batching and a Dockerfile, and deploy it to any cloud provider. This makes an excellent portfolio project or the core of a final year project, for example an AI resume screener or a news analytics dashboard.

Common Challenges in NER (and How to Solve Them)

1. Ambiguity

"Washington" can be a person, a city or a state. Solution: use context-aware models (transformers), and for critical applications add entity linking to connect mentions to a knowledge base such as Wikidata.

2. Domain shift

A model trained on news performs poorly on tweets, medical notes or code comments. Solution: fine-tune on a few hundred in-domain examples, or use a domain-specific model like scispaCy for biomedical text.

3. Nested and overlapping entities

In "Bank of America headquarters in Charlotte", "America" is a location nested inside an organization. Standard BIO tagging cannot represent nesting. Solution: use span-based models such as spaCy's SpanCategorizer or GLiNER, which score spans rather than individual tokens.

4. Low-resource languages

Urdu, Pashto and many regional languages have little labeled data. Solution: multilingual transformers (XLM-RoBERTa, mDeBERTa) transfer knowledge across languages surprisingly well, and LLM-assisted annotation can bootstrap a dataset quickly. Pay special attention to tokenization and script direction for right-to-left languages.

5. Noisy and informal text

Lowercase names, typos, slang, emojis and code-mixed text (for example, Roman Urdu mixed with English) confuse models trained on clean text. Solution: include noisy examples in training data and use data augmentation such as random lowercasing.

6. Inconsistent annotation

If two annotators label "Dr. Ahmed" differently, the model learns confusion. Solution: write a short annotation guideline with examples and edge cases, and measure inter-annotator agreement on a sample.

7. Long documents

Transformers have input limits. Solution: split documents into sentences or overlapping windows, run NER on each chunk, then map offsets back and merge duplicate entities at chunk borders.

Best Practices for Production NER

Start with the simplest thing that works. Try a pre-trained spaCy model and a few rules before training anything.
Define your label schema clearly. Fewer, well-defined labels beat many vague ones.
Build a gold test set early. Even 200 carefully labeled examples let you compare approaches objectively.
Combine methods. Rules for patterns, models for language, zero-shot for the long tail.
Post-process results. Normalize entities ("Pak", "Pakistan" and "Islamic Republic of Pakistan" can map to one ID), strip stray punctuation and merge duplicates.
Load models once. Keep them in memory at startup; loading per request is a common performance bug.
Batch your inputs. Use nlp.pipe() or batched pipelines for big throughput gains.
Monitor in production. Log confidence scores, sample predictions for human review, and watch for drift as your data changes.
Respect privacy. If you process personal data, prefer local models, redact PII before sending text to external APIs, and follow applicable data-protection laws.
Version everything. Keep your data, config and model versions together so results are reproducible.

Frequently Asked Questions (FAQ)

What is the difference between NER and POS tagging?

Part-of-speech tagging labels the grammatical role of every word (noun, verb, adjective). NER only labels spans that refer to specific named things (people, places, organizations) and assigns semantic categories. POS tags are often used as features for classic NER systems.

Which Python library is best for NER in 2026?

For most production applications, spaCy is the best starting point because it is fast, reliable and easy to deploy. Use Hugging Face Transformers when you need maximum accuracy or a specific pre-trained domain model, GLiNER when you need custom entity types without training data, and LLMs for prototyping or labeling data.

Can NER work for Urdu and other non-English languages?

Yes. spaCy offers pipelines for many languages, and multilingual transformer models on the Hugging Face Hub support over 100 languages, including Urdu, Arabic and Hindi. Accuracy is usually lower than for English, so fine-tuning on local data is highly recommended.

Is NER still relevant now that we have LLMs?

Absolutely. Dedicated NER models are much faster, cheaper, more private and more predictable than calling an LLM for every document. In fact, NER is a key component of many LLM systems, for tasks like RAG metadata, PII redaction and structured data extraction.

How much data do I need to train a custom NER model?

It depends on the difficulty of your labels, but a few hundred examples per entity type usually produce a usable model, and a few thousand produce a strong one. With GLiNER or LLM pre-labeling, you can reach that amount of data much faster.

What is a good F1 score for NER?

On standard benchmarks like CoNLL-2003, strong models score above 0.90 F1. In real-world domains, an F1 of 0.80 to 0.85 is often good enough for production, especially when combined with human review. The right target depends on the cost of errors in your application.

Conclusion: Your Next Steps with NER

Named Entity Recognition turns unstructured text into structured, searchable and actionable data. In this guide, you learned how NER works with tokenization and BIO tagging, how the field evolved from rules to transformers, GLiNER and LLMs, and how to apply five different methods in Python. You also trained a custom spaCy model, fine-tuned BERT, evaluated results with seqeval and built a deployable FastAPI service.

Key takeaways:

NER finds and classifies entity spans such as PERSON, ORG, GPE, DATE and MONEY.
spaCy is the best default for fast, production-ready NER on a CPU.
Fine-tuned transformers give the highest accuracy on a fixed label set.
GLiNER enables zero-shot NER with any labels you can describe in words.
LLMs are powerful for prototyping and labeling data, but costly at scale.
Always evaluate with entity-level precision, recall and F1 on a real test set.
The strongest real-world systems combine rules, trained models and zero-shot methods.

The best way to learn is by building. Pick a dataset that interests you, such as job postings, local news or product reviews, and extract entities from it using two or three of the methods above. Compare the results, study the mistakes and iterate. Within a weekend, you will have a practical NER skill that applies to almost every text-heavy project you will ever work on.

Happy coding, and if you found this guide useful, share it with a fellow developer who is starting their NLP journey!

References and Further Reading

Zaratiana et al., GLiNER: Generalist Model for Named Entity Recognition using Bidirectional Transformer (arXiv:2311.08526)
Stepanov et al., The Million-Label NER: Breaking Scale Barriers with GLiNER bi-encoder (arXiv:2602.18487, Feb 2026)
GLiNER on PyPI: pypi.org/project/GLiNER
spaCy documentation: spacy.io
Hugging Face Transformers token classification documentation: huggingface.co/docs/transformers

AI Tools & Artificial Intelligence | ChatGPT, AI News & Tutorials

NLP (2026): Complete Guide with Python Code