Named Entity Recognition

Named entity recognition pulls the proper nouns out of running text and says what kind each one is.

The standard types are person, organisation, location, date and money, though domains add their own.

A clinical system tags drugs and dosages; a legal one tags parties, statutes and case numbers.

Unlike text classification, the output is a set of spans, each with a start, an end and a type.

That is why it is modelled as tagging every token rather than labelling the document.

The usual scheme marks each token as the beginning of an entity, inside one, or outside any.

That encoding is what lets a model represent multi-word entities like New York University.

The central difficulty is ambiguity, and it is not solvable by looking words up.

Washington is a person, a city, a state and an organisation, all spelled identically.

Apple is a company in one sentence and a fruit in the next, with no spelling difference to help.

Only surrounding words disambiguate, which is why contextual models replaced gazetteers and rules.

Fine-tuned encoder models such as BERT became the standard, and large models can now do it by prompt.

Evaluation is by exact span match, so getting the type right but the boundary wrong still counts as wrong.

The applications are mostly plumbing: redacting personal data, filtering search, and populating databases.

It is also the first step of entity linking, which connects each span to a specific record in a knowledge base.