TracksStudyAbout
Sign in
Start studying
Module 3 · Computer Vision Module contents
Module 3  ·  Card 22 / 22intermediate

OCR

OCR is optical character recognition, turning images of text into machine text.

The pipeline usually detects text regions and then recognises the characters.

Classic systems used a CNN for features and an RNN with CTC loss for sequences.

Modern systems increasingly use transformer-based vision-language models end to end.

Handwriting, skew, low resolution and unusual fonts are the persistent hard cases.

Back Finish module
Learn in depth ↓
Module 03 · Computer Vision · 22 concepts
In depth  /  15 lines

OCR, in more detail

OCR turns an image containing text into text you can search and edit.

It is one of the oldest applied computer vision problems and still not fully solved.

The usual pipeline has two stages, detection and recognition.

Detection finds regions of the image that contain text.

Recognition reads the characters within each region.

The classical recogniser used a CNN for features and a recurrent layer for the sequence.

CTC loss made that trainable without knowing where each character sits.

That alignment problem is the hard part, since you have an image and a string and no mapping.

Modern systems increasingly use transformer models that read the crop end to end.

Vision-language models can now read a document and answer questions about it directly.

Clean printed text on a flat page is essentially solved.

Handwriting, low resolution, skew, glare and unusual fonts remain genuinely hard.

Tables and multi-column layouts fail in a particular way, by scrambling reading order.

That failure is a common and quiet source of corruption in document pipelines.

Post-processing against a known vocabulary fixes many of the remaining errors.

Want the full technical article? Wikipedia ↗
Deep dive

What is underneath OCR

The alignment problem

An image of a word is a strip of pixels. The answer is a string of characters. Nothing says which columns of pixels belong to which character.

T≫LT \gg L
  • TT: the number of frames, meaning columns of the feature map.
  • LL: the number of characters in the label.

Labelling every column by hand would solve it and is exactly what nobody has. The training data is a picture and its transcription, with no alignment between them.

CTC, which sums over every alignment

p(y∣x)=∑π∈B−1(y)∏t=1Tp(πt∣x)p(\mathbf{y} \mid \mathbf{x}) = \sum_{\boldsymbol{\pi} \in \mathcal{B}^{-1}(\mathbf{y})} \prod_{t=1}^{T} p(\pi_t \mid \mathbf{x})
  • π\boldsymbol{\pi}: one path, meaning one character or blank chosen at every frame.
  • B\mathcal{B}: the collapsing rule that turns a path into a string.

The network emits a distribution over the alphabet plus a blank symbol at every frame, and the probability of a transcription is the total probability of every path that collapses to it.

B(h-e-l-ϵ-l-l-o)=hello\mathcal{B}(\text{h-e-l-}\epsilon\text{-l-l-o}) = \text{hello}

Collapsing removes repeats first and then blanks, so the blank is what allows a genuine double letter to survive.

LCTC=−log⁡p(y∣x)\mathcal{L}_{\mathrm{CTC}} = -\log p(\mathbf{y} \mid \mathbf{x})

The number of paths is exponential in TT, and the sum is still computed exactly by a forward-backward recursion in O(TL)O(TL), which is the contribution of the method.

One line of text, in order

  1. Detect and crop the text region, then rectify it to a fixed height.
  2. Run a convolutional stack, producing a feature map one row tall and TT columns wide.
  3. Read those columns as a sequence through a recurrent or transformer layer.
  4. Project each frame to a distribution over the alphabet plus blank.
  5. Decode the frames into a string, and score the whole thing against the label with CTC during training.

Decoding, where a language model earns its place

y^=B(arg⁡max⁡π∏tp(πt))\hat{\mathbf{y}} = \mathcal{B}\left( \arg\max_{\boldsymbol{\pi}} \prod_t p(\pi_t) \right)

Greedy decoding takes the most likely symbol at every frame and collapses the result. It is fast and it is not the most likely transcription, because many paths can share a string.

score(y)=log⁡p(y∣x)+αlog⁡pLM(y)+β∣y∣\mathrm{score}(\mathbf{y}) = \log p(\mathbf{y} \mid \mathbf{x}) + \alpha \log p_{\mathrm{LM}}(\mathbf{y}) + \beta |\mathbf{y}|

Beam search keeps several candidate strings and can add a language model score, which is what turns a plausible character sequence into a real word.

The length term β\beta is there because adding a log probability per character otherwise biases the search towards short outputs.

What actually breaks in production

CER=S+D+IN\text{CER} = \frac{S + D + I}{N}

Character error rate is the edit distance to the truth over the number of characters, and it is the metric to report because word accuracy hides near misses.

CTC assumes the output is monotonic in the input, which holds for a line of text and fails for anything with layout, so pages need detection and reading order before recognition ever runs.

Curved, rotated or low-contrast text fails at the rectification step rather than the recognition step, and that is where most of the remaining error lives.

Attention-based decoders remove the monotonic assumption and handle curved text better.

They pay for it by being able to hallucinate a plausible word that is not in the image, which CTC structurally cannot do.

Test yourself

One question on this card. Every option explains itself once you pick, right or wrong.

Question 1
Which conditions remain hardest for OCR?
Computer Vision

22 concepts in this module, then a graded test with every answer explained.

Study this module →
↑↓ move↵ openesc close