These are privacy attacks that pull training data back out of a trained model.
Membership inference asks whether one specific record was in the training set.
Model inversion is more ambitious and reconstructs features of the data itself.
Both exploit a model behaving more confidently on examples it has already seen.
Adding noise during training, returning labels instead of scores and rate limiting reduce the signal.
Membership inference and model inversion are privacy attacks aimed at the training data rather than the application.
They matter because a trained model is, in part, a lossy record of everything it was trained on.
Membership inference asks one narrow question, which is whether a particular record was in the training set.
That alone can be a serious disclosure when the dataset is a list of patients, defaulters or subscribers.
The attack works because models behave measurably differently on data they have already seen.
Confidence tends to run higher and loss lower on training examples than on genuinely new ones.
An attacker compares the model's response on a candidate record against that expected pattern.
Model inversion is more ambitious and tries to reconstruct the underlying data itself.
Given a label and enough queries, it can gradually recover features typical of that class.
Overfitted models leak considerably more than well-regularised ones, because they memorised more.
Small training sets and rare records are the most exposed, since the model effectively saw them as individuals.
Adding calibrated noise during training bounds how much any single record can influence the result.
Returning a label instead of a full confidence score removes most of the signal an attacker relies on.
Rate limiting matters as well, because both attacks need a great many queries to work at all.
Deduplicate and minimise what you collect, since data you never held cannot be recovered from your model.
A model fits its training data better than anything else, and that gap is the whole attack surface.
Membership inference asks whether a specific record was in the training set, and the simplest version is a threshold on the loss.
The attack is only meaningful above chance, and it is measured as the true positive rate at a very low false positive rate rather than as an average accuracy.
Knowing that one person's record was used to train a medical model is a disclosure even when the record itself is never recovered.
The wider the generalisation gap, the better the attack works.
A model whose training and test losses are equal leaks nothing through this channel, because there is no signal to threshold.
The stronger version trains copies of the model on known data, learns what a member's loss looks like, and applies that classifier to the target.
Duplicated records are the worst case. A sequence appearing many times in the corpus is memorised rather than learned, and it can be extracted verbatim.
Inversion optimises an input to maximise a class score, reconstructing a representative example of that class rather than testing a specific record.
On a face recognition model that produces a recognisable average face for a named person, from the weights alone.
For language models the equivalent is simpler: a prefix from the training data often causes the model to continue it verbatim, including keys, addresses and personal details.
The guarantee is that the output distribution barely changes when any single record is added or removed, which bounds what any attack can learn about that record.
DP-SGD implements it by clipping each example's gradient and adding calibrated noise, so no single example can move the weights far.
The cost is accuracy and compute, and per-example gradient clipping is several times slower than ordinary training, which is why it is reserved for data that warrants it.
One question on this card. Every option explains itself once you pick, right or wrong.
16 concepts in this module, then a graded test with every answer explained.