Alignment means getting a model to do what people actually want.
A capable model is not automatically a helpful or a safe one.
The gap opens because training optimises a measurable proxy, never the real intention.
RLHF, DPO and constitutional methods are all attempts to close that gap.
It remains unsolved, because nobody can fully write down what we want.
Alignment means getting a model to do what people actually want it to do.
Capability and alignment are different axes, and progress on one does not deliver the other.
A very capable model pursuing the wrong goal is more dangerous than a weak one.
The difficulty starts with the fact that training needs a measurable objective.
What we actually want is rarely measurable, so a proxy is used in its place.
Optimising hard against a proxy pulls the model away from the thing the proxy stood for.
That is the same mechanism as reward hacking, seen from the outside.
RLHF was the first method to work at scale, using human rankings as the objective.
DPO reaches a similar result without a separate reward model or a reinforcement learning loop.
Constitutional methods replace some of the human labelling with a written set of principles.
All three optimise a stand-in for human preference rather than human preference itself.
Alignment also covers honesty, refusing genuinely harmful requests and admitting uncertainty.
Evaluation is the hard part, because the failures are rare, subtle and often adversarial.
Nobody can write down the full specification of what we want, which is why this stays open.
Treat every alignment method as a reduction of the problem rather than a solution to it.
One question on this card. Every option explains itself once you pick, right or wrong.
23 concepts in this module, then a graded test with every answer explained.