Output Parsing and Retries

The moment a model's output feeds code rather than a person, its shape becomes a contract.

Asking for JSON in the prompt is a request, not a guarantee, and requests are sometimes declined.

Typical failures are a prose preamble before the object, a trailing comma, or a missing required field.

So every production path needs a parsing layer between the model and whatever consumes it.

Parsing has three stages, and skipping any of them produces a different class of bug.

First, extract and load the structure, tolerating the code fences models like to wrap things in.

Second, validate against a schema so missing or misnamed fields are caught rather than defaulted.

Third, check semantics, since a valid schema can still carry an out-of-range date or a negative price.

When a check fails, the useful move is to feed the specific error back and ask for a correction.

Models are good at repairing their own output when told precisely what was wrong with it.

That retry must be bounded, because the second failure usually predicts the third.

Two attempts then an explicit error is a reasonable default for most applications.

Log every failure with the raw text, since a rising parse-failure rate is an early signal of drift.

The stronger option is to remove the possibility of invalid output at generation time.

Constrained decoding restricts the sampler to tokens that keep the output valid against a grammar or schema.