Running Experiments

An experiment should test one clearly stated hypothesis.

Write the hypothesis down before running anything.

Adding this component improves accuracy on this dataset is testable.

Trying some things and seeing what happens is not, and it produces unusable results.

Change one variable at a time and hold everything else fixed.

Changing two means you cannot attribute the difference to either.

Fix random seeds so a run can be repeated exactly.

Then run across several seeds, because a single run is one sample of a noisy process.

Report the mean and the spread rather than the best result you saw.

Reporting the best of five is a quiet and common form of cheating.

Log every configuration, since you will not remember what you ran by Thursday.

Include the failed runs, which is where most of the information sits.

Decide the metric before looking at the results, or you will choose the flattering one.

Negative results are informative and worth recording properly.

Most experiments fail, and a well-run failure still narrows the space.