Tuning is a loop: propose a change, run it, measure, keep or revert. Written as a program with the eval harness as the referee, it stops being a set of manual runs and becomes a search that a person can read the log of.
About 1 min of reading
Before this new lesson, one small question about the lesson you just finished. Recall it, do not look it up.
Track the best.
The log
Before you run it, say out loud what it will print. Then open the answer.
Read the first example one line at a time. A line that prints shows what appears on screen; the rest set things up quietly.
Config, seed, score, and time for each run. That log is the difference between a tuned model someone can rebuild and one nobody can explain six months later.
If several steps do not beat the best, stop. Continuing past a plateau spends budget on noise, and the noise is what makes a final number look better than it is.
Each step changes one thing: a knob, a subset of the data, a template. The log of steps with scores is the record of what was tried, which is what stops the same run being repeated for a week.
Track the highest held out score and its config. A loop that keeps the last run keeps whatever the last experiment happened to be, which is usually worse than the best.
Each run costs time and money. A budget of runs forces the search to be about the likely causes, and the budget is what makes the loop finish before it becomes a project.
Each one is defined here on this site. Open the entry for the full definition and the other words it points at.