One train test split measures one moment. A strategy that works only in that moment is a strategy that was fitted to it, and a single split cannot tell the difference between fitted and real.
About 1 min of reading
Run three folds and keep the spread
Write the fold results
Before you run it, say out loud what it will print. Then open the answer.
Print the mean and the worst
Before you run it, say out loud what it will print. Then open the answer.
Refuse on the worst fold
Before you run it, say out loud what it will print. Then open the answer.
A fold where the fit window overlaps the trade window is a leak, and it is the leak that makes every backtest look better than it is.
Folds cost the same as data. The count is chosen so the whole run fits in one change cycle, the same rule an eval set follows.
Fit on a window, trade the next window, move both. Each fold is one independent measurement of the same rule.
A long out of sample window is a window that contains several markets. The fold is sized to the market, not to the data.
Five folds with three wins and two losses is a distribution. That distribution is the result, and a mean alone hides it.
Each one is defined here on this site. Open the entry for the full definition and the other words it points at.