Tune without fooling yourself

Tuning means trying many values of a setting, for example every stop loss from 10% to 50%, and keeping the best. It is useful, and it is also the easiest way to fool yourself.

Why the best result flatters

Try enough settings and one of them will look excellent by luck: it happened to suit the exact days in your test. On new days that luck is gone.

Example: you try 200 combinations of stop loss, target and entry time on six months of data. The best one shows a Sharpe of 3. But even if none of the 200 had any real edge, the best of 200 random tries would still look good. The question is not “is the winner good?” but “is the winner better than luck?”

Five habits

1. Tune few things, over sensible ranges

Each extra setting multiplies the number of combinations and the room for luck. Tune the one or two settings you believe matter, over ranges a trader would actually use. See How many combinations.

Example: tuning stop loss (10 values) × target (10) × entry time (12) × wait (5) makes 6,000 combinations. Instead, tune just the stop loss from 20% to 40% in steps of 5: five combinations. Five honest tries leave far less room for a lucky winner than 6,000.

2. Look at the neighbours, not just the winner

A real edge is usually a plateau: nearby values do almost as well. A lone spike, where 25% is great but 20% and 30% are poor, is usually luck. Winner Parameter Sensitivity in AutoTune shows what happens one step either side.

Example: stop loss 25% made ₹80,000; 20% made ₹78,000 and 30% made ₹76,000. That is a plateau: any value from 20% to 30% would have worked, so the edge does not hang on one number. If 20% and 30% had made ₹5,000 and −₹12,000, the ₹80,000 at 25% would almost certainly be luck.

3. Read the overfitting numbers

Under Overfitting Diagnostics:

Prob. of Overfitting (PBO) near 50% means the ranking is noise. Lower is better.
The winner’s Sharpe should clearly beat the Selection-Bias Hurdle: the Sharpe you would expect from the best of that many tries with no skill at all.

Example: winner Sharpe 2.1, Selection-Bias Hurdle 1.9, PBO 44%. The winner barely beats what luck alone would produce, and the ranking is close to a coin toss. Do not trade it on this evidence.

4. Test on months the tuning never saw

Run Walk Forward. It tunes on some months, then tests the winner on the next month it has not seen, and repeats. Look at:

OOS PnL (True Profit): the unseen months added together. This is the number that matters, but it is before costs: brokerage and slippage are not in it (and only Walk Forward 2 takes off the government charges). Before you trust a setting, run it with 💰 Check real costs (in the New view of the results) and judge the result after costs.
Overfit Windows and the Institutional Verdict: many FAILED rounds, or OVERFIT_RISK, mean the tuned setting does not carry forward.
Param Consistency: if the winning setting changes every round, there is no stable answer.

In Saved Reports, the small label under Best P&L says whether a figure is in-sample or out-of-sample. Only compare out-of-sample with out-of-sample.

Example: a Grid Search report says Best P&L ₹1,20,000 in-sample; a Walk Forward on the same idea says ₹35,000 OOS, joined. The ₹35,000 is the honest figure: what the routine earned on months it had not been tuned on.

5. Paper trade before real money

Even a setting that passes all of the above has only been tested on the past. Run it in Paper mode in the Live Deployments for a few weeks first. See From idea to live trading.

Example: the backtest averaged ₹1,500 a day. After three weeks on paper the average is ₹600 a day: paper fills are simulated, but the gap is still worth understanding before you add real money. Compare the trades in the Trading Journal with the backtest’s trade log.