Please don’t backtest thousands of strategies
edit: as people are downvoting this I would like to clarify something. I am a full time trader, have been trading for 16 years and currently profitably run 7 algorithmic strategies. i am a big supporter of backtesting and this is post is not saying that backtesting is bad. It’s just explaining why blindly backtesting thousands of strategies will cost you money.
I recently saw multiple people on Reddit following this approach of automatically testing thousands or even tens of thousands of strategies to then afterwards live trade whatever survives the backtesting and demo trading so I just want to create this short post and explain why this won’t work and why it will cost you a lot of money.
Basically the idea when creating a strategy and when then testing this strategy on out of sample data during a backtest as well as during demo trading is that afterwards you can ask yourself “what’s the probability that a non-profitable strategy would perform this well?”. Then usually if you only did this with one strategy the answer would be that it’s highly unlikely that an unprofitable strategy would perform well on out of sample data as well as demo trading and you could confidently conclude that your strategy does indeed have an edge and that it will most likely continue working when live trading.
However when backtesting for example 10.000 strategies that changes the whole question. in that case if one of your strategies performs well the relevant question wouldn’t be what’s the chance that this specific strategy performed so well just by chance but instead “what’s the probability that at least 1 of 10.000 strategies would perform that well just by pure chance?”. And the answer is basically always that it’s very very high and that most likely your strategy won’t continue working when live trading.
There are also ways to simulate this exact probability which I can share in case this intuition isn’t enough but I just felt like sharing this because this approach seems to become more and more common and it will cost people a lot of money.