We’re currently living in the golden age of the backtest.
Things have never been better for quantitatively-inclined investors. There are algorithms on top of algorithms. We’ve never had so much historical market or corporate data available at the click of a button. Computing power is off the charts. Entry-level analysts can now perform calculations with Excel spreadsheets that the world’s smartest scientists and mathematicians could only dream about a generation or two ago.
For finance history buffs such as myself, these tools are enormously useful. I can check historical performance going back many decades, backtest every strategy under the sun, quickly and easily perform scenario analysis, run thousands of Monte Carlo simulations and dissect every type of market environment imaginable.
There is, however, a downside to all of this. Here are 10 things you can’t learn from a backtest:
1. How many bad backtests came before the good ones? I wonder how many millions of deceased backtests there right now are sitting in a recycle bin graveyard on computer desktops all across the globe? No one ever shows you a bad backtest because it’s much easier to date mine the past than the future.
2. Data availability at the time. The fact that we now have data that wasn’t available in the past changes the nature of that past data. There would have been ripple effects if investors knew then what we know now. Hindsight changes perception.
3. What the frictions were. It’s almost impossible in a backtest to completely account for costs and frictions such as taxes, commissions, market impact from trading, market liquidity, etc. Sure, you can estimate these frictions, but you never truly understand how these things will affect your bottom line until you actually have to execute buy and sell orders.
4. The difference between % returns and dollars invested. Returns on a spreadsheet are not the same thing as dollars gained or lost. It’s much easier to look back at annual return numbers as a percentage than to feel what those percentages would mean for your net worth in real time.
5. How to optimize life. Portfolio optimization is something that has never made much sense to me. You can always build a perfect portfolio using past data, but a flawless investment strategy on paper doesn’t take into account the fact that life is messy. Hard choices often have to be made that can’t be optimized.
6. How it feels to put real money to work. I still remember the first time I played blackjack at the casino when I was 18 years-old. It was maybe $50 but my adrenaline was flowing. I was equal parts nervous and excited. It wasn’t much money but I couldn’t help this natural reaction. Now imagine these feelings with real money at stake. Envy, fear, greed, doubt, and panic are impossible to plan for in a Monte Carlo simulation.
7. An itchy trigger finger. Murphy’s law of investing says that everything you buy will immediately fall, everything you sell will immediately rise and every new strategy you implement will immediately underperform. When things don’t go right in the real world, the first thing most investors look to do is change their model. Investors backtest investment strategies looking back almost 100 years, but then change it after a poor six-month stretch. It takes stone cold discipline to stick with a rules-based system when it’s not working. Most would rather make changes to relieve discomfort.
8. How it feels to lose money. My favorite quote from the classic investment book, Where Are the Customers’ Yachts, describes this perfectly:
Like all of life’s rich emotional experiences, the full flavor of losing important money cannot be conveyed by literature. You cannot convey to an inexperienced girl what it is truly like to be a wife and mother. There are certain things that cannot be adequately explained to a virgin by words or pictures.
The pain and emotions you feel from losing money is not something that can be simulated. No one knows how they will react until they are in that moment of maximum pain.
9. How to avoid overconfidence. It’s easy to feel like a world-beater when you’re able to put together an amazing track record in a backtest. The data, smooth-looking graphs, unreal risk-adjusted return metrics and insane performance numbers make you feel like you can conquer the markets. You have to be willing to play the role of “red team” against your own models, an uneasy position for many investors to put themselves in.
10. What’s going to happen in the future. As my colleague, Michael Batnick, put it recently, “Unfortunately there is no such thing as a front-test.” Every market environment is different than the last so you have to be able to accept that the future will never look exactly like your time-tested strategy.
Further Reading:
Torturing Historical Market Data
