10 Things You Can’t Learn From a Backtest

We’re currently living in the golden age of the backtest.

Things have never been better for quantitatively-inclined investors. There are algorithms on top of algorithms. We’ve never had so much historical market or corporate data available at the click of a button. Computing power is off the charts. Entry-level analysts can now perform calculations with Excel spreadsheets that the world’s smartest scientists and mathematicians could only dream about a generation or two ago.

For finance history buffs such as myself, these tools are enormously useful. I can check historical performance going back many decades, backtest every strategy under the sun, quickly and easily perform scenario analysis, run thousands of Monte Carlo simulations and dissect every type of market environment imaginable.

There is, however, a downside to all of this. Here are 10 things you can’t learn from a backtest:

1. How many bad backtests came before the good ones? I wonder how many millions of deceased backtests there right now are sitting in a recycle bin graveyard on computer desktops all across the globe? No one ever shows you a bad backtest because it’s much easier to date mine the past than the future.

2. Data availability at the time. The fact that we now have data that wasn’t available in the past changes the nature of that past data. There would have been ripple effects if investors knew then what we know now. Hindsight changes perception.

3. What the frictions were. It’s almost impossible in a backtest to completely account for costs and frictions such as taxes, commissions, market impact from trading, market liquidity, etc. Sure, you can estimate these frictions, but you never truly understand how these things will affect your bottom line until you actually have to execute buy and sell orders.

4. The difference between % returns and dollars invested. Returns on a spreadsheet are not the same thing as dollars gained or lost. It’s much easier to look back at annual return numbers as a percentage than to feel what those percentages would mean for your net worth in real time.

5. How to optimize life. Portfolio optimization is something that has never made much sense to me. You can always build a perfect portfolio using past data, but a flawless investment strategy on paper doesn’t take into account the fact that life is messy. Hard choices often have to be made that can’t be optimized.

6. How it feels to put real money to work. I still remember the first time I played blackjack at the casino when I was 18 years-old. It was maybe $50 but my adrenaline was flowing. I was equal parts nervous and excited. It wasn’t much money but I couldn’t help this natural reaction. Now imagine these feelings with real money at stake. Envy, fear, greed, doubt, and panic are impossible to plan for in a Monte Carlo simulation.

7. An itchy trigger finger. Murphy’s law of investing says that everything you buy will immediately fall, everything you sell will immediately rise and every new strategy you implement will immediately underperform. When things don’t go right in the real world, the first thing most investors look to do is change their model. Investors backtest investment strategies looking back almost 100 years, but then change it after a poor six-month stretch. It takes stone cold discipline to stick with a rules-based system when it’s not working. Most would rather make changes to relieve discomfort.

8. How it feels to lose money. My favorite quote from the classic investment book, Where Are the Customers’ Yachts, describes this perfectly:

Like all of life’s rich emotional experiences, the full flavor of losing important money cannot be conveyed by literature. You cannot convey to an inexperienced girl what it is truly like to be a wife and mother. There are certain things that cannot be adequately explained to a virgin by words or pictures.

The pain and emotions you feel from losing money is not something that can be simulated. No one knows how they will react until they are in that moment of maximum pain.

9. How to avoid overconfidence. It’s easy to feel like a world-beater when you’re able to put together an amazing track record in a backtest. The data, smooth-looking graphs, unreal risk-adjusted return metrics and insane performance numbers make you feel like you can conquer the markets. You have to be willing to play the role of “red team” against your own models, an uneasy position for many investors to put themselves in.

10. What’s going to happen in the future. As my colleague, Michael Batnick, put it recently, “Unfortunately there is no such thing as a front-test.” Every market environment is different than the last so you have to be able to accept that the future will never look exactly like your time-tested strategy.

Further Reading:
Torturing Historical Market Data



Download PDF

Full Disclosure: Nothing on this site should ever be considered to be advice, research or an invitation to buy or sell any securities, please see my Terms & Conditions page for a full disclaimer.

  • 11. Subtle look ahead bias (e.g., failure to account for stock splits in certain data sets)
    12. Data collection errors. Sometimes the data is just plain wrong.
    13. The possibility that non-recurring market conditions somehow influenced the backtest.
    14. Use of nonlinear models, which can be highly vulnerable to data mining.
    15. Randomness; i.e., if you try enough models, one will eventually “work” even if it has no grounding in any rational thought process.

    Thoughtful out-of-sample testing can solve many of these technical problems.

    Nick de Peyster

    • Ben

      Good point on the data. Many investors don’t take the time to understand the data sources they’re using.

  • You never see a bad backtest and investment consultants, advisors, and investors should be aware of this when being pitched investment ideas, especially new ideas based on factors. I’ve found the work of Marcos Lopez de Prado, who has written extensively on the problem of backtest overfitting, to be useful.

    Also, Cam Harvey warns against depending on out-of-sample (OOS) testing (@nickdepeyster:disqus ). as a panacea to backtest and overfitting biases. As Cam writes in “Evaluating Trading Strategies”: “First, often OOS is not really out-of-sample because the researcher knows what has happened in that period. Second, in dicing up the data, we run into the possibility that, with fewer observations in the in-sample (IS) period, we might not have enough power to identify true strategies… Finally, with few observations in the OOS period, some true strategies from the IS period may not pass the test in the OOS period and may be mistakenly discarded.”

    “This was our paradox: No course of action could be determined by a rule, because every course of action can be made to accord with the rule.”
    – Ludwig Wittgenstein

    • Ben

      This is true but I don’t think that invalidates using a rules-based approach. You just have to be ready to accept the bad with the good and understand exactly what you’re getting yourself into.

  • I’m a huge advocate of a rules-based approach. With that said, seeing back tests without realistic friction estimates is extremely common. It’s also impossible to properly estimate skid/friction and incorporate it into a back test. Fills will be much worse when investing in size or during certain periods. No back testing software that I know of is capable of estimating skid as it would have occurred. Also, having worked in the managed futures industry, it was common to see managers design a system that worked but was never scalable – they were capped at running $10m (or whatever their functional ceiling happened to be).

    • Ben

      good point. a lot of people don’t take into account liquidity when running these things

  • Simon Shaw

    “No one ever shows you a bad backtest because it’s much easier to date mine the past than the future”. The answer there is not to depend on others to do the backtesting for you. Do your own backtesting and you will certainly discover strategies with poor returns.

    Backtesting is simply a tool and like any tool it can be used well or it can used badly. Just because there are poor drivers it obviously doesn’t mean that cars are bad.

    Points 4, 6, 7 and 8 are well taken. One of the weaknesses of any backtest is that it ovbiously does not come anywhere near simulating the emotion-filled reality that the investor would have actually endured.

  • MG

    Hi Ben, just as an ego boost I thought you’d like to know that your article is featured on marketminder.com (on the right side of the homepage):

    10 Things You Can’t Learn From a Backtest

    By Ben Carlson, A Wealth of Common Sense, 01/11/2017

    MarketMinder’s View: Ah, the backtest. We have lost track of how many times we’ve seen a boast like “check out my awesome new strategy, which I thought up yesterday and backtested all the way to 1970, so I know it like totally works!” Conceptually, backtesting probably seems sound. Why not use actual historical market returns to test whether trading X at Y time would generate better returns? How else would you know if your investment model works or, more to the point, worked? But backtests, while useful to an extent, have limitations. They don’t tell you what will work, just what did (when properly implemented) in a controlled test environment. We’ve seen many fall prey to errors like survivorship bias. This article helps inoculate investors who might otherwise put too much stock in backtests. For more, see Elisabeth Dellinger’s commentary, “Monkeying About.”

    • Ben

      Very nice. thank you very much

  • Mark Lyck

    Hi I’m Mark from https://formulastocks.com

    While these are very valid points that “most” backtests suffer from. They’re not unfixable. We started using backtesting and machine learning in our strategies 14 years ago. Much much earlier than anyone else in the industry. Today we have simulations that perform extremely close to real life scenarios. We’ve tested many strategies in our simulation, most notably Warren Buffet’s and achieved similar results, same with Joel Greenblatt’s “magic Formula” (which by the way, vastly underperforms outside his selected year-range in his book)

    Furthermore we launched our strategies in 2009. Since then our best strategy, both on paper and in real life performance, is up +1601% as of last month. We’ve kept our 90% success rate, our avg. annual returns are as expected.

    There has been nothing to suggest that our backtesting was flawed in 2003-2009. And since we deployed our systems with real investments, our results have been on or near record levels.

    We’ve written an article here, explaining how we avoid the backtesting bias mistakes most other backtesting simulations suffer from: https://medium.com/@FormulaStocks/backtesting-bias-and-how-we-avoid-it-fe598930cb1#.v3oe8l3aa

    Don’t disregard all backtesting as useless. Because most people are just starting to get into quantitative investments now, and don’t really know what they are doing yet.