The backtesting process is an invaluable way to assess how well your current strategy or portfolio would have done in the past. In a world without backtest, you'd have to spend many years running a particular investment strategy to see whether it works (i.e., whether the strategy can beat the benchmark over many years).
Learning from your own experience is fine, but it takes years to draw some viable conclusions.
But here comes the really scary part. What if the conclusion is that the strategy doesn't work like it was supposed to? You can always calibrate some assumptions or even change the strategy completely, fair enough, but that means you need to spend another 2-3 years living through the different market and economic circumstances to see whether your adjustments were the right ones.
What if the second strategy doesn't beat the benchmark, either? You've already wasted a few years of your life and probably a lot of money just to start over again.
To avoid that, we've equipped SCRAB with two specific tools to do backtests in two different ways: Portfolio and Strategy Backtests. Especially the latter is extremely useful since it allows you to answer questions like these: What would have happened if I had used my current strategy over the last ten years? How would my portfolio have performed? How much gain would I book to this date? What would my biggest drawdowns have been?
The unbiased knowledge of the probable outcomes of different kinds of "what if" scenarios and situations will help you to calibrate and adjust your strategy even before putting the real money on the market. No need to say that it's extremely advantageous to run your portfolio profitably just from the first day on the stock exchange.
The bad way to do backtests
Most people who do backtests, are backtesting their current portfolio. That means they analyze and select a bunch of stocks that meet their criteria today and put them into backtest machine to see how well such a portfolio would have performed in the past. The results are often outstanding.
That's not because the investor has an exceptional investing strategy but rather because he has chosen today's winners to build his portfolio. That means he has chosen stocks that already did well (they fit his investment criteria today). The catch is if he was looking for companies meeting the same criteria a few years ago, he would probably end up with totally different companies altogether.
Hence, it doesn't make much sense to backtest portfolios made of stocks chosen today because this is not the portfolio you would have built many years ago.
That being said, we give you the Portfolio Backtest tool just in case you want to use it, for example, to validate how well the typical 60/40 Bonds/Stocks portfolio has done in the past compared to a 100% S&P 500 portfolio, etc.
But the real game-changer is the Strategy Backtest tool.
The good way to do backtests
The only way to draw viable conclusions from any backtests is to backtest not the particular stocks but the criteria (aka strategy) for choosing the right stocks. The whole process has three steps:
- Define your investing criteria (i.e., you decide to invest only in companies with Ohlson Default Probability < 1% and EPS Average Growth Estimates > 10%)
- Build an automated scoring model taking the above criteria into consideration when searching for stocks
- Put this strategy to the backtest machine to see which stocks have been meeting your criteria in the past and how well your portfolio would perform for many years if you were investing only in such stocks.
This is exactly how our Strategy Backtests tool works. First, it looks for a fixed set of fundamental, technical, or economic criteria you defined in one of your scoring models. The next step is to move back in time to mine through all the companies meeting your criteria during the last years (which means that they would probably end up in your portfolio back then).
As a result, you'll have presented not only the full list of stocks you would have had to buy a few years or a few months ago but also the summary of your P&L and other performance metrics such as drawdowns, volatility, and different ratios describing the characteristics of your hypothetical portfolio.
Basically, the Strategy Backtest tool tries to simulate your investing behavior over many years to assess whether your strategy makes sense in terms of beating the benchmark and bringing solid gains, so you won't have to test it with your real money for the next few years.
Survivorship bias and the S&P 500 rebalancing problem
The most vivid problem when doing backtest is a so-called survivorship bias. Trying to test your strategy within today's stock pool means choosing the stocks only from the group of companies that have already survived bankruptcy and are still traded today.
Hence, your simulated gains will always be higher than if you were choosing your stocks in the past. Our tool allows you to eliminate this problem because Strategy Backtest always tries to see the world (and pool of stocks) exactly how it was back then and not how it looks today. To put it another way, we mine through the companies' universe that was at our disposal at the time of simulating the trade, not at the moment of doing backtests.
That being said, there is often another typical problem when comparing your strategy results against benchmarks like S&P 500. Constituents of the S&P 500 are not constant. They rotate every quarter.
Performing a backtest, like many people do, in a way that simulates transactions only within the S&P 500 constituents will always be biased as long as we take our companies out from the actual S&P 500 index. Being in the S&P 500 today means that the company made it to one of the 500 biggest companies in the US stock market. Which means... that its market cap grew substantially, which means its stock price rose recently.
Thus, if we were conducting backtests selecting, for example, the best companies from the S&P 500 pool, we would almost always outperform the past results of the "old" real S&P 500, no matter what strategy we had applied. That is because the real S&P 500 gets rid of the underperforming companies and replaces them with outperforming companies. This process is known as rebalancing and is repeated every quarter. That means the real S&P 500 will always be lagging if we decide to pick stocks for our backtests out of today's S&P 500 squad and compare it to the old squad.
To avoid being biased when selecting stocks, a good backtest system should always judge whether the particular company was, in fact, in the S&P 500 index at the time of simulating the trade. This is known as a rolling backtest, which you won't find in many places outside of SCRAB.
It's a pity because this is the only way to guarantee that your backtests are worth anything.
Number of stocks, weights, and rebalancing periods
When doing a backtest, we allow you to toy a bit with different additional parameters that may or may not elevate your returns. Ask yourself questions like these: Should I have 20, 30, or 40 companies in my portfolio? How often should I replace them? Is it better to keep the equal weights between my holdings or rather set the weights according to some criteria, such as the amount of risk the particular stock carries? If so, which indicator should I use to measure the risk?
Well, you can read a few books on risk management and try to answer these on your own, but... the truth is that the correct answer will almost always depend on what kind of strategy you are trying to use and what kind of stocks you are buying. There is no one right approach that fits all strategies and all investors.
The only reasonable and objective way to answer the questions above is to backtest various settings and determine what fits your particular strategy best. The rule of thumb is you should have around 25 companies in your portfolio and rebalance them every month to keep their weights equal. But...
Sometimes you can reach for a few additional percentage points of gains if you just replace equal weights with some kind of risk-based-weights like Hierarchical Risk Parity. But, again, it depends on your strategy and your stocks.
You might have a lot of hypotheses, but these are only theoretical concepts until you validate them with real data in the real world.
Our backtests are designed to do just that in the most reliable way possible.
How not to interpret the backtests results
It should be obvious to most investors. Nevertheless, it's not going to hurt if we repeat it here one more time – past results are no guarantee of future performance. Past behaviors are indeed the best predictor of future behaviors, yet it's not a guarantee. Something that has been working for the last 40 years probably will also work for the next year or so, but it is not warranted.
The one thing you can do to make the odds in your favor is to seek a reasonable explanation of the simulated results to make sure they are sensible and well-founded. This is important because, in today's world, we have access to such a huge volume of data, that mining through it will always result in discovering some kind of correlations.
But correlation doesn't mean causation. Sometimes there is just no cause-and-effect relationship between two events or variables, so it won't make much sense to make our investment decisions solely on the basis of an observed association or correlation between two indicators.
Unless... we can explain that relationship in a reasonable fundamental way.