Reprinted from the CSI Technical Journal - Sept. 2000
By Bob Pelletier
Our Trading System Performance Evaluator™ (TSPE) has been getting a lot of attention lately. The widespread interest is proof positive that traders want to know the accuracy with which trading signals are issued and want to be able to evaluate trading systems before they buy. Now that system evaluation is available, traders want to know in advance if a given program will definitely produce profits. This brings up the question of whether any trading system or trading system evaluator can absolutely protect the trader from market losses or absolutely guarantee profits. Is there a "sure thing" for traders?
In the simplest of terms, for every trade in the commodity markets, there is a winner and a loser: someone who shorted when they should have gone long and vice-versa. For stocks, the simple "buy low, sell high" axiom is a great indicator of success or failure. However, the complexities of modern trading make this simple black-and-white proposition a veritable rainbow in shades of gray. Combining options, spreads, straddles, crop insurance and various forms of hedging can either amplify or mitigate your outcome, whether the recording ink is red or black.
Most trading systems do not address these alternative investment strategies, particularly when used in combination. Nor do they consider the cost of the trading system itself or the cost of lost revenue that might have been gained had the money been invested in other venues such as risk-free (in some cases tax- free) bonds. It would take a staff such as those employed by fund managers and investment firms to fully manage your investments using every available tool.
That said, our view is that there are certainly good trading systems out there that can help the individual investor improve trading results. A great many are sufficiently meritorious to be put to the test through TSPE, which is part of CSI's Unfair Advantage® (UA) software. This is the only program that, to our knowledge, will realistically compute the capital required to run a given trading system based on a given profit goal, and include in its output a known level of maximum drawdown.
We have mentioned TSPE in several recent Technical Journal issues, but to review the recommended trading system guidelines, your system's parameter count should be low, the number of derived samples should be large, and the nature of the system itself should be sound. By "system's nature" we mean the handling of dependent vs. independent variables. We believe that multiple independent variables should combine to contribute to the overall performance.
The necessity of providing an adequate sample size applies even to the example we used a couple of months ago when explaining the new Perl language provided within UA version 2.2.0. We reported on a simple breakout system that evaluated thousands of stocks with CSI's [MarketScanner]™ over a period of about one year. About 57% of the roughly four thousand stocks examined turned a profit, and the median profit was about 15% greater than the median loss. These results may be considered fair to good, especially when you consider that there were no attempts at parametric optimization. However, they don't give us the whole picture.
In order to provide a truly representative sample for analysis, we would have to test the system in an entirely different environment where the general markets were moving down or choppy instead of just up. Further testing on choppy or down markets should be tried before jumping to any conclusions. An adequate sample size must be measured both in number of trades and diversity of market trends.
In hindsight computer simulation, it is quite easy to be falsely convinced that you have found something unique and enormously profitable. Consider the solicitor who recommends by mail to hundreds of recipients the purchase of a random stock from among hundreds of stocks. Then in a second mailing, the would-be winners (not the losers, based on the initial stock suggestion) are told to buy a group of other stocks, etc., etc. After many mailings, the few remaining would-be winners would then be solicited to pay for the advice that might follow. Only the few winners who sustained an uninterrupted string of random picks that turned out to be winners would have been solicited for money in such a scam.
When searching for that profitable computer trading system, be careful that you are not similarly finding an ill-conceived scheme such as the above that will fool you into believing that you should risk your hard-earned capital. It is very easy to convince yourself that you have uncovered a viable plan, even though it may be statistically worthless. The above system could not be run through TSPE, of course, because of the fraudulent manner in which it was conducted. (Parameters would be equal to the number of stocks suggested to all participants.) TSPE's algorithm will always help you rule out unviable plans such as this.
Is there a way to evaluate potential trading systems to prove that a given system will definitely produce profits? We think that TSPE comes as close as possible to this goal. By consistently evaluating trading systems using individual trade profits, sample size and parameter count, TSPE gathers the necessary evidence to prove whether or not past performance is repeatable with a known probability of success. Even the best trading systems have some losing trades and uncertain futures because of the myriad variables affecting the markets. TSPE goes a step farther than the rest by letting you know the probability of losing your investment dollars. We think that acknowledging and quantifying the ever-present possibility of failure is eminently more valuable than touting a "sure thing" that overlooks the possibility.