By Charles Maley and Nick Pingitore
I don’t know how to sit outside myself and test against a hypothetical self who stayed home – THOM GUNN
I have had the great pleasure of working with Nick Pingitore for the last 10 years. After being a client of mine for over five years, Nick and I decided to team up and further research our ideas about building “real world” trading systems.
Nick is an engineer by degree and therefore has a great background for the mathematical challenges of “back testing.” In fact, of all the system developers that I have worked with (and there are plenty) I think Nick has one of the best grips on just how challenging it is to build a reliable trading system.
It’s not what you think though. It’s not because he has found ways around the limitations of “the back test,” it’s because he knows you can’t. Therefore he must build in strict money management to address the inevitable surprises of randomness.
Nassim Taleb, in his book The Black Swan, coined the phrases Mediocristan and Extremistan to distinguish between two types of randomness. All randomness is not created equal and must be dealt with, as it is what destroys most trading systems in the “real world”.
In Mediocristan, everything has boundaries and limits that can be easily measured. Things like IQ scores, height, and weight, and how much people smoke would be examples of Mediocristan.
If we were to randomly select 1000 people and calculate the mean weight, we can be reasonably assured that no one sample will dominate the distribution. In other words, you may find the average weight of 1000 men to be 200 lbs, but you won’t find a man weighing in at 200,000 lbs. Even the heaviest man in the sample will not materially affect the mean value of a distribution.
Let’s say that instead of weighing the 1000 men we find out their incomes. Now, even though the mean income might be $75,000 for 999 men, what if the final entry made 75 million a year? Now we have one event that’s 1000x the mean and also doubles the mean value.
This one sample (event) can change the entire distribution, and in some ways makes it meaningless to depend on. After all, if I am trying to get some idea of what to expect from sampling, and one event can ruin the whole thing, what good is it?
Joan Baez once said “hypothetical questions get hypothetical answers.” When building a trading system, we need to be aware of the limitations of the back test and not buy entirely into the hypothetical suggestions. We not only need to be aware of potential extreme randomness, but we also need ways of protecting ourselves when it happens.
In a recent article about building trading systems Nick said, “Traits that make a good system developer are often the opposite of those that make a good trader. A developer always wants to improve their methods and make them as efficient as possible in regard to risk and reward. A trader understands that there is no efficiency in making money and takes profits when the market gives it to them, while continually managing their risks.”
Most system developers constantly evolve their systems to produce better and better backtests (with hindsight, of course). Our strategies, however, were not developed prior to trading, but only after several successful years of trading and analyzing what consistently works in real-world trading.
In other words, we built our systems around our “real world” trading that was successful as opposed to creating something in the computer that “should work in the real world.”
Nick goes on to say, “Unfortunately, there is no perfect system that works in all market environments. The key in trading is to use the right system in the right market environment, but this is an art based on years of experience and insight. There are, however, some robust methods that can be used to implement commodity trading systems or strategies into the real world and make the crossover from hypothetical to reality as least painstaking and unexpected as possible.”
“We focus on two main priorities in implementing commodity trading systems into the real world. The first is on breaking down the robustness of the performance and the second is on managing risk and exposure, including keeping the maximum drawdown within reason.”
To break down the robustness of a system, we do two types of tests. The first is to obtain a set of robust metrics on the commodity trading system. The second is to test across a large set of data followed by several subsets.
The metrics that we feel are the most reliable are:
- High positive Mathematical Expectation
- Large average trade
- Low standard deviation of daily returns
- High MAR Ratio
- Low leverage and exposure
- Low or positive tracking error
- High profit /cost ratio
(Note: for any test to be of any value, testing on markets only liquid enough to trade and realistic slippage and commission assumptions must be used.)
In the trading world we have a duel edged sword. On one hand we are unfortunately held hostage to the same extremistan surprises that could blow us up. On the other hand these are the events that can make us wealthy. In trading, it is the same extreme moves that make the effort worthwhile.
It is one of those businesses that one or two big events (trades) can make your whole year. Just like authors or rock stars where a handful of stars generate most of the revenues, a handful of trades do the same thing.
The trick is staying alive long enough to be there when they come.
NEW – Weekly Blast from the Past!!