Elements of A Mechanical System (Success Series)

BarroMetrics View: Elements of A Mechanical System

Assuming a mechanical system is ideal your personality, the questions that arise are:

  • What are the qualitative elements of a reliable system?
  • Where can I test my system?
  • What are the quantitative measures to be aware of?

Turning to the qualitative elements…..whenever I am creating a new system, the first question I ask myself is what assumptions will the system be making? For example, in the system that is currently taught in Ultimate II, the underlying assumption ‘is the market is producing directional momentum’.

The next question is, what are the elements of the system? Then I ensure that I structure the system correctly so that it flows. Essentially the system has to answer four questions:

  1. The instrument to be traded
  2. The entry criteria
  3. The initial stop criteria
  4. The target criteria

Once the qualitative measures are completed, the next step is to have a system tested.

When I first started trading, there was limited software for testing and innumerable services that would do it for you. Now, the shoe is on the other foot. When I googled “back testing for mechanical trading systems”, I found innumerable referrals to testing software, and none in the way of testing services. I’m sure some there are some services, but you may need to perform some deep searching to find them.

Back testing and forward testing are not my areas of expertise. I am lucky enough to have the services of an excellent programmer who is well versed in probability theory. 

It looks as though if you are mechanical traders, you will need to acquire the programming expertise. One piece of advice I can pass on, my programmer says that all back testing and forward testing should include a Monte Carlo simulation. In this way, the testing will ensure that the result has not been caused by a favourable sequence of trades that understate the risk.

If any reader knows of back testing and forward testing services, please do drop a comment.

Once the back testing has been completed, you will need to analyse the results. Some of the things I would look at are:

  • The probability of risk of ruin
  • The number of consecutive losses
  • The number of consecutive wins
  • The average dollar win
  • The average dollar loss
  • The win rate
  • The loss rate
  • The average percentage return (on Capital)
  • The average percentage loss (on Capital)
  • Return divided by drawdown

Next week we’ll look at a discretionary trading plan and answer Baz’s questions on stop placement.  Tomorrow I’ll reply to Wenda’s query

26 thoughts on “Elements of A Mechanical System (Success Series)”

  1. hi Ray,
    Am I right to say that:
    No goals, no back and forward testing.

    Qualitatively,
    To maximize average profit?
    To minimize drawdown?
    To maximize yearly return?
    ….

    Quantitatively, what values/ranges are acceptable?

    How you decide 2 systems to trade based on the results?

    ____

    Though trading champion like Andrea Unger used Monte Carlo simulation, I have my reservation of using it for testing purpose.

    Ask your programmer, does a price movement path actually happened yesterday simulated by the Monte Carlo simulation?

    Is Financial Monte Carlo Simulation Dead?
    http://www.cbsnews.com/news/is-financial-monte-carlo-simulation-dead/

    Why Some Advisors Just Say No to Monte Carlo Simulations
    http://wealthmanagement.com/practice-management/why-some-advisors-just-say-no-monte-carlo-simulations

  2. Hi Paul

    Not clear on what your first question is driving at, perhaps you can elaborate.

    In respect of the qualitative and quantitive: the answers depend on your personality. You could say it boils down to your trading philosophy.

    Mine, for example, has, as the first tenet, the protection of capital. So, defensive measures take priority. I am not interested in making 100% pa if my risk of ruin stands at 33%. Other, more adventurous traders will beg to differ.

    Turning to the Monte Carlo issue…..

    The views expressed in your references (cbsnews and wealthmgt links) to Monte Carlo Simulations contain so many errors that I’ll need a blog to address them.

    The key thing to remember is: MCS are used to determine the robustness of the mechanical system by mixing the test sample’s occurrence sequences – that’s it, no more, no less. This idea is something the journos in the references failed to appreciate.

  3. 1.
    I just want to highlight on our goal setting, we should have goals that allow us to evaluate our trading results.

    2.
    My concern about MCS is whether it can simulate an actual real trading price path.

    It has to assume a price probability distribution curve?

    Have you compare your historical back testing results with the MCS results for the same period?

  4. Hi Paul

    Thanks for clarifying

    I agree with you on (1).

    Backtesting and forward testing are mere guides. The proof of the pudding lies in our results (our profits and losses are the only evaluation tools). After all, Method is but one aspect contributing to our success.

    (2) Monte Carlo “simulate an actual real trading price path”…..if you are looking for that, let me know when you find it (G).

    I think see your problem.

    Are you seeking certainty in an uncertain world?

    The best any tool (at least those I know of) can do is to provide a general idea that your Method (applied consistently) has an edge.

    If you are looking for a backtesting result that will produce the same equity curve in real-time trading – ‘it ain’t going to happen’.

    What backtesting and forward testing do is provide an acknowledgement (as long as the system’s assumptions hold) that your Method has an edge. All Monte Carol simulations do is seek to ensure that the edge is real, and not a result of the sequence in which the trades occurred.

    As for my trading. The answer is ‘no’ because MC does not apply to discretionary trading. I have found MC useful in assessing whether a setup and trigger provides an edge.

  5. hi Ray,
    I just want to highlight the problems/limitations on MCS, which can be readily available nowadays, but many could have blindly believe in its usefulness.

    Even Kevin Davey in his book

    Building Winning Algorithmic Trading Systems: A Trader’s Journey From Data Mining to Monte Carlo Simulation to Live Trading (Wiley Trading) 1st Edition

    acknowledged MCS limitations.

    ___

    Below is what I have written today before I read your blog:

    >
    Assume we play a game by tossing a coin. Head (H) – we win $5, tail (T) we lose $5. Let’s play 3 times in a row.

    Possible event paths :
    1. HHH Profit = $5 + $5 + $5 = $15
    2. HHT Profit = $5 + $5 – $5 = $5
    3. HTH Profit = $5 – $5 + $5 = $5
    4. HTT Profit = $5 – $5 – $5 = -$5
    5. THH Profit = -$5 + $5 + $5 = $5
    6. THT Profit = -$5 + $5 -$5 = -$5
    7. TTH Profit = -$5 – $5 + $5 = -$5
    8. TTT Profit = -$5 – $5 – $5 = -$15

    Average = $(15+5+5-5+5-5-5-15)/8 = 0

    For a correct Monte Carlo Simulation, it should produce 8 and only 8 event paths, no less, no more, no repetition.

    But if Monte Carlo Simulation uses a random number to generate a random number between 0 to 1, and assigns T if the number generated is between 0 and less than 0.5, H if the number generated is between more than 0.5 and 1, it may generate a path of HHH twice.

    Depended how the Monte Carlo Simulation is coded, it may generate
    a. less than 8 event paths
    b. more than 8 event paths
    c. repeated event paths

    For head and tail game, all the event paths are independent binary event which can all be listed correctly, no more, no less.

    But for trading, if the price change in the past is between -2% to +2%, the event paths are infinite. Also, in future, the price change in the future could be between -3% to +3%.

    In reality, the event may not be independent. Say the game of coin, it could be co-related/biased such that no HHH or TTT could happen in a row. In that case, there are only 6 possible event paths. If the Monte Carlo Simulation generates 8 event paths for evaluation, the results will be wrong.

    For all the event paths generated by Monte Carlo Simulation, is there a path which is or very near to the actual real event paths?!

  6. Hi Paul

    Your key words are:

    ‘For a correct Monte Carlo Simulation, it should produce 8 and only 8 event paths, no less, no more, no repetition.’

    The rest of the criticism is a ‘non-sequitur’.

    Let’s take a simple example:

    If I gave you a system that says “buy when ‘x’ occurs”, but,

    you buy when a, b, c occur,

    how can you then complain the system does not work?

    The same comment can be made when Davey (? haven’t read the book, so I assume he made the comment) criticises MC by saying:

    ……that its coding throws out a sum of the events greater than the population size!

    It’s worth repeating my comment that all MC does is test a system’s dependence on the sequence of a population. This means, as you pointed out, the MC coding must deal with the same number of events as the population.

    A robust system will be profitable even if the sequence changes. Sure precise dollar profits may change but the edge ought to remain.

    MC is not intended to predict the exact positive expectancy of a system. I don’t know of any tool that does that.

  7. hi Ray,
    If the simulated event/price paths are wrong, how can we test the edge and robustness of a trading system?

    Back and forward testing will be realistic.

  8. Hi Paul

    Did I mention ‘wrong?’ What I said was you cannot expect backtesting and forwarding results to be mirrored in real-time trading. That would imply ‘certainty’ – something that does not exist in trading.

  9. hi Ray,
    Since I have not used/tried any MCS Software before, all my thoughts/comments may change when I use it one day. 🙂

  10. Ray’s post included:

    “It looks as though if you are mechanical traders, you will need to acquire the programming expertise”.

    Programming will be essential. Any system that is built around out of the box indicators will be worthless IMO (as a general statement) therefore a system that can be built using a system generated mechanical system that does not require programming e.g. an MA crossover type system are a waste of time.

    Adam Grimes has some good posts on the coding options:

    http://adamhgrimes.com/blog/first-steps-python/

    http://adamhgrimes.com/blog/resources-learning-python/

    Chris

  11. Hi Chris

    Thanks for the Bandy references, Andrew’s posts and comment.

    I agree – programming skills are necessary

  12. It was from reading one of Ray’s posts that I think I heard about Adam Grimes.

    He has a free trading course.

    http://adamhgrimes.com/TAAS/the-course/

    Module 15 Introduction to Algorithmic Trading with Ernest Chan, Ph.D.

    covers mechanical/quantitative trading.

    The module has 8 lessons.

    Anyone considering back testing a mechanical system needs to know about these topics and more.

    I think it is covered in the module but for example even people who have been coding trading systems for over 20 years make mistakes like “forward look” errors (not sure if that is the correct term). That is where bad coding uses a piece of data that would not have been known when the signal was created e.g. tomorrow’s open when you are entering on today’s close 🙂

    So even people with massive experience check their coding and results and then check again. If the results look fantastic there is quite likely an error in your code 🙁

  13. IIRC I think it was Howard Bandy who said that when live trading a system you should assume the risk will be twice the risk shown in back testing and that the returns will be half of the returns from back testing.

    Another recollection I have is that he told me that when he worked for a CTA they computer tested 100s of breakout strategies.

    Chris

  14. hi Chris,
    Thanks for your reference.

    Please comment on/ correct my writing regarding Monte Carlo Simulation.

    Cheers!

    Paul

  15. Hi Paul,

    I am not sure what your post is conveying and I am probably not qualified to answer.

    Was it covered in the Bandy MCS video?

    Some other suggestions:

    Adam Grimes might have something on the topic.

    Bandy has some free chapters at his site and a whole book that is free.

    Bandy might have an article/s on it.

    Bandy’s book Modeling Trading System Performance book has a whole chapter (Ch 6 30 pages) on MCS. He even covers https://en.wikipedia.org/wiki/Mersenne_Twister

    there are add ins for Excel some of which are free. AmiBroker has a Mersenne Twister function.

    At his site Bandy’s says:
    “Which paper are you looking for?

    Some of the papers that had been on the website before its problems are being revised. If there is a particular one you were hoping to find, enter a Comment at the bottom of this page, or send an email using the Contact form on the About page”.

    He is very approachable and you could ask him if he has done a paper on MCS.

    Hope that helps.

  16. Hi Chris

    I can provide a summary of conclusions. But as far as the practical application is concerned, that’s another matter.

    I can say that I have more confidence in the backtesting results when MC is used.

    Remember all we do with MC is randomly redistribute the order for the population’s occurrence. That way the testing eliminates results that may be due the sequencing occurrence.

Leave a Reply

Your email address will not be published. Required fields are marked *