Improving the speed of your tick data backtests and optimizations

Modified on Sat, 21 May 2022 at 01:24 PM

The duration of a backtest is directly proportional to the number of ticks in the backtest. Since tick data backtests have a lot more ticks than regular backtests, they typically take a lot longer as well. However, in most cases the speed can be improved and this article aims to offer a few tips.

In order to get the best out of your hardware when running tick data backtests and optimizations you should:

Make several copies of your MT4 folder and run parallel backtests (or optimizations) in order to take advantage of all the cores of your CPU. Also see the support article Shared MQL4 folder for multiple MT4 instances.
Keep your tick data repository on a SSD drive - this will provide a major speed boost, especially when running parallel backtests or optimizations.
In order to ensure that no swapping can possibly occur, your computer should have about 4 GB RAM per running MT4 instance.
If concerned about disk speed when reading lots of small files (disk read-ahead favors large files), you can always use the Misc tab of the Tick data settings dialog to prepare an FXT file and use it in subsequent backtests to avoid the data decompression which can take a total of 1-2 minutes on an i7 with the data on a SSD for a full tick data backtest using the Dukascopy EURUSD data from 2003 to 2017. If you wish to save disk space, you can also share the tester\history folder between your MT4 instances by slightly modifying the procedure described in the support article linked above.
Note that by default tick data optimization will use FXT files for caching so there's no need to do anything special in that case.

However, the typical problem when it comes to backtesting speed is not the hardware limitations but rather the lack of optimization in the EA code. You should benchmark your backtesting setup by measuring the duration of a Dukascopy EURUSD 2003 to today tick data backtest of the MACD Sample EA (the duration is displayed at the end) and comparing it against the duration of backtesting your EA on the same data. More complex EAs will naturally take more time, but there is almost always room for improvement, especially if the duration of the backtest of your EA is larger by several orders of magnitude.

There are quite a lot of things that can be done to speed up an EA during backtesting and I'm going to enumerate a few here (note that you must have the EA source code in order to apply these and you must be very familiar with MQL programming):

Make sure there are no errors that keep repeating in the backtest journal. If there are, take steps to ensure they no longer occur.
Ensure that there are no Comment(), Print() or Alert() functions used. Furthermore, any string building used for these functions should be deactivated during non-visual backtests and optimization. For all the steps in this section, you can initialize a global isTesting boolean variable in the OnInit() function as follows:
```
isTesting = isOptimization() || (IsTesting() && !IsVisualMode());
```

No chart objects should be used during a backtest. Even more, chart objects that are automatically created for the trades (even in optimizations and non-visual backtests) should be deleted every time the number of trades has changed:

int orderCount = OrdersTotal();
if (prevOrderCount != orderCount)
{
   prevOrderCount = orderCount;
   for(int i = ObjectsTotal() - 1; i >= 0; i--)
   {
      string objectName = ObjectName(i);
      ObjectDelete(0, objectName);
   }
}

Cache absolutely everything you can. If you have functions that are called several times per tick with the same parameters, cache the results and return the cached values.
If you have indicators that have a non-zero shift, you only need to perform all the calculations involving them at the start of a bar. If you have indicators with a non-zero shift running on timeframes other than the backtest timeframe, you can use a common divisor timeframe as the trigger (e.g. if you have indicators running on H1 and M15 it's safe to update all calculations that use these indicators at the start of an M15 bar).
Naturally, if you have indicators that use a shift of 0 you must update the calculations that involve them at every tick but - if possible - it's a good idea to try to migrate your EA to using a shift of 1 instead.
If possible, skip processing ticks. For example, if your EA opens a trade and never closes below 5 pips, you can set some price thresholds for ignored ticks at 5 pips above and below the entry price of the trade. If your EA is based on a breakout concept, for every new bar you can set the price limits that trigger tick processing. If your EA is a grid or martingale, you typically know the next price level(s) at which you need to open positions and there is no point to process ticks until these price levels are reached.
Many order errors that come up on a live account cannot show up during backtests so if you have complex order retry functions with intricate error handling, convert them to functions and use different, less complex code during backtests.
For example, calling IsTradeContextBusy() is completely useless during a backtest; similarly, in all likelihood more than 50% of the possible order error codes do not appear during backtests. Also, your EA should take into consideration all the relevant factors prior to placing a trade so that it should never be possible to receive errors such as ERR_NOT_ENOUGH_MONEY, ERR_INVALID_PRICE, ERR_INVALID_STOPS or ERR_INVALID_TRADE_VOLUME.
Speaking of checking these conditions, it's worth noting that some configuration settings are guaranteed not to change during backtests so you can grab their values in the OnInit() functions and store them in variables. For example, you can read the minimum stop level - MarketInfo(Symbol(), MODE_STOPLEVEL) - and store it in a stopLevel variable that never needs to be updated during the backtest. In contrast, on a live server you would want to update the variable prior to placing a trade because some brokers change this during periods of increased market volatility. The same goes for leverage, swap, lot settings, margin requirements and so on.
Do not go through the current orders more often than needed. Typically, it's enough to do it once per processed tick. With some smart programming, you can completely skip going through them most of the time - if your EA isn't using pending orders, you don't have to enumerate the positions if their count does not change and if your previous enumeration set some clever price thresholds. For example, if your EA has a single position with a trailing stop you don't need to recheck the order unless the market price has moved in the right direction by at least the minimum threshold required by your EA for updating the trailing stop. Speaking of trailing stops, if they are placed at a distance larger than ~10 pips it's not a good idea to update them at every sub-pip price movement - your broker will hate you and your backtest duration will increase; it's much better to update such trailing stops when the market has moved at least 1 pip. When using pending orders, it's very easy to identify the next price level that will trigger opening a position and wait until that level has been reached or until an existing trade was closed, changing the number of open orders.
If your EA needs to check historical (closed) positions, do this as rarely as possible and try to cache the result. If there's the same number of trades in the backtest history, the function that goes through the closed positions is guaranteed to return the same result if it's called with the same parameters. This also applies to live operation - closed trades can get modified by the broker but this only happens in very special conditions, an event which is rare enough that you can safely ignore the possibility.
If you are using custom indicators, make sure the prev_calculate parameter supplied to OnCalculate() is taken into account and only the necessary number of bars is updated. This also applies to live operation.
If your EA has any functions that reside in a DLL (e.g. used for protection), always implement these functions directly in MQL code in the version that you use for backtesting. Also, it's generally a good idea to disable all protection checks in that version assuming you're the only one using it. If your EA has a DLL with virtualized code, that is typically the one thing that contributes the most to the speed decrease.