The Realization feature in Investor/RT refers to the process of walk-forward out-of-sample testing (or walk-forward optimization). In essence, Investor/RT optimizes (or backtests) over a given period, then tests using the optimized variables (or best performing symbols, or both) over a subsequent period immediately following the optimization period. In this way, we are seeing historically what we could have achieved in reality by trading our optimized results or best performing symbols. This process can give us a good gauge of whether the system should be trading going forward, or shelved/re-evaluated.
Before reading more about Realization, it's recommended that the user have some general knowledge of how Trading System Tools and Optimizations work.
The Realization window can be seen below:
- Name - Name of Realization. Generally, it's recommended to use the same name as the optimization and trading system it's built upon.
- Optimization - Optimization on which the realization is based. If optimization is not needed for your realization, you must still create an optimization that uses the trading system you want to realize upon, and specify this optimization in the realization window.
- Optimization Period - Number of days which will be optimized or sampled before performing the subsequent "real-testing".
- Realization Period - Number of days over which to perform the "real-testing".
- Test X Days Ending mm/dd/yy - Controls the period used for the realization. In most circumstances, this period should encompass the period for which you have data.
- Realize Best X Symbols By [APS/Net Profit] - The user may choose to test/trade only the best symbols (based on average profit per share or net profit).
- Optimize Only the Best X Symbols - When this option is checked, a "pre-test" will be run over the optimization/sampling period on all the symbols, and then only the best X symbols will be optimized and the "real" tested.
- Run Scan _____ Before [Every/First] Iteration - This option is available primarily in order to reset variables between each iteration, or before the first iteration.
- Show Intermediate Optimization and Backtest Reports - When checked, the optimization and backtest reports will appear just as if you were running them independently.
There are three primary methods in which Realization can be used:
- Survival Of The Fittest (SOTF) Test a large group of symbols over a given period of time (optimization period). Take only the best (fittest) symbols of this group in respect to either Net Profit or Average Profit / Share. Test these best symbols over a subsequent period of time (realization period). Repeat. This method is based on the premise that the symbols that have performed best most recentlywill continue to perform best over the near future. The "Skip Optimization" checkbox should generally be checked when using this method.
- Forward Optimization Optimize one or more parameters/variables over a given period of time (optimization period). Use the optimal set of parameters (based on Net Profit or Average Profit / Share) and test over a subsequent period of time (realization). Repeat. This method can be used with one or many symbols. This method is based on the premise that the parameters which worked best over a recent period will continue to work best over a subsequent future period. Parameters may be optimized in either a "per symbol" method (each symbol gets it's own optimal set of parameters) or a global method (all symbols share same set of parameters).
- Hybrid A hybrid of the two methods above. Use a combination of the best symbols and optimal parameters when forward testing.
The Realization window asks for an "Optimization" on which to operate. When performing SOTF realization, there is no need to optimize, however, an optimization must be created involving the trading system which the user wants tested, and that optimization should be specified in the realization window. In this scenario, the "Skip Optimization" checkbox should be checked as well.
When a Realization is in process, a "Status" area at the bottom of the Realization window provides the user with useful information regarding the progress and state of the Realization. The information provided includes which iteration is currently being processed, which process is currently running (pre-testing, post-testing, optimization, or real-testing), the status of the process currently running, the approximate amount of time left in realization, the Net Profit and APS overall, the Net Profit of Last iteration, and the date range of the data currently being tested.
When the Realization is competed, a report is provided which looks very similar to the :Day by Day" backtest report. It provides a variety of cumulative statistics for the realized testing, including Net Profit, Average Profit / Share, Total Trades, Best Day, Worst Day, etc. It also breaks down each day. If SOTF realization was performed, the "best" symbols used for each iteration, along with the statistical value (Net or APS) of each symbol, is listed for each iteration.
Taking a look at a specific example, let's assume we have a quotepage with 100 symbols (Nasdaq 100) and we have the 500 days of daily data ending 3/23/05 which can be used for testing. To keep things simple, let's look at a very "vanilla" system which buys at the open of each day and sells at the close of each day when the previous day was up (CL.1 > OP.1) and shorts at the open of each day and covers at the close when the previous day was down or flat (CL.1 <= OP.1). This particular system will make one trade (100 shares) every day for every symbol tested. The system exits each position at the close of the day.
If we simply test this system as is over the 500 day period, the Nasdaq 100 gives us a relatively marginal loss of $0.0019 / share (Net $-17,913.42). Based on these results, we certainly wouldn't be excited about trading this system. If anything, we would probably choose to trade the reverse (buy when previous day was down), but even that would result in a very marginal profit that would get sucked up by commissions and slippage. Before discarding this system however, we might want to put it through some Realization runs to see if there potential opportunities.
Using Method 1 as mentioned above, we might choose to trade only the top 50 symbols based on their performance over the most recent 40 days (~2 months), and change this list of 50 each 20 days (~1 month). This would require an optimization period of 40 days, and a realization period of 20 days. The realization window would be setup similar to what is seen below:
Here are a sample of results on the Vanilla system. Opt Period Real Period Symbols Net Profit/Share 0 500 100 (All) $(17,913.42) $(0.0019), 40 20 50 (1/2) $6,568.97 $0.0014 report 40 20 25 (1/4) $10,836.89 $0.0045 report 40 20 10 (1/10) $7,707.49 $0.0079, 40 20 5 (1/20) $7,405.26 $0.0153 report 40 20 2 (1/50) $5,316.20 $0.0274, 40 5 2 (1/50) $11,063.30 $0.0570 report 20 5 2 (1/50) $11,873.15 $0.0589 report 10 5 2 (1/50) $6,151.63 $0.0299.
Looking at the results, it's clear that at least for this system in this environment, the symbols that did well in the recent past, seemed to continue to perform into the future. Using an optimization period of 20 days with a realization period of 5 days, and testing only the best 2 symbols, produced a strong 5.9 cents/share for a net profit of $11,873. This is compared with a $17,913 loss when testing the whole list of 100 stocks. This net profit was also produced trading only 2% of the shares.
Now, let's really give realization a test by turning the Vanilla system on it's head. Instead, we'll buys at the open of each day and sells at the close of each day when the previous day was down or flat (CL <= OP) and short at the open of each day and cover at the close when the previous day was up (CL > OP). Here are some realization results using the VanillaRev system.
Opt Period Real Period Symbols Net Profit/Share 0 500 100 (All) $17,913.42 $0.0019, 20 5 25 (1/4) $18,796.21 $0.0075 report 20 5 10 (1/10) $13,913.98 $0.0138, 20 2 10 (1/10) $3,673.91 $0.0165 report 20 5 2 (1/50) $3,673.91 $0.0182 report
Now, let's throw a variable into the system via a target so that we can involve optimization (on this variable target) in the realization. We'll start by testing with a target of 0.40. Going back to our original Vanilla system, we get a net profit of $3,398 and an APS of $0.0004. This is a pretty nice improvement over our "no target" original system already (Net: $-17,913.42).
Now, we'll setup a "per symbol" optimization on the target, going from 0.20 to 1.00 and stepping 0.20.
Opt Period Real Period Symbols Net Profit/Share 0 500 100 (All) $3,398 $0.0004 40 20 25 (1/4) $15,507.32 $0.0064, 20 5 10 (1/10) $9,150.70 $0.0091 report