Pre-processing data, filters, and the frequency domain

Login or Register

Please login or register in order to view and post in the Forum.

1 post / 0 new
galtstein
Last seen: 1 year 10 months ago
Joined: 05/02/2012 - 00:00
Pre-processing data, filters, and the frequency domain

Hey traders, couple caveats real quick:

Trying to come from a humble, honest, and helpful place

No agenda, not affiliated with IRT, respect Chad and grateful for the help he’s given me in implementing some of my ideas over more than 10 years. Excited to see rapid pace of advancement in capabilities, tools, and data at IRT.

Trying to give something back. Wish somebody would’ve shared with me what I’m about to share. Would have saved me a lot of money and time.

LOVE that Chad has made lots of John Ehlers contributions to trading readily accessible in IRT.

Also, this helps me clarify some of my thoughts and thinking, and maybe someone points out mistakes I’m making or suggests improvements

But always remember these are one man’s thoughts. My OPINIONS. Please be skeptical of the ideas and voices you let inside your mind when it comes to short-term trading and making predictions about prices. Most of the voices are wrong. Above all, be skeptical of every word of mine because ultimately these are things you must discover yourself.

---

I’m going to explain how to implement something John Ehlers calls a “Hybrid Filter” in Investor RT. Search those terms and you can find info that goes into much more detail than I will with some of the concepts and math. In fact, read and trust John Ehlers. He’s one of the most intellectually honest voices there ever was in this domain we call trading. It takes a little bit of work to understand some of his concepts, but once you grasp them, the how’s and why’s, it’s valuable, universal, fractal, and customizable. And this one technique/tool in particular will immediately improve your trading results.

Please read on if you are NOT pre-processing your data or if you don’t know what this means, or if you’re not filtering your data, or if you’ve never heard of the frequency domain.

---

An HFT guy I worked with taught me the critical importance of pre-processing your data BEFORE ever attempting to make a prediction, particularly a prediction about price.

One of the simplest ways to pre-process your input data, and immediately gain benefits in your outputs like indicators, is by taking the mid-point of price ( High + Low / 2 ) rather than the last or close price. Two inputs, more and better information, than one, such as the LAST price. Pretty simple concept.

I don’t use LAST or CLOSE prices anywhere. In fact, I don’t use raw market prices to make any buying or selling decisions.

Only place I see any value in using a LAST or CLOSE market price is in the base calculation of the Internal Bar Strength indicator – one of the few price-based indicators I’ve found use for. Outside of that, almost all of my inputs use the midpoint in my calculations, whether it’s price, or some other data, because the output is smoother and provides a better estimate.

An easy way to apply this, and improve the results in your indicators and outer outputs, is to use a Simple Moving Average of 1 bar and the H/L as the input. Now, feed THIS into your indicators and you should immediately see smoother trends and crisper turns at the highs and lows, compared to using the LAST or CLOSE input. You’d implement this with a Custom Indicator with something like:

RSI ( SMA )
CCI ( SMA )

You could use the midpoint as inputs in the RSI and CCI and achieve the same result, but the key point is taking one indicator, and using it as an input to another. This is the basic, quick way to start filtering raw, market-generated data.

I haven’t looked into the formulas behind some of the other periodicities like Renko, or TLB, or Heiken Ashi for a long time, but I would guess that they all, sort of, filter price in one way or another.

This idea of a filter – Why do we filter air or water? To make it cleaner. Clearer. Make it better. I filter data like prices of futures or stocks to make it cleaner. To make it “cleaner” means easier to track and predict.

What I’m doing is twisting, contorting, turning inside-out, looking at from different angles, and under different lights, all the raw market-generated data. Ultimately I’m making my OWN DATA because I’ve found it’s easier to measure, track, and ultimately predict, than raw market-generated data.

Because let’s face it, predicting prices of financial instruments is difficult, especially when you look at these changing prices in the time domain. It is my belief that any attempt at predicting ANYTHING had better be done as far away from the time domain as possible. The time domain is full of static. Noise. It’s where the academics and index funds are stuck, believing in the efficient market hypothesis. It’s my belief the time domain is the reason I would knee-jerk react out of FOMO, buying the high, and getting stopped out at the low.

But you have the volume domain to work with, and other periodicities at your disposal, like Tick, Renko, P&F, etc. It’s up to you to figure out which is right for you and your style or system. But in my view they are all better than the time domain.

(FWIW…There are only 2 intervals of price data that I find any use for whatsoever from the time domain, and that’s Daily and 1-sec. Everything else, to me, is utterly useless when it comes to having anything to do with trying to predict price.)

I believe the volume domain itself serves as a sort of filter for price and some kind of pre-processing of your data. That’s because it’s recording prices in units of volume and I believe volume is the life-blood that moves prices. Like the old HFT guy liked to say, the passage of time does not motivate changes in price. It’s actual buying and selling that motivates changes in price, particularly, imbalances in buying and selling. And you need look no further than the Volume Breakdown indicator to know EXACTY the balance, or imbalance, of actual buying and selling. This Volume Breakdown is what brought me to IRT at those years ago.

The volume domain filters because it’s adapting to changing market conditions and this introduces some non-linearity in your data stream, which I believe is essential to successfully predicting prices in the short-term.

So, once in the volume domain, here’s where the filtering begins. This is a Custom Indicator you need to build for Ehlers Hybrid Filter:

0.67 * ( 2 * MED – MED.1 ) + 0.33 ( EMA (3) ( WMA ( 4 ) )

Let me breakdown what’s going on here, starting with the left side of the equeation:

( 2 * MED – MED.1 )

This is a quick, dirty, efficient, and effective way to SPEED-UP your data. You can think of this like making a forecast 1 bar into the future.

The 0.67 is a weighting that you’re giving the SPEED component of this filter on the left side of the equation. The other side, with the moving averages, let’s talk about that:

EMA (3) ( WMA ( 4 ) )

I’m using a 4-bar Weighted Moving Average as the INPUT to a 3-bar Exponential Moving Average for a reason. Little bit different than what Ehlers used in the PDF I referenced earlier because I wanted this filtered price to be a bit more raw and faster than the output from the MAs he used in the example. Now, I have VERY CAREFULLY, chosen a 4 bar WMA. A 3 or 5-bar WMA wouldn’t work as well. There’s a reason for that. There is a reason I chose the EMA over other moving average types as the output for this half of the filter. There is a reason I chose a 3-bar window for the EMA.

Until you understand all the reasons, I suggest keeping as is. Like Picasso said something like: first learn the rules, so you know how and when to break them.

Back to the above equation and Hybrid Filter -- The right side of the equation is smoothing and filtering out some of the randomness, chaos, and excess that still exists in raw prices, even in the volume domain. Let’s call it SMOOTH.

Now, in Ehlers original calculation and PDF, he gives SMOOTH a weighting of 0.33. to balance out that 0.67 weighting given to the SPEED component. So it’s like this:

0.67 SPEED + 0.33 SMOOTH = Hybrid Filtered Price

Like Ehlers says, this 67% and 33% weighting is a matter of personal preference and data types you’re using. Generally, faster is better, but I’ve experimented with other weightings like 50/50 or even given more weighting to the smoothing side.

For instance, TICK indicators might require more smoothing than speeding up, because they are already pretty fast just given the nature of the data. But something like an advance/decline line can be pretty smooth to begin with, but kind of slow, or same goes for an Accumulated Volume Breakdown.
So, you might want to speed up those date inputs even more, or maybe NOT even smooth them at all, and use just the SPEED component. You can customize ll this by changing the weightings to SPEED vs. SMOOTH

Can you see how you can start to take these concepts, these techniques, of massaging the data, of pre-processing, in so many different directions? I can say, without a doubt in my mind, that no other trader on this planet is looking at the same set of data inputs or outputs as I have on my screens. It's because I have learned this process for approaching the market. Focus on this process.

Back to the Hybrid Filter and I’ll move forward with Ehlers’ recommendation of 67% weighting toward the SPEED side and the 33% weighting to the SMOOTH side:

0.67 * ( 2 * MED – MED.1 ) + 0.33 ( EMA (3) ( WMA ( 4 ) )

The output of this Custom Indicator is now a filtered price. In the chart below, I show you the difference between a filtered price and the raw, market price. There’s a huge difference and less noise, but the tradeoff is the filtered price is a little slower, but I’m okay missing, or not even trying for, the exact tick or low/high, because it’s so much easier to watch this smoother/filtered price than the market price, and to understand its movements.

Now, from here, you could wrap a moving average around this Hybrid Filter output with something like:

AMA (3) of Hybrid Filter (0.67 * ( 2 * MED – MED.1 ) + 0.33 ( EMA (3) ( WMA ( 4 ) ), where AMA = Adaptive moving average

Or maybe a:

Sine-weighted_MA (4) of Hybrid Filter ( 0.67 * ( 2 * MED – MED.1 ) + 0.33 ( EMA (3) ( WMA ( 4 ) )

You could use this moving average of the Hybrid Filter as a tool to TRACK where price has been and where it is. This word “TRACK” is NOT predict. Remember this.

But where things get real interesting is using the OUTPUT of the Hybrid Filter as an input to other indicators. You can do this by declaring the Hybrid Filter output as a variable (e.g. V#XX) right inside your Custom Indicator. This is how I generated the chart below of the raw bars for the Hybrid Filter output. You can do this by using the Variable you just declared to create a Custom Instrument out of the Hybrid Filter and apply all the other tools and indicators already built in IRT.

And there’s one in particular that I’d like to draw your attention to and that’s the BandPass filter – another Ehlers nod.

You can find the BandPass filter as one of the components of the Empirical Mode Decomposition indicator. I recommend sticking pretty close to the default settings because if you stray too far, it turns BAndPass output into complete garbage.

But if you use it as intended, it could improve your trading right away as a standalone indicator OR, like I’ve been explaining with the Hybrid Filter, you can use the BandPass as an input.

For our purposes, I’m doing this:

Hybrid Filter ---> BandPass

But you could do something like this:

BandPass ---> CCI, RSI, Stochastics…whatever

There’s a lot of room for creativity in short-term trading.

Back to the BandPass – remember all that ranting about the time domain? Well, the BandPass transforms your data and places it in the frequency domain, where prices are much easier to predict, and the market is orderly most of the time, and the outputs are mostly minus 1s, zeros, and plus 1s. Pretty cool. Some real Matrix-type-shit goes on in the frequency domain. But don’t take my word it. Observe for yourself.

Let me summarize:

1, We’re creating a Custom Indicator called the Hybrid Filter courtesy of Mr. John Ehlers. Super simple to do using Investor RT, but really cool and powerful. It makes data easier read and it's like looking into the future.

2. Applying this Hybrid Filter to price in the Volume domain

3. Using a variable to turn the output into a Custom Instrument

4. Charting this instrument as better, smoother, easier to track version of raw market price

5. Using this better version of price as an input to a very useful indicator called a BandPass filter

6. Tracking price in the frequency domain

Another tool you can use to transform your data into the frequency domain is the FISH indicator. This is a Fisher transformation and one of the most important and often-used tools in my systems/indicators/tools that I create.

The FISH gives raw price data a close-to normal distribution, which makes all the difference when feeding it to common indicators. So like the BandPass, the FISH can be used as an indicator unto itself, or better yet, as an input to other indicators. Doing something like:

RSI ( FISH )

…results in a dramatic improvement in indicator performance.

Bare minimum, shortest FISH input you can get away with is 2. Probably don’t ever need to go above 10. I usually use 4 or 8.

When to use BandPass and when to use FISH?

I tend to use the BandPass on data that drift, that trend, and the FISH seems to work better on data that are stationary like a NYSE TICK. In fact, take a look at a volume-based NYSE TICK and run it through a FISH transformation and then chart that against the ES or SPY. Just be mindful of the volume units in each symbol.

Okay…back to our little project. Let’s see what all this actually looks like.

Below is a 150 volume bar chart of the NQ continuous contract from Thursday, 08/30, late in the day. That’s a pretty fast chart when the market is moving, like it was Thursday afternoon. Can be hard to keep up with the machines or sometimes impossible.

Pane 1 is a raw, unfiltered, market-generated price of the NQ.

Pane 2 is the Hybrid Filter of the NQ. This is a variable that I’m calling as a Custom Instrument..

In pane 3, I’m using the Hybrid Filter as an input to the BandPass filter with the default settings and I sped it up a little bit.

The chart below is of a range-bound, cycling market, where it would seem you could just use the BandPass by itself and make money.

I’m going to post a second post in the forum with a second example of this same chart set-up, from the same day, right before this cycling period, but of a different, trending phase, and you’ll see why I don’t use the BandPass as a standalone indicator to predict price.

But this is simple: This application of the Hybrid Filter as an input to the BAndPass is a better way to TRACK where the market was and where it is. But it’s not predicting where the market is going.

Now, you might infer some rules or logic from the BandPass, that could maybe assist in predicting price, but I don’t find it to be predictive by itself, when using price as the input. But, if you were to apply the BandPass to some other data, besides price, that’s where you might start to see some predictive, leading qualities in your outputs, in the frequency domain. I’ve found the Volume Breakdown as an input that has predictive qualities.

But, when applied to price in the way I’m doing, I find the BandPass to be helpful to define timing. WHEN to buy. WHEN to sell. But since we’re in the frequency domain, WHEN really doesn’t matter. It’s more like: If, and only if, a certain set of conditions are met, only THEN it’s probably a good time to buy or sell.

I’ve given you some loose guidelines and ways to think about these questions of when to buy and when to sell on the chart, using the BandPass. You can fill in the spaces. But I’ll give a hint: Watch the BandPass in real time for a couple of weeks, and pay attention to what it does and how it reacts in relation to price, especially around zero and +/- 1, and +/- 2. Then, apply it to data other than price.

In summary, I humbly suggest you pre-process your data and whatever you do, whoever you are, please avoid predicting price in the time domain.

0