Smart Tick Compression

If you are running Investor/RT 14 or higher, do not use the smart compression feature anymore which was designed to cope with possible database size limitations of earlier 32 bit versions of Investor/RT

Smart Compression of Tick Data can be turned on by choosing Setup > Preferences > Historical Data from the main menu and checking the check box titled "Use Smart Compression of Tick Data."

During our testing, Smart Compression reduced the tick data for typical futures contracts (number of ticks) consistently by over 85%. Users can now store approximately 7 times as much tick data in the same amount of space. This compression has a number of very positive impacts on the performance and capability of IRT such as:

  • users can store more tick data
  • charts load and process faster
  • databases verify and backup more quickly
  • faster back testing

When Smart Compression is on, there a few aspects of IRT that may be adversely affected, however. Charts with Tick based periodicities are one example. If tick bar charts are a big part of your analysis, then Smart Compression might not be for you since tick bar charts will not function properly with compressed ticks. However, ALL other periodicities (Volume Bars, Range Bars, Second Bars, etc) will be identical with Smart Compression turned on. Also, advanced volume functionality like the Profile Indicator (Volume at Price / Delta at Price), the Volume Breakdown Indicator (Delta per Bar), and VWAP, will also be unaffected by Smart Compression. Certain results of the Volume Breakdown Indicator can be affected.

When Smart Compression is on, ticks are stored in the database in a compressed format by combining similar ticks which share the price, bid, ask, and timestamp. These similar consecutive ticks are stored as one tick with a volume that represents the cumulative volume of the component ticks. Once this option is turned on, existing historical data is compressed, and all new tick data that flows into the database will be compressed. This includes ticks that result from downloads/backfills and live ticks that are flushed to the database at the end of each session and each time the user quits the program.

What aspects of Investor/RT can be adversely affected when Smart Compression is turned on?

Any aspect of IRT that relies upon knowledge of the actual number of raw ticks over a given period, will not work accurately with Smart Compression. These include:

  • Volume Filtering Functionality
    • Volume Breakdown Indicator with "Consider Trades with Size >/</=" volume filtering turned on.
    • Instruments/Bars with Size Filtering turned on within "Load Volume at Price"
  • Tickbar Charts or Periodicities
  • Specific Volume Breakdown Indicator Options including:
    • VB is setup to be based on "Trades" instead of "Volume" (top right setting in Prefs window)
    • VB setup with any "Result" that is based on tick counts including:
      • Total Trades/Ticks
      • Average Size
      • Big Trade%
      • Average Buy Size
      • Average Sell Size
      • Buy Trades/Ticks
      • Sell Trades/Ticks
      • VB setup with a volume filter "Consider Trades with Volume" above, below, or between a given level.