JTOOLS INTERVAL FILE GENERATOR - FUTURES SPECIFIC


JTools Table of Contents



Project Location: \JTools_IntervalFileGenerator_Futures


The Interval File Generator for Futures and Future Options is very similar to the Standard Interval File Generator but contains some modifications and special features. Like the Standard Interval Generator, the data is true interval data. That is to say that if the timing interval is set to 1 minute then the values generated are for 1 specific minute in time.

The JTools_IntervalFileGenerator for Futures allows you to dump trade information in almost any fashion you desire. The system gathers information from NxCore Trade messages for interval data. The application can be run from state files generated just before or at the desired time. Fields generated and field order are determined by the usage strings (see below for a complete list of possible fields). It will dump the output to the console (if specified to) and to a dated text file(s).

The following interval data can be produced with the The JTools_IntervalFileGenerator for Futures:
  • One specific future.
  • One specific future and all underlying future options.
  • All futures belonging to a specific root symbol.
  • All futures belonging to a specific root symbol plus all underlying future options.
Data is output for each day to one file per symbol, or to a data-time series file containing all symbols (unlike the Standard Interval Generator which outputs data for all symbols only into date-time series files).


Data is also collected using a different method than the Standard Interval Generator, in that each symbol has it's own composite structure with 1440 possible data points (the number of minutes in one day). This is possible as the symbol scope (and hence memory usage) is more limited than the Standard Interval Generator (which can contain all symbols for all data types, equities, futures, indexes, etc).

Another feature not possible with the standard interval generator is the ability to correct bar data based on cancellations and trade corrections by capturing Category 25 Bar Correction Messages.

The JTools Interval Generator for Futures also combines the functionality of the JTools Batch Processor to make processing data which spans days, months or years trivial.



Interval data timestamps will mark the beginning of the interval. For instance, if 5 minute interval data is chosen, an interval at 9:30 will contain data from 9:30:00 through 9:34:59. The Open price reflects the first price recorded for the specific interval. The last price will represent the last price recorded for the specific interval. The high and low price reflect the highest price and lowest price during the specific interval. Volume will reflect the total volume accumulated during the specific interval.


Request Data Parameters:

  • Future Symbol or Root - The specific futures symbol or the root symbol to be output. For example, fES.M14 is a specific symbol for the June 2014 Emini contract, and only that symbol would be output. fES however is the root symbol and all fES Emini contracts will be output.

  • Generate Options Data - If checked, underlying options will be generated.

  • Listed Exg Filter - "0"=Include all exchanges. If only specific exchanges are desired, enter those exchange here, comma separated. IE for NYSE and NQEX - 3,12

Time and Interval Period Parameters:

  • Start Time (HHMM) - Time that the application will start collecting/generating interval data. Example - "09:30".

  • Stop Time (HHMM) - Time that the application will stop collecting/generating interval data. Example - "16:00".

  • Bar Interval (1-60) - Amount of minutes each interval represents.

Data Parameters:

  • Data Usage/Output String - Possible values:
    (s) Symbol (e) Listed Exg (d) Date (t) Time (1) Open (2) High (3) Low (4) Last (5) NetChg (6) Total Volume (7) Tick Volume (a) Ask Price (b) Bid Price (8) Bid Size (9) Ask Size (E)Expiration Date (O) Open Interest (/) Blank Field.

    Additional Values for Future Options Only:

    S) Strike (U) Underlying Symbol (C) Call/Put

    The field order of the output data will match the field order of the Data Usage String supplied.

Misc Parameters:

  • Print Headers - Print headers in output file if checked.

  • Exg Codes as Text - If checked, exchange codes are output as text, otherwise they are output as numeric.

  • Process Empties - If checked issues with no data for the interval will be included and zeroed. If unchecked, only intervals where trading data was recorded will be output.

  • Carry Through OHLC on Empty- If checked, a bar that has no trade data will carry the previous bar values for the Open,High,Low,Last values.

  • Carry Through Bid Ask on Empty - If checked, a bar that has no new quote data will carry the previous bar values for the Best Bid, Best Ask, Best Bid Size and Best Ask Size.

  • Output Symbol Prefix - If checked symbol prefixes will be output with the symbol.

  • Decimal precision - ("0" - "5", "-1" for auto based on price type).

  • Price Filter Level - NxCore halt built in analysis on every trade that can help determine possible erroneous pricing (which may cause spikes when charting the data). One such item is the Filter Level field within the NxCTAnalysis structure. You may see the full documentation for this HERE.

    For example, if you wish to for eliminate all trades with a filter level of 6 or higher, set the Price Filter Level to 6. Note that the volume for these trades will still be included in the total interval volume, but the pricing will be ignored.

  • Process Saturday - If checked files will be generated on Saturday.

  • Process Sunday - If checked files will be generated on Sunday.

Output File Parameters:

  • Write On Interval - If checked, data files will be generated when each time interval is passed. No corrections are available for data generated in this manner. May be used in conjunction with Write On End-Of-Tape.

  • Write On End-Of-Tape (or Stop Time) - If checked, data files will be generated either a) when the specified Stop Time has been reached or b) at the end of the tape (see below). May be used in conjunction with Write On Interval.

  • Continue Until Complete Msg - If checked and Write On End Of Tape is also specified, data will be written when the tape completes as opposed to the specified Stop Time. This allows any corrections to be processed as corrections do not occur until after the actual trade data, sometimes much later.

  • Symbol Based Files - If checked, files will be named and output according to the stock symbol. If unchecked, all symbols will be named and output to a single date-time based file.

  • Append - One File Per Symbol - If checked, data will be written to one file per symbol, continuously. Symbol Based Files and Write On End-Of-Tape options must also be enabled (checked on).

    Note: If appending to one file per symbol, the files are never overwritten with new data, but the existing data is appended to. Therefore, should you change parameters in the application and wish to regenerate the data from scratch, you must first Delete the existing files. Use the Delete Files button to accomplish this, or manually delete them through Windows Explorer.

  • Default File Extension - The default file extension to use for filenames.

  • Field Separator - A single character used to separate data fields within the files.

  • Data Folder - Folder data files are created in.

    If left blank, the output files will be generated in the sub-folder JTools_OutData\IntervalData\Stocks. The sub-folder will be located in the same folder that the application runs from.

  • Delete Files - Deletes all files in the Data Folder with filenames ending with ".IFGSIO.<Default File Extension>" . You will be asked when pressing this button if you are sure, before the deletion occurs.

Batch Mode Parameters:

  • Run in Batch Mode - If checked, batch parameters become enabled and allow the application to process files in a "Batch Mode". If unchecked, individual files or realtime mode may be chosen in the standard settings window.

  • Start Date - Date of the first historical file to be processed.

  • Stop Date - Date of the last historical file to be processed.

  • Tape Set - The NxCore 2 or 3 letter tape set code.

  • Tape Ext - The NxCore tape extension (nxc, nx2 or nx3).

  • Tape Path - Path to the historical tapes.

  • Use Year Subfolders - Many users will organize their historical tapes by year (such as Data\2016, Data\2015, etc). If 'Use Year Subfolders' is enabled then the application will search in the Tape Path folder + the year sub-folder for the specific NxCore historical tape file.

  • Continue on Errors - If checked and an error occurs (such as a missing file in the sequence) processing will continue with the next file. If unchecked processing will stop on the first error encountered.

  • Run in Reverse Direction - When selected, Batch processing will occur from the stop date to the start date. If not selected batch processing occurs from the start date to the stop date.

  • Output Data to Activity Monitor - Output of the actual interval data will be displayed in the activity monitor if checked. While this can be useful to determine if data is actually being generated, it can severely impede performance of the application.

  • Output File Status to Activity Monitor - Output of the file write status will be displayed in the activity monitor if checked. Note any file errors will be reported regardless of this setting.

Project Location: \JTools_IntervalFileGenerator_Futures



JTools Table of Contents