Skip to end of metadata
Go to start of metadata

CME Group provides market data recovery services and related recovery methods for recovering missed market data or synchronizing client systems to the latest state. Client Systems must support the Recovery Services and may use the Recovery Methods in conjunction with the Recovery Services.

Recovery Services include:

  • Market Recovery - Snapshot loop to recover most recent market state, all book levels, and statistics per instrument per channel.
  • Instrument Recovery - Instrument loop to recover most recent set of instruments.
  • TCP Recovery - Establish TCP connection to recover all missed packets.

Recovery Methods for use concurrent with Recovery Services include:

  • Instrument Level Sequencing
  • Natural Refresh

MDP 3.0 - Channel Reset provides a process for synchronizing order books and trade session statistics.

  • CME Group strongly recommends that client systems process both the A and B Incremental UDP feeds. UDP Feed A and UDP Feed B provide the first level of protection against missed market data messages.

  • CME Group recommends Market Recovery in conjunction with Natural Refresh as a primary recovery option.

  • All customers must certify for Market Recovery functionality.

Processing Overview

This topic describes CME Group recommended recovery processes. Based on the recovery and supplemental methods, client systems can choose to implement those methods as applicable. In general, two paths can be followed: processing while recovering, and queuing while recovering.

The recovering data process applies to affected channels only. Unaffected channels can continue normal processing.

The following diagram illustrates the two paths to recovery. Note that processing while recovering employs either Instrument Level Sequencing or Natural Refresh logic.

Recovery Process Overview

The Market Recovery feed should be used for large-scale data recovery (i.e. major outage or late joiners). TCP replay is not a performance-based recovery option and should only be used for small-scale data recovery.

Large Scale Outage Using Market Recovery – Queuing while Recovering

This section describes the process to follow for a large scale outage in which the client system is out of sync. This process uses Market Recovery - queuing of the Incremental Market Data feed and Market Recovery feed until the client system is synchronized to the latest state transmitted by CME Group. To avoid an excessive number of queued messages, it is recommended that the client system process snapshots and apply the applicable incremental feed as the snapshots arrive.

If the Market Recovery method is used, client systems also need to subscribe to the Instrument Definition feed to determine whether any new instruments were defined. Client systems will recover the most recent statistics on the Market Recovery feed. Any intermediary statistics will not be recovered.

 

  1. Identify channel(s) in which the client system is out of sync.

  2. Listen to and queue the Incremental Market Data feed for the affected channel(s).

  3. Listen to the Market Recovery feed for the affected channel(s).
 
Make sure the snapshot loop messages contain tag 369-LastMsgSeqNumProcessed with a value greater than sequence numbers from the queued messages. If tag 369-LastMsgSeqNumProcessed is lower than the lowest sequence number in the queue, wait until the next snapshot loop.

Optional

  • Tag 779-LastUpdateTime in the Snapshot (tag 35-MsgType=W) message provides the time of the last instrument add/update or delete on the channel.
  • During recovery, the Client System can compare the value in tag 779-LastUpdateTime of the Snapshot to the Client System record of when the last Security Definition (tag 35-MsgType=d) message was processed on the channel to determine if there is a need to recover the instruments from Replay loop.
  • This method is optional; the Client System can recover all instruments without checking this value.

4. Verify that all snapshots have been received for a given Market Recovery feed.

Tag 911-TotNumReports in a Market Data Snapshot Full Refresh (tag 35-MsgType=W) message contains the total number of messages that have been sent. Listen to the Market Recovery feed until the indicated number of Market Data Snapshot Full Refresh (tag 35-MsgType=W) messages has been received.

5. Apply all incremental data in sequence, where EITHER:

  • the last processed packet sequence number is greater than the lowest value for tag 369-LastMsgSeqNumProcessed OR
  • tag 83-RptSeq from the Market Data Incremental Refresh (tag 35-MsgType=X) message is greater than the lowest value for tag 83-RptSeq on the Market Recovery feed.

6. Using queued data, restart normal processing.

Large Scale Outage Using Market Recovery - Processing while Recovering

This section describes the process to follow for a large scale outage using Market Recovery while continuing to process the Incremental Market Data feed and obtaining snapshots from the Market Recovery feed. Once books are recovered, they can resume normal processing even if other books are still being recovered.

If the Market Recovery method is used, client systems also need to subscribe to the Instrument Definition feed to determine whether any new instruments were defined. Client systems will recover the most recent statistics on the Market Recovery feed. Any intermediary statistics will not be recovered.

  1. Identify channel(s) in which the client system is out of sync.

  2. Listen to the Incremental Market Data feed for the affected channel(s) and optionally attempt a natural refresh.

  3. Listen to the Market Recovery feed for the affected channel(s).

 

Optional

  • Tag 779-LastUpdateTime in the Snapshot (tag 35-MsgType=W) message provides the time of the last instrument add/update or delete on the channel.
  • During recovery, the Client System can compare the value in tag 779-LastUpdateTime of the Snapshot to the Client System record of when the last Security Definition (tag 35-MsgType=d) message was processed on the channel to determine if there is a need to recover the instruments from Replay loop.
  • This method is optional; the Client System can recover all instruments without checking this value.

4. Confirm that EITHER:

    • the last processed packet sequence number is greater than the lowest value for tag 369-LastMsgSeqNumProcessed OR
    • tag 83-RptSeq from theMarket Data Incremental Refresh (tag 35-MsgType=X) message is greater than the lowest value for tag 83-RptSeq on the Market Recovery feed.

5. Restart normal processing.

  • No labels