Page tree
Skip to end of metadata
Go to start of metadata

This page describes the market data recovery process for Market by Price (MBP) and Market by Order (MBO) order books. This method:

  • should be used for large-scale data recovery (e.g. major outage or late join)
  • synchronizes client order books with the current state maintained by CME Group

The Market Recovery Feed:

  • loops the Market Data Snapshot Full Refresh (tag 35-MsgType=W) message.
    • at the beginning of the week, a Snapshot message is generated each time book/trade activity occurs on an instrument
      • no Snapshot message is sent for an instrument with no book activity
    • more than one instrument can be updated in a packet
  • supports primary (A) and backup (B) feeds
    • Feed A - disseminates market data Snapshot messages for all books having activity since beginning of the week
    • Feed B - acts as a backup in the event that Feed A becomes inoperable
CME Group strongly recommends that the Market Recovery feeds be used for recovery purposes only. Once the client system has retrieved the recovery data, the client system should stop listening to the Market Recovery feeds.
Market Recovery support is required for client systems that utilize MDP 3.0.

This topic includes: 

Market Recovery Feed Messaging

This section describes the messaging and templates for MBP and MBO Market Recovery feeds.

MBP Recovery Feed Messaging

The MBP Market Recovery feed:

  • loops Snapshot messages
  • implements Template SnapshotFullRefresh52 for recovery processing

Template SnapshotFullRefresh52 contains:

  • Top of Book MBP Quotes – Bids and Offers
  • Top of Book Implied Quotes – Bids and Offers
  • Last Trade
  • Opening Price
  • Session High and Low Trade Prices
  • Session High Bid and Session Low Offer
  • Fixing Price
  • Settlement Price 
  • Cleared Trade Volume
  • Open Interest
  • Electronic Volume
  • Threshold Limits
For MBP books, client systems have the option to concurrently recover instrument books via MBP Natural Refresh logic.

MBO Recovery Feed Messaging

The MBO Market Recovery feed:

  • loops Snapshot messages
  • implements Template SnapshotFullRefresh52 and Template SnapshotFullRefreshOrderBook53 for recovery processing

In addition to Template SnapshotFullRefresh52, Template SnapshotFullRefreshOrderBook53 contains recovery data in the following order:

  • MBP information
  • market state
  • statistics
  • MBO book information

The MBO Market Recovery feed also supports:

  • recovery chunk tag 37709-NoChunks and tag 37710- CurrentChunk
    • allows client systems to determine the number of packets required to recover an MBO book.
    • tag 37709-NoChunks - total number of packets that constitute a single instrument order book
    • tag 37710-CurrentChunk - current chunk in the sequence

The example below shows two different instruments updated in an MBO Market Recovery feed.

MBO Chunks 4

Market Recovery Price Interleaving (MBO)

To optimize recovery time for the highest priority orders, the Snapshot message alternates bid and ask orders from highest to lowest priority (tag 37707-MDOrderPriority).

MBO Snapshot Alternate Prices4

Market Recovery Processing (MBP & MBO)

When packets are missed on both UDP Incremental feeds, packet loss is indicated by a gap in message packet sequence numbers.

  • the tag 369-LastMsgSeqNumProcessed value on the Snapshot message corresponds to the packet sequence number on the Incremental feed
  • tag 60-TransactTime communicates the start transaction time of the last event for the instrument

    Market Recovery Feed - Snapshot (tag 35-MsgType=W) Message
    Incremental Feed
    369-LastMsgSeqNumProcessed corresponds topacket sequence number
    tag 60-TransactTime tag 60-TransactTime
The order of each Snapshot message iteration is not guaranteed; client systems must process one full iteration of a Market Recovery Snapshot message starting at sequence number 1 to ensure full recovery.

When this large-scale market recovery method is used, client systems must subscribe to the Instrument Definition feed to determine if any new instruments have been defined. A client application can determine if recovery is necessary using the tag 779-LastUpdateTime timestamp on the Market Recovery feed to detect if instrument data has changed.

Market Recovery Processing Workflow

This section describes the process to follow for large-scale recovery concurrently processing the Incremental feed and Snapshot messages from the Market Recovery feed. Once a book is recovered, client systems can resume normal processing for that instrument even if other books are still being recovered. Client systems will not recover any missed statistics on the Market Recovery feed.

Recovery processing steps are as follows:

  1. Identify channel(s) in which the client system is out of sync.
  2. Listen to the Incremental feed and queue real-time data for the affected channel(s), and optionally begin MBP natural refresh.
  3. Listen to the Market Recovery feed for the affected channel(s).
  4. For a given instrument, compare the Market Recovery Snapshot message tag 369-LastMsgSeqNumProcessed to the Incremental feed Market Data Incremental Refresh (35=X) message packet sequence number.

    • If the SecurityID appears in both the Incremental feed and Market Recovery feed updates during the comparison, then compare the Market Recovery feed tag 60-TransactTime to the Incremental feed tag 60-TransactTime. The instrument with the unequal 60-TransactTime must be recovered via the next market recovery cycle or optional concurrent natural refresh processing.

    • Drop all cached Incremental feed updates with a packet sequence number < 369-LastMsgSeqNumProcessed.

  5. Once all instruments are recovered via market recovery or natural refresh, start normal processing and disconnect from the market recovery feed. 

Market Recovery Feed Processing Examples

The following section provides examples for Market Recovery feed processing.  

Example 1 – Standard Recovery

The following example shows a standard recovery scenario using the following workflow:

  1. Identify the channel that is out of sync.
  2. Connect to the Incremental feed and begin queuing data.
  3. Connect to the Market Recovery feed. In this example, the first Market Recovery Snapshot message sequence number = 4.
  4. Compare the Market Recovery Snapshot message tag 369-LastMsgSeqNumProcessed to the Incremental feed packet sequence number. In this example, the Incremental feed sequence number compared is 104.
    • If the instrument (tag 48-SecurityID) appears in both the Incremental and Market Recovery updates during the comparison, then compare the Market Recovery feed tag 60-TransactTime to the Incremental feed tag 60-TransactTime.
    • In this example, the tag 60-TransactTime comparison is not required due to different instruments between the feeds.
    • For Market Recovery Snapshot message processing, drop all cached Incremental feed updates with a packet sequence number < tag 369-LastMsgSeqNumProcessed from the Market Recovery feed.
    • Begin normal processing for recovered instrument 13205.
    • Recover subsequent instruments via the Market Recovery feed (8087, etc).
    • Additionally, as tag 369-LastMsgSeqNumProcessed increases in subsequent Market Recovery feed updates, continue to drop cached market data where the packet sequence number < tag 369-LastMsgSeqNumProcessed from the Market Recovery feed.
  5. Disconnect from the Market Recovery feed once processing is complete for all instruments.

Incremental Feed

Market Recovery Feed

Incremental Packet Sequence Number

Tag 35-

MsgType

Tag 48-

SecurityID(s)

Tag 60-TransactTime(s)

Snapshot Packet

Sequence Number

Tag 369-

LastMsgSeq

NumProcessed

Tag 60-TransactTime

 

Tag 48-SecurityID

Tag 911-

TotNumReports

100 - drop

X

13205

20191120-18:36:41.000000000


101 - drop

X

5522

20191120-18:36:41.000278478

904

20191120-18:36:41.000293451

102 - drop

X

904

20191120-18:36:41.000320480

103 - drop

X

1741

20191120-18:36:41.000353097

104 - recover instrument 13205 information via Market Recovery Snapshot message and begin normal processing

X

7720

20191120-18:36:41.000368253



4 – Connect to Market Recovery Feed

104

20191120-18:36:41.000000000



13205

10

105 – apply update to 13205 information gathered in the Snapshot

X

13205

20191120-18:36:41.000383610

5 – Next instrument to be processed

108

20191120-18:32:22.080423217

8087

10

106

X

5522

20191120-18:36:41.000412613

Next Market Recovery Iteration 

107

X

12121

20191120-18:36:41.000423217

1

112


1741

10

….




….





Example 2 - Same Transaction Time for Instrument

In the following example, the instrument (tag 48-SecurityID=15212) appears in both the Incremental and Market Recovery feed updates when comparing tag 369-LastMsgSeqNumProcessed. Therefore, client systems must compare the Market Recovery feed tag 60-TransactTime to the Incremental feed tag 60-TransactTime. In this example the tag 60-TransactTime values match and the instrument can be recovered. 

Incremental Feed

Market Recovery Feed

Incremental Packet Sequence Number

Tag 35-

MsgType

Tag 48-

SecurityID(s)

Tag 60-TransactTime(s)

Snapshot Packet

Sequence Number

Tag 369-

LastMsgSeq

NumProcessed

Tag 60-TransactTime

 

Tag 48-SecurityID

Tag 911-

TotNumReports

100 - drop

X

13205

20191120-18:36:41.000000000


101 - drop

X

5522

20191120-18:36:41.000278478

904

20191120-18:36:41.000293451

102 - drop

X

904

20191120-18:36:41.000320480

103 - drop

X

15212

20191120-18:36:41.000368253

104 - recover instrument 15212 information via Market Recovery Snapshot message and begin normal processing


X

15212

20191120-18:36:41.000368253


TransactTime matches with Snapshot message

4 – Connect to Market Recovery Feed

104

20191120-18:36:41.000368253

 

TransactTime matches with Incremental message

15212

10

Example 3 - Different Transaction Time for Instrument

In the following example, the instrument (tag 48-SecurityID=13205) appears in both the Incremental and Market Recovery feed updates when comparing tag 369-LastMsgSeqNumProcessed, however the tag 60-TransactTime values do not match. Therefore, client systems must process the next Market Recovery feed iteration for the instrument.

Incremental Feed

Market Recovery Feed

Incremental Packet Sequence Number

Tag 35-

MsgType

Tag 48-

SecurityID(s)

Tag 60-TransactTime(s)

Snapshot Packet

Sequence Number

Tag 369-

LastMsgSeq

NumProcessed

Tag 60-TransactTime

 

Tag 48-SecurityID

Tag 911-

TotNumReports

100 - drop

X

13205

20191120-18:36:41.000000000


101 - drop

X

5522

20191120-18:36:41.000278478

904

20191120-18:36:41.000293451

102 - drop

X

904

20191120-18:36:41.000320480

103 - drop

X

13205

20191120-18:36:41.000353097

104 

X

13205

20191120-18:36:41.000353097


4 – Connect to Market Recovery Feed

104

20191120-18:36:40.103368841

 

TransactTime does not match with Incremental message, must process next loop to recover 13205

13205

10

105

X

13205

20191120-18:36:41.000353097

4 – Next instrument to be processed

107

20191120-18:36:22.000423217

8087

10

106

X

5522

20191120-18:36:41.000412613

Next Market Recovery Iteration 

107

X

12121

20191120-18:36:41.000423217

1

110


1741

10

….




….





170

X

13205

20191120-18:36:43.121427232


TransactTime matches with Snapshot message, recover instrument 13205

4

170

20191120-18:36:43.121427232


TransactTime matches with Incremental message, recover instrument 13205

13205

10

Example 4 - New Market Recovery Snapshot Message Added

In the following example there is activity for an instrument (tag 48-SecurityID) that adds a new Market Recovery Snapshot message to the Market Recovery feed. Therefore, to ensure all new instruments are recovered, client systems must process an additional Market Recovery iteration starting from the Snapshot message with a packet sequence number = 1. Client systems can determine if a new Snapshot message has been added by comparing tag 911-TotNumReports with the previous iteration. The order of each Snapshot message iteration is not guaranteed, therefore the newly created Snapshot message may be placed mid-loop in the next iteration. 

Incremental Feed

Market Recovery Feed

Incremental Packet Sequence Number

Tag 35-

MsgType

Tag 48-

SecurityID(s)

Tag 60-TransactTime(s)

Snapshot Packet

Sequence Number

Tag 369-

LastMsgSeq

NumProcessed

Tag 60-TransactTime

 

Tag 48-SecurityID

Tag 911-

TotNumReports

105 - drop

X

12171 – New instrument activity generates a Market Recovery Snapshot message

20191120-18:36:41.000383610

4  - Recovery in process

107

20191120-18:36:41.000013217

8087

10

106 - drop

X

5522

20191120-18:36:41.000412613

Next Market Recovery Iteration 

107 - drop

X

13205

20191120-18:36:41.000423217

1

110

20191120-18:36:41.000353097

1741

11 – incremented to reflect new Snapshot message for instrument 12171

….




….





170 – begin normal processing for instrument 12171 from Market Recovery Snapshot due to the Incremental update for packet sequence number < to 369-LastMsgSeqNumProcessed


X

904

20191120-18:36:41.001223443

3 – new Snapshot placed mid-iteration

170 – recovery instrument via Snapshot

20191120-18:36:41.000383610

12171

11

171 – apply update to 12171 information gathered in the Snapshot

X

12171


….





….




  • No labels