Page tree
Skip to end of metadata
Go to start of metadata

The Market by Price (MBP) and Market by Order Limited Depth (MBOLD) Market Recovery for the CME MDP Conflated UDP market data group should be used for large-scale data recovery (i.e. major outage or late joiners) to synchronize client systems to the latest state maintained by CME Group. 

Market Recovery support is required for client systems that utilize the CME MDP Conflated UDP market data group.

MBP & MBOLD Market Recovery Messaging

The MBP & MBOLD market recovery feed constantly iterates and sends the Market Data Snapshot Full Refresh (tag 35-MsgType=W) message via templates SnapshotFullRefresh and SnapshotFullRefreshOrderBook. Template SnapshotFullRefreshOrderBook53 communicates Market by Order Limited Depth (MBOLD) book information. Template SnapshotFullRefresh52 communicates the following information:

  • Market by Price (MBP) 
  • Implied book information
  • Last trade
  • Electronic Volume
  • Session Statistics
  • Market States

A Market Data Snapshot Full Refresh (35=W) message will be generated only if the instrument has had book/trade activity since the beginning of the week. Once the instrument has had at least one order, it will stay in the loop until the end of the week, even though the order is cancelled. Market Data Snapshot Full Refresh (35=W) messages will not be sent for the instruments that have only price bands, or thresholds, but have had no book activity since the beginning of the week. For an instrument MBP, market state, and statistics information is sent first then the MBOLD book information is sent. More than one instrument can be updated in a packet. The example below shows the message structure of three different instruments updated in an MBP & MBOLD Conflated UDP market recovery feed across two packets.

MBOLD-Packets

MBOLD Market Recovery Price Interleaving

To optimize recovery times for the highest priority orders, the MBOLD Market Data Snapshot Full Refresh (tag 35-MsgType=W) message will alternate bid and ask orders from highest to lowest priority (tag 37707-MDOrderPriority).

MBOLD-Price-Format

MBP & MBOLD Market Recovery Processing

When packets are missed on both UDP incremental feeds, a recovery process is required to take place. Packet loss is detected using the message packet sequence numbers. The packet sequence number is incremental; therefore, if a gap is detected between packets, it indicates a packet has been missed. In such a case, it should be assumed that all books maintained in the client system may no longer have the correct, latest state maintained by CME Group. Client systems must resynchronize all books to the latest state maintained by CME Group, and determine whether any new instrument definitions were published.

CME Group strongly recommends that client systems process both the A and B Incremental UDP feeds. UDP Feed A and UDP Feed B provide the first level of protection against missed market data messages.

Tag 369-LastMsgSeqNumProcessed value on the Market Recovery feed Snapshot (tag 35-MsgType=W) message corresponds to the packet sequence number on the Incremental Market Data feed. Additionally, tag 60-TransactTime communicates the start transaction time of the last event the security participated in. Therefore, tag 369-LastMsgSeqNumProcessed in conjunction with tag 60-TransactTime from the Market Recovery feed Snapshot (tag 35-MsgType=W) are used to link to the incremental feed packet sequence number and tag 60-TransactTime respectively. The order of each snapshot iteration is not guaranteed, therefore client systems must process one full iteration of a market recovery snapshot iteration starting at sequence number 1 to ensure full recovery.  

When the Market Recovery method is used, client systems also need to subscribe to the Instrument Definition feed to determine whether any new instruments were defined. A client application can use the tag 779-LastUpdateTime timestamp on the market recovery feed to determine if instrument data has changed, and if recovery is necessary, from the Instrument Replay iteration in addition to the Market Recovery iteration. 

MBP & MBOLD Market Recovery Processing Workflow Overview

This section describes at the process to follow for a large scale outage using Market Recovery while continuing to process the Incremental Market Data feed and obtaining snapshots from the Market Recovery feed. Once books are recovered, client systems can resume normal processing even if other books are still being recovered. Client systems have the option to concurrently recover Market by Price (MBP) instrument books via MBP Natural Refresh logic. Additionally, MBOLD books can be naturally refreshed on the incremental feed due to the overlay full book snapshot updatesConcurrent processing of Market Recovery and natural refresh provide the best book recovery performance. 

Client systems will not recover any missed statistics on the Market Recovery feed.

  1. Identify channel(s) in which the client system is out of sync. 
  2. Listen to the Incremental Market Data feed and queue real-time data for the affected channel(s), and optionally begin MBP or MBOLD natural refresh. 
  3. Listen to the Market Recovery feed for the affected channel(s). 
  4. For a given instrument, compare the market recovery snapshot tag 369-LastMsgSeqNumProcessed to the incremental packet sequence number. Next, if the security id appears in both the incremental and market recovery updates during the comparison, then compare the market recovery Tag 60-TransactTime to the incremental Tag 60-TransactTime. The instrument with the unequal 60-TransactTime must be recovered via the next market recovery cycle or optional concurrent natural refresh processing. Additionally, drop cached all incremental updates with a packet sequence number < 369-LastMsgSeqNumProcessed.

  5. Once all instruments are recovered via market recovery or natural refresh, start normal processing and disconnect from the market recovery feed. 

UDP Market Recovery Processing Examples

The following section provides examples for UDP market recovery feed processing.  

Example 1 – Standard Scenario

The following example shows a standard recovery scenario using the following workflow:

  1. Identified channel that is out of sync.  

  2. Connect to the incremental feed and begin queuing data.

  3. Connect to the market recovery feed. In this example, the first market recovery snapshot has a sequence number is equal to 4. 

  4. Compare the market recovery snapshot tag 369-LastMsgSeqNumProcessed to the incremental packet sequence number. In this example, the incremental sequence number compared is 104.
    • If the instrument (tag 48-SecurityID) appears in both the incremental and market recovery updates during the comparison, then compare the market recovery tag 60-TransactTime to the incremental tag 60-TransactTime.
    • In this example, the tag 60-TransactTime comparison is not required due to different instruments between the feeds.
    • For market recovery snapshot processing, drop all cached incremental updates with a packet sequence number < 369-LastMsgSeqNumProcessed.
    • Begin normal processing for recovered instrument 13205.
    • Recover subsequent instruments via the market recovery feed (8087, etc).
    • Additionally, as 369-LastMsgSeqNumProcessed increases in subsequent market recovery updates, continue to drop cached market data where the packet sequence number < to 369-LastMsgSeqNumProcessed.  

  5. Disconnect from the market recovery feed once processing is complete for all instruments.

Incremental Feed

Market Recovery Feed

Incremental Packet Sequence Number

Tag 35-

MsgType

Tag 48-

SecurityID(s)

Tag 60-TransactTime(s)

Snapshot Packet

Sequence Number

Tag 369-

LastMsgSeq

NumProcessed

Tag 60-TransactTime

 

Tag 48-SecurityID

Tag 911-

TotNumReports

100 - drop

X

13205

20191120-18:36:41.000000000


101 - drop

X

5522

20191120-18:36:41.000278478

904

20191120-18:36:41.000293451

102 - drop

X

904

20191120-18:36:41.000320480

103 - drop

X

1741

20191120-18:36:41.000353097

104 - recover instrument 13205 information via market recovery snapshot message and begin normal processing

X

7720

20191120-18:36:41.000368253



4 – Connect to Market Recovery Feed

104

20191120-18:36:41.000000000



13205

10

105 – apply update to 13205 information gathered in the snapshot

X

13205

20191120-18:36:41.000383610

5 – Next instrument to be processed

108

20191120-18:32:22.080423217

8087

10

106

X

5522

20191120-18:36:41.000412613

Next Market Recovery Iteration 

107

X

12121

20191120-18:36:41.000423217

1

112


1741

10

….




….





Example 2 - Same Transaction Time Equal For Instrument

In the following example, the instrument (tag 48-SecurityID=15212) appears in both the incremental and market recovery updates when comparing tag 369-LastMsgSeqNumProcessed. Therefore, client systems must compare the market recovery tag 60-TransactTime to the incremental Tag 60-TransactTime. In this example the tag 60-TransactTime values match and the instrument can be recovered. 

Incremental Feed

Market Recovery Feed

Incremental Packet Sequence Number

Tag 35-

MsgType

Tag 48-

SecurityID(s)

Tag 60-TransactTime(s)

Snapshot Packet

Sequence Number

Tag 369-

LastMsgSeq

NumProcessed

Tag 60-TransactTime

 

Tag 48-SecurityID

Tag 911-

TotNumReports

100 - drop

X

13205

20191120-18:36:41.000000000


101 - drop

X

5522

20191120-18:36:41.000278478

904

20191120-18:36:41.000293451

102 - drop

X

904

20191120-18:36:41.000320480

103 - drop

X

15212

20191120-18:36:41.000368253

104 - recover instrument 15212 information via market recovery snapshot message and begin normal processing


X

15212

20191120-18:36:41.000368253


TransactTime matches with snapshot message

4 – Connect to Market Recovery Feed

104

20191120-18:36:41.000368253

 

TransactTime matches with incremental message

15212

10

Example 3 - Different Transaction Time For Instrument

In the following example, the instrument (tag 48-SecurityID=13205) appears in both the incremental and market recovery updates when comparing tag 369-LastMsgSeqNumProcessed, however the tag 60-TransactTime values do not match. Therefore, for the instrument client systems must process the next market recovery feed iteration.

Incremental Feed

Market Recovery Feed

Incremental Packet Sequence Number

Tag 35-

MsgType

Tag 48-

SecurityID(s)

Tag 60-TransactTime(s)

Snapshot Packet

Sequence Number

Tag 369-

LastMsgSeq

NumProcessed

Tag 60-TransactTime

 

Tag 48-SecurityID

Tag 911-

TotNumReports

100 - drop

X

13205

20191120-18:36:41.000000000


101 - drop

X

5522

20191120-18:36:41.000278478

904

20191120-18:36:41.000293451

102 - drop

X

904

20191120-18:36:41.000320480

103 - drop

X

13205

20191120-18:36:41.000353097

104 

X

13205

20191120-18:36:41.000353097


4 – Connect to Market Recovery Feed

104

20191120-18:36:40.103368841

 

TransactTime does not match with incremental message, must process next loop to recover 13205

13205

10

105

X

13205

20191120-18:36:41.000353097

4 – Next instrument to be processed

107

20191120-18:36:22.000423217

8087

10

106

X

5522

20191120-18:36:41.000412613

Next Market Recovery Iteration 

107

X

12121

20191120-18:36:41.000423217

1

110


1741

10

….




….





170

X

13205

20191120-18:36:43.121427232


TransactTime matches with snapshot message, recover instrument 13205

4

170

20191120-18:36:43.121427232


TransactTime matches with incremental message, recover instrument 13205

13205

10

Example 4 - New Market Recovery Snapshot Added

In the following scenario there’s activity for an instrument (tag 48-SecurityID) which causes a new market recovery snapshot message to be added to the market recovery feed. Therefore, to ensure all new instruments are recovered client systems must process an additional market recovery iteration starting from snapshot packet sequence number equal to one. Client systems can determine if a new snapshot as been added by comparing tag 911-TotNumReports with the previous iteration. The order of each snapshot iteration is not guaranteed, therefore the newly created snapshot may be placed mid-loop in the next iteration.  

Incremental Feed

Market Recovery Feed

Incremental Packet Sequence Number

Tag 35-

MsgType

Tag 48-

SecurityID(s)

Tag 60-TransactTime(s)

Snapshot Packet

Sequence Number

Tag 369-

LastMsgSeq

NumProcessed

Tag 60-TransactTime

 

Tag 48-SecurityID

Tag 911-

TotNumReports

105 - drop

X

12171 – New instrument activity generates a market recovery snapshot message

20191120-18:36:41.000383610

4 - Recovery in process

107

20191120-18:36:41.000013217

8087

10

106 - drop

X

5522

20191120-18:36:41.000412613

Next Market Recovery Iteration 

107 - drop

X

13205

20191120-18:36:41.000423217

1

110

20191120-18:36:41.000353097

1741

11 – incremented to reflect new snapshot message for instrument 12171

….




….





170 – begin normal processing for instrument 12171 from market recovery snapshot due to the incremental update for packet sequence number < to 369-LastMsgSeqNumProcessed


X

904

20191120-18:36:41.001223443

3 – new snapshot placed mid-iteration

170 – recovery instrument via snapshot

20191120-18:36:41.000383610

12171

11

171 – apply update to 12171 information gathered in the snapshot

X

12171


….





….




  • No labels