Page tree
Skip to end of metadata
Go to start of metadata

This page information about CME MDP 3.0 SBE market data on CME Smart Stream on Google Cloud Platform (GCP). CME Smart Stream on GCP leverages Google Cloud Pub/Sub technology for market data distribution. For complete details regarding Google Cloud Pub/Sub please refer to Google Pub/Sub documentation.

Within CME Smart Stream, CME MDP refers to the CME Group Market Data Platform, which provides the complete array of CME Group market data. 

CME MDP 3.0 SBE is the CME Globex event-based bid, ask, trade, and statistical market data format.

  • Real-Time - Real-time market data via CME Globex
  • Delayed (10 minute delay) - Allows customers to use CME Group data under delayed licensing terms. Traditionally, customers licensing delayed data had to self-manage the throttling of real-time feeds from exchanges per licensing policies. This cloud-based delivery mechanism provides customers with low cost, flexible, and reliable access to data without the requirement to delay the content so they can focus on its use. Receipt of the data and licensing removes the requirement, complexity, and cost of an intermediary service.

Contents

Revision History

DateDescription
TBD
  • "Testing and Certification" - Updated section. Added note about test environment data rates.
  • "Market Data Support Services" - Added section.
  • "Google Cloud Pub/Sub Processing Overview" - Updated section with more links to Google's documentation.
    • "CME MDP on Google Cloud Pub/Sub Message Flow" - Updated step 3.
      • "Market by Order (MBO) Market Recovery Snapshot Topic" - added mention of real-time and delayed data
      • "Market by Price (MBP) Market Recovery Snapshot Topic" - added mention of real-time and delayed data
      • "Instrument Recovery Topic" - Added description of topic.
    • "CME MDP Included Attributes" - Added note about Saturday test data.
  • "Additional Product and Instrument Information" - Removed section.
November 22, 2019Client impact assessment rolled in under the CME Market Data topic.
October 14, 2019Initial publication.

Market Data Data Offerings via Google Cloud

CME Smart Stream on GCP includes MDP 3.0 channels (CME, CBOT, COMEX, NYMEX, DME, MGEX), Streamlined SBE channels, Data Insights, and BrokerTec U.S. Treasuries as follows:

  • Equity indices - S&P and Bloomberg
  • Settlements and Valuations
  • Block trades – DME
  • OTC daily
  • BrokerTec U.S. Treasury Market Data

The following table outlines all market data channels supported on CME Smart Stream on GCP.  

CME Group Channel IDChannel NameMarket Data Platform SourceDCM/Venue/Market
200 CME Credit OTCStreamlined - OTC DailyCME
201 CME Cleared OTCStreamlined - OTC DailyCME
210S&P IndexesStreamlined - Equity IndicesS&P
211Dow Jones IndexesStreamlined - Equity IndicesDJIA
213CME CF Rates and IndexesStreamlined - CME CF Cryptocurrency IndicesCME
215BrokerTec US Treasury DataBrokerTec US Treasury DataBrokerTec/NEX Data
230CME Block TradesStreamlined - Block TradesCME
231CBOT Block TradesStreamlined - Block TradesCBOT
232NYMEX Block TradesStreamlined - Block TradesNYMEX
233 COMEX Block TradesStreamlined - Block TradesCOMEX
234 DME Block TradesStreamlined - Block TradesDME
251CME Settlement and ValuationsData Insights - Settlement and ValuationsCME
252CBOT Settlement and ValuationsData Insights - Settlement and ValuationsCBOT
253NYMEX Settlement and ValuationsData Insights - Settlement and ValuationsNYMEX
254COMEX Settlement and ValuationsData Insights - Settlement and ValuationsCOMEX
261CME Benchmark Administration PremiumData Insights - CME Benchmark Administration PremiumCME
310CME Globex Equity FuturesGlobex MDP 3.0CME
311CME Globex Equity OptionsGlobex MDP 3.0CME
312CME Globex Interest Rate FuturesGlobex MDP 3.0CME
313CME Globex Interest Rate OptionsGlobex MDP 3.0CME
314CME Globex FX FuturesGlobex MDP 3.0CME
315CME Globex FX OptionsGlobex MDP 3.0CME
316CME Globex Commodity FuturesGlobex MDP 3.0CME
317CME Globex Commodity OptionsGlobex MDP 3.0CME
318CME Globex Equity Futures - excludes E-mini S&P 500Globex MDP 3.0CME
319CME Globex Equity Options - excludes E-mini S&P 500Globex MDP 3.0CME
320CME Globex FX Futures IIGlobex MDP 3.0CME
321CME Globex FX Options IIGlobex MDP 3.0CME
340CBOT Globex Commodity FuturesGlobex MDP 3.0CBOT
341CBOT Globex Commodity OptionsGlobex MDP 3.0CBOT
342CBOT Globex Equity Index FuturesGlobex MDP 3.0CBOT
343CBOT Globex Equity Index OptionsGlobex MDP 3.0CBOT
344CBOT Globex Interest Rate FuturesGlobex MDP 3.0CBOT
345CBOT Globex Interest Rate OptionsGlobex MDP 3.0CBOT
346CBOT Globex Commodity Futures IIGlobex MDP 3.0CBOT
348CBOT Globex Interest Rate Futures IIGlobex MDP 3.0CBOT
360COMEX Globex FuturesGlobex MDP 3.0COMEX
361COMEX Globex OptionsGlobex MDP 3.0COMEX
380NYMEX Globex Emissions FuturesGlobex MDP 3.0NYMEX
381NYMEX Globex Emissions OptionsGlobex MDP 3.0NYMEX
382NYMEX Globex Crude & Crude Refined FuturesGlobex MDP 3.0NYMEX
383NYMEX Globex Crude & Crude Refined OptionsGlobex MDP 3.0NYMEX
384NYMEX Globex Metals, Softs, & Alternative Market FuturesGlobex MDP 3.0NYMEX
385NYMEX Globex Metals, Softs, & Alternative Market OptionsGlobex MDP 3.0NYMEX
386NYMEX Globex Nat Gas & other Non-Crude Energy FuturesGlobex MDP 3.0NYMEX
387NYMEX Globex Nat Gas & other Non-Crude Energy OptionsGlobex MDP 3.0NYMEX
440DME Globex FuturesGlobex MDP 3.0DME
441DME Globex OptionsGlobex MDP 3.0DME
460MGEX Globex FuturesGlobex MDP 3.0MGEX
461MGEX Globex OptionsGlobex MDP 3.0MGEX
See CME Smart Stream Topic Names for a complete listing of Certification, New Release, and Production topics.

Testing and Certification

The following section outlines testing and certification for GCP.

Certification

Certification via Autocert+ is required for all client systems that will support CME Smart Stream on GCP. The following market data offerings have separate test suites:

  • Globex MDP 3.0
  • Data Insights - Settlement and Valuations
  • Streamlined - Block Trades
  • Streamlined - OTC Daily
  • Streamlined - CME CF Cryptocurrency Indices
  • Streamlined - Equity Indices
  • Streamlined - OTC Daily
  • BrokerTec U.S. Treasury Data

CME Globex Test Environments

The New Release test environment allows product and new functionality testing prior to release in production. Use the New Release environment to perform:

  • New product testing
  • Development testing on new functionality
  • Certification testing on new functionality

The New Release test environment has significantly lower data rates than production.

Getting Started with CME MDP Concepts

This section lists current CME MDP 3.0, Streamlined, Data Insight, and BrokerTec U.S. Treasury market data processing concepts applicable to CME MDP on Google Cloud Pub/Sub. If you are new to CME MDP technology or wish to better understand how your current system aligns with cloud based technology, CME Group recommends starting here.

Google Cloud Pub/Sub Market data recovery processing differs from the current MDP multicast offering and is outlined in this document.

MDP 3.0

Clients planning to process CME MDP CME Globex market data on Google Cloud Pub/Sub should be familiar with the following CME MDP 3.0 concepts:

Simple Binary Encoding (SBE)

Trade Summary

Book Processing

Statistics

Message Specification

Streamlined SBE

Clients planning to process Streamlined SBE on CME Smart Stream on GCP should be familiar with the following Streamlined SBE functional descriptions:

Equity Indices

OTC Daily

Block Trades

CME CF Cryptocurrency Indices

The message specifications for all streamlined systems are available here

Data Insights: Settlements and Valuations 

Clients planning to process CME Data Insights Settlements and Valuations market data should be familiar with the functional specification available here

BrokerTec U.S. Treasury Data

Clients planning to process BrokerTec U.S. Treasury market data on CME Smart Stream on GCP should be familiar with the functional specification available here.  

Market Data Support Services

Market data services provide the external data required to process CME Group market data.

GCP SBE MD Support

Support Services 

There are two MDP services available for markets on CME Globex: Core Globex SBE Schema and CME Reference Data API. An FTP site is used to store the schema files for all environments.

Service

Description

Core Globex SBE Schema 

MDP is a template-based SBE protocol wherein a given message is interpreted by means of its corresponding template. Each message contains a unique Schema ID that references the template to use to interpret the message. The core CME Globex SBE schema is used across all market data groups.

CME Reference Data API 

Product and instrument referential data for CME Smart Stream on GCP can be obtained via CME Reference Data API.

  • MDP 3.0 tag 48-SecurityID maps to the Reference data API instrument field globexSecurityId.
  • Additionally, client systems can map Settlements and Valuations tag 7178-ProductGUID to CME Reference Data API fields and map Settlements and Valuations tag 37513-InstrumentGUID to CME Reference Data API GUID fields.

FTP Site Information

CME provides an FTP (ftp://ftp.cmegroup.com) and SFTP (sftpng.cmegroup.com) site to disseminate MDP SBE schemas and market data configuration information. This FTP/SFTP site contains the Schema and Configuration files for all events. 

Information applies as follows in the table:

  • Environment - specific environment (i.e., Certification, New Release, Production)
  • Service - Core Globex SBE Schema
  • FTP Site - address of the FTP/SFTP site
  • Directory Location - identifies directory
  • Client System Update Schedule - client systems should download updates according to schedule specified
EnvironmentServiceFTP/SFTP SiteDirectory LocationClient System Update Schedule

New Release

Core Globex SBE Schema 

ftp.cmegroup.com or sftpng.cmegroup.com

/SBEFix/NRCert/Templates

Sunday prior to market open

New Release Autocert+

Core Globex SBE Schema 

/SBEFix/NRAutoCertPlus/Templates

Sunday prior to market open

Production

Core Globex SBE Schema 

/SBEFix/Production/Templates

Sunday prior to market open

Google Cloud Pub/Sub Processing Overview 

This section provides an overview of CME MDP processing for Google Cloud Pub/Sub. For complete details regarding Google Cloud Pub/Sub technology, refer to Google's online Pub/Sub documentation. For Pub/Sub libraries, refer to Google’s Pub/Sub Client Libraries. Common GCP SDK Pub/Sub processing concepts include the following:

Terminology

  • Topic: A named resource to which messages are sent by a publisher (CME Group).
  • Subscription: A named resource representing the stream of messages from a single, specific topic, to be delivered to the subscribing application (client system). 
  • Message: The combination of Google data and CME SBE data sent to a topic and delivered to subscribers.  

CME MDP on Google Cloud Pub/Sub Message Flow

The following diagram outlines the Publisher (CME Group) relationship with the Subscriber (Client System) for Google Cloud Pub/Sub technology.   

Pub-Sub

  1. CME Group will create a topic in the Cloud Pub/Sub service and send messages to the topic. To process CME MDP on Google Cloud Pub/Sub, client systems must subscribe to a topic to receive data. The Google Cloud message will contain CME Group messages.
  2. Message store ensures that published messages are retained on behalf of subscriptions. 
  3. Cloud Pub/Sub forwards messages from a topic to each individual subscription. Each subscription receives messages by pulling them from the service. For CME MDP on Cloud Pub/Sub there are two types of topics for a given channel ID that client systems can subscribe to:
  • Incremental Topic - CME Group incremental market data using encoded packets containing the following FIX market data message types:
    • Security Definition (tag 35-MsgType=d)
    • Market Data Security Status (tag 35-MsgType=f)
    • Market Data Incremental Refresh (tag 35-MsgType=X)
    • Request for Quote (tag 35-MsgType=R)
  • Market by Order (MBO) Market Recovery Snapshot Topic - Market by Order (MBO) Market Recovery method used for real-time and delayed data large-scale data recovery (i.e. major outage or late joiners) to synchronize client systems to the latest state maintained by CME Group. This topic is only available for CME Globex MDP 3.0 channels.   
  • Market by Price (MBP) Market Recovery Snapshot Topic - Market by Price (MBP) Market Recovery method used for real-time and delayed data large-scale data recovery (i.e. major outage or late joiners) to synchronize client systems to the latest state maintained by CME Group. This topic is only available for CME Globex MDP 3.0 channels.   
  • Instrument Recovery Topic - The Instrument Replay feed constantly replays the current week's Security Definition (tag-MsgType=d) messages, and will reflect any additions, modifications, or deletes to available instruments. This topic is only available for real-time and delayed data CME Globex MDP 3.0 channels.

4. The subscriber receives pending messages from its subscription and acknowledges each one to the Cloud Pub/Sub service.  A message contains the CME Group Simple Binary Encoded (SBE) messages.

5. When a message is acknowledged by the subscriber, it is removed from the subscription's message queue.

Message Structure

The following section outlines the CME MDP messaging within a Google Pub/Sub Message structure.  

CME MDP Included Data

The Google Pub/Sub data field provides the full CME Group UDP data packet.  

The following diagram outlines CME data with an Google Pub/Sub message:

PubSub Message

CME MDP Included Attributes 

In addition to the full CME Group UDP data packet, Smart Stream on GCP provides the following quick access data in the Google Pub/Sub attributes field:

SendingTime - Sending time from CME before the message is sent to GCP. 

Client systems may receive Saturday test data upon connection. Test data can be filtered using the SendingTime attribute.  

MsgSeqNum - CME Group Packet sequence number.

Channel - CME Group Channel ID defined in the Market Data Data Offerings section. Each feed has a unique channel ID identification methodology:

Feed Type

Example Channel Identifier

Incremental Feed314
Market by Price (MBP) Market Recovery Feed314SA
Market by Order (MBO) Market Recovery Feed314SAMBO

CME Group MDP UDP Packet Headers

The following table outlines CME Group MDP UDP packet header concepts in the diagram above:

Packet Header

(12 bytes)

Message Header

(10 bytes)

Sequence: 4 bytes

  • Packet sequence number.
  • A unique sequence number given to each packet sent.
  • Each channel will have its own separate set of sequence numbers that will increment sequentially with each packet and reset weekly.

Message Size: 2 bytes

  • Overall message size including headers to support framing

Sending Time: 8 bytes

  • UTC Time of message transmission by the Gateway.
  • UTC Timestamps are sent in number of nanoseconds since Unix epoch synced to a master clock to microsecond accuracy.

Block Length: 2 bytes

  • The total space reserved for the root level of the message

TemplateID: 2 bytes

  • Message SBE Template Identifier

SchemaID: 2 bytes

  • Identifier of the SBE Message Schema that contains the Template


Version: 2 bytes

  • Version of the SBE Message Schema in which the message is defined

Message Ordering

Google Pub/Sub does not guarantee message order. Therefore, the client system must ensure the correct processing of messages.

CME Group offers open source java code to assist customers to accomplish this task (https://github.com/CMEGroup/CMESmartStreamonGCP_BinaryData).

Additionally, CME Group offers a CME Smart Stream on GCP tutorial (https://github.com/CMEGroup/CMESmartStream-on-GCP-Tutorial).

Topic Naming Methodology

CME Smart Stream on Google Cloud Platform (GCP) has an incremental topic per channel. Additionally, Market by Order (MBO) Market Recovery, Market by Price (MBP) and Instrument Recovery topics are available for MDP 3.0 channels. Topic names have the following methodology:  

NameValid ValuesDescription
EnvType

PROD = Production

NR = New Release

CERT = Certification 

Environment
ServiceNameSSCL = Smart Stream Cloud LinkCloud service provider
Provider

GCP = Google Cloud Products

Name of the cloud provider

ContentTypeMD = Market DataType of content
Content Timing

RT = Real Time

DLY = Delayed 

Time data is sent
ContentOwner

CMEG = CME Group Owned Market Data

SPDJI = S&P DowJones

USTBTEC = BrokerTec US Treasury Data

MGEX = Minneapolis Grain Exchange

DME = Dubai Mercantile Exchange

Reference to exchange
Format

FIXBIN = MDP 3.0 Globex Data Channels

STRBIN = Streamlined

SVBIN = Settlements & Valuations

Feed Type
Version

v01000

CME Group version of cloud solution
Topic Type

INCR = Incremental

RPLY = MBP Recovery

MBORPLY = MBO Recovery

INST = Instrument Recovery

Defines the type of topic
Channel Number

Examples:

230 = CME Block Trades

340 = CBOT Globex Commodity Futures

CME Group assigned channel id

Topic Name Examples

The following are topic name examples:

  • Production MDP 3.0 Globex Equity Futures Incremental Real-Time Feed (Channel 310): PROD.SSCL.GCP.MD.RT.CMEG.FIXBIN.v01000.INCR.310
  • Production MDP 3.0 Globex Equity Futures Recovery Real-Time Feed (Channel 310): PROD.SSCL.GCP.MD.RT.CMEG.FIXBIN.v01000.MBORPLY.310
  • New Release Streamlined S&P Index Real-Time Feed (Channel 211): NR.SSCL.GCP.MD.RT.SPDJI.STRBIN.v01000.INCR.210
  • Production MDP 3.0 BMD Futures Delayed Feed (Channel 430): PROD.SSCL.GCP.MD.DLY.BMD.FIXBIN.v01000.INCR.430

Recovery Processing

The following section outlines recovery options for CME Smart Stream on GCP.

Google Cloud Pub/Sub Message Replay

Google Cloud Pub/Sub includes the ability to replay messages. For more information on recovering messages, please refer to the Google quick start guide. (https://cloud.google.com/pubsub/docs/replay-qs).  

Google Cloud Pub/Sub replay is available for all CME MDP channels on Google Cloud Pub/Sub. Client systems that require data recovery from the beginning of the week on CME Globex can determine the weekly start via the incremental topic when the CME Group sequence number is equal to one.

Large-Scale Market Recovery

This section describes Market by Price (MBP) & Market by Order (MBO) recovery processing for MDP 3.0 channels via Google Cloud Pub/Sub. Market Recovery should be used for large-scale data recovery (i.e. major outage or late join) to synchronize client systems to the latest state maintained by CME Group.  Market by Price (MBP) & Market by Order (MBO) recovery have separate topics.

Once a subscriber requests to recover from a market recovery topic all Market Data Snapshot Full Refresh (tag 35-MsgType=W) messages are looped 3 times for an applicable channel.  

The Pub/Sub market recovery topic is different the from the MDP 3.0 implementation where the the market feed continually iterates.

The following is an example of the messaging from the market recovery 3x topic loop:

Market Recovery Topic

CME Snapshot Sequence Number

Tag 369-LastMsgSeqNumProcessed

Tag 60-TransactTime

 

Tag 48-SecurityID

Tag 911-TotNumReports

First Market Recovery Iteration (1)

1  - First message from market recovery topic

107 

20191010-18:36:41.000013217

8087

3

12020191010-18:36:41.0000330417733
313120191010-18:36:41.000054910121713

Next Market Recovery Iteration (2)

140

20191120-18:36:41.000071886

8087

3

2

152

20191120-18:36:41.000093224

773

3

3

157 

20191120-18:36:41.000111773

12171

3

Final Market Recovery Iteration (3)

166

20191120-18:36:41.000129878

8087

3

2

174

20191120-18:36:41.000150215

773

3

3 - Last message from market recovery topic

179

20191120-18:36:41.000172319

12171

3

A Market Data Snapshot Full Refresh (35=W) message will be included if the instrument has had book/trade activity since the beginning of the week. Once the instrument has had at least one order, it will stay in the recovery topic, even if the order is cancelled. Market Data Snapshot Full Refresh (35=W) messages will not be sent for instruments having no book activity. More than one instrument can be updated in a packet. The following information is available in the Market Data Snapshot Full Refresh (35=W) message:

  • Book Quotes – Bids and Offers
  • Implied Book Quotes – Bids and Offers
  • Last Trade
  • Opening Price
  • Session High and Low Trade Prices
  • Session High Bid and Session Low Offer
  • Fixing Price
  • Settlement Price 
  • Cleared Trade Volume
  • Open Interest
  • Electronic Volume
  • Threshold Limits

When the Group Status Message is sent with tag 1174-SecurityTradingEvent=4 (Change of Trading Session/Reset Stats), the following statistics will be cleared until new statistics are received from the Incremental topic:

  • Last Trade
  • Opening Price
  • Session High Trade Price
  • Session Low Trade Price
  • Session High Bid
  • Session Low Offer
  • Electronic Volume

Large Scale Market Recovery Processing

When recovery is required, client systems must re-synchronize all books to the latest state maintained by CME Group, and determine whether any new instrument definitions have been published.

  • the tag 369-LastMsgSeqNumProcessed value on the Snapshot message corresponds to the packet sequence number on the Incremental topic
  • tag 60-TransactTime communicates the start transaction time of the last event for the instrument

    Market Recovery Topic - Snapshot (tag 35-MsgType=W) Message
    Incremental Topic
    tag 369-LastMsgSeqNumProcessed corresponds topacket sequence number
    tag 60-TransactTime tag 60-TransactTime
The order of each Snapshot message iteration is not guaranteed; client systems must process one full iteration of a Market Recovery Snapshot message starting at sequence number 1 to ensure full recovery.

CME Smart Stream on GCP utilizes the same SBE template processing methodology as CME Globex. 

Market Recovery Processing Workflow

This section describes the process to follow for large-scale recovery concurrently processing the Incremental topic and Snapshot messages from the Market Recovery topic. Once a book is recovered, client systems can resume normal processing for that instrument even if other books are still being recovered. Client systems will not recover any missed statistics on the Market Recovery topic.

Recovery processing steps are as follows:

  1. Identify channel(s) in which the client system is out of sync.
  2. Listen to the Incremental topic and queue real-time data for the affected channel(s), and optionally begin MBP natural refresh.
  3. Listen to the Market Recovery topic for the affected channel(s).
  4. For a given instrument, compare the Market Recovery Snapshot message tag 369-LastMsgSeqNumProcessed to the Incremental topic Market Data Incremental Refresh (35=X) message packet sequence number.

    • If the Security ID appears in both the Incremental topic and Market Recovery topic updates during the comparison, then compare the Market Recovery topic tag 60-TransactTime to the Incremental topic tag 60-TransactTime. The instrument with the unequal 60-TransactTime must be recovered via the next market recovery cycle or optional concurrent natural refresh processing.

    • Drop all cached Incremental topic updates with a packet sequence number < 369-LastMsgSeqNumProcessed.

  5. Once all instruments are recovered via market recovery or natural refresh, start normal processing and disconnect from the market recovery topic. 

Market Recovery Processing Examples

The following section provides examples for market recovery topic processing.  

Example 1 – Standard Scenario

The following example shows a standard recovery scenario using the following workflow:

  1. Identify the channel that is out of sync.
  2. Connect to the Incremental topic and begin queuing data.
  3. Connect to the Market Recovery topic. In this example, the first Market Recovery Snapshot message sequence number = 4.
  4. Compare the Market Recovery Snapshot message tag 369-LastMsgSeqNumProcessed to the Incremental topic packet sequence number. In this example, the Incremental topic sequence number compared is 104.
    • If the instrument (tag 48-SecurityID) appears in both the Incremental and Market Recovery updates during the comparison, then compare the Market Recovery topic tag 60-TransactTime to the Incremental topic tag 60-TransactTime.
    • In this example, the tag 60-TransactTime comparison is not required due to different instruments between the topics.
    • For Market Recovery Snapshot message processing, drop all cached Incremental topic updates with a packet sequence number < tag 369-LastMsgSeqNumProcessed from the Market Recovery topic.
    • Begin normal processing for recovered instrument 13205.
    • Recover subsequent instruments via the Market Recovery topic (8087, etc).
    • Additionally, as tag 369-LastMsgSeqNumProcessed increases in subsequent Market Recovery topic updates, continue to drop cached market data where the packet sequence number < tag 369-LastMsgSeqNumProcessed from the Market Recovery topic.
  5. Disconnect from the Market Recovery topic once processing is complete for all instruments.

Incremental Topic

Market Recovery Topic

CME Snapshot Sequence Number

Tag 35-

MsgType

Tag 48-

SecurityID(s)

Tag 60-TransactTime(s)

Snapshot Packet

Sequence Number

Tag 369-

LastMsgSeq

NumProcessed

Tag 60-TransactTime

 

Tag 48-SecurityID

Tag 911-

TotNumReports

100 - drop

X

13205

20191120-18:36:41.000000000


101 - drop

X

5522

20191120-18:36:41.000278478

904

20191120-18:36:41.000293451

102 - drop

X

904

20191120-18:36:41.000320480

103 - drop

X

1741

20191120-18:36:41.000353097

104 - recover instrument 13205 information via market recovery snapshot message and begin normal processing

X

7720

20191120-18:36:41.000368253



1 – Connect to Market Recovery topic

104

20191120-18:36:41.000000000



13205

10

105 – apply update to 13205 information gathered in the snapshot

X

13205

20191120-18:36:41.000383610

2 – Next instrument to be processed

108

20191120-18:32:22.080423217

8087

10

106

X

5522

20191120-18:36:41.000412613

….





Next Market Recovery Iteration 

107

X

12121

20191120-18:36:41.000423217

1

112


1741

10

….




….





Example 2 - Same Transaction Time Equal For Instrument

In the following example, the instrument (tag 48-SecurityID=15212) appears in both the Incremental and Market Recovery topic updates when comparing tag 369-LastMsgSeqNumProcessed. Therefore, client systems must compare the Market Recovery topic tag 60-TransactTime to the Incremental topic tag 60-TransactTime. In this example the tag 60-TransactTime values match and the instrument can be recovered. 

Incremental Topic

Market Recovery Topic

Incremental Packet Sequence Number

Tag 35-

MsgType

Tag 48-

SecurityID(s)

Tag 60-TransactTime(s)

Snapshot Packet

Sequence Number

Tag 369-

LastMsgSeq

NumProcessed

Tag 60-TransactTime

 

Tag 48-SecurityID

Tag 911-

TotNumReports

100 - drop

X

13205

20191120-18:36:41.000000000


101 - drop

X

5522

20191120-18:36:41.000278478

904

20191120-18:36:41.000293451

102 - drop

X

904

20191120-18:36:41.000320480

103 - drop

X

15212

20191120-18:36:41.000368253

104- recover instrument 15212 information via market recovery snapshot message and begin normal processing


X

15212

20191120-18:36:41.000368253


TransactTime matches with snapshot message

1 – Connect to Market Recovery topic

104

20191120-18:36:41.000368253

 

TransactTime matches with incremental message

15212

10

Example 3 - Different Transaction Time For Instrument

In the following example, the instrument (tag 48-SecurityID=13205) appears in both the Incremental and Market Recovery topic updates when comparing tag 369-LastMsgSeqNumProcessed, however the tag 60-TransactTime values do not match. Therefore, client systems must process the next Market Recovery topic iteration for the instrument.

Incremental Topic

Market Recovery Topic

Incremental Packet Sequence Number

Tag 35-

MsgType

Tag 48-

SecurityID(s)

Tag 60-TransactTime(s)

Snapshot Packet

Sequence Number

Tag 369-

LastMsgSeq

NumProcessed

Tag 60-TransactTime

 

Tag 48-SecurityID

Tag 911-

TotNumReports

100 - drop

X

13205

20191120-18:36:41.000000000


101 - drop

X

5522

20191120-18:36:41.000278478

904

20191120-18:36:41.000293451

102 - drop

X

904

20191120-18:36:41.000320480

103 - drop

X

13205

20191120-18:36:41.000353097

104 

X

13205

20191120-18:36:41.000353097


1 – Connect to Market Recovery topic

104

20191120-18:36:40.103368841

 

TransactTime does not match with incremental message, must process next loop to recover 13205

13205

10

105

X

13205

20191120-18:36:41.000353097

2 – Next instrument to be processed

107

20191120-18:36:22.000423217

8087

10

....



106

X

5522

20191120-18:36:41.000412613

Next Market Recovery Iteration 

107

X

12121

20191120-18:36:41.000423217

1

110


1741

10

….




….





170

X

13205

20191120-18:36:43.121427232


TransactTime matches with snapshot message, recover instrument 13205

4

170

20191120-18:36:43.121427232


TransactTime matches with incremental message, recover instrument 13205

13205

10

Example 4 - New Market Recovery Snapshot Added

In the following example there is activity for an instrument (tag 48-SecurityID) that adds a new Market Recovery Snapshot message to the Market Recovery topic. Therefore, to ensure all new instruments are recovered, client systems must process an additional Market Recovery iteration starting from the Snapshot message with a packet sequence number = 1. Client systems can determine if a new Snapshot message has been added by comparing tag 911-TotNumReports with the previous iteration. The order of each Snapshot message iteration is not guaranteed, therefore the newly created Snapshot message may be placed mid-loop in the next iteration.

Incremental Topic

Market Recovery Topic

Incremental Packet Sequence Number

Tag 35-

MsgType

Tag 48-

SecurityID(s)

Tag 60-TransactTime(s)

Snapshot Packet

Sequence Number

Tag 369-

LastMsgSeq

NumProcessed

Tag 60-TransactTime

 

Tag 48-SecurityID

Tag 911-

TotNumReports

105 - drop

X

12171 – New instrument activity generates a market recovery snapshot message

20191120-18:36:41.000383610

1  - Recovery in process

107

20191120-18:36:41.000013217

8087

10





....



106 - drop

X

5522

20191120-18:36:41.000412613

Next Market Recovery Iteration 

107 - drop

X

13205

20191120-18:36:41.000423217

1

140

20191120-18:36:41.000353097

1741

11 – incremented to reflect new snapshot message for instrument 12171

….




….





170 – begin normal processing for instrument 12171 from market recovery snapshot due to the incremental update for packet sequence number < to 369-LastMsgSeqNumProcessed


X

904

20191120-18:36:41.001223443

3 – new snapshot placed mid-iteration

170 – recovery instrument via snapshot

20191120-18:36:41.000383610

12171

11

171 – apply update to 12171 information gathered in the snapshot

X

12171


….





….




Instrument Recovery 

Client systems can use the Instrument Recovery topic on each Globex channel to determine if any new instruments have been defined through the Security Definition (tag 35-MsgType=d) message. The Instrument Recovery topic sends two replays of the current week's Security Definition (tag-MsgType=d) messages, and will reflect any additions, modifications, or deletes to available instruments. Tag 911-TotNumReports indicates the total number of messages to process for a complete set of Security Definitions. CME Group does not guarantee the order of the messages being sent.  The Instrument Recovery topic can have multiple messages per packet.

The Pub/Sub instrument recovery topic is different the from the MDP 3.0 implementation where the market feed continually iterates.


The following is an example of the messaging from the instrument recovery 2x topic loop:

Instrument Recovery Topic

CME Instrument Recovery Sequence Number

Tag 48-SecurityID

Tag 911-TotNumReports

First Instrument Recovery Iteration (1)

1  - First message from instrument recovery topic

8087

3

7733
3121713

Final Instrument Recovery Iteration (2)

8087

3

2

773

3

3

12171

3

Contact Information

For technical development support, contact Certification Support for Electronic Trading (CSET).

For production requests, please contact the Global Command Center (GCC).

For all other inquiries, please contact Global Account Management (GAM).

  • No labels