Traditional approaches to integrate with exchange market data feeds normally include managing data over two UDP streams and a TCP replay endpoint. Complex logic is required to switch between the two UDP feeds to ensure no data packets are dropped and any snapshot or message replay happens over the TCP transport (reference:https://www.cmegroup.com/confluence/display/EPICSANDBOX/MDP+3.0+-+Dissemination).

A custom implementation of such low-level protocols requires certification against the exchange to ensure the client software can correctly handle various scenarios. To reduce development time, pre-certified software can be purchased which manages all the low level transport complexities (see https://www.onixs.biz/cme-mdp-3-0-market-data-handler. html). In addition to the transport protocol complexities, access to these endpoints would normally include setting up cross-connection or even co-location in the exchange’s data center which incur a large lead time and cost.

Following the native connectivity approach provides the least latency as the client software interacts with the lowlevel transport but, as described above, comes with a connectivity and implementation cost. Small companies and startups, such as skew, may not have the technical resources to pursue this route. Additionally, many clients do not have single digit millisecond latency-sensitive requirements

CME Group Market Data on Google Cloud provides a cloud native solution to access CME Group market data feeds without having to implement any low-level transport protocols or setting up dedicated connectivity. Using the standard GCP SDK, available in a variety of popular programing languages, software can be developed which can easily access the data from any client location over a single transport layer.

Once the initial integration with Google Cloud PubSub is complete, it becomes easy to subscribe to additional topics containing additional CME Group product data without the need to manage additional transport layer interactions. This puts skew in a position to quickly release new products in response to customer demand.

Large scale market data recovery is also simplified, since it is just another Cloud PubSub subscription to connect to, with the same delivery guarantees.

Before processing CME Group market data, client systems must certify with Autocert+. CME Group provides Cloud PubSub topics to accomplish this. Using these certification topics, and thanks to the integration of CME’s Autocert+ with the cloud solution, skew was able to certify with CME Group in a very short period of time.

To bootstrap the development process, CME Group provides a sample project which consumes data from the topics and ensures in-order processing. The project can be found at https://github.com/CMEGroup/CMESmartStreamonGCP-BinaryData

The service is also easily accessible cross-cloud. skew hosts AWS production and staging environments in Ohio with Pub/Sub latency of 13ms at the 95% level with a median of 2ms.

There is no requirement however to use AWS and, indeed, the skew stack has been run on GCP as a proof of concept. There are plans additionally to link resources in GCP and AWS using Consul Mesh Gateways

skew uses 41 subscriptions to nine unique topics across its development, staging and production environments. These can be configured via the GCP Console, the gcloud SDK or using Terraform. In July 2020, skew’s subscriptions made 500 million requests at a cost of $500.

Google Pub/Sub and, more generally GCP, can be monitored via a Datadog integration. skew uses Datadog to monitor its AWS and GCP infrastructure including its GCP subscriptions across different environments. In addition, monitoring alerts are triggered when configured thresholds are breached.

Google PubSub provides guaranteed delivery but does not guarantee in-order delivery. skew requires in-order processing of CME Group market data in order to show clients correct and up-to-date pricing. The lack of in-order delivery from PubSub added some complexity to the client design, especially given the high volume of messages produced by CME Group. We would have faced a similar challenge using traditional solutions. Traditional UDP does not guarantee deliver or ordering.

This was slightly compounded by CME Group’s certification environment not matching the throughput of its production environment. CME Group’s certification PubSub topics have a throughput of a few hundred messagesper-second, whereas in production, throughput is closer to 15,000 messages-per-second for a single topic. skew’s client systems had to be reworked to handle the increased throughput while also ensuring in-order processing of messages. Again, we would have experienced this with traditional solutions, but the cloud eliminated the potential of having to order and provision additional hardware once we optimized our code.