Traditional approaches to integrating with exchange market data feeds normally include managing data over two UDP streams and a TCP replay endpoint. Complex logic is required to switch between the two UDP feeds to ensure no data packets are dropped and any snapshot or message replay happens over the TCP transport (reference: https://www.cmegroup.com/confluence/display/EPICSANDBOX/MDP+3.0+-+Dissemination)

The cloud-based solution used by Grão Direto had a much simpler process: after the required licenses were established, CME Group granted access to Pub/Sub topics to Grão Direto’s Google Cloud Platform account. This scenario made it possible for the technology team to build a subscriber client in a matter of days.

On a regular day, the commodities markets generate a lot of data. Thousands of bids, asks, trades and other actions are occurring every minute, and CME Group makes all this information available on specific Pub/Sub topics. Handling this volume of information was the main concern of Grão Direto engineers when building the subscriber.

How it works is that CME Group’s cloud solution sets up a publisher on the Pub/Sub GCP service. This publisher has the function of, for every relevant action in the specific market, “push” a JSON message to the specific topic. CME Group has divided the topics based on certain characteristics, such as information type (Top of Book, Trades, Level 1 Bids and Asks, Level 2 Bids and Asks, …) and product (Soybean Futures, Corn Futures, etc.). Even so, as market activity increases, some topics can have thousands of messages pushed to it in a couple of seconds.

On the other side, there is Grão Direto’s subscriber. This client must connect to the topic and get the CME Group messages. Every time the client pulls the information, all the previous JSON messages not yet received by it are collected. In the physical price calculation case, the only relevant messages were the latest-traded ones. To make all this work, the client was set to make pull requests on a one-minute basis, find the most recent message for each commodities type, discard the rest and go on with the processing.

The platform chosen for the subscriber was Google Cloud Functions. These functions are micro services that run on the cloud, with total abstraction of bypassing server setup, configurations, availability and other aspects that regular software needs to control. Even better, since the service is on GCP, it completely and easily integrates with the Pub/Sub service.

The physical price calculation tool relies on the information gathered and processed by the subscriber. The only relevant information is the current price for each commodities type and contract due date. For example, if the service is calculating a price for soybeans that will be delivered on March, 2021, all it needs to know is the current value.

To accomplish that, Grão Direto has set up an in-memory map, that stores only the latest values. When the service needs a quotation, it gets the value from the map. When the subscriber finds a new value for that product/month, it updates the in-memory map. The map is always online with the latest quotes.

After getting the specific quotation from the IN-MEMORY-MAP, the physical price calculation uses other relevant data (such as the buyer’s basis, locations and freights, and client information) to calculate the final price. Furthermore, the service needs to make some measurement conversions (bushel, tons, bags) and deal with currency exchange (U.S. dollar, Brazilian reais).

During a physical market grain trade, the value informed by the subscriber may change. When this happens, the physical price calculation tool has the option to refresh/cancel a not-yet-confirmed transaction that was priced with the old information. This communication was built using the socket protocol, which establishes a synchronous flow of information. When the subscriber makes a change on the map, it also broadcasts a message on the socket connection/channel. Depending on the configuration, the physical price calculation tool listening on the same channel can warn or even rollback a transaction.

As mentioned before, end users can benefit from the physical price calculation tool in several ways. Adding real-time CME Group market data to it increases the trustworthiness of the final prices, and enables sellers to trade their grains online with little friction. These sellers, using Grão Direto services, access dozens of prices that are constantly changing. Users benefit from the experience of a real-time online market, where prices vary and deal opportunities can emerge during the trading day.

The experience for buyers is similar. Trading desks can set up specific conditions for the price calculation, which helps streamline negotiations. Beyond that, when a deal is closed, the “price breakdown” is automatically available. Trading desks can proceed with hedge operations with no friction, as all the information is already organized beforehand. Having this customized price also streamlines the process of hedging physical grain using futures, since buyers have greater confidence in the spot price and the quote is right there on their screen.

CME Group Market Data for GCP solution is a major factor in Grão Direto’s successes. Using a familiar cloud provider made the development and adaptations go smoothly for the engineering team. Using Pub/Sub is simple, guarantees that the data is reliable and avoids having to use a much more complex platform to consume the data. The topics architecture is also great: by separating the information on specific topics based on product and information type, CME Group enabled Grão Direto to use only the information they needed, avoiding dealing with a large volume of data that would not be relevant.

Regarding the end-user, Grão Direto had to conduct UX research and tests in order to find the optimal refresh time. Although real-time information brings confidence and price transparency, constant pop-ups or refreshes are not a great experience for the customers.

Another relevant fact was the effect on standardization and compliance on the buying side. Using physical price calculation with real-time quotes enabled all prices to be registered, with all variables determined. That way, the system can identify any outliers and prevent errors when calculating the final prices.

Since Grão Direto was able to quickly and easily meet its market data needs using CME Group, it was able to focus on scaling and growing its user base.

The company has seen significant growth in the number of clients using its online pricing tools. Because the tool creates greater transparency among farmers, it encouraged dealmaking with clients, even those who had not bought or sold in some time.

It’s important to highlight the amount of information that is used during the price estimation on a large scale. The physical price calculation tool factors in many different variables for all types of transactions, including crops and contract due dates, for each farmer or warehouse, updating constantly. This amount of “big data” can be analyzed in a variety of ways and has the potential to provide an enormous quantity of relevant insights. It’s also possible to build and train models, using Artificial Intelligence and Machine Learning, to better understand the behavior of the physical grain market, make predictive analysis and improve origination strategies.

Grão Direto is now expanding its physical price calculation tool to its base of buyers, adding new crops and providing real-time calculated physical grain prices for a larger set of farmers.