Automatic S3 Transfer

The Automatic S3 Transfer service is a feature that enables the automatic transfer of CME Group historic files to a client’s S3 bucket daily for subscription orders and one-time transfers for historical orders. This service is scalable, traceable and manageable by admin users.

Configuration

The client must first configure their S3 Bucket Policy by using the “S3 Transfer Settings” page on the ‘My Account’ tab. Users input their bucket name and optional subfolder name to generate their S3 Policy. Once generated, users can paste their policy in their S3 Bucket Policy section on their AWS account to start receiving files. Users can then test their configuration by going back to the “S3 Transfer Settings” page and clicking the “Test Configuration” button.

S3 Transfer Settings Page

Generate a policy from your Datamine account

Paste your policy into your AWS S3 Bucket Policy page and save it

Test Configuration

A test file should now appear in your bucket

Personal Transfer Progress Page

Each client has access to a “S3 Transfer History” page. This page keeps track of all your transfers jobs with a progress bar to see how many files have successfully been transferred.

When an S3 Transfer job is initiated, this page will show a real-time progress bar estimating how many files have already been transferred displayed with how many total files still need to be transferred and how many transfer failures there have been. Clients can click on the link in the “Diagnosis” column to see what kind of errors may have occurred if there are failures in the S3 Transfer job.

S3 Transfer product purchase

S3 Transfer History page

Diagnosis column to debug errors in case they occur

Transfer failures may occur from Amazon machine faults, users deleting their S3 bucket policy mid-transfer, or from admin accounts changing client entitlements to underlying products being transferred.

User Email Feedback

Users will receive a transfer summary email per transfer job. The email contains the following information:

  • Dataset
  • Exchange
  • Symbol
  • Date From/Date To
  • Expected number of files, success count, and failure count
  • Transfer start time and end time