Frequently Asked Questions: Orbital Insight

What is Orbital Insight?

Orbital Insight is a company that develops geospatial data analytics to help its clients unlock societal and economic trends at a global scale. Orbital Insight sources petabytes of satellite, drone, balloon and other unmanned aerial vehicle (UAV) data. Using computer vision and machine learning technologies, Orbital Insight processes and interprets this data to create intelligence, enabling businesses, governments, and NGOs to make better informed decisions.

What is the Energy Product from Orbital Insight?

The Orbital Insight Energy Product provides a daily indication of crude oil inventory estimates on a global-, regional- and country-wide scale. The data is also available at the padd-level in the U.S. and province-level in China.

The dataset presents a time series of estimates that track billions of barrels of crude oil stored in tens of thousands of floating-roof tanks (FRTs). Orbital Insight’s database of worldwide FRTs is generated by using artificial intelligence to locate and dimension floating-roof tanks.

Then, an estimate of the aggregate volume of global crude oil storage is calculated by estimating the fill volume of crude oil in each tank using shadows identified by trigonometric projection calculations and advanced computer vision algorithms. The time series reflects the estimated volume of crude oil in FRTs (floating-roof tanks) only, and does not include crude oil volume stored using other methods such as fixed-roof tanks, underground caverns, very large crude carriers (VLCCs), railcar, truck or pipeline storage.

What type of data does Orbital Insight Energy provide?

Orbital Insight Energy is a global and daily view into estimates based on floating-roof tank storage of  crude oil in the petroleum industry. Orbital Insight Energy monitors all the top oil storage regions in the world, including OPEC, China, U.S. and EMEA, which results in data collection from over 5 billion barrels of crude oil storage capacity analyzed by tank and region.

What is the file format of the Orbital Insight data?

Data is provided in CSV files (comma separated values). An example of the data is shown below:

Do you have a data dictionary which better describes the columns?

  • The columns provided in the data are: Date - Date of estimate, measurements are typically between 10 a.m. and 2 p.m. local time
  • Volume.estimate - Orbital Insight daily estimate for storage in millions of barrels
  • Smoothed.estimate - 20-day zero lag exponential moving average
    • Orbital Insight suggests using this number for trend analysis as it removes quite a bit of the volatility found in the volume.estimate
  • Volume.estimate.stderr - Error model, this is incomplete and is not applicable
  • Storage.capacity.estimate - Estimate of total storage capacity (millions of barrels) of floating-roof tanks in the respective region
  • Total.available.tanks - Total number of floating-roof tanks in the respective region
  • Sampled.tanks - Total number of individual tanks observed at least once in the last 15 days
  • Truth_value_mb - Government estimate

Do you have a description for all the regions and categories within oil?

The regions found in Orbital Insight Energy data include:

  • China (CHN) - Province levels
  • OPEC - Algeria, Angola, Ecuador, Iran, Iraq, Kuwait, Libya, Nigeria, Qatar, Saudi Arabia, United Arab Emirates, Venezuela
  • U.S. Total - PADDs 1-5 and Cushing
  • OECD Europe - All European-member OECD countries we track: Austria, Belgium, Czech Republic, Denmark, Finland, France, Germany, Greece, Hungary, Ireland, Italy, Poland, Portugal, Slovakia, Spain, Sweden, Switzerland, The Netherlands, Turkey, United Kingdom
  • Other countries - Universal country codes (e.g. QAT = Qatar)

What is the average daily file size?

The files are small in size - just a few hundred KBs at most.

How many files are available per day?

There are two files available per region, per day: one with complete historical information and one with the most recent 60 days.

What is the delivery frequency of the data?

The data is delivered on a daily cadence and estimates are provided on a T-1 basis.

What time is delivery time of the data each day? When does the data update?

Data is provided at 7 a.m. Pacific Time/9 a.m. Central Time/10 a.m. Eastern Time.

Are files compressed?

No. They are not compressed.

Are sample files available?

How far back historically is data available?

The datasets begin in January 2014. For some regions the datasets start a bit later due to lack of satellite imagery.

How do Orbital Insight data science algorithms work? How does Orbital Insight analyze tank capacity? How does Orbital Insight take observation-level data and aggregate it to a country-level reading on a daily cadence?

Observation Window/Decay – Orbital Insight gradually decays older observations to derive its estimates.  The weight for older observations decay with a half-life of five days.

In other words, after five days, an observation gets half as much weight as it did the day it was new. After 10 days, the weight is (0.5 * 0.5), or 1/4 as much. After a few five-day intervals, the observation has very little effect on the final estimate. Because this weight decays quickly, Orbital Insight uses a limited backward-looking window. In effect, after about four intervals (20 days), old observations have very little effect.

Modeling of unobserved tanks – Past observations are never truly excluded from the estimate, but their effect gets down-weighted. Error increases when more old observations are used on a given day to create the volume estimate.

Region-level aggregation is performed by aggregating individual tank measurement estimates. These individual tank measurement estimates are based on a combination of the most recent image of each tank and the fill percentage estimates of other tanks in the region. As more time passes without seeing an individual tank, the weight of the most recent observation of that tank is decreased while the weight of the other observations of tanks in the region are increased, to impute the percentage-filled value estimate for that tank.

Why does oil data sometimes change in the first seven days?

Orbital Insight will receive imagery late and incorporate it into the estimate several days after the fact.

How do clouds come into play and how does Orbital Insight deal with them? Certain regions of the world (parts of China, for example) are perpetually cloudy at certain times of the year, hindering the ability to see an image of these tanks. How does Orbital Insight impute values for these tanks? Or is their capacity excluded from the estimate altogether?

Cloudy images are filtered out of the measurements by a computer vision algorithm designed to detect clouds, so all images used to create measurement estimates are cloudless.

Orbital Insight creates a daily estimate for storage in millions of barrels in each region by aggregating individual tank measurement estimates. These individual tank measurement estimates are based on a combination of the most recent image of each tank and the estimated fill percentages of other tanks in the region. As more time passes without seeing an individual tank, the weight of the most recent observation of that tank is decreased while the weight of the other observations of tanks in the region are increased to impute the percentage-filled value estimate for that tank.

Orbital Insight gradually decays older observations to derive its estimates.  The weight of each tank observation decays with a half-life of five days.

For sub-regions (padds in U.S., provinces in China), does Orbital Insight normalize at the sub-region/province-level and then simply aggregate up to country-wide? Or are the two done in parallel?

This is done in parallel. Orbital Insight treats each sub-region as its own entity. For this reason, if you sum up the individual Chinese region numbers it will not be equal to the overall China number. The imputed tank value estimates are not the same.

What type of smoothing is done to the "smoothed.estimate" for oil?

This is a 20-day moving average of our daily estimate data in column “volume.estimate”

List Criteria and Syntax

You can query a list of titled files using this API call, with optional criteria to limit the number of results returned.

Criteria

Name Description Required Type
dataset Dataset the user is querying. In this case, for monthly subscriptions it should always equal “orbitalinsight60” and for yearly subscriptions or one-time complete historical purchases it should always equal “orbitalinsight”   Yes String
exchangecode The country’s files that the user is wanting to list.   No String
yyyymmdd Report date. Accepted format is yyyymmdd.   Example: 20180312 No String
limit Amount of results per request.   A paging information will return with offset information. No int
offset Pagination mechanism   Use the offset returned by previous query for previous or next page. No int
firstfid Alternative pagination mechanism Use the firstfid returned in the paging section to query previous page No String
lastfid Use the lastfid returned in the paging section to query next page No String

Syntax:

curl --user {{UNO_API_KEY}}:{{UNO_API_PASSWORD}} 'https://{{URL}}/cme/api/v1/list?{{criteria1}}={{val1}}&{{criteria2}}={{val2}}&{{criteriaN}}={{valN}}'

curl --user API_JOHNSMITH:12345

'https://datamine.cmegroup.com/cme/api/v1/list?dataset=orbitalinsight60&exchangecode=USA

 

OR

 

curl --user API_JOHNSMITH:12345

'https://datamine.cmegroup.com/cme/api/v1/list?dataset=orbitalinsight&exchangecode=USA

 

Depending on whether you purchased a monthly (former) or yearly/one-time historical (latter).

Response and Samples

Name Description Type
dataset Dataset of the file String
exchangecode Country of the file String
url Fully qualified download URL String
expiration Expiration date of file access String
productcode Will always be a * String
yyyymmdd TellusLabs report date String
checksum MD5 Checksum for data String
size File size in KB int
fid Unique identifier for file. For monthly subscriptions, these files will have a 60 immediately after the ISO code. String
orderid Order number String

Sample response (monthly subscription):

{

    "files": [

        {

            "dataset": "ORBITALINSIGHT60",

            "yyyymmdd": "20180318",

            "url": "https://datamineuat.cmegroup.com/cme/api/v1/download?fid=20180318-ORBITALINSIGHT60_USA_padd1_60_0",

            "fid": "20180318-ORBITALINSIGHT60_USA_padd1_60_0",

            "orderid": "3950",

            "exchangecode": "USA",

            "productcode": "*",

            "checksum": "9a8764d5a1967b6642bbad395cddb2b4",

            "size": 2477,

            "expiration": "2018-04-18",

            "s3url": "cme-antivirus/uat/clean/ophirh/20180318/ORBITALINSIGHT60_USA_padd1_60_3_20180318.csv"

        },

        {

            "dataset": "ORBITALINSIGHT60",

            "yyyymmdd": "20180318",

            "url": "https://datamineuat.cmegroup.com/cme/api/v1/download?fid=20180318-ORBITALINSIGHT60_USA_padd1_60_0",

            "fid": "20180318-ORBITALINSIGHT60_USA_padd1_60_0",

            "orderid": "3950",

            "exchangecode": "USA",

            "productcode": "*",

            "checksum": "9a8764d5a1967b6642bbad395cddb2b4",

            "size": 2477,

            "expiration": "2018-04-18",

            "s3url": "cme-antivirus/uat/clean/ophirh/20180318/ORBITALINSIGHT60_USA_padd1_60_3_20180318.csv"

        }

    ],

    "paging": {

        "previous": "",

        "next": "https://datamineuat.cmegroup.com/cme/api/v1/list?dataset=orbitalinsight60&exchangecode=usa&limit=2&lastFid=20180318-ORBITALINSIGHT60_USA_padd1_60_0&page=1"

    }

}

Sample response (yearly subscription or one-time complete historical):

{

    "files": [

        {

            "dataset": "ORBITALINSIGHT",

            "yyyymmdd": "20180318",

            "url": "https://datamineuat.cmegroup.com/cme/api/v1/download?fid=20180318-ORBITALINSIGHT_USA_padd1_0_0",

            "fid": "20180318-ORBITALINSIGHT_USA_padd1_0_0",

            "orderid": "3950",

            "exchangecode": "USA",

            "productcode": "*",

            "checksum": "9a8764d5a1967b6642bbad395cddb2b4",

            "size": 2477,

            "expiration": "2018-04-18",

            "s3url": "cme-antivirus/uat/clean/orbitalinsight/20180318/ORBITALINSIGHT_USA_padd1_0_3_20180318.csv"

        },

        {

            "dataset": "ORBITALINSIGHT",

            "yyyymmdd": "20180318",

            "url": "https://datamineuat.cmegroup.com/cme/api/v1/download?fid=20180318-ORBITALINSIGHT_USA_padd1_0_0",

            "fid": "20180318-ORBITALINSIGHT_USA_padd1_0_0",

            "orderid": "3950",

            "exchangecode": "USA",

            "productcode": "*",

            "checksum": "9a8764d5a1967b6642bbad395cddb2b4",

            "size": 2477,

            "expiration": "2018-04-18",

            "s3url": "cme-antivirus/uat/clean/orbitalinsight/20180318/ORBITALINSIGHT_USA_padd1_0_3_20180318.csv"

        }

    ],

    "paging": {

        "previous": "",

        "next": "https://datamineuat.cmegroup.com/cme/api/v1/list?dataset=orbitalinsight&exchangecode=usa&limit=2&lastFid=20180318-ORBITALINSIGHT_USA_padd1_0_0&page=1"

    }

}

Download Criteria and Syntax

You can download a file that you have access to using this api call.

Criteria

Name Description Required Type
fid File Id, available from the list api, case insensitive. For monthly subscriptions, these files will have a 60 immediately after ORBITALINSIGHT.

 

 

The fid format for monthly subscriptions is: yyyymmdd-ORBITALINSIGHT60_[ISO CODE]_[REGION]_60_0

 

The fid format for yearly subscriptions and one-time historical purchases is:
yyyymmdd-ORBITALINSIGHT_[ISO CODE]_[REGION]_0_0

 

Example:

20180308-ORBITALINSIGHT60_USA_padd1_60_0

20180308-ORBITALINSIGHT_USA_padd1_0_0

Yes String

Syntax:

curl -J -O --user {{UNO_API_KEY}}:{{UNO_API_PASSWORD}} 'https://{{URL}}/cme/api/v1/download?fid={{val}}'

curl -J -O --user API_JOHNSMITH:12345

'https://datamine.cmegroup.com/cme/api/v1/download?fid=20180308-ORBITALINSIGHT60_USA_padd1_60_0'

 

You can use “-o” option to name your own file:

curl --user API_JOHNSMITH:12345

'https://datamine.cmegroup.com/cme/api/v1/download?fid=20180308-ORBITALINSIGHT_USA_padd1_0_0' -o my_orbitalinsight_file.gz

How do I know if Orbital Insight has the information I need?

A trial subscription is available upon request. Contact Market Tech Sales to inquire about a trial.