Data is available via the detailed endpoint in “near real time”, meaning it is made available to users once it is processed and quality controlled in our system, rather than being available all at once at the reportDate time
As of Q2 2020, data is available via the detailed endpoint in “near real time”, meaning it is made available to users once it is processed and quality controlled in our system, rather than being available all at once at the reportDate time. The availability of data can be variable depending on when images are collected over given tanks and the speed of processing and quality assurance.
This does not change the fundamental nature of the API (see example call above for detailed endpoint). This simply allows for some data to be accessible sooner than other data within a reporting window, which should be more convenient for the end user.
Update: the ‘resultTime’ field is now queryable, similar to how a user would input qry_reportTime_start and qry_reportTime_end to access all tank measurements for a given period of time, based on reportDate. By using this field to query on, a user can get individual tank measurements for a given geospatial entity (e.g. country, terminal, location, etc.) within a more tightly defined time window if those measurements exist.
An example is provided:
https://platform.ursaspace.com/api/data/detailed/find?qry_Country=<Value>&qry_ResultTime_start=2020-02-06T14:00:00Z&qry_ResultTime_end=2020-02-08T14:00:00Z
Output includes:
{
“tankEntityId”: null,
“reportTime”: “2020-02-13T14:00:00.000Z”,
“resultTime”: “2020-02-07T03:43:21.000Z”,
“record”: {
“coordinates”: {
“axis_Company”: “Baniyas Refining Company”,
“axis_Country”: “Syria”,
“axis_CrudeConfidence”: “Yes”,
“axis_IsOecd”: “false”,
“axis_LocationName”: “Baniyas”,
“axis_Region”: “Middle East”,
“axis_ReportTime”: “2020-02-13T14:00:00.000Z”,
“axis_StorageType”: “Refinery”,
“axis_SubscriptionType”: “All Regions,MENA”,
“axis_TankEntity”: “BAN001”,
“axis_TerminalName”: “SCOT (Baniyas)”,
“axis_Territory”: “Tartus”
},
“values”: {
“capacityBarrels”: 218594,
“cylinderHeightM”: 17.7,
“fillBarrels”: 66851,
“heightDropM”: 12.287,
“latitude”: 35.2189,
“longitude”: 35.9657,
“radiusAverageM”: 25
}
}
},
Some important things to keep in mind when using this update of the detailed endpoint going forward:
-
Because data is made available as it is processed, queries can still return “records: []” in the middle of a reporting week for a given query.
-
Data will come in periodically, so users may need to make consistent pulls in order to get updates in near real time.
-
Ursa may publish partial results for a given geospatial entity (e.g. country, terminal, location) due to image coverage. Data will not be “carried forward” in the detailed endpoint, so users will need to account for this in processing and comprehension of data.
An example to illustrate this concept is shown below:
On February 9th, 2020, Sally uses the API to query for Zhoushan, China tank measurements from the last 3 days. She could use this API call:
https://platform.ursaspace.com/api/data/detailed/find?qry_Location=Zhoushan&qry_ResultTime_start=2020-02-06T15:00:00Z&qry_ResultTime_end=2020-02-09T15:00:00Z
She is aware that Ursa tracks ~144 million barrels of capacity in Zhoushan, China. She receives measurements from yesterday, but only sees results for a portion of the data – 98 million barrels of capacity. Sally is a bit confused – did these missing tanks go out of commission? Are they empty?
No. It is likely that Ursa received an image that covered a portion of tanks only. After immediately processing and quality checking these results, these data are made available to the user. It is quite possible that the next day, the remaining tanks will receive a measurement that can be queried via the API – in order to complete a new measurement for the full set of tanks in Zhoushan.
It is recommended that users of the updated detailed endpoint utilize data at the tank level. If a user requires the most recent total aggregate update for a given geospatial entity, the recommended courses of action are:
-
Use the aggregate endpoint of the API, which is updated only at reportDate times.
-
Perform queries in such a way that you receive the most recent measurement per tank in a set (likely done in code).
-
Use the API to update another internal database where each measurement is added, then use any internal tools to aggregate these data however needed.
Update 2020: “Near Real Time” Detailed Endpoint Update FAQs
Q: Why has there been a change to the detailed endpoint?
A: Ursa has updated processing capabilities in order to distribute data more quickly to customers. This data, provided at a tank-by-tank level, will be processed, quality-controlled, and delivered in a fluid manner in near real-time.
Q: How do I use these changes?
A: A user does not have to change anything about API calls made to the detailed endpoint. However, users should be expecting to see data made available in small increments, rather than one large publish on Thursdays at 14:00 UTC.
Q: When will I receive my data?
A: If you are receiving our weekly emailed reports, nothing will change for you. The only feature that is being updated is the tank-by-tank (detailed) API endpoint. Data will come in as it is processed, pulled via our internal systems every 5 minutes. This means you could have new data made accessible quite often, but it will be up to you as a user to make a request for this data.
Q: Is it true that I can receive only some tank measurements for a given location?
A: Yes, this is the case via the updated detailed endpoint. If imagery captures half of a location, measurements will be made and provided, even if the full location has not been imaged yet. Missing data may be filled in soon after or at the time of report publishing (Thursday at 14:00 UTC).
Q: How should I be querying the data now?
A: You can still use the ‘reportTime’ field to query for tank measurements, but now you can also use the ‘resultTime’ and ‘publishTime’ fields to access data. This means that you can query and receive measurements for any period you require. For example, you could utilize the API to get the most recent measurements for every tank globally, and then perform analytics on the data for more up-to-date intelligence. For reference, these three date fields can be defined as follows:
-
resultTime = The moment in time the image was taken.
-
publishTime = The moment in time the data was made available via API.
-
reportTime = Thursdays at 14:00 UTC each week, a normalizing date to standardize measurements from a week.