Customer push
Send your sites' data to the DUET ingest API on your own schedule.
In this option, your team builds and runs the pipeline, and sends data to the DUET ingest API on your own schedule. Choose it if your organization prefers to own the integration layer.
Before you begin
- Create a Platform API token. The token's user needs permission to edit each site you send data for.
- Get the project ID for each site from List projects.
- Agree the tag names you send with your account manager, so DUET can map each tag to the right device and measure.
Choose an ingest method
| Method | Best for | Format |
|---|---|---|
| File upload | Periodic batches, such as a daily export. | CSV or Parquet files, of any size. |
| Arrow stream | Frequent, smaller deliveries with lower latency. | An Apache Arrow IPC stream, up to 50 MB per request. |
File upload
A file upload takes three steps, so large files upload in parts directly to storage:
- Call Start a file upload with the project ID, file name, content type, and file size. The response gives an upload ID, the part size, and a URL for each part.
- Upload each part of the file to its URL with an HTTP
PUT, and keep theETagheader from each response. - Call Complete a file upload with the upload ID and the list of part numbers and ETags.
DUET then imports the file in the background.
Arrow stream
Send an Apache Arrow IPC stream to Ingest an Arrow stream, with the Content-Type header set to application/vnd.apache.arrow.stream. Each row is one reading:
| Column | Type | Contents |
|---|---|---|
dt_timestamp | Timestamp, microseconds | When the reading was taken, in UTC. |
tag | String | The tag name. |
float_value | Float64 | The reading. |
Each request can be up to 50 MB. To send more, split the data across several requests.
Rate limits and errors
Ingest requests count toward the Platform API's rate limit of 100 requests a minute. For error codes and retries, see Requests and responses.