Peak Platform Ingestion API : Peak Support

TABLE OF CONTENTS

This article will explain what Peak Platform Ingestion API is and how to use it to ingest data into the Peak Platform.

What is Peak Platform Ingestion API?

The API gives you the capability to ingest data into the data warehouse. This API will provide the ability for technically mature users to be self-sufficient with data ingestion and also improves the platform's connectivity with different data platforms/IPaaS solutions in order to fetch data from complicated sources.

The Ingestion API will allow you to control fetch(from source), transformation, sync-frequency, and error management aspects of data ingestion on your own. This will make the solution more technically mature and reduce the need to use back routes/workaround to ingest data into the platform.

Step-by-step guide on how to use the Ingestion API Endpoints:

Prerequisites:

Authentication:
The APIs support multiple authentication mechanisms.
1. Tenant API Key: The user needs to obtain the API key from the Peak Platform to execute these APIs. You can read the article Setting up an API key to know more about this.

2. OAuth: The user will need the client credentials to generate an access token for the APIs. Users can request the client credentials with the Peak support team. Once the user has the client_id and client_secret, the user can generate the access token to use the ingestion APIs.
Access tokens can be generated using the API in the Swagger link available here. The audience parameter is optional.

An example response looks like this:

The token is valid for 24 hours.

Swagger Document link: You can get details about the API specification at this Swagger document (link)

Step 1: Creating a Table Schema

Before you can start ingesting data into the platform, you need to define the table schema. In the first version of this API, we will only support Data warehouses(Snowflake and Redshift) as Ingestion destinations, not data lakes(S3). You need to define the table schema in the default input schema of the respective warehouse before you can start ingesting data into the platform. You can also use the existing table schema which is created using a Feed.

You can use the following Endpoints to create and get a table schema

Create a table schema for Ingestion: In order to create a table schema in Peak platform, you have to use POST /api/v1/tables endpoint. In the payload, you have to define the schema by mentioning the Name of the table, columns name and data type.

Note: All the column types supported by the data warehouse will be supported by this endpoint i.e string, boolean, Json, integer, numeric, date and timestamp. Additionally, you can define date format for Date and timestamp columns and precision and scale for all numeric columns

On successful response, you will get 201 success response. Now, the table schema is ready for ingesting data into it.

The table schema will be created in default ingestion schema for the data warehouse

Limitations:

Currently, you can not update the table schema.
You can not delete the schema, which is created using the API.

2. Getting a table Schema: In order to check the table schema for a specific table you can use

GET /api/v1/tables/{tableName} endpoint. You will need to pass on the table name for which you want to retrieve the table schema as a Parameter.

In the response, you will get the schema of the specified table:

3. Getting the list of tables: You can also get the list of all the table schema present in the Platform using

GET /api/v1/tables. The table list will be in descending order of the date of creation of the table schema. You may pass the limit and NextToken(in case of multiple pages)to control the number of results and offset

Once the request is sent you will receive the list of tables and NextToken(if the total number of tables present is greater than the limit; default is 20)

Step 2: Ingesting data into a Table Schema

To ingest data using the API, you will need to prepare your data in JSON data Array. You can club multiple records into one request, although 1 request can only have a maximum of 500 records and request size should be less than 1 MB. The API will validate the schema of incoming data against the table schema in the system. It will not allow partial ingestion, so even if one record has a data error complete request will be failed.

You can use the POST /api/v1/tables/{tableName} endpoint to ingest data. You will need to provide the table name as a Parameter and send data as a Request body.

Once ingested you will see a successful response and the total number of records ingested