This article describes how to add a new or edit an existing REST API Connector connection for your Peak organization.


Contents

Process Overview

There are four stages that need to be completed when adding or editing a REST connection:

  • Connection
  • Import Configuration 
  • Destination
  • Trigger

To find out how to create new and edit existing data feeds, see Data Sources overview.


Connection

When setting up a connection, you can either use an existing connection or create a new one.

If you are using an existing connection, you can leave the configuration parameters as they are or edit them.


Using an existing connection

To use an existing connection:

  1. Select your required connection from Select Connection dropdown menu.
    The Select Connection dropdown menu contains any connections that have already been established. These can be used as they are, edited or deleted.
    Note: The connection name and authentication type cannot be edited.
  2. Click NEXT.
    The Import Configuration screen appears.

New connections

If you are making a new connection, the connection name and authentication type are first saved. They will then appear in the Select Connection dropdown ready for you to continue.

To make a new connection:

  1. Click NEW CONNECTION.
    The New Connection window appears.
  2. Enter the required connection parameters.
    See below for details.
  3. Click SAVE to save the parameters and move to the next step.


Connection parameters

  1. To make a connection to your chosen REST API or edit the details of an existing connection, complete these details:


  • Connection Name
    Enter a suitable name for the connection using alphanumeric characters.
  • Authentication Type
    Choose one of the following:
    • No Auth
      Used if the REST API does not require any authorization to access the data.
      If selected, you will not have to enter any further information and the connection will be saved ready to be selected from the Select Connection drop-down menu.
    • Basic Auth
      Used if the REST API requires basic authorization to access the data.
      If selected, you will be prompted to enter a username and password.
      The connection will then be saved ready to be selected from the Select Connection drop-down menu.
    • Custom Auth
      Used if the REST API requires customised authorization information to access the data.
      If selected, you will be prompted to enter a specific authorization URL along with request headers and a JSON body. The connection will then be saved ready to be selected from the Select Connection drop-down menu.


Import Configuration

The Import Configuration screen enables you to define what the connection will retrieve via the API, test the logic of your requests and specify where the data will be stored.

After completing the required fields and testing your fetch script, click NEXT to proceed to the Destination screen.


Install Dependency (Optional)

If you want to install any Python packages on the feed instance, add them here so that they can can be consumed in your script. Multiple packages can be added by pressing the tab or enter keys.


Script Editor (Python)

This area enables you to write Python code containing your fetch logic and then test it.

Example scripts

import requests
from lib import outputStream, metadata

### Write your fetch logic below.
### The feed run details can be saved in the metadata object as follows:
### metadata.save(object)
### This variable can be utilised during API calls in order to fetch data incrementally.
### metadata will be an empty object {} initially.
### The outputStream variable can be used as follows:
### outputStream.write(obj)

r = requests.get("https://api.publicapis.org/entries")

for i in r.json()["entries"]:
  outputStream.write(i)
import requests
from lib import outputStream, metadata

### Write your fetch logic below.
### The feed run details can be saved in the metadata object as follows:
### metadata.save(object)
### This variable can be utilised during API calls in order to fetch data incrementally.
### metadata will be an empty object {} initially.
### The outputStream variable can be used as follows:
### outputStream.write(obj)

r = requests.get("http://52.213.225.80/get-data?from=1&limit=1000")
data = r.json()["data"]
for record in data:
  outputStream.write(record)


Logs

Logs will appear here after testing the script.

To fetch live logs once they have been generated, click the refresh icon.

Preview

A sample preview will appear here after testing the script.

Two tabs are available:

  • JSON
    This shows the data as it has been received in JSON format.
    A maximum of 10 records can be shown in the preview.

  • TABULAR
    This shows the data as it has been received in tabular format.
    A maximum of 10 records can be shown in the preview.


Load Type

This will always be Incremental.

Truncate and Insert will be available in a future release.

Feed Name

Requirements:

  • Must be alphanumeric
  • Must start with a letter
  • Must not end with an underscore
  • Can have up to 50 characters


Destination

The Destination stage enables you to choose a where your data will be stored.

If there are multiple options available, you can choose multiple destinations. The destinations available will depend on the data warehouse you use. For example, Redshift users will see Redshift and S3 as options, while users of Snowflake will see Snowflake only.

For more information, see Choosing a destination for a data connector.


Setting a trigger

For a guide to setting triggers for your data feeds, see How to create a trigger for a data feed.