When configuring a Data Connector, the Import Configuration screen enables you to specify the source that you are importing data from and how the data will be formatted and loaded.


There are three load types that you can use to specify how the data will be ingested:


There are two configurations that you need to set at the time of choosing a load type:

  • Primary key: This column or set of columns serves the purpose of uniquely identifying and accessing the individual records.
  • Last run key: This column can either be one of the two types, a timestamp value that detects new and modified rows or a strictly incrementing value that provides a unique ID for updates.


The last run key configuration while choosing a load type is only available for Database type connectors.

Truncate and Insert

The truncate and insert feed is mostly used for small-size data tables.

When this load type is selected, the entire data table that currently exists is replaced with the new data table that is retrieved from the feed. 


Last run key

This is not available for truncate and insert. As we completely replace the old data with the new data.


Primary key

The primary key is optional for truncate and insert because the whole data table is replaced, not specific records.

Incremental

The incremental load type is frequently used for data sets that are of an event type. 

With incremental feeds, records are inserted but not updated. This means that a request to append the data is generated.


Last run key

This is mandatory for incremental feeds because, at the time of fetching the data, we will only fetch the records that have been added after the last run in terms of the column value mentioned in the last run key.


For example, if the last run key is DOB, at the time of fetching we will only fetch the new records which have the DOB greater than the DOB present in the last feed run.


Primary key

The primary key is optional for this type of feed because only the new records are inserted into the existing data table.

Upsert

The upsert load type is most commonly used for transactional data. When selected, the new records are both updated and inserted into the existing data table.


Last run key

This is optional for upsert. As we may need to fetch the complete data in some cases if we want to update some records in the ingested table.


Primary key

The primary key is mandatory as it is required to identify which old records need to be updated after retrieving the new records.