This article describes how to ingest data via a command line or terminal.


Contents

Overview

To ingest a file via the command line, a two-step process needs to be followed. 

Step 1: Generate a secure URL with Peak

Prerequisites: Tenant API Key


  1. Request Url: https://api.peak.ai/file/signedurl/client
  2. Method: POST
  3. Headers:                                   
    • Content-Type: application/json
    • Authorization: Tenant API key
  4. Parameters: 
    • { “fileName” :  “{filename}{Timestamp}{version}.csv/.txt/.json/.xml/.gz” }
    •  Example - customers2018010111/12/13.gz


     cURL EXAMPLE

  • Command

    curl -i -H "Content-Type: application/json" -H "authorization: d098192e-06df-471a-846c-890700e53062" -X POST -d '{"fileName": "salesrecords2018062712/15/37.gz"}' https://api.peak.ai/file/signedurl/client

  • Successful Response

    "https://kilimanjaro-test-datalake.s3.eu-west-1.amazonaws.com/mercure/data/salesrecords/salesrecords20180627121537.zip?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=ASIAJZYTEZ4AN5RM5NCA%2F20180629%2Feu-west-1%2Fs3%2Faws4_request&X-Amz-Date=20180629T092849Z&X-Amz-Expires=900&X-Amz-Security-Token=FQoDYXdzEOv%2F%2F%2F%2F%2F%2F%2F%2F%2F%2FwEaDHxoKBm7TsfXc7EecyKBAmtIOniGoUm3bTh8dhL4P%2FpGLcsAQHQXdmurTYtfVJtETs8qMnEwoXUgMpFOzKK%2Bx6%2Br6%2Fd%2Bi6F9g3nGVkLy87KiCTX%2Frejuo%2F%2BIPXVup%2Fg0RK8oWYmSTOpNXtUsmohVgGQM4pO5B%2FMOww2HsRom0wKvOXWq8GhcZX2PQDbBKgvIPP4xeqpGCArsMaX9R9a3SZyUoTzb6MpUHlV8KY62JN59ojRHG5uJF7zPga62aq4bZboHknt3GZse35vwUgeUlw%2BUi8KXRlNa6k4vO3xiw00%2BNFjoC11T5jUuWFdphTZIXbqjQpDmoaBdewm4oymStbTJQTrjURh3PFIEWdjGWoVjKM%2F319kF&X-Amz-Signature=0c7f2687fccacea733f0967f9b3a6b4d4f07285f22a7464800ba8c05c95beebe&X-Amz-SignedHeaders=host%3Bx-amz-server-side-encryption&x-amz-server-side-encryption=aws%3Akms"

             


Once generated, the URL will be valid for 900 seconds (15 minutes). It will have to be generated again post that.


In case of error, the response would be:

  1. Invalid file extension. Allowed file extensions .gz/.csv/.txt/.json/.xml
  2. Invalid filename. Filename should be like <name><yyyymmddhh/mm/ss><N>


Step 2: Upload the file 


  1. To open the directory, enter the command ls
  2. Enter the location of the file as a command. For example, if the file is in downloads, type in cd downloads.
  3. Formulate the curl command.
    1. REQUEST HEADERS-
      1. Content-Type: application/json
    2. METHOD: PUT
    3. API_URL - Response URL from the above request


      cURL EXAMPLE

  • Command

    curl -i -H "Content-Type: application/json" -X PUT --upload-file salesrecords20180627121537.csv "https://kilimanjaro-test-datalake.s3.eu-west-1.amazonaws.com/mercure/data/salesrecords/salesrecords20180627121537.zip?X-Amz-Algorithm=AWS4-HMAC-SHA256&amp;X-Amz-Credential=ASIAJZYTEZ4AN5RM5NCA%2F20180629%2Feu-west-1%2Fs3%2Faws4_request&amp;X-Amz-Date=20180629T092849Z&amp;X-Amz-Expires=900&amp;X-Amz-Security-Token=FQoDYXdzEOv%2F%2F%2F%2F%2F%2F%2F%2F%2F%2FwEaDHxoKBm7TsfXc7EecyKBAmtIOniGoUm3bTh8dhL4P%2FpGLcsAQHQXdmurTYtfVJtETs8qMnEwoXUgMpFOzKK%2Bx6%2Br6%2Fd%2Bi6F9g3nGVkLy87KiCTX%2Frejuo%2F%2BIPXVup%2Fg0RK8oWYmSTOpNXtUsmohVgGQM4pO5B%2FMOww2HsRom0wKvOXWq8GhcZX2PQDbBKgvIPP4xeqpGCArsMaX9R9a3SZyUoTzb6MpUHlV8KY62JN59ojRHG5uJF7zPga62aq4bZboHknt3GZse35vwUgeUlw%2BUi8KXRlNa6k4vO3xiw00%2BNFjoC11T5jUuWFdphTZIXbqjQpDmoaBdewm4oymStbTJQTrjURh3PFIEWdjGWoVjKM%2F319kF&amp;X-Amz-Signature=0c7f2687fccacea733f0967f9b3a6b4d4f07285f22a7464800ba8c05c95beebe&amp;X-Amz-SignedHeaders=host%3Bx-amz-server-side-encryption&amp;x-amz-server-side-encryption=aws%3Akms"


  • Successful Response


If the response returned is HTTP/1.1 100 Continue, wait a few seconds for the correct response to return as shown above. The system is just waiting for the file to upload.


Next Steps:

Once the data has been ingested into the Peak data lake, a feed can be created and scheduled.