Create and Configure an Import API Source with the Connect API

Note: This guide focuses on creating an Import API source using Stitch Connect. Refer to the Managing Import API access tokens in Stitch guide for instructions on generating API access tokens in the Stitch web app.

Using the Stitch Connect API, create a new Import API source and generate an access token. You’ll also learn how to push data to the Import API after the source has been configured.

Prerequisites

Access to Stitch Connect and valid Connect API credentials. Connect access is a Stitch Advanced or Premium feature. Refer to the Connect API reference for more info on obtaining API credentials.

Step 1: Get the Import API's report card

When preparing for source creation, the first step is to get the report card for the source you want to create. The report card contains information about the steps required to fully configure a source.

Use GET /v4/source-types/{source_type} to get the report card for source type: import_api:

GET /v4/source-types/{source_type}

curl "https://api.stitchdata.com/v4/source-types/import_api" \
     -H 'Authorization: Bearer [ACCESS_TOKEN]' \
     -H 'Content-Type: application/json'

The response will be a Source object with a Connection step object:

Response for GET /v4/source-types/{source_type}

{
  "type": "import_api",
  "current_step": 1,
  "current_step_type": "form",
  "steps": [
    {
      "type": "form",
      "properties": []
    },
    {
      "type": "fully_configured",
      "properties": []
    }
  ],
  "details": {
    "pricing_tier": "standard",
    "pipeline_state": "released",
    "default_start_date": "-1 year",
    "default_scheduling_interval": 60,
    "protocol": "import_api",
    "access": true
  }
}

For Import API sources, the only step to being fully_configured is to complete the form step.

Step 2: Create the source and complete the form step

Use POST /v4/sources to create the Import API source. The request body must include the following top-level properties:

type: This must be import_api.
display_name: A descriptive name for the source. This will be used to dynamically generate the name corresponding to the schema name or dataset name that the data from this source will be loaded into.

For example: A display name of Import API will create a destination schema named import_api.

This request will complete the form step outlined in the source’s report card, which you retrieved in Step 1:

POST /v4/sources

curl -X "POST" "https://api.stitchdata.com/v4/sources" \
     -H 'Authorization: Bearer [ACCESS_TOKEN]' \
     -H 'Content-Type: application/json' \
     -d \
'{
  "type": "import_api",
  "display_name": "Import API"
}

The response will be a Source object containing the ID, report card, and current configuration status of the Import API source, which will be fully_configured:

Response for POST /v4/sources

{
  "properties": {},
  "updated_at": "2020-01-02T15:49:27Z",
  "schedule": null,
  "name": "import_api",
  "type": "import_api",
  "deleted_at": null,
  "system_paused_at": null,
  "stitch_client_id": 116078,
  "paused_at": null,
  "id": 126890,
  "display_name": "Import API",
  "created_at": "2020-01-02T15:49:27Z",
  "report_card": {
    "type": "import_api",
    "current_step": 2,
    "current_step_type": "fully_configured",
    "steps": [
      {
        "type": "form",
        "properties": []
      },
      {
        "type": "fully_configured",
        "properties": []
      }
    ]
  }
}

Note the id value - you’ll need it to complete the next step.

Step 3: Generate an Import API access token

Requests made to the Import API must include an access token associated with the Import API source. In this step, you’ll generate an access token for the Import API.

Using the Import API source’s ID, make a request to POST /v4/sources/{source_id}/tokens:

POST /v4/sources/{source_id}/tokens

curl -X "POST" "https://api.stitchdata.com/v4/sources/126890/tokens" \
     -H 'Authorization: Bearer [ACCESS_TOKEN]' \
     -H 'Content-Type: application/json'

The response will be an Import API access token object with an access_token property. The value of this property is the access token you’ll need to include in requests made to the Import API:

Response for POST /v4/sources/{source_id}/tokens

{
  "id": 828792559,
  "access_token": "[IMPORT_API_ACCESS_TOKEN]"
}

Note: The API will only return the Import API access token once, immediately after generation. Store the access token and its ID somewhere secure, as you’ll need the access token ID to revoke the token.

Step 4: Push data to the Import API

Now that the Import API source is fully_configured, you can start pushing data to it.

While you used the Connect API to create the Import API source, to actually push data, you’ll need to use the Import API.

In this section:

Step	Action	Endpoint
1	Retrieve the correct Import API base URL for your region
2	Build the request header
3	Submit the request	POST /v2/import/batch

Step 4.1: Retrieve the correct Import API base URL for your region

Next, you’ll identify the data pipeline region your Stitch account is in. You’ll use this to retrieve the correct Import API base URL for your account’s region.

The base URL is used in requests submitted to the Import API and is similar to https://api.stitchdata.com.

To identify your region and get your base URL:

Use these instructions to locate your account’s data pipeline region.
Refer to the Import API base URL reference to locate the base URL for your region.

Step 4.2: Build the request header

Pushing data to the Import API is accomplished by making a request to POST /v2/import/batch. The request header must include:

The correct base URL for your Stitch data pipeline region
Your the Import API access token
A supported media type of Content-Type: application/json

For example:

POST /v2/import/batch

curl -X "POST" "https://api.stitchdata.com/v2/import/batch" \
     -H 'Authorization: Bearer [ACCESS_TOKEN]' \
     -H 'Content-Type: application/json'

Step 4.3: Submit the request

Note: Instructions for structuring the data in request bodies for the Import API is outside the scope of this guide. Refer to the Structuring data for the Import API guide for instructions and examples.

To push data to Stitch, use the Create a batch endpoint. This endpoint uses a JSON schema to validate and type the data in the records sent to the Import API.

Once the request is processed, data will be loaded into the destination connected to your Stitch account.

In the example below, the request will send a single record for the customers table to the Import API:

curl -X "POST" "https://api.stitchdata.com/v2/import/batch" \
     -H 'Authorization: Bearer [ACCESS_TOKEN]' \
     -H 'Content-Type: application/json' \
     -d \
'{
   "table_name":"customers",
   "schema":{
      "properties":{
         "id":{
            "type":"integer"
         },
         "name":{
            "type":"string"
         },
         "age":{
            "type":"integer"
         },
         "has_magic":{
            "type":"boolean"
         }
      }
   },
   "messages":[
      {
         "action":"upsert",
         "sequence":1565880017,
         "data":{
            "id":1,
            "name":"Finn",
            "age":15,
            "has_magic":false
         }
      }
   ],
   "key_names":[
      "id"
   ]
}'

If successful, the Import API will return a 2xx status and a Batch Status object.

If the status is 201, this means that the request was accepted and will be processed later. New data will be added once the request is processed. The response body will be:

Response for (201 status)

{
  "status": "OK",
  "message": "Batch Accepted!"
}

If the status is 202, this means that the request was accepted, but cannot currently be processed due to an internal error. Data will be automatically re-processed. The response body will be:

Response for (202 status)

{
  "status": "Accepted",
  "message": "The batch is queued to be processed."
}

Note: Due to the structure of Stitch’s replication process, data pushed to the Import API will not immediately be available in the destination. The successful response in this section refers only to Stitch accepting the data, not it being loaded.

Step 5: Verify the data in the destination

After you’ve pushed a batch of data to the Import API, Stitch will queue it for processing.

Stitch’s replication process consists of three distinct steps: Extraction, preparation, and loading. Each step occurs independently and takes a bit of time to complete, which means you won’t immediately see data in your destination after it’s been pushed to the Import API.

When Stitch loads the data into the destination, it will be in the schema or dataset associated with the Import API source you created. In this example, Stitch would create a table named customers with a single record in a schema named import_api:

id	name	age	has_magic
1	Finn	15	false

Note: How data is structured in your destination depends on how attributes are typed in Import API requests and the type of destination Stitch loads data into. Refer to the Structuring data for the Import API guide for more info.

Next steps

Congratulations on configuring your Import API source! Next, we recommend checking out:

Structuring data for the Import API: Learn how to structure and type data in your Import API requests.
Sequencing data for the Import API: Learn how the Import API considers data points for loading, which affects how data is updated in your destination.
Managing and revoking Import API access tokens via the Connect API: Learn how to manage and revoke Import API access tokens using the Connect API.