Post-load hooks allow you to configure a webhook that fires each time data is loaded into your existing destination.

Using post-load hooks, you can extend Stitch and automate dependent processes. For example: Trigger downstream processing in SQL, an Amazon Web Services Lambda function, Talend Cloud jobs, or any other system that can be controlled with an HTTP request.


Set up a post-load hook

You can configure a post-load hook in two ways:

  1. In the Stitch app
  2. Via the Connect API, if your Stitch plan includes API access

You can configure up to 10 post-load hooks.


Webhook triggers

Post-load hooks are sent on a per-integration, per-table basis to each configured post-load webhook URL. The post-load webhook must have a status of Enabled in order for Stitch to send requests.

If multiple tables are set to replicate for an integration, Stitch will send a request for each table every time data is successfully loaded or rejected.

If the load for a table fails, a post-load webhook will not be sent. For example: The Stitch database user has insufficient database privileges. In this scenario, Stitch will send an email notification.


Security

Webhook URL requirements

To be compatible with Stitch post-load hooks, the service must provide a properly formatted HTTPS webhook URL. For example: https://your-webhook-provider.com/webhooks

If the URL doesn’t use HTTPS, you’ll receive an Invalid URI error when you attempt to add the webhook in Stitch.

IP whitelisting

Post-load notification webhook requests will originate from one of the IP addresses for your Stitch account’s data pipeline region.

To add an additional layer of security, or if the service you’re using requires it, you can whitelist the Stitch IP addresses for your account’s data pipeline region. This ensures that only requests sent from Stitch will be accepted and processed by the webhook service you’re using.


Retries

If the service you’ve configured to receive post-load hooks encounters problems, Stitch will attempt to re-send the notification up to five times. Stitch will re-try when the following occurs:

  • The request fails to connect
  • The request doesn’t receive a response after 30 seconds (timeout)
  • The request recieves a response code other than 2xx

If the notification can’t be delivered after the maximum number of retries, Stitch will send an email notification immediately after the last retry fails. Replaying post-load hooks isn’t currently supported.

Note: A load is still considered successful even if the post-load notification webhook fails.


Webhook payload

Header

Post-load notification webhook requests are POST requests with a User-Agent of StitchLoadingWebhook. For example:

Example request header
curl -X "POST" "https://your-webhook-service.com/webhook" \
     -H 'Content-Type: application/json' \
     -H 'User-Agent: StitchLoadingWebhook'

Note: At this time, custom request headers are not supported.

Request format

Post-load notification webhook requests have a JSON payload in the body, similar to the following example.

Refer to the Request body properties section for attribute descriptions:

{  
   "version":"1.0",
   "minimum_sdc_batched_at":"2019-07-15T14:45:08.026Z",
   "integration_name":"shopify",
   "new_version":false,
   "source_root_table_name":"orders",
   "destination_root_table_name":"orders",
   "table_version":0,
   "tables":[  
      {  
         "source_table_name":"orders",
         "destination_table_name":"orders",
         "num_rows":100,
         "num_rows_rejected":11,
         "s3_objects":[  
            "stitch-s3-bucket/orders/0_1563294550038.jsonl"
         ],
         "primary_keys":[
            "order_id"
         ],
         "bookmark_metadata":[  
            {  
               "key":"updated_at",
               "max_value":"2018-12-31T00:00:00Z",
               "min_value":"2018-12-31T00:00:00Z"
            }
         ],
         "max_time_extracted":"2019-07-15T14:43:09.030Z"
      }
   ]
}

Request body properties

A post-load notification webhook object is composed of the following:

Top-level properties

A post-load notification webhook object contains the following top-level properties:

Attribute name Type Description
version string

The version of the post-load webhook.

minimum_sdc_batched_at timestamp

Timestamp indicating when Stitch loaded the batch the record was a part of into the destination.

integration_name string

The name of the integration.

new_version boolean

If true, the load involved a TRUNCATE command for the table.

source_root_table_name string

The root name of the table in the data source.

destination_root_table_name string

The root name of the table in the destination.

table_version integer

The version of the table.

tables array

An array of table objects.

Table object properties

The tables object contains info about the table being loaded. This list will contain at least one object, but may contain more if there are subtables.

Note: If using a destination that doesn’t natively support nested structures, this list will include subtables created as a result of denesting JSON arrays. Refer to the Nested JSON data structures guide for more info.

A table object contains the following properties:

Attribute name Type Description
bookmark_metadata array

An array of bookmark_metadata objects.

Note: If all rows for the table have been rejected, this property will be null.

destination_table_name string

The name of the table in the destination.

Note: If this is a subtable created from denesting an array, this value will be the source_root_table_name appended with the source_table_name.

For example: If orders.order_items, this value would be orders__order_items.

max_time_extracted timestamp

A timestamp indicating the time that the last row was extracted for the table.

num_rows integer

The total number of rows that were successfully loaded into the destination for the table.

Note: If all rows for the table have been rejected, this value will be 0.

num_rows_rejected integer

If rows were rejected, this value will indicate the total number of rows that were rejected for the table. Refer to the Rejected records log guide for more info.

primary_keys array of strings

Applicable only if using an Amazon S3 destination. The names of the table’s Primary Keys columns.

s3_objects array of strings

Applicable only if using an Amazon S3 destination. The table’s corresponding Object Key in the S3 bucket. This is determined by the Object Key setting in the Destination page of Stitch.

For example: stitch-s3-bucket/orders/0_1563294550038.jsonl

source_table_name string

The name of the table in the data source.

Note: If this is a subtable created from denesting an array, this value will be the array key. For example: If orders.order_items, this value would be order_items.

Bookmark metadata object properties

If the table uses Key-based or Log-based Incremental Replication, the bookmark_metadata array will contain info about the table’s Replication Key.

A bookmark_metadata object contains the following properties:

Attribute name Type Description
key string

The name of the column the table uses as a Replication Key.

min_value string

The minimum value of the Replication Key column that was extracted.

max_value string

The maximum value of the Replication Key column that was extracted. Stitch will use this value during the next replication job to identify new and updated data.