Incoming Webhooks feature snapshot
A high-level look at Stitch's Incoming Webhooks integration, including release status, useful links, and the features supported in Stitch.
|Singer GitHub repository||
|Configurable Replication Methods||
Stitch’s Incoming Webhooks integration provides a simple and flexible method to integrate dozens of webhook APIs with Stitch. If Stitch doesn’t have a native integration for the webhook you want to integrate, then Stitch webhooks is your best bet. Webhooks are handy for anything that requires a notification and a potential follow-up response, including:
- Email events, such as bounces or spam
- Incoming support requests
- Incoming leads from a marketing website
- Important updates to product docs
- And so on!
To use Stitch Incoming Webhooks:
- You must have a Stitch account.
- The Webhook API you’re integrating with meets the following requirements:
- The data sent by the webhook API must come to Stitch in JSON format. This is currently the only format Stitch supports.
- The payload (or delivery) of the data must come via a
- Request bodies must be less than 20MB
You can determine if the webhook API you want to use meets the above criteria by checking out that provider’s webhook API documentation.
Connecting Incoming Webhooks
Connecting Incoming Webhooks to Stitch is a four-step process:
- Add Incoming Webhooks as a Stitch data source
- Define the Primary Key
- Generate a Webhook URL
- Set up webhooks in your provider's app
Add Incoming Webhooks as a Stitch data source
- Sign into your Stitch account.
On the Stitch Dashboard page, click the Add Integration button.
Click the Incoming Webhooks icon.
Enter a name for the integration. This is the name that will display on the Stitch Dashboard for the integration; it’ll also be used to create the schema in your destination.
For example, the name “Stitch Incoming Webhooks” would create a schema called
stitch_incoming_webhooksin the destination. Note: Schema names cannot be changed after you save the integration.
Define the Primary Key
Next, you’ll define the Primary Key for the data that your Incoming Webhooks integration will send to your data warehouse. A Primary Key is a field that’s used to uniquely identify each message in the webhook data.
Primary Keys must:
- Uniquely identify each message in the webhook data
- Have field names with 50 or less characters
Be top-level fields. This means that Primary Key fields can’t be part of a nested object or array.
If you need help determining if a field is part of a top-level record, we recommend looking for examples of the JSON record in your app’s API documentation. We’re always happy to help if you still aren’t sure.
Primary Keys and Replication Methods
Defining a Primary Key is optional, but the Incremental Replication Method Stitch uses to update your webhook integration’s data will depend on whether a Primary Key is provided or not.
- If a Primary Key is provided, replication to webhooks data will take place via an upsert. This means that existing rows can be updated.
- If a Primary Key is NOT provided:
- Replication will be Append-Only.
- Stitch will auto-generate a field called
__sdc_primary_keyto be used as a Primary Key. This is because Stitch requires a Primary Key to replicate and de-dupe data.
Additionally, we don’t recommend changing a Primary Key once the integration has been set up and data has replicated. This could result in fields not being updated properly.
Enter the Primary Key into Stitch
In the Primary Key field, enter the name of the field you want to use as a Primary Key. Note that this field isn’t required, but if left blank Stitch will auto-generate a Primary Key instead.
Generate a Incoming Webhooks Webhook URL
Once Stitch has successfully saved and created the Incoming Webhooks integration, you’ll be redirected to a page that displays your Incoming Webhooks webhook URL and token (which is blurred in the image below):
Click the Copy button to copy it.
Note that this particular URL won’t display in Stitch again once you click Continue. Think of this URL like you would your login or API credentials - keep it secret, keep it safe. You can, however, generate another URL should you need it.
Once you’ve copied your webhook URL, click Continue to wrap things up in Stitch.
Set up webhooks in Your Provider’s App
The last step is to setup webhooks in your your provider’s app account.
For instructions on how to setup webhooks in your provider’s app, refer to their documentation or reach out to their support team, if necessary.
After you’ve successfully connected your Incoming Webhooks integration, Stitch will continuously replicate your webhook data into your destination.
- Webhooks and historical data
- How data is loaded into your destination
- How to query for the latest data
Webhooks and historical data
Because Incoming Webhooks data is sent to Stitch in real-time, this means that only new records are replicated to your destination. Most webhook-based integrations don’t retain historical data due to this as-it-happens approach.
In the event that our webhook service experiences downtime, you may notice some lag between an event occurring and the data appearing in your destination.
This version of Stitch’s Incoming Webhooks integration loads data in an Append-Only fashion. When data is loaded using the Append-Only behavior, records are appended to the end of the table as new rows. Existing rows in the table aren’t updated even if the source has defined Primary Keys. Multiple versions of a row can exist in a table, creating a log of how a record has changed over time. Data stored this way can provide insights and historical details about how those rows have changed over time.
Refer to the Understanding loading behavior guide for more info and examples.
Query for the latest data
If you simply want the latest version of the object - or objects, if you elected to track more than one during the setup - in the integration’s table (
data), you’ll have to adjust your querying strategy to account for the append-only method of replication. This is a little different than querying records that are updated using
updated_at Incremental Replication.
To do this, you can use the
_sdc_sequence column and the table’s Primary Key. The
_sdc_sequence is a Unix epoch (down to the millisecond) attached to the record during replication and can help determine the order of all the versions of a row.
Note: If you didn’t define a Primary Key while setting up the integration, the Primary Key for the table will be
If you wanted to create a snapshot of the latest version of this table, you could run a query like this:
SELECT * FROM [stitch-redshift:stitch-incoming-webhooks.data] o INNER JOIN ( SELECT MAX(_sdc_sequence) AS seq, [primary-key] FROM [stitch-redshift:stitch-incoming-webhooks.data] GROUP BY [primary-key]) oo ON o.[primary-key] = oo.[primary-key] AND o._sdc_sequence = oo.seq
This approach uses a subquery to get a single list of every row’s Primary Key and maximum sequence number. It then joins the original table to both the Primary Key and maximum sequence, which makes all other column values available for querying.
In v1 of the Stitch Incoming Webhooks integration, Stitch will create a single table - called
data - in the webhook integration schema (this will be the name you enter in the Integration Schema field when you set up Incoming Webhooks) of your data warehouse.
The schema of this table will contain two “types” of columns:
- Columns used by Stitch (prepended with
- Columns sent by the provider’s webhook API
Aside from the Stitch columns, the schema of this table will depend entirely on that provider’s webhook API. With the exception of the
_sdc fields (and a Primary Key if one isn’t provided), Stitch does not augment Incoming Webhooks data nor does it have any control over the fields sent by the webhook provider.
For example: if the webhook API you integrate with only sends
value fields, these are the only fields that will be created in the
NULLs in Webhook Data
When working with webhook data, you’ll likely notice that your data contains quite a few
NULLs. There are two reasons for this:
- The general nature of webhook data, and
- The design of the Incoming Webhooks integration
The Stitch Incoming Webhook integration is designed to create only one top-level table. This single table will contain all the data that Stitch receives from your webhook, which can include several types of records. For example: if you connect an email service like SendGrid using Incoming Webhooks, the table Stitch creates may contain records for
bounce events, and so on.
While each record type may contain some similar attributes (
created_at, etc.), it’s likely that each record type will have its own set of attributes, which will only be populated for that record type.
As such, some columns in some records will contain
NULLs. Here’s an example of a message send event, a whitelist add event, and a click event:
| id | timestamp | event | action | type | ip | url | |----+------------+-------+--------+-----------+-------------+---------------| | 1 | 1481730892 | send | NULL | NULL | NULL | NULL | | 2 | NULL | NULL | add | whitelist | NULL | NULL | | 3 | 1481732465 | click | NULL | NULL | 220.127.116.11 | http://hi.com | // record 1 is a message send event // record 2 is a whitelist add event // record 3 is a click event
Webhook URLs & Security
Stitch allows you to generate up to 2 Incoming Webhooks webhook URLs at a time. These URLs contain security access tokens and as such, have access to your Incoming Webhooks account.
If you ever need to change your webhook URL, you can do so in the Integration Settings page after the integration has been created:
- Click into the integration from the Stitch Dashboard page.
- Click the Settings button.
- In the Webhook URLs section, click the Generate Webhook URL button.
- A new webhook URL will display. Press the clipboard icon to copy it.
- Follow the steps in the Setting Up Webhooks in Incoming Webhooks section to update the webhook URL in Incoming Webhooks.
- After you’ve updated the webhook URL in Incoming Webhooks, click the Revoke button next to the oldest webhook URL in Stitch. This will invalidate the token and revoke access.