For each integration that you add to Stitch, a schema specific to that integration will be created in your data warehouse. The integration’s schema is where all the data Stitch replicates from the data source will be stored.
In this guide, we’ll cover:
In Stitch, we use the term
schema to refer to a location in a destination where integration data is loaded. Depending on the destination you’re using, this might mean different things.
- In destinations that are traditional databases, data is loaded to integration-specific schemas.
- In Google BigQuery destinations, data is loaded to integration-specific datasets.
- In Amazon S3 destinations, data is loaded into integration-specific folders. The integration names and S3 Object Keys determine the exact location in the S3 bucket.
Integration schema names
Create the schema name
When you create an integration, you’re asked to provide a name for the integration. This name is used to create the integration’s schema in your destination.
For example: An Integration Name of
Recurly Stitch would create a schema named
recurly_stitch in the destination.
If you want the schema name to be different than the display name when initially creating an integration:
Click the Use a different schema name? link below the Integration Name field:
Then, enter a schema name in the Integration Schema Name field that displays:
- Complete the integration setup.
- Save and create the integration.
Change a schema name
After an integration is initially saved and created, its schema name can’t be changed.
Changing an integration’s schema name requires you to create a new integration and re-replicate all historical data.
Re-use a schema name
Schema names from deleted integrations can be re-used. However, if a naming collision occurs (two schema names canonicalize to the same name) the destination may reject the data. This is because deleting an integration in Stitch won’t delete that integration’s schema or data from your destination.
Note: Free historical loads are allowed only once for each integration namespace, or schema name. Refer to the Billing FAQ for more info and examples.
Integration schema composition
Integration schemas created by Stitch will contain two types of tables:
The tables that Stitch creates in an integration schema depends on whether the integration supports data selection.
If the integration supports data selection, the integration schema will contain only the tables (and columns, if column selection is supported) that you set to replicate.
Otherwise, all available tables and columns in the integration will be replicated to your destination.
Refer to the Integration table schemas section below more info on how individual integration tables are structured.
Stitch system tables
In addition to the integration tables, Stitch will create additional tables in the integration schema. These tables are prepended with
Every integration schema will contain an
_sdc_rejected table, which serves as the integration’s log for data loading issues. Refer to the guide for more info.
If using a Microsoft Azure SQL Data Warehouse destination, every integration schema will also contain a table named
_sdc_primary_keys. This table contains the Primary Keys for the tables in the integration schema. Refer to the guide for more info.
Integration table schemas
The integration tables in the schema contain the replicated data from tables set to replicate. Note: If you de-select a table from replication, doing so won’t remove that table’s data from your destination.
How your data is stored in the schemas, tables, and columns created by Stitch depends on a few things:
- How data is structured in that particular data source (For example: Use of nested data structures),
- Any changes you might make to the data source (For example: adding/removing a column),
- Stitch-specific data handling rules (For example: entirely
- How your destination handles data (For example: Columns with mixed data types, nested data structures)
Stitch will encounter dozens of scenarios when replicating and loading your data. Familiarizing yourself with these scenarios and the nuances of your destination will enable you to better understand your data’s structure and efficiently troubleshoot if issues arise.
To learn more about how handles these scenarios, check out the Data Loading guide for your destination.
Stitch system (_sdc) columns
In addition to the columns set to replicate in these tables, there are also a few columns prepended with
_sdc. Stitch uses these columns to replicate your data. Don’t remove these columns, as doing so will cause replication issues in Stitch.
For descriptions of the system columns used by Stitch, refer to the System tables and columns guide.