Amazon S3 Setup Requirements

To set up Amazon S3 in Stitch, you need:

  • An Amazon Web Services (AWS) account. Signing up is free - click here or go to https://aws.amazon.com to create an account if you don’t have one already.

  • Permissions to create and manage S3 buckets in AWS. Your AWS user must be able to create a bucket (if one doesn’t already exist), add/modify bucket policies, and upload files to the bucket.

  • An up and running Amazon S3 bucket. Instructions for creating a bucket using the AWS console are in Step 1 of this guide.


Step 1: Create an Amazon S3 Bucket

Adding an Amazon S3 bucket policy in the AWS console

  1. Sign into AWS.
  2. Click Services near the top-left corner of the page.
  3. Under the Storage option, click S3. A page listing all buckets currently in use will display.
  4. Click the + Create Bucket button.
  5. On the first screen, Name and region, complete the following:
    • Bucket name: Enter a DNS-compliant name for the bucket.
    • Region: Select the region you want the bucket to be located in.
  6. When finished, click Next.
  7. As Stitch doesn’t require any particular configuration, how you define the settings in the Set properties and Set permissions screens are up to you. Follow the on-screen prompts to complete these steps.
  8. When you reach the Review screen, verify that all the bucket’s information and settings are correct.
  9. When ready, click Create bucket.

Step 2: Configure Stitch Settings

In this step, you’ll define the file and object key format Stitch will use to load data into Amazon S3.

First, enter the name of the bucket in the Bucket Name field. Enter only the bucket name: No URLs, https, or S3 parts.

For example: this-is-a-test-bucket-stitch-dev

Step 2.1: Select Data Storage Format

The data storage format defines the type of file Stitch will write to your Amazon S3 bucket. Supported options are:

  • JSON: Data will be stored as JSON files (.jsonl)
  • CSV: Data will be stored as CSV files (.csv)

For examples of how data in each format will be stored, click here.

CSV-specific Settings

If CSV is selected, there are a few additional configuration options for the files Stitch will write to your bucket:

  • Delimiter: Select the delimiter you want to use. Stitch will use the comma (,) option by default, but you may also use pipes (|) and tabs (\t).
  • Quote all elements in key-value pairs: If selected, Stitch will place all elements of key-value pairs in quotes. For example: Numerical fields will appear as "123" instead of 123.

Step 2.2: Define Webhook Loading Notifications

Webhooks allow external services to be notified when an event happens. If you choose, you can configure a webhook for Stitch to notify you when data is successfully loaded into your bucket.

Webhook notifications are sent on a per-integration basis. This means that every time Stitch successfully loads data for an integration, a summary webhook will be sent to the URL you define.

To enable this feature, check the Post to a webhook URL each time loading to S3 completes box and paste a webhook URL in the Webhook URL field.

More info about webhook loading notifications, including a list of attributes and sample use cases, can be found here.

Step 2.3: Define S3 Object Key

In Amazon S3, Object Keys are used to uniquely identify objects in a given bucket.

The S3 Key setting determines the convention Stitch uses to create Object Keys when it writes to your bucket. For example: If the default Key is used:

[integration_name]/[table_name]/[table_version]_[timestamp_loaded].[csv|jsonl]

This could create an object with an Object Key of:

salesforce-prod/account/1_1519235654474.[csv|jsonl]

You can opt to use the default Key, which is pre-populated, or define your own using the elements in the next section.

S3 Key Elements

The following elements are available to construct an S3 Key:

Required Elements Optional Elements
All of the following elements must be included in the S3 Key, in any order:
  • [integration_name]
  • [table_name]
  • [table_version]
  • [timestamp_loaded] - Note: This is a Unix timestamp.
The following elements are optional:
  • [year_loaded]
  • [day_loaded]
  • [month_loaded]
  • [hour_loaded]
  • [Arbitrary text]

Additionally, keep in mind that Keys cannot exceed 500 characters or include spaces or special characters (!@#$%^&*).

As you update the values in the S3 Key field, Stitch will validate the entry. If the Key doesn’t include all required elements or contains spaces or special characters, you will be prompted to make corrections.

After you’ve finished defining the Key, click Continue.


Step 3: Grant and Verify Bucket Access

Next, Stitch will display a Grant & Verify Access page. This page contains the info you need to configure bucket access for Stitch, which is accomplished via a bucket policy. A bucket policy is JSON-based access policy language to manage permissions to bucket resources.

Note: The policy Stitch provides is an auto-generated policy unique to the specific bucket you entered in the setup page.

For more info about the top-level permissions the Stitch bucket policy grants, click the link below.

Permission Name Operation Operation Description
s3:PutObject PUT Object

Allows the addition of objects to a bucket.

POST Object

An alternate form of PUT Ojbect, this allows the addition of objects to a bucket using HTML forms.

Initiate Multipart Upload

Allows a multipart upload and return of an upload ID.

Upload Part

Allows for the upload of a part in a multipart upload.

Complete Multipart Upload

Allows for the completion of a multipart upload by assembling previously uploaded parts.

PUT Object - Copy

Allows for the upload of a part by copying data from an existing object as the data source.

s3:GetObject GET Object

Allows for the retrieval of objects from Amazon S3.

HEAD Object

Allows for the retrieval of metadata from an object without returning the object itself.

s3:ListBucket GET Bucket (List Objects)

Allows for the return of some or all (up to 1,000) of the objects in a bucket.

HEAD Bucket

Used to determine if a bucket exists and access is allowed.

Step 3.1: Add the Stitch Bucket Policy

To allow Stitch to access the bucket, you’ll need to add a bucket policy using the AWS console.

Adding an Amazon S3 bucket policy in the AWS console

  1. Sign into AWS in another tab, if you aren’t currently logged in.
  2. Click Services near the top-left corner of the page.
  3. Under the Storage option, click S3.
  4. A page listing all buckets currently in use will display. Click the name of the bucket you want to connect to Stitch.
  5. Click the Permissions tab.
  6. In the Permissions tab, click the Bucket Policy button.
  7. In the Bucket policy editor, paste the bucket policy code from Stitch.
  8. When finished, click Save.

Leave this page open for now - you’ll come back to it in the next step.

Step 3.2: Verify Bucket Access

Next, to ensure that Stitch can access the bucket, you’ll create a blank file that Stitch will use to test the permissions settings.

  1. In Stitch, just below the bucket policy code, is the Verify your bucket section. In this section is a field containing the unique name of the test file you need to create:

    Amazon S3 challenge file field in Stitch

    Note: This file name will only display once. If you navigate out of this screen without saving the file name, you’ll need to start over.

  2. Create a blank file using the name displayed in this field. Do not save the file with an extension (file type) like .csv or .txt. In the image below, notice that there isn’t any kind of file extension after the challenge file name:

    Saving the Amazon S3 challenge file without a file extension

  3. Switch back to the AWS console and click the Overview tab.
  4. Click the Upload button and follow the prompts to upload the file.
  5. After the file has been uploaded to the bucket, switch back to where you have Stitch open.
  6. Click Check and Save to save and test the connection to Amazon S3.

Troubleshooting connection errors

If you receive an error message, we recommend trying these troubleshooting steps for common destination connection problems before reaching out to support.

We’ve found that the majority of the time, these resources can resolve most issues.


Learning about Stitch and Amazon S3

Now that your Amazon S3 data warehouse is up and running, we recommend learning about how Stitch will load your data into it.


Questions? Feedback?

Did this article help? If you have questions or feedback, feel free to submit a pull request with your suggestions, open an issue on GitHub, or reach out to us.

Tags: