CircleCI (v1) | Stitch Documentation

CircleCI is in beta
CircleCI is currently in beta. The info in this guide is subject to change.

CircleCI is supported by Stitch
This integration is powered by Singer's CircleCI tap and certified by Stitch. Check out and contribute to the repo on GitHub.

For support, contact Stitch support.

CircleCI integration summary

Stitch’s CircleCI integration replicates data using the CircleCI API v2. Refer to the Schema section for a list of objects available for replication.

CircleCI feature snapshot

A high-level look at Stitch's CircleCI (v1) integration, including release status, useful links, and the features supported in Stitch.

STITCH
Release status	Beta	Supported by	Stitch
Stitch plan	Standard	API availability	Available
Singer GitHub repository	singer-io/tap-circle-ci
REPLICATION SETTINGS
Anchor Scheduling	Supported	Advanced Scheduling	Supported
Table-level reset	Unsupported	Configurable Replication Methods	Unsupported
DATA SELECTION
Table selection	Supported	Column selection	Supported
Select all	Supported
TRANSPARENCY
Extraction Logs	Supported	Loading Reports	Supported

Connecting CircleCI

Step 1: Create CircleCI API token

Log into your CircleCI application.
Go to your User settings.
Click Personal API Tokens.
Click the Create New Token button.
The the Token name field, type in a name you will remember - like Stitch Integration.
Click the Add API Token button.
Copy your new API token and paste it in a safe location, as you will not be able to view it in the application again.

Step 2: Create your project slugs

A project slug is a triplet of the componenets:

Project type - which can either be github or bitbucket
Organization - the name of your organization on GitHub or Bitbucket
Repository - the name of the repository

The project slug takes the form of <project_type>/<org_name>/<repo_name>. For example, the project slug for this CircleCI Singer tap is github/singer-io/circle-ci.

Determine which repositories you would like to include in your Stitch CircleCI integration and list them, separated by spaces, in the project slug format. Keep this list in a safe location.

Step 3: Add CircleCI as a Stitch data source

Sign into your Stitch account.
On the Stitch Dashboard page, click the Add Integration button.
Click the CircleCI icon.
Enter a name for the integration. This is the name that will display on the Stitch Dashboard for the integration; it’ll also be used to create the schema in your destination.

For example, the name “Stitch CircleCI” would create a schema called stitch_circleci in the destination. Note: Schema names cannot be changed after you save the integration.
In the Project Slugs field, paste the list of project slugs you created in step 2.
In the Token field, paste the API token you created in step 1.

Step 4: Define the historical replication start date

The Sync Historical Data setting defines the starting date for your CircleCI integration. This means that:

For tables using Key-based Incremental Replication, data equal to or newer than this date will be replicated to your destination.
For tables using Full Table Replication, all data - including records that are older, equal to, or newer than this date - will be replicated to your destination.

Change this setting if you want to replicate data beyond CircleCI’s default setting of 1 year. For a detailed look at historical replication jobs, check out the Syncing Historical SaaS Data guide.

Step 5: Create a replication schedule

Replication schedules affect the time Extraction begins, not the time to data loaded. Refer to the Replication Scheduling documentation for more information.

In the Replication Frequency section, you’ll create the integration’s replication schedule. An integration’s replication schedule determines how often Stitch runs a replication job, and the time that job begins.

CircleCI integrations support the following replication scheduling methods:

Replication Frequency
Anchor Scheduling
Advanced Scheduling using Cron (Advanced or Premium plans only)

To keep your row usage low, consider setting the integration to replicate less frequently. See the Understanding and Reducing Your Row Usage guide for tips on reducing your usage.

Step 6: Set objects to replicate

Is an object missing or not replicating? Verify that the object meets the requirements for selection and replication.

The last step is to select the tables and columns you want to replicate. Learn about the available tables for this integration.

Note: If a replication job is currently in progress, new selections won’t be used until the next job starts.

For CircleCI integrations, you can select:

Individual tables and columns
All tables and columns

Click the tabs to view instructions for each selection method.

In the integration’s Tables to Replicate tab, locate a table you want to replicate.
To track a table, click the checkbox next to the table’s name. A blue checkmark means the table is set to replicate.
To track a column, click the checkbox next to the column’s name. A blue checkmark means the column is set to replicate.
Repeat this process for all the tables and columns you want to replicate.
When finished, click the Finalize Your Selections button at the bottom of the screen to save your selections.

Important: Using the Select All feature will overwrite any previous selections. However, selections aren’t final until Finalize Your Selections is clicked. Clicking Cancel will restore your previous selections. Refer to the Select All guide for more info about this feature.

Click into the integration from the Stitch Dashboard page.
Click the Tables to Replicate tab.
In the list of tables, click the box next to the Table Names column.
In the menu that displays, click Track all Tables and Fields:
Click the Finalize Your Selections button at the bottom of the page to save your data selections.

Initial and historical replication jobs

After you finish setting up CircleCI, its Sync Status may show as Pending on either the Stitch Dashboard or in the Integration Details page.

For a new integration, a Pending status indicates that Stitch is in the process of scheduling the initial replication job for the integration. This may take some time to complete.

Initial replication jobs with Anchor Scheduling

If using Anchor Scheduling, an initial replication job may not kick off immediately. This depends on the selected Replication Frequency and Anchor Time. Refer to the Anchor Scheduling documentation for more information.

Free historical data loads

The first seven days of replication, beginning when data is first replicated, are free. Rows replicated from the new integration during this time won’t count towards your quota. Stitch offers this as a way of testing new integrations, measuring usage, and ensuring historical data volumes don’t quickly consume your quota.

Replication will continue after the seven days are over. If you’re no longer interested in this source, be sure to pause or delete the integration to prevent unwanted usage.

CircleCI table reference

Schemas and versioning

Schemas and naming conventions can change from version to version, so we recommend verifying your integration’s version before continuing.

The schema and info displayed below is for version 1 of this integration.

This is the latest version of the CircleCI integration.

Table and column names in your destination

Depending on your destination, table and column names may not appear as they are outlined below.

For example: Object names are lowercased in Redshift (CusTomERs > customers), while case is maintained in PostgreSQL destinations (CusTomERs > CusTomERs). Refer to the Loading Guide for your destination for more info.

jobs

View table schema on GitHub

Replication Method :	Full Table	Primary Key :	id : _workflow_id
API endpoint :	Get a workflow’s jobs

The jobs table contains information about jobs from your CircleCI workflows.

jobs table schema

id
STRING

The job ID.

_workflow_id
STRING

The workflow ID.

Reference:

workflows.id
jobs._workflow_id

_pipeline_id
STRING

The pipeline ID.

Reference:

pipelines.id
jobs._pipeline_id
workflows.pipeline_id

approval_request_id
STRING

approved_by
STRING

canceled_by
STRING

dependencies
ARRAY

Click to expand dependencies

items
STRING

jobs (table), dependencies (attribute)

job_number
INTEGER

name
STRING

project_slug
STRING

started_at
DATE-TIME

status
STRING

stopped_at
DATE-TIME

type
STRING

pipelines

View table schema on GitHub

Replication Method :	Key-based Incremental	Replication Key :	updated_at
Primary Key :	id	API endpoint :	Get all pipelines

The pipelines table contains information about piplines from your CircleCI projects.

pipelines table schema

id
STRING

The pipeline ID.

Reference:

pipelines.id
jobs._pipeline_id
workflows.pipeline_id

updated_at
DATE-TIME

The time the pipeline was updated.

created_at
DATE-TIME

errors
ARRAY

Click to expand errors

type
STRING

message
STRING

pipelines (table), errors (attribute)

number
INTEGER

project_slug
STRING

state
STRING

trigger
OBJECT

Click to expand trigger

received_at
DATE-TIME

type
STRING

actor
OBJECT

Click to expand actor

login
STRING

avatar_url
STRING

pipelines (table), actor (attribute)

pipelines (table), trigger (attribute)

trigger_parameters
OBJECT

vcs
OBJECT

Click to expand vcs

origin_repository_url
STRING

target_repository_url
STRING

revision
STRING

provider_name
STRING

commit
OBJECT

Click to expand commit

body
STRING

subject
STRING

pipelines (table), commit (attribute)

branch
STRING

tag
STRING

pipelines (table), vcs (attribute)

workflows

View table schema on GitHub

Replication Method :	Key-based Incremental	Replication Key :	created_at
Primary Key :	id	API endpoint :	Get a pipeline’s workflows.

The workflows table contains a list of workflows from your CircleCI projects, sorted by pipeline ID.

workflows table schema

id
STRING

The workflow ID.

Reference:

workflows.id
jobs._workflow_id

created_at
DATE-TIME

The time the workflow was created.

canceled_by
STRING

errored_by
STRING

name
STRING

pipeline_id
STRING

The pipeline ID.

Reference:

pipelines.id
jobs._pipeline_id
workflows.pipeline_id

pipeline_number
INTEGER

project_slug
STRING

started_by
STRING

status
STRING

stopped_at
STRING

tag
STRING

Questions? Feedback?

Did this article help? If you have questions or feedback, feel free to submit a pull request with your suggestions, open an issue on GitHub, or reach out to us.

Related	Troubleshooting
Destination & Integration Compatibility Replication Scheduling Syncing Historical SaaS Data Resetting Replication Keys Nested Data Structures & Row Count Impact	Third-Party Downtime Understanding & Reducing Your Usage Re-Authorizing Integrations Replication Issues