Data is everywhere: It’s produced by almost everything you interact with in increasing volumes every day. This is great for anyone who’s attempting to make decisions about running a business, but historically, it’s created a headache for developers. Do you want to build and maintain a Salesforce integration so that your VP of Sales can measure your funnel? How about building integrations with Facebook Ads and Google AdWords so that your advertising team can optimize their campaigns?
Building data plumbing like this is deceptively hard. Accuracy, availability, and consistency in streaming environments are not easy guarantees to make, and writing code like this takes developers away from building user-facing products.
All too often, developers get stuck in the role of data janitors: They’re responsible for moving data from Point A to Point B to support the business. Today, developers at companies around the world are solving this problem over and over again, and few of them are happy about it. This is bad.
Developers shouldn’t write ETL
Data platform executive Jeff Magnusson goes even further:
In case you did not realize it, nobody enjoys writing and maintaining data pipelines or ETL. It’s the industry’s ultimate hot potato. It really shouldn’t come as a surprise then that ETL engineering roles are the archetypal breeding ground of mediocrity.
Engineers should not write ETL. For the love of everything sacred and holy in the profession, this should not be a dedicated or specialized role. There is nothing more soul sucking than writing, maintaining, modifying, and supporting ETL to produce data that you yourself never get to use or consume.
Fortunately, ETL no longer needs to be a part of your job description. Today we’re officially launching Stitch, a fully managed ETL service built for developers.
Stitch connects to all your data sources — from databases like MongoDB and MySQL to SaaS tools like Salesforce and Zendesk — and streams that data to your warehouse. With Stitch, developers can provision data for analysts and other team members in minutes, not weeks.
A fully managed ETL service
The hard part about ETL isn’t moving data from one place to another. You can write a Python script to move data from an API to a database in less than 100 lines. The hard thing is making sure that the system can handle whatever you throw at it. You need ETL that:
is highly available so that you don’t miss any critical events,
scales to billions of events so that you don’t have to rebuild it as your data volume grows,
updates its integrations with dozens of APIs every time specs change,
is built with industrial-grade security, and
notifies you proactively when there are any problems so that you can correct them.
Building that system isn’t a 100-line Python script. In fact, based on research we’ve done, it’s a project that typically takes between 8,000 and 15,000 development hours to get off the ground and then one or two developers to maintain on an ongoing basis.
Stitch, on the other hand, is available for free for fewer than 5 million events a month and can be spun up in less than five minutes.
Currently improving the lives of hundreds of developers
Stitch has been in beta under the name RJMetrics Pipeline* for the past nine months. In that time, we’ve had hundreds of developers decide to use our ETL service instead of building their own.
Andy Mockli, development lead for Wevorce, says:
Stitch has helped Wevorce gather data from all of the various services that we use with point-and-click ease. It’s saved my team countless hours of dev time, helped us create valuable insight into how we can better serve our customers, and it just works.
Matt Kent, head of engineering at Sprig, agrees:
We have a very lean technical team, and wanted our data scientists running predictive models on food costs and optimizing our menu, and our developers focused on the Sprig app. With Stitch, we were able to get our Stripe, Zendesk, and Heroku data streaming to Redshift in minutes, and, even better, it requires zero engineering maintenance.
Khuram Malik, founder at Stratagem.io, piled on:
Getting Stitch set up was really easy. Everything has just worked beautifully and magically. Using Stitch is a no-brainer.
We really appreciate the kind words and dedication to using a beta product. Without our users, we would not have been able to build Stitch to where it is today.
Give Stitch a try today
We built Stitch to be the best way for developers to provision ETL for their internal users so that they could get back to building their products. We’d love it if you’d give Stitch a try: for the first 14 days, sync unlimited data for free. After that, sync 5 million rows a month for free, forever. Sign up now.
. . .
- You read that right! Stitch was recently spun out of RJMetrics when RJMetrics was acquired by Magento.