How ezCater implemented enterprise ETL with Singer and Stitch
ezCater is a nationwide online marketplace for business catering, offering connections to more than 60,000 local caterers and restaurants in more than 20,000 US cities and towns. The company relies on a number of ad networks and online services in its marketing and sales efforts – Facebook Ads, Bing Ads, Google AdWords, SendGrid, and Zapier among them – and needs to incorporate information from these and other tools, including in-house databases, to get a full picture of how well its marketing efforts are working.
"You have questions that you can’t answer without all the raw data," explains Jeff Dwyer, ezCater's software engineering manager. Dwyer says the company tried a number of different approaches that didn't work well before they turned to a solution they're happy with – using Stitch as the pipeline to their data warehouse.
When the company first set out to establish a data warehouse to integrate the data from its multiple sources, it tried one data pipeline provider, but the pricing was "bananas, so we stopped using them," Dwyer says. The company also evaluated analytics service provider Amplitude, but that proved too expensive for getting data to Redshift. Kinesis Firehose, another alternative, was cheap, but took significant effort to implement and wasn’t resilient to schema changes.
Now, ezCater uses a combination of native Stitch integrations and Stitch's Import API to send data to its warehouse. "Stitch handles all the messy bits in between," Dwyer says.
ezCater's relationship with Stitch began well. "The onboarding experience was quite good," Dwyer says. "We didn't have to talk to a salesperson – I could see it work first."
"Stitch eliminated the cost of confusion and helped us get running quickly," Senior Software Engineer Jon Gherardini says.
Gherardini also appreciates Stitch's support team. "Having chat on the site has been helpful. When something goes wrong, Stitch helps us resolve it quickly, and that creates value."
For ezCater, the value of a data pipeline comes down to three questions:
With Stitch, they can check off all three considerations.
With all of the products and services ezCater uses, eventually it ran into a data source that Stitch didn't support – but that wasn't an obstacle for long. Stitch integrations, or taps, are written on top of Singer, an open source standard for writing scripts that move data. Using Singer, anyone can develop an integration that can run on the Stitch infrastructure. "We have engineering talent here," Gherardini says. "With Singer, we have the ability to fiddle with things and make something that serves us. We don't have to wait for something to roll out on Stitch's roadmap."
ezCater used the Singer platform to write a tap for AppsFlyer from scratch. "When you buy an ad that drives users to a mobile app, there's a weird disconnect and you can’t see where the user came from. AppsFlyer tries to solve that problem and tell us how much it costs to get a converted mobile user. To extract data from AppsFlyer without Singer would have meant writing a script that runs on a schedule, pulls from the API, and pushes to Kinesis. The tap we wrote is just as simple, but because it's built with Singer it can be run inside Stitch, and now we don’t have to worry about running it, and we have no ongoing operations costs."
ezCater's tap is now available for all Stitch users as a community integration.
ezCater also used Singer to enhance Stitch's HubSpot integration. "With HubSpot, the existing tap didn't work for all of our use cases. Some issues only rear their heads when you get into the depths of a really complicated account, which is what we have.
“Because of Singer we were able to contribute directly to the code for the HubSpot tap and get the data we needed. We also went to HubSpot and asked them to expose fields we needed in their API so that could get access to them for the first time.
"Now, with the enhancements we made through Singer, your HubSpot integration is the best puller of this data in the world, and that's because you let people contribute to it.
"We told HubSpot and AppsFlyer – when you build a new API, please consider how it can work with Stitch."
With all its in-house engineering talent, why doesn't ezCater just write its own data pipeline? "We could do it ourselves," Dwyer says. "But without Stitch and Singer, it would have taken multiple engineers multiple months to create something, and then we would need to maintain it. That’s a large investment in a platform that's not part of our core business."
The middle of the solution should not be anything novel – it should just work. Stitch finishes the job once it has the data and keeps us safe from things like third-party blips. It gives us resiliency. That made it easy for us to make the business case. Stitch just worked, right off the bat, which made it easy to tell my management team I wanted to use it. I didn’t have to explain anything about functionality and pricing.
Software Engineering Manager
Dwyer and Gherardini take advantage of the Singer Slack channel, which has more than 250 members who bounce ideas around and help each other troubleshoot issues. "I'm always lurking," Gherardini says. "Whenever there are new messages, I will read them. It's super-valuable as a resource and a community-builder.”
ezCater plans to use Singer and Stitch in future projects, and continue to build and contribute to taps that then run inside Stitch. In the final analysis, Dwyer says he appreciates Stitch and Singer and how well their people work with ezCater. "You're not only selling enterprise software, you're selling a partnership."
Using Singer, anyone can add code that helps us drive our business without needing to have in-depth business knowledge.
Software Engineering Manager