App-ocalypse now: How to master data integration in today's application ecosystem

Not that long ago, the strategic IT end goal for large corporations was an all-encompassing enterprise resource planning (ERP) system — one central brain and one source of the truth to inform every employee, providing everything from individual customer insights for front-line customer service reps to on-demand reporting for the executive suite. All the information on the enterprise, from orders to invoice, from service requests to spare parts inventory, from employee onboarding to retirement packages, would be managed in one monolithic system. That, at least, was the vision, and it came with an '80s mullet.

Such monolithic systems promised great advantages, especially for IT: a single system to maintain, a single version of the truth, and with all critical data in one place, reporting could be centralized. True, each system would create its own data silo — and data silos come with their own costs — but then again, building an internal data integration process should be relatively simple. With just several core systems, pulling the data together should be straightforward.

But then the technology landscape changed.

The arrival of the World Wide Web in the '90s, along with cloud data storage and processing in the '00s, could have made this data nirvana a reality — but instead it led us away from monolithic systems. Today we have an explosion of independent best-of-breed apps, each deployed for a specific narrow purpose.

For instance, where corporations once managed one master human resources information system (HRIS), they now deploy a plethora of cloud-based apps, each handling one piece of the puzzle: expenses, benefits, travel, employee surveys, retirement services, collaboration, and community building.

In a similar fashion, sales and marketing teams today may rely on one central customer relationship management (CRM) platform, but such systems have ecosystems of connected apps for managing email automation, data augmentation, and lead tracking. Cloud-based platforms large and small now hold business-critical data. According to research by security firm Thales, 48% of all corporate data is now hosted on cloud servers.

A move to SaaS and mobile ...

The explosion of web-native applications has made it easy for businesses to rapidly (and independently) deploy solutions that promise

  • rapid time-to-value (get up and running in hours not months!) and
  • all of the benefits of the SaaS model (month-to-month contracts, no need to involve those pesky IT folks!).

Overnight, new SaaS solutions popped up that departments could deploy easily and see benefits from almost immediately.

Furthermore, everything went mobile. The growth of easy-to-consume apps started in the consumer space. Today the iOS store counts 1.8M apps. And the mobile trend is accelerating; experts project close to 260 billion downloads from the main app stores in 2022, according to Statista. Business applications followed the consumer trend, and represent 10% of mobile apps running today.

… led to an explosion of apps

SaaS adoption has been so rapid that "68% of organizations say they are mostly or all SaaS-driven at this point, with nearly 23% saying they operate solely using SaaS apps today" according to a report from SaaS app management firm Blissfully.

Most companies underestimate the number of apps in use inside their own organizations, Blissfully says. Their research suggests that businesses use nearly twice as many apps as they think they do, with companies of 1,000+ employees having more than 200 apps in use and 98 individual SaaS tool billing owners.

Adding to the chaos, each organization's SaaS app ecosystem is far from static. Many SaaS subscriptions are either orphaned (no owner, but still being billed) or have duplicate instances running across an organization. Companies find themselves running apps with overlapping and often underutilized functionality.

"App turnover is very common. In fact, the typical midsized company saw 39% of their SaaS stack change last year," Blissfully says. App turnover in most companies exceeds staff turnover.

Soon enough IT teams were facing an "app-ocalypse." Apps were popping up everywhere, and access control was managed — often poorly — in the business functions. Technology spend was fragmented, often duplicative, and far from efficient. And data was spread across hundreds of systems.

Where did the monoliths go?

So are the monoliths really gone? Yes and no. As business applications, the monolithic ERPs, HRISs and CRMs of the '80s are struggling to remain relevant. But ERP, HR management, customer management as processes — along with data-driven business insights — remain critical tools that businesses need to compete.

Since the dawn of the industrial revolution organizations have pursued efficiency — think steam power, assembly-line manufacturing, just-in-time inventory management, kanban, Six Sigma process improvement, up to today’s automation and use of artificial intelligence and connected devices.

Like the technology ecosystem, the business landscape has become more competitive, and runs on ever-shortening cycles. Having reliable and up-to-data intelligence on the enterprise is crucial to managing in today’s "VUCA" environment: volatile uncertain, complex, and ambiguous.

ETL — the essential foundation for healthy data flow

Given this shift from monoliths to ecosystems, data-driven organizations struggle to create a unified picture from disparate systems. The process of unifying the data begins with ETL (extract, transform, and load), a collection of technologies, processes, and business rules to harmonize, integrate, and consolidate data.

In a fast-changing environment, traditional home-grown ETL processes are increasingly difficult to maintain. In recent years the number of possible data sources has exploded, and at the same time, the most efficient platforms for running these consolidated efforts have moved to the cloud.

Technology giants like Amazon, Google, and Microsoft evolved their cloud computing platforms to provide analytics databases to help centralize all of this information. These new cloud data warehouses brought performance improvements but still required engineering work to get them connected to the many apps in a company’s ecosystem.

Home-grown ETL can create a bottleneck in the process of gaining data-driven insights. CIOs faced a dilemma: At what point did it make sense to transition the ETL process from the internal team to a third-party service? What were the security and compliance challenges? How could they handle their proprietary data sources alongside standardized commercial SaaS solutions?

As with many other areas, smaller digitally native firms led the way, embracing new tools and approaches and setting up scalable data flows that generated business intelligence in close to real time, and at a fraction of the cost of home-grown ETL.

What's your team’s time worth?

A qualified data engineer or data warehouse expert may cost an organization $150,000 in salary and benefits, and average 2,080 work hours in a year[1]. That puts internal data development and management costs based on labor alone at $72 an hour.

Typically, an organization spends 100 hours on managing, monitoring, and troubleshooting each individual integration, or $7,200 per connection.

If you're pulling data from three or more business-critical applications, the economics of building your own ETL system don’t stack up.

Tools like Stitch provide a better solution by connecting the organization to more than 100 data sources, and provide options for scheduling and configuration. Even a single integration managed through the Stitch platform can be cheaper than developing and maintaining the same capability in-house.

When API management firm Postman recognized their need for a data pipeline, they were faced with the same conundrum many companies face: build or buy?

Postman’s engineering team quickly realized they would need to invest a lot of time and energy to build and maintain an internal data pipeline, which would take time away from their improving their own product.

That was when they decided to go with Stitch. At the time Postman Co-founder and CTO Ankit Sobti said, "The best part of Stitch was that I could set it up and see the benefits on my own. It’s a product that doesn’t need an explanation. When you’re building an entire tech stack and are stuck trialing and demoing a ton of tools, it’s a big deal when you see something that just works."

If you want to master the app-ocalypse and set your organization up for success, sign up today for Stitch and get a 14-day free trial.

Notes


  1. Source: Salary.com & assumption of 260 working days at 8 hours per day. Salary can vary significantly between geographies. ↩︎