The Stitch guide to choosing a business intelligence tool

You want to choose the best business intelligence (BI) program for your organization. The problem is that there are so many products and platforms available — how do you choose the best one?

Here's our guide to choosing a BI or data analytics platform, developed through what we've learned from working with thousands of companies to power their data infrastructure, as well as discussion with data analytics professionals and experts at companies that offer BI products and services. There are three high-level steps in the decision process: gathering information by asking questions, creating a scorecard, and performing evaluations.

1. Ask questions about data usage

The first phase of the process involves learning about your organization and the way it uses data. Ask questions about who, what, when, where, how, and why.

Who handles the data?

In your organization today, who works with data? Do you have a team of data analysts who build all your visualizations? Do business users create their own reports? Are data scientists building models with the data? Who's responsible for data transformations?

Why you need this information: Data scientists and other sophisticated users have different needs and expectations than business users. Talk to everyone who creates reports (or, if you're in a large organization, a representative sample) and learn about how they use data and what their requirements are. Are they viewing dashboards daily, or do they need hourly updates? Do they print reports? Do they collaborate with other users? Do they need to create ad hoc visualizations?

The question of data transformations is as significant as the choice of data warehouse. Chances are you're going to want to replicate your raw data into your data warehouse, and transform some of the data once it gets there to make it more useful for reporting. Some BI tools can perform transformations on raw data, but many organizations prefer to run transformations as a separate step. Tools designed to do transformations include dbt, Talend Open Studio for Data Integration, and Talend Pipeline Designer.

How you'll use this information: The information about who uses analytical data will help you compile evaluation criteria for screening potential products and, eventually, trying them out in-house.

If non-technical employees need self-service analytics, then you'll want a tool with an intuitive point-and-click interface. If you plan to have only technical workers use the tool, then you can explore tools that use code as input. In some cases, only expert users can use a tool to its full potential. Then you have to ask what's the opportunity cost of having expert data analysts build and update all reports, compared against a tool that enables anyone to get analytics on their own.

One tool may be able to meet the needs of both constituencies. Many organizations feel it's better to avoid multiple tools across different departments. Other organizations choose to select different tools for different types of users.

What kind of questions do you need to answer?

Write down the use cases for your organization's BI tool, with as many specifics as you can, including what users expect to accomplish with your reports, visualizations, and analytics. Some common needs include:

  • operational reporting snapshots
  • data exploration
  • spreadsheet integration (both directions)
  • data analysis
  • self-service custom reports, charts, and graphs

Why you need this information: BI tools are complex products with many capabilities. It's hard to know how to start evaluating them. You're better off starting from your needs than a product's capabilities, because some of those super shiny features won't be useful for your organization.

How you'll use this information: Having use cases and users' goals will help you build the evaluation criteria you'll use.

Where is the data?

Where does the data you're reporting on live? Is it in in-house databases? SaaS platforms? A corporate data warehouse?

Why you need this information: You have to consider your BI tool in the context of an entire data analytics infrastructure. Some tools read directly from a variety of data sources — but often not all the ones you use. Some connect with cloud data warehouses, and these data warehouses need to be filled with data replicated from all of the sources you want to analyze. That's where Stitch comes in. Stitch connects to more than 90 data sources and replicates data to any of the the popular cloud data warehouses, from which any BI tool can access data from all of your sources.

Try Stitch for free

How you'll use this information: Knowing where your data is helps you consider how to round out your analytics infrastructure. Many important analytic tasks are impossible without a data warehouse. If you don't already have one, you can evaluate a data warehouse simultaneously with BI tools, and thereby ensure you have a data stack that really suits the needs of your business.

What does the data look like?

You also have to take into account the state of the data being queried. How much data prep is necessary before someone can use the visualization tool? If self-service users need to have access to the data, but the data is messy or poorly modeled, then you need to factor in a transformation tool, or your BI tool needs to be able to make transformations. You have more options if the data is modeled cleanly and there are no logical calculations being done, other than generic aggregations of available fields.

Why you need this information: If your data is poorly prepared, you'll need to clean it up before you can use it. The alternative is to limit report creation to experts who can work around the flaws.

How you'll use this information: You'll have to consider the opportunity cost of having expert data analysts build and update all reports vs. enabling all kinds of users to get their analytics themselves.

What's your budget?

Before you bring in any BI tool, you need to know how much you can afford to pay for it. And, as we've seen, your budget may have to cover more than just the BI tool — it may also have to encompass implementing a data warehouse and an ETL tool. For each of those components you must consider licensing costs, operational costs, and, if you use on-premises resources, hardware costs. (Hint: Don't use on-premises hardware. A cloud-based data warehouse can scale elastically to handle virtually any level of data processing. That makes it attractive to do any necessary data cleaning or transformation once the data is already in the data warehouse, which in turn makes the ETL pipeline faster and easier to use.)

Then there are expenses on top of what you pay for the software. You may need technical experts to transform data and maintain data models, or data engineers to write and manage home-brewed ETL pipelines. You may need to provide training for your staff, conducted by a peer leader, if you're lucky, or by a vendor or a third party.

Do your costs have to be predictable? If not, you can take advantage of a cloud platform's usage-based pricing to save money during slack periods, and spend more during periods of heavy usage, such as year end.

The number and kind of users also affects pricing. Some tools don't charge anything for dashboard users. Others require licenses for anyone who creates or consumes BI.

2. Make a scorecard to evaluate BI tools

At this point you should have a good feeling for how your organization uses analytics and how your colleagues want to use new BI tools going forward. The next step is to create a list of criteria on which to evaluate each BI platform.

Criteria can be as general or specific as you find useful. Here are some points to consider (not a comprehensive list). Does the tool have the ability to:

  • Run on your client platforms (Windows or Mac) or in a web browser
  • Access the specific data sources you use
  • Integrate data from multiple sources, databases, and tables
  • Create custom measures for reporting
  • Create specific visualization types (line graphs, scatter charts, maps, etc.)
  • Perform interactive filtering
  • Aggregate visualizations in dashboards
  • Drill down into visualizations for in-depth reporting
  • Create mobile-friendly visualizations, or let users work on data from a mobile device
  • Meet your organization's governance requirements — for example, will the vendor of a BI platform sign a BAA for an organization bound by HIPAA?

You should also think about how well the tool will perform in the future. The volume of queries against your data will almost certainly grow once you have a tool in place. Look for functionality that may not be useful for your organization today but could come in handy eventually.

Create a rubric for each criterion so that everyone involved in the evaluation process understands how they should evaluate all the tools. The rubric also serves as documentation for managers who aren't involved in the evaluation process.

Next, take the list and divide your criteria into two groups: must have and nice to have. If a factor is truly must-have, and a tool you're considering doesn't support it, you can cross that tool off your list.

Once you've considered all of the yes/no points, you can turn to more nuanced questions, such as:

  • How easy is it to administer the tool? and
  • How much time does it take to build a report?

You can score these questions on a numeric (1 – 10) or qualitative scale (poor, mediocre, good, great). Some people like to give an even number of choices (0 – 3) to force people to score something as either above or below a middle value. You can also attach adjectives to factors, such as intuitive or simple or clunky or deficient.

You may better organize your thinking if you group your criteria into categories, such as:

  • Analytics capabilities
  • Reporting and data visualization capabilities
  • Consumption/sharing of reports

Add up the scores in each of your categories, or weight them according to their importance, to get an overall ranking. Look for one to three products or platforms to rise to the top.

3. Conduct hands-on evaluations

Now you can bring in the top contenders and try them with your data. All product demos look slick, but how well does the tool perform in practice?

Use a small project to see how quickly stakeholders can use the tool. Watch how your users work, and find out how close they can get to their ideal solution.

During this part of the evaluation you can judge how well the developer supports its tool. Does the vendor provide adequate online documentation? Is tech support available, and if so, how quickly can you get attention? If you have a difficult problem, will you have to pay extra for consulting?

How well does the new technology fit in with your existing infrastructure? How much hand-holding do users need? If they need a lot during the evaluation, it may be an indicator of what to expect later when you have more complex requirements — and it may foreshadow how much support you'll have to provide to internal customers.

Reach out for peer evaluations

Hands-on evaluations are critical, but you can also ask other data professionals about their opinions of the tools you're considering. If you don't already know anyone who can give you an opinion, go to a local user group meeting and ask your questions there.

Give special emphasis to any positive assessments; people remember negative experiences they had and are happy to talk about them, but people tend to have fewer good things to say about products and services.

Pull the trigger

When you reach the end of this long process you're ready to buy. We'll leave negotiating the best deal to you. Rolling out the solution is also a topic for another day.

Finally, once you've made the purchase decision, don't stress about whether the product you picked was the absolute best one you could have chosen. Chances are that if a product made it to the top of your list, it will do fine for you.

Thanks to the experts who contributed their advice for this post: Jennifer Hudiono and Alex Poulos for Chartio; Erin Franz at Looker; Adam Bonefeste and Lauren Gimmillaro at Periscope Data; Redditors opendataalex, triiimit, olavhellothere, and secretWolfMan; Amy McQuaid at Talend; and Roshan Bhalla, Rob Moore, and Qun Wei.

Image credit: Growthlakes