It’s been nearly three years since Dan Ariely declared that big data is like teenage sex. His logic? “Everyone talks about it, nobody really knows how to do it, everyone thinks everyone else is doing it, so everyone claims they are doing it.”
You might think that as we approach 2016, this is no longer the case; we now have Kafka, Spark, and Hadoop. Engineers from Spotify, Buffer, Netflix, and Braintree have all written in detail about their process of building their big data tech stacks. Surely today’s companies are actually “doing big data” and not just talking about it – right?
We turned to the data to find out. Using the same data set that we used for our latest benchmark, The State of Data Science, we set out to answer the question, “How many companies have big data skills?” Our criteria was this — how many companies have at least one data scientist? This isn’t perfect, there are companies like Google who (obviously) have big data skills, but don’t hire people under the title “data scientist.” But it turns out that even that rough cut of the data can teach us quite a lot.
Big data skills by company size
In our first round of analysis on this data set we saw that the top employers of data scientists have 50+ data scientists on their teams. Microsoft has the most data scientists (close to 227 by our estimates), followed by Facebook (132), and IBM (98). But while these big players are snapping up data scientists in the double digits, most companies don’t employ a single one.
It’s not surprising that small companies with 1–200 employees don’t have big data skills in the form of a data scientist. But it’s a little more surprising to see how little this trend changes even for bigger companies. Only 6% of large companies (companies employing 10k+ employees) have a single data scientist on staff.
While some companies, the Microsofts and Facebooks, are hiring lots of data scientists, most companies don’t have any at all.
Big data skills by company location
This same trend holds true when you look at how many companies have a single data scientist by country. The term ‘data scientist’ was created in the US and over 6,000 data scientists, 55% of the total number, work in the US. But only .20% of US companies have a single data scientist on staff.
French and Israeli companies are the most likely to have a single data scientist on staff, followed by Singapore and Finland.
Big data skills by industry
In The State of Data Science, we looked at what industries employ the most data scientists. Here are the top three:
Information Technology & Services employ 19% of all data scientists
Internet companies employ 13% of all data scientists
Software companies employ 13% of all data scientists
But when you look at this at the company level, things get much less impressive. Only 0.3% of Information Technology & Services companies employ a single data scientist, and about 0.5% of Internet and Computer Software companies employ a single data scientist.
If only a tiny percentage of companies are hiring data scientists, why all the hype about big data? There are two reasons why big data makes headlines but is still so underrepresented in the general population:
1. Companies that use big data effectively are building the most valuable businesses in the world. Google, Facebook, Tesla, Uber, Spotify, and more are some of the world’s largest and fastest-growing businesses, and they’re creating the majority of that value with algorithms running on top of massive data sets.
Your average drive-time radio DJ can’t compete with Spotify and a taxi driver can’t compete with Uber. Humans delivering services without the benefit of data are quickly losing their edge and entire industries are being disrupted as a result. This disruption makes for great headlines, even if there are a small total number of companies doing it.
2. We’re still at the very beginning of this trend. As the numbers indicate, only a tiny number of companies employ a data scientist today, but that’s growing incredibly quickly, with the number of data scientists doubling in the past four years.
With the success of data-driven business models, there’s no reason to anticipate that growth slowing any time soon. So, while the absolute number of companies with big data skills may be less than impressive today, expect that to change in the coming years.
So … is anyone really “doing big data?”
The answer is, yes, but not that many. Data scientists are in high-demand, and while some companies can afford to hire large teams of data scientists, and are doing just that, the vast majority of companies still don’t have any big data skills.
And that’s OK! Big data makes headlines, but there’s an enormous amount of value that can be derived from normal sized data sets. We learned (also from LinkedIn data) that there are 1.1 million light quants in the US, and 365,000 of them work for companies with fewer than 500 employees.
We define light quants as people that list SQL as a skill on their LinkedIn profile, but don’t list “big data skills” like Python, R, or machine learning. People with these skills don’t go by the title of data scientist, but they still have the analytical abilities to derive enormous value from data. Combine SQL skills with tools like Looker, Mode, and Stitch and a single analyst can help a company make smarter decisions about everything from customer acquisition to product development.
If you liked this, you should check out these posts: