Not long ago, the term "data science" meant nothing to most people -- even the those who worked in data. A likely response to the term was: "Isn't that just statistics?".
These days, data science is hot. The job of "data scientist" was referred to by the Harvard Business Review as the "Sexiest Job of the 21st Century." Why did data science come to exist? And just what is it that distinguishes data science from statistics?
The very first line of the American Statistical Association's definition of statistics is "Statistics is the science of learning from data..." Given that the words "data" and "science" appear in the beginning fragment of this definition, one might assume that data science is just a rebranding of statistics. A number of Twitter humorists certainly have:
While there's a grain of truth in these jokes, the reality is more complicated. Data science, and its differentiation from statistics, has deep roots in the history of computers.
Statistics was primarily developed to help people deal with pre-computer data problems like testing the impact of fertilizer in agriculture, or figuring out the accuracy of an estimate from a small sample. Data science emphasizes the data problems of the 21st Century, like accessing information from large databases, writing code to manipulate data, and data visualization.
A Computer from the 1960s.
The arrival of the personal computer revolutionized access to data and what could be done with that data. It can be argued that data science is simply a response to this new technology.
The first well known appearance of the term data science is from legendary computer scientist Peter Naur's 1974 book Concise Survey of Computer Methods . In this book, Naur defines data science as "The science of dealing with data...." Right from the start, data science was not just about "analyzing" data (the bread and butter of classical statistics), but about "dealing" with it, using a computer. In Naurs's book, "dealing" with data includes all of the cleaning, processing, storing and manipulation of data that happens before the data is analyzed, and the subsequent analysis.
Though the term data science did not catch on from Naur's usage, in the 1980s and 90s an innovative community started to blossom of people who used computers to "deal with" data. Groups like the International Association for Statistical Computing and KDNuggets came up with new ways to use computers to find meaning in data.
This innovation was prompted by a few things: (1) The need to work with datasets larger than pre-computational statisticians could have conceived of. These datasets would later come to be known asbig data. And, (2) an increased focus in industry on prediction -- of markets, of resources, of customer behavior, what have you -- for commercial uses. The inventors of data science borrowed from statistics, machine learning and database management to create a whole new set of tools for those working with data.
Statistics, on the other hand, has not changed significantly in response to new technology. The field continues to emphasize theory, and introductory statistics courses focus more on hypothesis testing than statistical computing.
Within the field of statistics, there were a few who believed that the discipline should transform itself to fit the changing landscape. In 2001, the influential statistician William Cleveland wrote a paperwhich suggested expanding the field of statistics and renaming it "data science." This new field would include a greater focus on real world "data analysis" and "computing." Cleveland's dream never came to pass, but many universities do now have data science departments -- in addition to their statistics departments.
Perhaps the most accurate Twitter quip about data scientists is the following:
"A data scientist is someone who is better at statistics than any software engineer and better at software engineering than any statistician."
Statistician and data visualizer Nathan Yau of Flowing Data suggests that data scientists typically have 3 major skills:
1. They have a strong knowledge of basic statistics and machine learning (at least enough to avoid misinterpreting correlation for causation, or extrapolating too much from a small sample size).
2. They have the computer science skills to take an unruly dataset and use a programming language (like R or Python), to make it easy to analyze.
3. They should be able to present that data and their analysis in a way that is meaningful to somebody less conversant in data, through visualization and summary.
Andrew Gelman, a statistician at Columbia University, writes that it is "fair to consider statistics... as a subset of data science" and probably the "least important" aspect. He suggests that the administrative aspects of dealing with data like harvesting, processing, storing and cleaning are more central to data science than hard core statistics.
The academic backgrounds of Udemy users who take data science and statistics courses demonstrates both the similarities and differences between the disciplines. The following table shows the ten most common academic backgrounds of Udemy users who took one of our statistics or data science courses.
Data Via Udemy
For courses in both statistics and data science, the most common backgrounds are Computer Science and Economics. The differences appear lower down the list. More of our data science students have a background in computationally heavy disciplines like electrical engineering, mathematics and accounting. In contrast, those taking statistics course are more likely to have focused on a less mathematical discipline, like graphic design, marketing or psychology.
It's not just hype, data science really is in the ascendancy. According to data from the job search website Indeed.com, there were barely any job postings for data scientists before 2011, but by 2015 the demand for data scientists had surpassed the demand for statisticians. The chart below displays the percentage of all jobs posted for data scientists and for statisticians over the last ten years.
Data scientist jobs are on the rise while statistician positions are on the decline. It is likely that some of the positions that, in the past, would have been listed for statisticians are now listed for data scientists (some firms use the terms interchangeably). But it is not just a tradeoff. For data scientists and statisticians combined, there were more than twice as many jobs listed in early 2015 than there were in early 2012.
Data science jobs are not just more common that statistics jobs, they are also more lucrative. According to Glassdoor, the national average salary for a data scientist position is $118,709 compared to $75,069 for statisticians.
Arguments over the differences between data science and statistics can sometimes get contentious. When the term "data science" came to prominence around 2011 there was a backlash. At that time, one well-known statistician referred to the position of a data scientist as "just the hip new name for statistician that will probably sound stupid 5 years from now."
But data science and statistics both continue to exist and there is no indication that either will go away. Although there is a great deal of overlap between the disciplines, data science developed for a very good reason. For the most past, statisticians chose not take on the data problems of the computer age.