Understanding data, analytics and decision making
A brief introduction incorporating some of the better resources, perspectives and case studies that I’ve come across.
- Section 1: What is Big Data?
- Section 2: Practically, how do I use analytics and big data?
- Section 3: What are the barriers
- Section 4: How to integrate data into a decision making process
- Section 5: The (big) difference between data driven and data informed
Section 1: What is Big Data?
Start here: It’s important to understand that “big data” is seriously overused (and overextended): Big data was the most overused buzzword of 2013 (@jmcduling)
To quote from this article: “‘big data’ is often misused to mean the kind of data analysis that could be done on a single desktop computer, when it should really be reserved for applications requiring serious processing power. That hasn’t stopped it from being used in the corporate world with increasing frequency.”
The terminology is incredibly confusing – but easy to understand once you know what you’re looking for.
- In business buzzword speak, “big data” means data analytics. It has nothing to do with the size of the data.
- In technical buzzword speak, “big data” means data that is too big to be processed by a normal computer. Incredibly large data sets on the scale of Google and Facebook. But this isn’t what most of the business press is referring to.
This article provides a good summary of the issue: Most data isn’t big and business are wasting money pretending it is (@mims)
Here’s the more technical view of the big data landscape. This article focuses on the definitions, tools and applications, but is written in plain english and easily accessible. An introduction to the big data landscape (@edd)
This is all interesting, but why is it valuable?
This article provides some great historical context: “Breakthroughs in innovation often rely on breakthroughs in measurement.” The Big Data Boom Is the Innovation Story of Our Time (@amcafee and @erikbryn)
And finally, This is probably the best piece on the long term impact of true big data and analytics.It’s a huge, well considered report by McKinsey. Summed up in a sentence, “Leaders in every sector will have to grapple with the implications of big data, not just a few data-oriented managers.” Big data: The next frontier for innovation, competition, and productivity. (@mchui, @BradfordBrown, @richard_dobbs, @CharlesRoxburgh and others)
Section 2: Practically, how do I use analytics and big data?
It’s important to start with the differences between big data and data analytics.
From Wikipedia, “Analytics is the discovery and communication of meaningful patterns in data.” Done well, data analytics is built on a firm grasp of statistics and probability. These skills are used to discover useful information, suggest conclusions, and support decision making.
Big data just scales this up. You still need the analytics to get to understanding and decisions. You’re just scaling up the field of investigation. Put simply, big data is useless unless you have a firm analytics foundation to build on.
This is a good series of videos from McKinsey about the three challenges of data analytics in application:
- “Deciding which data to use (and where outside your organization to look)
- Handling analytics (and securing the right capabilities to do so)
- And using the insights you’ve gained to transform your operations.”
Making data analytics work: Three key challenges (Tim McGuire)
This HBR piece expands on this view, diving into the practical application of big data and analytics as management tools: Big Data: The Management Revolution (@amcafee and @erikbryn)
In thinking about analytics maturity, this is a nice simple four level model that can help in thinking about your organisation: Talent Analytics Maturity Model (@Josh_Bersin)
In practical examples, a well cited (and somewhat creepy) piece about how Target uses data analytics to predict if you’re pregnant: How Companies Learn Your Secrets (@cduhigg)
Section 3: What are the barriers
The biggest challenge to moving to big data and advanced analytics is the lack of analytical talent.
As this AMA research piece identifies, The challenge isn’t the tools it’s learning the analytical approach: Companies See Need to Build Analytical Skills in Their Organizations: A Study of Analytical Skills in the Workforce (@AMAnet)
Many commentators have identified a significant talent crush: “A significant constraint on realizing value from Big Data will be a shortage of talent, particularly of people with deep expertise in statistics and machine learning, and the managers and analysts who know how to operate companies by using insights from Big Data”. Big Data’s Big Problem: Little Talent (@benjrooney)
But the problem isn’t always that the talent doesn’t exist. It’s that the specialised talent has always existed outside of the mainline business process.
Quoting from the following HBR blog by Paul Barth and Randy Bean: “Data analytic talent is not new. Statisticians, database marketers, and PhD quantitative analysts have long been a staple of sophisticated marketing organizations and have played critical roles in financial engineering. But all too often these same individuals have been relegated to the sidelines.” There’s No Panacea for the Big Data Talent Gap (@NVPBigData)
This challenge has led to the creation of the new job role of “Data Scientist”. The Data Scientist is a hybrid role that combines the management consultant, analyst and programmer.
This 2012 HBR piece introduced the term “Data Scientist” and is still the best place to start: Data Scientist: The Sexiest Job of the 21st Century (@tdav and @dpatil)
I also like this ZDnet debate, which contrasts the pros and cons on the role of the Data Scientist: Business Analytics: Do we need data scientists? (@ldignan)
And to finish up there’s some nice visuals here that support the difference between statistics, data science and business intelligence: Statistics vs Data Science vs BI (@revodavid)
Section 4: How to integrate data into a decision making process
All the data in the world is only as good as decisions that it allows you to make.
A nice introdution to the challenge. In a sentence: “Big data is a measurement revolution that must be accompanied by a similar management revolution.” Big data calls for data-driven decision-making skills (@JackVaughanatTT)
As this piece argues, all data has a job, and that job is always about making a decision. The Power to Decide (@antonioregalado)
A nice, simple, plain english five step model for making data informed decisions. Five Steps For Making Data-Driven Decisions (@AnalyticsQueen)
Most people are uncomfortable with data. Here’s how to dispel your team’s fear of data. (@thedatadoc1)
But it’s worth remembering that getting data incorporated into a decision making process is a significant and hard change management exercise. Good Data Won’t Guarantee Good Decisions (@ShvetankShah)
It’s easy to be lulled into forgetting the weaknesses of decision models. Here’s some of the common mistakes: The benefits—and limits—of decision models (Phil Rosenzweig)
A nice example of good decision making: KeyBank Moves To Data Driven Decision Making (@Tomgroenfeldt)
And finally, one of the best pieces on making the cultural shift. Seven Steps to Creating a Data Driven Decision Making Culture (@avinash)
Section 5: The (big) difference between data driven and data informed
There’s a big difference between a data informed and data driven making decision process. Some good pieces that explore the issue:
Like most other topics in the big data space, this conversation started at Facebook (or Google). Here’s the story of how Facebook learnt the lesson of being data informed rather than data driven. Data Informed, Not Data Driven (@mosseri)
This is a great Andrew Chen post, that builds on this video in the context of product design. In a sentence: data will always allow incremental improvement, but data alone will never show the best outcome in the biggest market. This post is worth reading because it hits the nail on the head in terms of the limitations of data. It’s easy to extend the analogy to other applications of data analytics. Know the difference between data-informed and data-driven (@andrewchen)
A good NYT piece on the use of data in large, multifaceted strategic decisions. What Data Can’t Do (@nytdavidbrooks)
I like this approach from Bob Sutton, integrating intuition led hunches with data supported quantification. Intuition vs. Data-Driven Decision-Making: Some Rough Ideas (@work_matters)
Taking data-driven to its logical conclusion. The quote that sums up the piece follows: “if we let the data drive editorial, all you will read about at CBS News is Paris Hilton’s breasts and Lindsay Lohan’s drinking problem.” The Myth of the “Data-Driven” Business (@erictpeterson)