Wednesday, January 25, 2017

Avoid the Unintended Consequences of Analytic Models

Cathy O’Neil’s Weapons of Math Destruction is a must-read for analytics professionals and data scientists.

In a world where it is acceptable for people to say, “I’m not good at math,” it’s tempting to lean on analytic models as the arbiters of truth.

But like anything else, analytic models can be done poorly. And sometimes, you must look outside your organization to spot the damages.

The Nature of Analytic Insights

Traditional OLAP focuses on the objective aspects of business information. “The person who placed this $40 order is 39 years old and lives in Helena Montana.” No argument there.

But analytics go beyond simple descriptive assertions. Analytic insights are derived from mathematical models that make inferences or predictions, often using statistics and data mining.1

This brings in the messy world of probability. The result is a different kind of insight: “This person is likely to default on their payment.” How likely? What degree of certainty is needed to turn them away?

When you make a decision based on analytics, you are playing the odds at best. But what if the underlying model is flawed?

Several things can go wrong with the model itself:
  • It is a poor fit to the business situation
  • It is based on inappropriate proxy metrics
  • It uses training data that reinforces past errors or injustices
  • It is so complex that it is not understood by those who use it to make decisions.
And here is the worst news: whether or not you manage to avoid these pitfalls, a model can seem to be “working” for one area of your business, while causing damage elsewhere.

The first step in learning to avoid these problems is knowing what to look for.

Shining a Light on Hidden Damages

In Weapons of Math Destruction, Cathy O’Neil teaches you to identify a class of models that do serious harm. This harm might otherwise go un-noticed, since the negative impacts are often felt outside the organization.2 She calls these models “weapons of math destruction.”

O’Neil defines a WMD as a model with three characteristics:
  • Opacity – the workings of the model are not accessible to those it impacts
  • Scale – the model has the potential to impact large numbers of people
  • Damage – the model is used to make decisions that may negatively impact individuals
The book explores models that have all three of these characteristics. It exposes their hidden effects on familiar areas of everyday life – choosing a college, getting a job, or securing a loan. It also explores their effects on parts of our culture that might not be familiar to the reader, such as sentencing in the criminal justice system.

Misaligned Incentives

O’Neil’s book is not a blanket indictment of analytics. She points out that analytic models can have wide ranging benefits. This occurs when everyone’s best interests line up.

For example, as Amazon’s recommendation engine improves, both Amazon and their customers benefit. In this case, the internal incentive to improve lines up with the external benefits.

WMD’s occur when these interests conflict. O’Neil finds this to be the case for models that screen job applications. If these models reduce the number of résumés that HR staff must consider, they are deemed “good enough” to use. They may also exclude valid candidates from consideration, but there is not an internal incentive to improve them. The fact that they harm outside parties may even go unnoticed.

Untangling Impact from Intent

WMD’s can seem insidious, but they are often born of good intentions. O’Neil shows that it is important to distinguish between the business objective and the model itself.  It’s possible to have the best of intentions, but produce a model that generates untold damage.

The hand-screening of job applications, for example, has been shown to be inherently biased. Who would argue against “solving” this problem by replacing the manual screening with an objective model?

This may be a noble intention, but O’Neil shows that it fails miserably when the model internalizes the very same biases. Couple that with misaligned incentives for improvement, and the WMD fuels a vicious cycle that can have the precisely the opposite of the intended effect.

Learning to Spot Analytic Pitfalls

The first step to avoiding analytics gone awry is to learn what to look for.

“Data scientists all too often lose sight of the folks at the receiving end of the transaction,” O’Neill writes in the introduction. This book is the vaccine that helps prevent that mistake.

If you work in the field of analytics, Weapons of Math Destruction is an essential read.


1. OLAP and Analytics are two of the key service areas of a modern BI program. To learn more about what distinguishes them, see The Three Pillars of Modern BI (Feb 9, 2005).

2. But not always. For example, some of the models explored in the book have negative impacts on employees.

Friday, December 2, 2016

Is Your Team Losing the Spirit of the Agile Manifesto?

As the adoption of agile BI techniques spread, it is easy to become wrapped up in methodology and lose site of the agile spirit.  

This year is the 15th anniversary of the Agile Manifesto. Mark the occasion by refocusing on the agile principles.

The Age of “NO”

Fifteen years ago, most business software was developed using rich but complex methodologies. Businesses had spent years refining various waterfall-based methods, which were heavily influenced by two decades of information engineering.

The result seemed to make sense from IT’s point of view, ensuring a predictable process, repeatable tasks, and standard deliverables.

To the people who needed the systems, however, the process was inscrutable.1 It seemed like a bewildering bureaucracy that was managed in a foreign language. They were always being told “no.”

Can we add a field to this screen?
No, you would have had to ask that before the requirements freeze.
Can we change the priority of this report?
No, that decision must be made by the steering committee.
Can we add a value to this domain?
No, that is the province of the modeling group.

The groups were looking at software development from completely different perspectives. They were not really collaborating.

Business people were asking for functionality. IT was beating back the requests by appealing to methodology.

Enter Agile

In 2001, a group of developers met in Colorado to talk about what was working and not working in the world of software development. They produced the Agile Manifesto:

Manifesto for Agile Software Development

We are uncovering better ways of developing
software by doing it and helping others do it.
Through this work we have come to value:

Individuals and interactions over processes and tools
Working software over comprehensive documentation
Customer collaboration over contract negotiation
Responding to change over following a plan

That is, while there is value in the items on
the right, we value the items on the left more.

Kent Beck, Mike Beedle, Arie van Bennekum, Alistair Cockburn, Ward Cunningham, James Grenning, Jim Highsmith, Andrew Hunt, Ron Jeffries, Jon Kern, Brian Marick, Robert C. Martin, Steve Mellor, Ken Schwaber, Jeff Sutherland, Dave Thomas

© 2001, the above authors
this declaration may be freely copied in any form, but only in its entirety through this notice. 

The thrust of the manifesto is on collaboration between business and technical personnel, with an emphasis on visible business results. That is, while methods are important, results are more important.2

The Agile Manifesto helped refocus software development on the product: functional business applications.

Agile Today: The Danger of Cooptation

Fifteen years later, it is safe to say that “Agile” has permeated the mainstream. This is largely a good thing. But whenever something moves from being a new alternative to wide acceptance, there is a potential dark side. As new ideas spread, they can be misunderstood or corrupted; adoption becomes cooption.

I frequently see signs of back-sliding to The Age of “No.” Even among developers who follow agile-based approaches, there is a tendency to lose sight of the agile principles. Here are two examples from my recent experience:

A team had decided to implement an unusual data model. It could easily return incorrect results if not queried in specific ways. The recommendation was to write some extra documentation explaining the pitfalls, and a short tutorial for developers. The response: “We cannot do that. We are an Agile shop. We cannot produce documentation that is not auto-generated by our tools.”

On another occasion, a team was developing the scope for several work streams. One set of needs clearly required a three-week collaborative activity. This was expressly forbidden. “Our Agile approach requires everything be broken down into two-week sprints.”

In both cases, the response was couched in methodological concerns, with no focus on the business benefit or value.3 

This is precisely the kind of methodological tunnel-vision against which the Agile Manifesto was a reaction.

Keeping the Faith

It is hard to disagree with the agile principles, regardless of your organization’s commitment (or lack thereof) to an agile process.

You can take one simple step to ensure that you are not losing touch with agile principles:

     Whenever you are tempted to say “no,” pause and reflect.

Why are you denying the request? Is it simply based on process? Think about what the person actually wants. Is there value? Is there a way to address the business concern?

Sometimes “no” is the right answer. But always be sure your evaluation places business capabilities and benefits ahead of process and procedure.

Learn More:

Read more posts about applying agile principles to BI and analytics:


1. These people were often referred to as “users.” Over time, this became a derogatory term.
2. Agile is often misinterpreted as emphasizing speed.
3. Luckily, both of these teams saw fit to make exceptions to their processes, prioritizing business value over method.

Thursday, November 17, 2016

Probability and Analytics: Reactions to 2016 Election Forecasts

Reactions to the 2016 election forecasts suggest we don’t do a good job communicating probability and risk.

In a September 2016 post, I suggested readers check out the discussions of analytic models at FiveThirtyEight. One of the links led to their forecast model for the 2016 presidential election.1

In the past week, I have received quite a bit of email suggesting I should take down the post, given that the model “failed.” For example, one emailer wrote:
How can you continue to promote Nate Silver? The election result proved the analytics wrong.
These reactions expose a real issue with analytics: most people do not understand how to interpret probability.

An analytic failure?

On November 7th, the  final prediction of the FiveThirtyEight “Polls Only Model” gave Hillary Clinton a 71% chance of winning. As things turned out, she lost.

Those emailing me were not alone in believing the model failed. The day after the election, there were many stories suggesting FiveThirtyEight and the other aggregators were wrong.2

But were they?

Nate Silver discusses the FiveThirtyEight Model
(If the video above does not play, you can access it here.)

Understanding probability

The FiveThirtyEight model gave Clinton a 71% chance of winning the election. That’s about a 7 in 10 chance. To understand how to interpret this probability, try the following thought experiment:

Suppose you are at Dulles airport, and are about to board a plane. While you are waiting, you are notified that there is a 7 in 10 chance your flight will land safely.  Would you get on the plane? 

I know I wouldn’t.

When the probability of something happening is 70%, the probability of it not happening is 30%. In the case of the airline flight, that’s not an acceptable risk!

Now suppose the flight lands safely.  Was the prediction right?

Maybe, but maybe not.  The plane landed safely, but were the odds with the passengers?  Was there actually a greater danger that was narrowly avoided? Was there no danger at all?

When a single event is assigned a probability, its hard to assess whether the assigned probability was “correct.”

Suppose every flight departing Dulles was given a 7 in 10 chance of landing safely, rather than just one. The next day, we check the results and find that all flights landed safely. Was the prediction correct?

In this case, we are able to say that the model was clearly wrong. About 1,800 flights depart Dulles airport each day. The model predicted that thirty percent, or about 540 flights, would not land safely. It clearly missed the mark, and by a wide margin.

Probabilistic predictions are easier to evaluate when they apply to a large number of events.

Explaining probability

In the days and weeks leading up to the election, the FiveThirtyEight staff spent a good deal of time trying to put the uncertainty of their forecast in context.  As the election drew closer, these became daily warnings:
  • November 6:  A post outlined just how close the race was, and how a standard polling miss of 3% could swing the election.
  • November 7:  An update called a Clinton win “probable but far from certain.”
  • November 8: The final model discussion outlined all the reasons a Clinton win was not a certainty, and explored scenarios that would lead to a loss.
Despite all this, many people were unable to interpret the probabilistic model, and the associated uncertainty.

Avoiding unrealistic expectations

If  a research scientist at Yale and the MIT Technology Review misunderstood a probabilistic forecast, how well are people in your business doing?
  • Are people in your business making decisions based on probabilistic models? 
  • Are they factoring an appropriate risk level into their actions?
  • Are you doing enough to help them understand the strength of model predictions?
It's important that decision makers comprehend the predictive strength of the models they use. And it’s everyone’s responsibility to make sure they understand.

We have a long, long way to go.


1. See the post Read (or Listen to) Discussions of Analytic Models.  The model discussion I linked to is: A User’s Guide To FiveThirtyEight’s 2016 General Election Forecast

2. “Aggregators" is a term used by the mainstream press to describe data scientists who build models based on polling data.  Here are a few stories that suggested these models were wrong: The WrapVanity FairThe New YorkerQuanta MagazineMIT Technology Review.