Thought Leadership


Subscribe to our blog

Your email:

Connect with Ness

Software Engineering Services Blog

Current Articles | RSS Feed RSS Feed

Big data versus instinct fight continues


Data versus intuitionA debate we're likely to see rage throughout 2013 is data versus intuition. As Big Data sources gives us better and larger stores of data, can it produce better results than the human brain or the human gut instinct?

It depends who you ask.

A recent New York Times article, Sure Big Data is Great, But So is Intuition, suggested that maybe Big Data can't solve every problem, and that sometimes human intervention isn't a bad thing. The article says the weak link is actually the human creating the model, which is in itself a simplification designed to measure something easily.

Beyond that, the article, says we have too few qualified professionals. 

This is all true, but when I saw Andrew McAfee, who is referenced in this article, speaking at the Enterprise 2.0 Conference in Boston last June, he suggested that one of the problems with business today is that managers think they know better. He said too many trust their gut over the data sitting in front of them. 

And publisher and thinker Tim O'Reilly put it this way in a Forbes interview we reported on last Spring: "The guy with the most data wins." O'Reilly sees a future where data defines success and he believes the more data you have, the more utility you will find for it.

And course, we've all heard Money Ball references enough to know that data matters. Before Oakland As general manager Billy Beane decided to measure the quality of a ball player based on metrics, scouts trusted their eyes and their instincts to make multi-million gambles on whether a player would be good or not. 

By now, we know data matters in baseball, but there is not an exact science for measuring a ball player's success or failure. Just because we can measure his on-base percentage doesn't mean we can look in his heart and measure his ability to deal with the constant failure that is baseball. You can't measure a guy's personality like you can his fastball or time to the plate, and therein lies the problem.

And as we've learned in baseball, some things are easier to measure than others. When looking at enterprise social networks for example, we could count the number of people on the network, but that wouldn't really tell you how it's helping the organization. How do you measure, for instance, how the network makes it easier to share ideas, to increase innovation or to find experts more easily. All of these are clear benefits of a successful enterprise social network, but it's hard to create metrics to measure these outcomes. 

Sid Probstein, who is CTO and co-founder at Attivio, a company that helps make sense of large amounts of data, wrote on Twitter that intuition should certainly drive experimentation and analysis of all that data. In other words, we need the part of us that is uniquely human to come up with creative ways to deal with the data.

Ultimately, I'm not sure there is a right or wrong answer here. Clearly, the more data we have, the better decisions we can make, but it doesn't mean that we can measure everything or use technology to make every decision. The human factor, for better or worse, will always come into play whether we choose to acknowledge that or not.

Photo Credit: (c) Can Stock Photo
blog comments powered by Disqus