By now, no doubt, you've been hearing a ton about the term 'Big Data' -- and you may even be a little sick of hearing about it -- but like many hype cycles, while there is a tendency to throw the term at just about anything that's data-intensive, there is really more to 'Big Data' than a new moniker for marketing teams to bandy about.
In fact, last week at CeBIT
, the enormous European technology and trade show held each year in Hanover Germany, Werner Vogels
, the chief technology officer at Amazon.com spoke about the challenges companies face as they move into this world of big data. As Vogels pointed out, just as the cloud is leaving the hype cycle, along comes big data and fills the breach.
Vogels says we have "moved to a world of data-centric decision making" and as with the cloud before it, people are struggling mightily to understand and define just what that entails.
He defines big data in simple terms as "the collection and analysis of large amounts of data to create a competitive advantage."
But of course there's much more to it and than that and it gets tricky because of what he calls the 3 Vs: volume, velocity and variety. Vogels explained the problems arise because the volume will be an order of magnitude higher than anything you had ever had to deal with before. It will come at you in speeds you have never experienced before and finally it will involve many different data structures, rather than one or two.
As Vogels explained, "When your data sets become so large, you have to start innovating on how to collect, store, organize, analyze and share it."
And this is where the challenge begins to come into play. You have all this data and you have to be able to manage it and use it, and the larger the data sets the more tricky that becomes.
One of the ironies of big data, however, is that the more data you have, the better and the more refined results you can generate. Vogels explained that Amazon has been at this for 15 years and it still makes plenty of mistakes such as when you buy an oil filter and they recommend you buy an album by the Jackson 5 or you buy Windows 7 and they suggest Japanese steak knives.
How can they get it so wrong?
Vogels says it's not because they have too much information, but too little. "The more information you have the better you can make recommendations," he said.
He says, it's important to understand as you enter this world of big data, that there should be no limits on storage and processing -- and this is a good thing for his company because Amazon Web Services
would be happy to sell you as much server and storage space as your company requires.
That said, it left me wondering where does small data end and big data begin? Practically all of us use data in some way every day. As a journalist for instance, I use analytics to measure how many people have viewed my story and I can get a wealth of other details too, but is this big data or just regular data?
There are no easy answers, but Amazon spokesperson Matt Lambert says there is not really a hard line that defines when you cross over into Big Data. Ultimately, it really depends on your company's requirements. "Big Data can mean many things to many different people. For one company 20 gigabytes might be what they consider Big Data and for others, as is the case with Yelp which stores around 100GB of logs per day, Big Data is substantially larger."
He adds, "With this in mind there isn’t an arbitrary limit that is set that defines when something changes from being regular data and suddenly becomes Big Data."
Whether there is a clear line or not, however, it's obvious that we are entering a world in which data will be a big business driver and being able to collect, store, analyze and manage that data is going to be a big differentiator moving forward -- and you need to understand this and be ready.
And that's not hype. It's reality.