What is Big Data?

What exactly is “big data”? Two inter­est­ing posts on the def­i­n­i­tion. First Andrew Brust on ZDNet offers a com­mon definition:

Big Data is about the tech­nolo­gies and prac­tice of han­dling data sets so large that con­ven­tional data­base man­age­ment sys­tems can­not han­dle them effi­ciently, and some­times can­not han­dle them at all.

?He goes on to note that the term “Big Data” is entrenched, even if the def­i­n­i­tion is not set­tled. He also men­tions the terms busi­ness intel­li­gence, deci­sion sup­port, and data mining.

Now the sec­ond arti­cle… Robert Hilliard of Deloitte in an arti­cle It’s Time for a new def­i­n­i­tion of Big Data, notes that

[Wikipedia defines Big Data in this way] “In infor­ma­tion tech­nol­ogy, big data con­sists of datasets that grow so large that they become awk­ward to work with using on-​hand data­base man­age­ment tools”. This approach to describ­ing the term con­strains the dis­cus­sion of big data to scale and fails to realise the key dif­fer­ence between reg­u­lar data and big data.

Hilliard goes on to make two points:

  1. Somee­times big data is small: If you had 100K sen­sors on an air­plane and each took a read­ing every sec­ond for a 1 hour flight, you have big data but it is only 3GB of data. So com­mon data­base and stor­age tech­nol­ogy would work for this.
  2. Large datasets are actu­ally small: If you look at somet­ing like tele­phone calls or inter­net con­nec­tions, there is a lot of data, but it is quite struc­tured and sim­ple to parse using RDBMS (data­base technology).

?He then makes the crit­i­cal point: Big data is more about the com­plex­ity of the data sets, espe­cially as large num­bers of dis­crete data points interact.

For me, I think big data has to do with data sets so large and intri­cate that it’s dif­fi­cult to get mean­ing out of the data in a rea­son­able time frame to act on it. In short, as the quan­tity of data goes up, the abil­ity to ana­lyze and under­stand it at a deep level often decreases. What do you think?

No Comments

Leave a Reply

Your email is never shared.Required fields are marked *