Big Data Explained

Big Data explained! Big Data is everywhere. Everyone knows it, everyone is talking about it. But what exactly is Big Data?

“How big, really, is Big Data? This is actually a very intriguing question whose answer seems to lack consensus at the moment but whose ambiguity has not stopped the use of the term. A common misconception, however, is that big data refers solely to the size of the data: if it is data and it is big then it must be big data. While size is certainly an element of the equation, there are other aspects or properties of big data not necessarily associated with size.”

Big Data is used in the singular and refers to a collection of data sets so large and complex, it’s impossible to process them with the usual databases and tools. Because of its size and associated numbers, Big Data is hard to capture, store, search, share, analyse and visualise.

The phenomenon came about in recent years due to the sheer amount of machine data being generated today – thanks to mobile devices, tracking systems, RFID (definition to follow), sensor networks, social networks, Internet searches, automated record keeping, video archives, e-commerce, etc. – coupled with the additional information derived by analysing all this information, which on its own creates another enormous data set.

Companies pursue Big Data because it can be revelatory in spotting business trends, improving research quality, and gaining insights in a variety of fields, from IT to medicine to law enforcement and everything in between and beyond.

Big Data explained: Who’s Generating Big Data

Social media and networks – all of us are generating data eg. Social media streams, web log files, customer sentiment

Mobile devices – tracking all objects all the time eg. Vehicle tracking, smartphone data collectors, workforce tracking, geofencing

Scientific instruments– collecting all sorts of data eg. Satellites, frame cameras, drones

Sensor technology and networks – measuring all kinds of data eg. Weather sensors, stream gauge measurements, heavy equipment monitors

Big data initiatives span four unique dimensions

Volume: Nowadays’large-scale systems are awash with ever-growing data, easily amassing terabytes or even petabytes of information

Velocity: Time-sensitive processes, such as bottleneck detection and service QoS prediction, could be achieved as data stream into the system

Variety: Structured and unstructured data are generated in various data types, making it possible to explore new insights when analyzing these data together

Veracity: Detecting and correcting noisy and inconsistent data are important to conduct trustable analysis. Establishing trust in big data presents a huge challenge as the variety and number of sources grows

2.7 Zetabytes of data exist in the digital universe today.
Facebook stores, accesses, and analyzes 30+ Petabytes of user generated data.
Akamai analyzes 75 million events per day to better target advertisements.

Big Data explained: A zettabyte is equal to 1 billion terabytes

A 2011 study predicted that roughly 1.8 zettabytes (say what? A zettabyte is equal to 1 billion terabytes. A terabyte is equal to 1024 gigabytes) of data would be generated in that year alone. That’s the same amount of data that would be created if everyone in the U.S. posted 3 Tweets every 60 seconds for a little under 27,000 years.

A health care consultancy has made the data coming out of medical practices the focus of its thriving business. The company collects billing and diagnostic code data from 10,000 doctors on a daily, weekly and monthly basis to create a virtual clinical integration model. The consulting company analyzes the data to help the groups understand how well they are meeting the FTC guidelines for negotiating with health plans and whether they qualify for enhanced reimbursement based on offering a more cost-effective standard of care.

It also sends them automated information to better take care of patients, like creating an automated outbound calling system for paediatric patients who weren’t up to date on their vaccinations.

“Big data is not about the data.” – Gary King Harvard University
Director, Inst. For Quantitative Social Science
Making the point that while data is plentiful and easy to collect,
the real value is in the analytics

Big Data explained: 1 Million Customer transaction per hour

Walmart handles more than 1 million customer transactions every hour, which is imported into databases estimated to contain more than 2.5 petabytes * of data — the equivalent of 167 times the information contained in all the books in the US Library of Congress.

A comprehensive and summarised view of each individual case on her dashboard to see her workload and completed assessments against specific timelines. Open assessments or referral details directly from the dashboard, with electronic storage of documents to upload to the system.

FICO Credit Card Fraud Detection System protects 2.1 billion active accounts world-wide.

The volume of business data worldwide, across all companies, doubles every 1.2 years, according to estimates.
(1 Petabyte = 1000000000000000B = 10005 B = 1015 B = 1 million gigabytes)

* Think of the hard drive on your computer at home having 500 gigabytes. Now multiply that by 2,000!

Summary: What is Big Data

Big data is both a marketing and a technical term referring to a valuable enterprise asset—information.

Big data represents a trend in technology that is leading the way to a new approach in understanding the world and making business decisions.

Big Data is high-volume, high-velocity and/or high-variety information assets that demand cost-effective, innovative forms of information processing that enable enhanced insight, decision making, and process automation.

People send more than 144.8 billion Email messages sent a day.
People and brands on Twitter send more than 340 million tweets a day.
People on Facebook share more than 684,000 bits of content a day.
People upload 72 hours (259,200 seconds) of new video to YouTube a minute

About Datanova: Over the last 12 years Datanova has gained a wealth of experience working with various social service providers. All have varying requirements and localised approaches to executing and delivering care. Our collaborative approach to developing our cloud based data systems in conjunction with our end user community means we always evolve our systems with the direct input of the industry. Creators of FlowLogic a Case Management Solution for Social Services CRM.