big opportunities in big data

Picture the scene: it's a rainy Tuesday in New York, and before Sidney gets up to start his job, he has a call from his doctor telling him he urgently needs to pick up some medication. If he doesn't, he will start feeling poorly in about four hours.

 

How did the doctor know this? Mining of big data provides the answer. Earlier that morning, transit numbers from Sidney's usual metro station were much lower. Analysis by station staff found that the mobile phones normally used to purchase tickets at that station in the morning were all located in Sidney's building. At the same time, a check of the water system in his building found a contaminant in his building's plumbing. Most of the residents had contacted the doctor about their stomach bug symptoms. 

 

Through analysis of all this data collated across various touchpoints, the doctor was able to determine that Sidney would be next if he did not start precautionary treatment. Evidently, big data has a huge role to play in the future for both consumers and enterprises.

 

Tony Jebara, director of the Machine Learning Laboratory at Columbia University, defines big data as information collected from every consumer touchpoint with a brand or service and not just traditional business analytics: 

 

"It's generated from social interactions and social data, primarily. The photographs, the tweets, the blogs, the emails, the check-ins, the communication data, and the mobile phone data."

 

Big data huge will have huge ramifications for enterprises. It will be driven by a desire from firms to ask difficult questions from all of their data. Joe Kelly, CEO of Infochimps, which provides social data sets and big data services for clients, explains how new technology will enable this change. 

 

"[Before] you'd have to buy a bunch of boxes and then have expensive database administrators guarding it. These days, with open platforms, the speed is higher and the cost is lower."

 

Just as how big data helped prevent Sidney from getting a stomach bug in our earlier example, enterprises can mine data and help realize business objectives or influence marketing campaigns in an effective and efficient way. 

 

For example, consumer-facing services like Netflix is using big data to try and understand what movies a customer might like and Clickfox uses big data to analyze information about customer interactions and provide companies with insights about customer satisfaction, segmentation, and revenue generation.

 

Figures suggest that these eye-opening realizations could increase revenues by 10% quarter-on-quarter. 

 

Kelly predicts that over the next few years, the cost of storing and processing that data will continue to fall, opening up opportunities for mid-range and smaller businesses to ask questions of their own.

 

Major vendors such as IBM are helping businesses to realize these benefits. It has worked with law enforcement agencies to understand all the different social data input methods and the relationship information that arises from these unstructured data searches. This is helping to identify fraudulent and other criminal activity. 

 

The biggest promise for big data, though, lies not in analyzing the past, but rather, in predicting the future. Newtonian physics relied on determinism, i.e. once something had been set in motion, its future path is predictable based on the forces that acted upon it. Big data takes this idea to social information; if you have enough of it, you can get a better picture of what will happen next.

 

Jebara cites the pharmaceutical sector as an example of this. "Companies could see how people are taking medication, and look at the joint effects that they haven't been able to explore. So if someone is taking a combination of three medications, then you could figure out that it's bad [for them]." 

 

Academics are already testing this out. Researchers at the University of California have developed an algorithm to predict what the stock market will do next, based on Twitter data. 

 

Online analysis tools such as Amalgmood and Recorded Future are already helping firms mine thousands of social text extracts and other data sources around the web, to collate information about what is happening now, and to predict what may happen in the future.

 

As these analytic and predictive capabilities come together, we will begin to realize even more opportunities in big data. One of the biggest challenges will be sourcing data science experts who know not only how to find the answers in this data - but also how to identify the right questions.