The concept of big data is one that gives us the ability to harness structured and unstructured data, ask provocative questions about it, and start attacking some of the world's most difficult challenges.
The Whatis? definition of big data is a general term used to describe the voluminous amount of unstructured and semi-structured data a company creates -- data that would take too much time and cost too much money to load into a relational database for analysis.
Vendors and web providers have clamoured around it, bidding to make it a core part of their IT offerings. IBM describes it like this: “Big data is more than a challenge; it is an opportunity to find insight in new and emerging types of data, to make your business more agile, and to answer questions that, in the past, were beyond reach.”
It's so important, argues consultants Deloitte, organisations must jump on the big data bandwagon now to protect themselves from falling behind competitors or perish.
how much of a challenge is it to enterprises?
Market research firm IDC has released a new forecast that shows the big data market is expected to grow from $3.2 billion in 2010 to $16.9 billion in 2015.
Its Worldwide Big Data Technology and Services 2012-2015 Forecast is a new frontier in IT where data sets can grow so large that they become awkward to work with using traditional database management tools. It projects that the increased investment needed to support Big Data represents a compound annual growth rate (CAGR) of 40%, or about seven times that of the overall information and communications technology (ICT) market.
It is for this reason that enterprises need new and more tools, frameworks, hardware, software and services to handle this emerging issue.
Over half a billion dollars in venture capital has already been invested in new big data worldwide and IDC predicts that the growth in appliances, cloud computing and outsourcing deals for big data technology will likely mean that over time end users will pay increasingly less attention to technology capabilities and will focus instead on the business value arguments.
new challenges for enterprises from big data
New experience will be required to cater for this new boom; a further challenge enterprises must equip themselves for. Add to this the fact that tooling up to handle this stuff requires major investment in computer hardware and software, and it seems the cost of the investment could reach proportionally high ranges.
Research by MGI and McKinsey's Business Technology Office, found that it will become a key basis of competition, underpinning new waves of productivity growth, innovation, and consumer surplus. Leaders in every sector will have to grapple with the implications of big data, not just a few data-oriented managers.
governments leading the way
It cites the example of how US healthcare could create more than $300 billion in value every year by mining the data correctly. Indeed, the Obama administration recently announced plans to spend hundreds of millions of dollars on a new "big data initiative" for research and development into technology to "access, store, visualize, and analyze" massive and complicated data sets - Learn more about ongoing Federal government programs that address the challenges of, and tap the opportunities afforded by, the big data revolution in its Big Data Fact Sheet.
Tom Kalil, deputy director for policy at the White House Office of Science &Technology Policy issued a clarion call for support in big data: “Clearly, the government can’t do this on its own. We need what the President calls an “all hands on deck” effort.”
Progress is slowly being made in encouraging this. Some 75% of enterprise decision makers polled in a recent survey believe that leveraging data will help their companies dramatically improve their business, yet more than half feel they currently lack the tools to mine true customer insights from the data generated by digital and offline efforts.
What do you think? Is it time to invest in big data or are you taking a wait-and-see approach?
image © rubysoho - Fotolia.com
Joe Fernandez is a technology writer and blogger for Futurity Media. As a journalist, he was an editor on Computer Weekly and Microscope magazines and worked as a deputy editor for Marketing Week and its sister title Pitch covering online marketing and social media developments. Joe has also appeared in titles including New Media Age, Guardian Computing, Computing Magazine, The Inquirer and Mobile Magazine.