Page Impressions Ltd Blogcetera: October 2014

Wednesday, October 08, 2014

Big Data - the coming revolution

Big Data is rapidly becoming the latest big thing in computing. The issue for us all is that its impact will be far beyond the world of computing and will effect every aspect of our lives from retail to health and everything in between. Static databases are becoming dynamic sources of unimaginable insight. The amount of structured and unstructured data that is being produced is just phenomenal. Where once databases were being compiled by user input into structured forms that companies used to provide basic trend and financial information, now we are met with terabytes of unstructured data being accumulated and stored as a result of machine generated interactions whether its in every transaction through supermarket terminals to every image stored by the millions of CCTV cameras that have proliferated in our public and private spaces. What has changed is that the cost of storage has plummeted and combined with the infinite connectivity offered by the Internet, stored data has just snowballed.
With the advent of free database search tools developed by Google such as Hadoop and MapReduce it has become possible for machines to begin to analyse this huge warehouse of unstructured data. Hadoop is a free, Java-based programming framework that supports the processing of large data sets in a distributed computing environment and MapReduce is a programming model and an associated implementation for processing and generating large data sets with a parallel, distributed algorithm on a cluster. These tools enable almost anyone to begin to analyse their data for hidden insights into their business activity or the world around them. More importantly,Big Data has the potential to alter the economics of some of our most important industries.. A recent report by McKinsey suggested that if US health care could use big data creatively and effectively to drive efficiency and quality, the potential value from data in the sector could be more than $300 billion in value every year, two-thirds of which would be in the form of reducing national health care expenditures by about 8 percent. Furthermore they suggested that in the private sector, a retailer using big data to the full has the potential to increase its operating margin by more than 60 percent.  In the public sector, across the developed economies of Europe, government administration could save more than €100 billion ($149 billion) in operational efficiency improvements alone by using big data.
These are figures that should make every CEO, Politician and citizen sit up and take notice.  Big Data has enormous power to change the lives of everyone it touches in ways we cannot begin to understand as yet.
In addition, whilst in the past the analysis and evaluation of data-sets was the preserve of the expert, that is also changing as computers increasingly take on the task of analysis and evaluation and learning.  Computers are not only evaluating the data, they are increasingly "learning" how to improve and and extend our understanding of what the data means.  For example, a computer was given the task of analyzing a vast database of cancer biopsy results and duly endeavored to identify twelve key traits that might suggest cancerous cells.  The issue was that only nine traits had been previously identified in the published medical literature.  The use of machine  evaluating Big Data had moved the science of medical diagnosis on significantly and in doing so potentially advanced our detection of cancer and improving survival rates.
So Big Data cannot be ignored by anyone and the trend is to enable access to such tools to a wider and wider audience enabling every business and public sector body the opportunity to benefit from this key aspect of the third industrial revolution.
Is there a downside? Well the primary impact in the medium term will be to render unemployed many of the professional classes once seen as having jobs for life.  Big Data has the potential to impact on highly skilled job roles which had relied upon experience and expertise built up over many years.  For example, taking the cancer biopsy analysis a stage further, computers will be able to calculate radiology treatment quicker and more accurately for treatment than a highly trained and experienced consultant radiologist.  On the other hand there has already identified an enormous of shortage of data analysts to drive this Big Data revolution.  One thing is clear, where industrial automation and IT capabilities eliminated manual labour and secretarial jobs in the 1980's and 1990's, it will be the highly skilled white collared jobs of consultants across a range of professions in medicine, banking, insurance and engineering which will become vulnerable to Big Data.
Equally there is no going back, just as with previous industrial revolutions the genie is out of the bottle and we need to adapt to take advantage of the opportunities offered by these developments.