Big Data - the coming revolution

Big Data is rapidly becoming the latest big thing in computing. The issue for us all is that its impact will be far beyond the world of computing and will effect every aspect of our lives from retail to health and everything in between. Static databases are becoming dynamic sources of unimaginable insight. The amount of structured and unstructured data that is being produced is just phenomenal. Where once databases were being compiled by user input into structured forms that companies used to provide basic trend and financial information, now we are met with terabytes of unstructured data being accumulated and stored as a result of machine generated interactions whether its in every transaction through supermarket terminals to every image stored by the millions of CCTV cameras that have proliferated in our public and private spaces. What has changed is that the cost of storage has plummeted and combined with the infinite connectivity offered by the Internet, stored data has just snowballed.

With the advent of free database search tools developed by Google such as Hadoop and MapReduce it has become possible for machines to begin to analyse this huge warehouse of unstructured data. Hadoop is a free, Java-based programming framework that supports the processing of large data sets in a distributed computing environment and MapReduce is a programming model and an associated implementation for processing and generating large data sets with a parallel, distributed algorithm on a cluster. These tools enable almost anyone to begin to analyse their data for hidden insights into their business activity or the world around them. More importantly,Big Data has the potential to alter the economics of some of our most important industries.. A recent report by McKinsey suggested that if US health care could use big data creatively and effectively to drive efficiency and quality, the potential value from data in the sector could be more than $300 billion in value every year, two-thirds of which would be in the form of reducing national health care expenditures by about 8 percent. Furthermore they suggested that in the private sector, a retailer using big data to the full has the potential to increase its operating margin by more than 60 percent.  In the public sector, across the developed economies of Europe, government administration could save more than €100 billion ($149 billion) in operational efficiency improvements alone by using big data.

These are figures that should make every CEO, Politician and citizen sit up and take notice.  Big Data has enormous power to change the lives of everyone it touches in ways we cannot begin to understand as yet.

In addition, whilst in the past the analysis and evaluation of data-sets was the preserve of the expert, that is also changing as computers increasingly take on the task of analysis and evaluation and learning.  Computers are not only evaluating the data, they are increasingly "learning" how to improve and and extend our understanding of what the data means.  For example, a computer was given the task of analyzing a vast database of cancer biopsy results and duly endeavored to identify twelve key traits that might suggest cancerous cells.  The issue was that only nine traits had been previously identified in the published medical literature.  The use of machine  evaluating Big Data had moved the science of medical diagnosis on significantly and in doing so potentially advanced our detection of cancer and improving survival rates.

So Big Data cannot be ignored by anyone and the trend is to enable access to such tools to a wider and wider audience enabling every business and public sector body the opportunity to benefit from this key aspect of the third industrial revolution.

Is there a downside? Well the primary impact in the medium term will be to render unemployed many of the professional classes once seen as having jobs for life.  Big Data has the potential to impact on highly skilled job roles which had relied upon experience and expertise built up over many years.  For example, taking the cancer biopsy analysis a stage further, computers will be able to calculate radiology treatment quicker and more accurately for treatment than a highly trained and experienced consultant radiologist.  On the other hand there has already identified an enormous of shortage of data analysts to drive this Big Data revolution.  One thing is clear, where industrial automation and IT capabilities eliminated manual labour and secretarial jobs in the 1980's and 1990's, it will be the highly skilled white collared jobs of consultants across a range of professions in medicine, banking, insurance and engineering which will become vulnerable to Big Data.

Equally there is no going back, just as with previous industrial revolutions the genie is out of the bottle and we need to adapt to take advantage of the opportunities offered by these developments.

Is BT Infinity worth the hassle?

Today BT installed BT's flagship product BT Infinity to our property.  I was looking forward to superfast broadband as promised by BT's adverts.  Unfortunately the service comes with a number of faults.
1. The range of the wireless router appears to be rather less than 4m.  This has resulted in the need to have the router balanced on a bookcase around the corner from the master socket in order to set up a "line of sight" connection to my internet enable TV.  Very poor.
2. The installing engineer from BT Openreach who installed the new master socket has disconnected all my home extensions.  This fault didn't become apparent until after he had left.  Despite numerous calls to BT Customer Service, they seem totally unable to stay on the line and have repeatedly dropped the connection.
When I finally did get through I was told that they could not schedule an engineer visit until next week! To cap it all Gnesh then dropped the line again.

So if I have any advice for potential customers of BT Infinity is go elsewhere.  BT are just not fit for purpose. I would not have "upgraded" had I known what a total fiasco BT would make of a simple installation.

Google to institute only pay if viewed!

Fraud has plagued the on-line advertising market since its initiation. Back in 2009, Click Forensics estimated that for advertisers and ad networks, 14.1% of the clicks on their ads were bogus, and costs them money.

All PPC (pay per click) ad providers are keen to combat click fraud and have sophisticated methods for doing so although it still represents a significant proportion of their income since it is the advertiser who will always end up paying.

 However, on Thursday (13th December 2013) Google announced that it was introducing a new system to deal with how ads are viewed and consequently charged for. “If you are an advertiser and a human being didn’t see your ad, then frankly nothing else matters,” said Neal Mohan, Google’s vice-president of display advertising products at Google. “If you are a marketer, why pay if a human being did not see the ad?”

The global on-line advertising industry is worth $117bn and it is estimated that as many as half of the digital ads that marketers buy are not seen at all, with a large portion only being viewed if a website user scrolls all the way down to the bottom of a web page. This issue of how effective advertising is has always been an issue for all forms of advertising be it TV of Bill Posters.

 Now Google intends to introduce an approach called Active View. Google's new Active View offering is based on an emerging industry standard called IAB/3MS, which states that an ad is only “viewable” if more than 50 per cent of it is visible on the screen for one second or longer. Advertisers will be able to see a report of how many viewable impressions they have received for any given campaign, and this data can be used to inform future campaigns. The Active View system ensures that if your ad is buried "below the fold" and doesn't get seen then you will not be billed and if your on-line ad has been seen for at least one second then you will be billed for that impression.

I have always been rather sceptical about the value of such viewed ads which do not have any "call to action" and certainly attempts to at least bill for a viewing by a human being is a step in the right direction, but it really doesn't make for very effective use of ones advertising dollar, pound or euro. Billing for ads that didn't get seen in the past was always a very dodgy practice and this approach should have been introduced a long time ago. Whilst this may not be click fraud, I do think it continues to call into question the billions spent on adverts which are just about brand awareness on-line and fail to trigger a genuine sales lead. Maybe advertisers and agencies placing the ads need to think rather more carefully about what they are attempting to achieve with their on-line campaigns.

This is a good start, but Google and the other major on-line players need to go much further to clean up this industry and ensure advertisers get the value for money they pay for.

Further reporting:-
Financial Times , The Daily Telegraph , BBC

Has the Mobile Internet finally emerged as a primary retail channel in 2013

Back in 2000, I worked with a former colleague from business school to raise investment for what we now refer to as a location based mobile application. The application was known as ZagMe enabled customers to send offers to users of the service in their location such as a shopping mall  (BBC Zagme News report). Unfortunately the technology platform was based on WAP which proved unreliable and costly to use. Today companies such as Foursquare have taken the concept to a mobile broadband customer base happy to use and act on location based offers. However, 2013 has seen the break through of the mobile device including tablets etc. as a primary channel for online purchasing. Retailers have rushed to adapt to the new format and even bring out their own optimised devices such as Tesco and Aldi launching bespoke low cost tablets. According to mobile experience management platform Artisan (www.useartisan.com ), 77% of consumers intend to make purchases through a mobile and/or tablet app this holiday season.

This statistic suggests that using mobile devices to purchase through sites such as Amazon, John Lewis etc. is more likely to be the norm rather than the exception.  In a recent survey, OFCOM forecast the number of mobile broadband subscribers to be in the region of 5.1 million.  With sales of Tablet computers of every type likely to be the most sought after gift this Christmas, that number is sure to grow dramatically and with it the opportunity for retailers to grow their online sales channel as well as location based sales by offering incentives to bring users into High Street stores.

Sadly all this furious activity in mobile sales comes to late for Zagme which succumbed to the dotcom bust when trying to raise a second round of finance.  Todays location based apps have a much brighter future!

UK ISP, Mobile Internet and Cable Subscriber Numbers - December 2013

Here is an update of the UK ISP market covering DSL and Cable Access market as well as the Mobile Dongle market in the UK.  I have also added in the published figures for 3G and 4G mobile access devices. I have used ITU published data for Broadband usage numbers and Neilson Ratings and ISP published figures to get an accurate picture as well all the reports and disclosures for each of the companies shown below. I believe these figures represent a reasonably accurate representation of the genuine adoption of broadband either via DSL, Cable or mobile dongle. Broadband connections included in this data cover download speeds equal to or faster than 256kbit/s.

Table 1. Broadband ISPs (Source: ITU, Neilsen, OFCOM & ISP Published Figures)

No.	ISP/Provider	User Numbers	% of UK Market (inc Mobile)
1	BT Retail	6,961,000	26.07%
2	Sky Broadband	5,017,000	18.79%
3	Virgin Media	4,488,600	16.81%
4	TalkTalk Group	4,076,000	15.27%
5	Orange	714,000	2.67%
6	Kingston Comms	178,200	0.67%
7	Zen Internet	94,000	0.32%
8	Vodafone UK	85,000	0.31%
9	Thus Group	80,000	0.30%
10	Entanet	70,000	0.26%

BT continues to grow its user base at the expense of the smaller players.  Sky Broadband has continued to grow, primarily by acquisition having taken over O2’s broadband user base.  However, although BT Sport is still in its early days, it is clearly impacting Sky’s organic growth.

The total UK Internet connectivity market is not only made up of Broadband ISPs, but connection is now frequently through 3G and a growing 4G user base. Consequently I have taken the total market as being made up of both the fixed-line broadband and mobile internet access as I think it would be erroneous to suggest that they are not part of the same competitive market. The Broadband ISP market size is estimated to be just under 22 million broadband connections. OFCOM estimates the mobile internet connectivity to be in the region of 5.0 million mobile broadband users in addition the 21.7 million fixed line users.  So the total market is estimated at 26.7million subscribers.  Of these super broadband users make up just 3.7million.

Google Announces New Search Algorithm

Google has just announced their new search algorithm, codenamed Hummingbird.  It is the first major upgrade for three years and is a major step towards semantic web search.

Launched quietly about a month ago, the new algorithm affects about 90% of Google searches. The update is designed to provide more accurate results when faced with natural prose questions from web searchers according to senior vice president of search Amit Singhal.

Google stressed that a new algorithm is important as users expect more natural and conversational interactions with a search engine - for example, using their voice to speak requests into mobile phones, smart watches and other wearable technology.

Hummingbird is focused more on ranking information based on a more intelligent understanding of search requests, unlike its predecessor, Caffeine, which was targeted at better indexing of websites.

 “We just changed Google's engines mid-flight - again” Amit Singhal Senior VP, Google Search.

It is more capable of understanding concepts and the relationships between them rather than simply words, which leads to more fluid interactions. In that sense, it is an extension of Google's "Knowledge Graph" concept introduced last year aimed at making interactions more human.

In one example, shown at the presentation, a Google executive showed off a voice search through her mobile phone, asking for pictures of the Eiffel Tower. After the pictures appeared, she then asked how tall it was. After Google correctly spoke back the correct answer, she then asked "show me pictures of the construction" - at which point a list of images appeared.

SEO just the same – Content is king!

As regards developing successful SEO programmes, really very little has changed.  In order to be successful in SEO the key is still to create relevant and interesting content that delivers real value for their consumers. 

However, a subtle change in how the algorithm views the content is happening whereby the new Google’s ranking algorithm focuses on the context of where content appears against search queries rather than traditional keyword matching.  Hummingbird tries to match documents based on the underlying user intent.  

The key to successful SEO will thus require much better content editing and site writing to ensure that the content is answering a question not just stuffing as many keywords and phrases a text will take.  The semantic web is coming of age with a search engine to match.

Apple on top, but Android's position is growing stonger!

Millennial Media, the Mobile ad platform, has released their latest Mobile Mix report, covering Q2 2013.  The report lists the top 20 mobile devices adding tablets into the mix along with smartphones and feature phones. Apple made devices dominate the rankings taking three of the top four places, however, the increasing importance of Android based machines is crucial. Samsung occupies the number two spot with its Galaxy S phones and gains some additional presence in the top ten thanks to the appearance of the Galaxy Tab and Galaxy Note in the top 10 list. HTC and Motorola lose some representation on the list, however and Amazon debuts with the Kindle Fire ranking number eight overall for impressions.  The fall of alternatives to iOS and Android make the market a two horse race.  Blackberry is clearly in trouble and Windows is failing to make any impression on the market.  Microsoft’s purchase of Nokia is unlikely to turn Windows prospects around.

Top 20 Devices - Ranked by Impressions

The fact that Google Play is now outpacing Apple App store for downloads suggest that the Android is increasingly eating Apple’s lunch.

iOS Continues to dominate mobile market for mobile ads, but for how long?

Most recent reports suggest that Apple's iOS continues to dominate the mobile advertising market in terms of impressions and revenue generation delivering almost 44% of all ad impressions and almost 50% of revenue (Figure 1).  The mobile market is increasingly looking like a two horse race between Apple iOS and Android and whilst on the face of it Apple continues to dominate, the decline of any alternative operating system other than Android suggests that in the near term Android will soon pass iOS as it becomes the de-facto alternative operation system.

Traffic share (mobile phone OS)

OS Share	% of Traffic	% of Revenue
Android	31.24%	28.08%
Phone	30.58%	27.76%
Tablet	0.66%	0.32%
iOS	43.75%	49.36%
IPhone	30.88%	36.44%
iPad	8.04%	10.21%
iTouch	4.83%	2.71%
RIM	3.37%	5.41%
Symbian	5.16%	1.56%
Windows	0.26%	0.30%
Other	16.21%	15.27%

Source: Opera  mediaworks

The challenge presented by Android is strengthening as Google and its partners increase the rate of innovation.  Apple is clearly showing signs of innovation fatigue.  A recent report by Goldman Sachs pointed to several concerns, including “delayed product cycles, supply chain difficulties, product price erosion, and a slower pace of product innovation.” Apple’s ability to continue innovating at the breakneck pace it maintained over the past few years remains a major concern.  The recent performance of Google Play surpassing Apple's App Store downloads particularly in emerging markets underline the direction of travel.

This rebalancing of the market suggests that in the longer term the Android market is set to become the most important mobile advertising market.