Privacy fears grow - Are Government and Industry ignoring civil liberties

Google’s Street View service has contributed massively to the concerns of privacy on the internet.  Not only content with publishing views of individuals going about their legitimate business on line, Google recent announcement to use our search activity on their search engine to target advertisements takes the privacy debate to a new level.

Search information is valuable, allowing firms neatly to target ads to a person’s interests to generate billions in additional advertising revenue. Google, the industry leader, stores personal information for 18 months, and they are not alone as Microsoft’s search engine, Yahoo and AOL all retain search requests for 13 months.

But they are not the only people retaining information.  Today’s revelations by the Joseph Rowntree Reform Trust, that a quarter of all government databases are illegal and should be scrapped or redesigned, throws into sharp relief the amount of information that is now available for a whole range of purposes.

The Trust says that storing information leads to vulnerable people, such as young black men, single parents and children, being victimised.

It says the UK's "database state" wastes billions from the public purse and often breaches human rights laws.

But the government says the report contains "no substantive evidence" on which to base its conclusions. A Ministry of Justice spokesman said the government was "never losing sight" of its obligations under the data protection and human rights acts.

"It takes its responsibilities seriously and will consider any concerns carefully, adapting existing safeguards where necessary," he added.

The government spends £16bn a year on databases and plans to spend a further £105bn on projects over five years but does not know the precise number of the "thousands" of systems it operates, the trust claims.

In the wake of numerous data loss scandals, the cross-party trust - which campaigns for civil liberties and social justice - examined 46 public sector systems.

It said 11 were "almost certainly" illegal under human rights or data protection laws.

And what is the Information Commissioner, Richard Thomas, doing about this? Well apparently nothing.  Maybe it is time for the Commissioner to start to earn his salary and start a formal investigation as to exactly what is being stored and retained by both Government and Industry. 

The trouble with civil liberties is that you don’t miss them until they are gone!

Another Googly hits GMail

Following on from last months Google Search problems, users of Google's popular e-mail service, Gmail, were hit by an outage yesterday.  The service went offline at 0930 GMT with Google saying it was "working hard to resolve this problem".  The problem covered both their general consumers and the paid for business users.  Professional users of Google mail are covered by a service level agreement that promises to be 99.9% operational in any calendar month.  To put that commitment in context, GMail could be down up to 72 hours per month and still meet their commitments! Most commercial email services are run on 99.999% or 5 nines availablity or just 72 minutes of unplanned downtime per month.

According to comScore there are more that 113 million users of Google’s “GMail” service worldwide.  In a statement, Google said "a number of users" were having problems with Google Mail. The problem wasn’t universal with some users unaware of the problems. Users accessing GMail through a third-party email clients configured to send and receive e-mail using the IMAP email protocol were unaffected.

Outages are nothing new to email services and all Google’s competitors have had similar experiences as they developed and certainly Google has invested significantly in their Gmail infrastructure.   According to Google, its e-mail service suffered an average of 10 to 15 minutes of downtime per month in 2008.  The last major outage was in August 2008 when users were unable to use Google Mail for "a couple of hours".

According to comScore, Google has the world's third most popular web mail service behind Hotmail with 283 million users and Yahoo with 274 million e-mail users.

Google will need to improve their target availability if they are to offer the GMail service as a credible alternative to competitive email services.

Time to cut the Internet Bureaucracy.

I am frequently struck by how Internet businesses seem to grow staff numbers on an exponential basis. Employing new people - preferably MBA types and under 35 - in endless non-jobs with little or no impact on the creation of new and exciting areas.  

Yahoo! are a interesting example.  Anybody who comes into contact with that organisation can see that at least 50% of their 15,000 staff could be made redundant and the underlying business would not suffer.  They are hugely overmanned and it seems that the management is either too inexperienced to see how to change things or view a the number of direct reports relates to the bonus they can demand.

Take Google, they now employ over 20,000 people worldwide.  The only business line that is really driving the revenue of the business is the original search business and the associated AdSense revenue.  Today the product is much the same and just as brilliant as it was when I met with Brin and Page when the company had just 7 employees back in 1999.  I suspect that percentage of today's 20,000 employees directly involved in that search and AdSense business line is fewer than 10% of the total workforce.  A friend of mine was telling me about a meeting he recently had with Google.  A recently appointed Director spent 45 minutes trying to explain precisely what her roles and responsibilities were within the company.  Anyone who takes 45 minutes to communicate such a basic aspect of their job is probably trying to desperately cover up the fact it is a non-job.  The same stories can be found in Yahoo!, Microsoft, and eBay.

The major Internet business have frequently fallen into the same trap that ultimately caused the demise of major telcos, steel companies and car manufactures back in the 1980s.  They failed to understand their markets and had grown into huge bureaucracies. This recession maybe a good time for the major Internet businesses to look critically at their businesses and start to cut the deadwood and also look to fund the brightest of their exiting alumni to start new businesses and generate real dividends.

Google to allow bidding on Brands in UK

Google announced today that, as of May 5 2008, it will allow open keyword bidding on all terms in the UK and Ireland, bringing these territories in line with the US and Canada.

Simply put, this means that the search giant has abolished its previous system in which a registered trademark could not be bid on by competitors.

Open keyword bidding was rolled out across the US and Canada in 2004 and means that competitors can now bid on a registered trademark, although they will not be able to use trademarked terms in their anchor text or text snippets.

For many, this is unlikely to be welcome news, although some major price comparison sites could see advantages. Until now, clients in the UK and Ireland have benefited from a large amount of relatively cheap traffic from searches on their brand name. The new move is likely to mean trademark holders and competitors will need to significantly increase their bids for the top spots on crucial brand keywords.

The roll out looks on the face of it a very clear revenue boosting idea by Google and no apparent benefit to the leading brands, users or ISPs with all the revenue flowing to Google. The biggest impact is likely to be in developing keyword navigation from the browser address bar which has traditionally been a source of significant revenue for both MSN via Internet Explore and Google via both Internet Explorer and more explicitly via Firefox.

This technique of Direct Navigation was first pioneered by RealNames between 1999 and 2002 before they closed at the behest of MSN. Simply typing a keyword such as “Amazon” into the browser address bar would take you directly to the Amazon site applicable to the territory setting in your browser. After the demise of RealNames, MSN has taken this traffic to MSN search and benefitted from the sponsored listings in a relative benign way although generating millions of dollars in PPC revenue. Google were quick to spot the opportunity and have similarly “hijacked” this traffic from the address bar to their search results. They took this a further step forward when they signed an exclusive deal with Firefox to send all keyword traffic to either search results or on “exact match” to the site in question. Now they are introducing bidding on these terms where will the “exact match” branded keyword go now.

You may feel that this is an excellent use of the web and Google are performing a fair service. However, the browser address bar is not owned by Google or MSN and any interference of outgoing traffic is the same as ease dropping on a phone call being dialled and redirecting the requested call to an alternative location of their choosing. Typing a keyword into a browser address bar is a “DNS error”. Under law you can only redirect it after a NXD or Non-existent domain error has been generated by the Authoritative Name Servers (ANS). So why has MSN and Google got away with this abuse? Well MSN won the right to be able to do this activity in the US and indeed all such search requests are indeed redirected to their servers in Redmond cost the ISPs significant transit costs. Google on the other hand have no such “get out of jail free card”, but have continued to redirect keyword traffic straight from the browser bar. This is not a cost free service since the ISPs naturally end up picking up the transit costs. Now Google wants a larger slice of the cake by forcing Brand names to bid up their own trademarks.

Two bodies who could deal with this issue, ICANN and the ITU, seem strangely silent on this activity and seem content to let Google and MSN to make unreasonable profits paid for by the ISPs.

Direct Internet Navigation and Error Correction

Direct Navigation and error correction are becoming significant areas of web development, enhancing the Internet surfing experience for users, enabling ISPs to monetise hitherto error traffic, helping brands to grow traffic through keywords and building advertising revenues for PPC and CPA players through the delivery of qualified direct navigation traffic. Today the PPC sponsored click through or the pay for transaction approach is the fiscal model behind all of the current error and keyword redirection companies such as Golog, Barefruit and Paxfire.

Originally pioneered by RealNames in 1998, the user entered a keyword and was taken directly via a paid-for route to a specific web page. Hence ‘jobs’ would be bought by Jobs.com for $1m. Each time the word ‘jobs’ was typed in the browser address bar of IE4, the user was taken to jobs.com website via an interstitial advising the user that they were navigating by a RealNames keyword. RealNames had a database of over 1.5m keywords, all of which were paid for. They were both brand names and a limited number of paid-for generic terms. The user of generics was restricted by MSN to use by them to navigate to MSN results pages. Other examples of direct navigation technology existed including the AOL keywords and Common Names.

The raison d’être of direct navigation was that a study by MIT in 2000 revealed that each time a user was forced to click to the next page to uncover a particular location, 50% of the audience was lost. Hence if a deep site location was 5 clicks away for every 1000 users searching for the initial site only 31 users would follow through. A subsequent study by the University of Boston proved that direct navigation improved eCommerce conversion by between 65% and 300% depending on category .

RealNames was forced into Chapter 7 by MSN because they viewed the rapid growth of keyword use as damaging their search business. Between October 2001 and May 2002 RealNames PPVs rose from 2m per month to 30m per month. The growth was 75% per month.

The direct navigation approach used by RealNames was successful because it had near 100% of user coverage through the embedding with Microsoft IE3 and IE4. This was also its main Achilles heel; since MSN were able to terminate this exclusive contract.

Since 2002 there have been a number of direct navigation solutions based on keywords that have been deployed. Most notably AOL continues with AOL keywords (although their use is restricted AOL subscribers). Yahoo! has deployed their own version on Yahoo! enabled browsers, as have Firefox and Opera.

Two years ago Google moved to introduce a ‘paid-for’ keyword direct navigation system based on the RealNames approach with the so-called “Browse by Name” approach, as well as a browser based spelling correction service using stored URL data. Google offered both Brands and Generics with direct navigation. Since it used the Google enable browser they have dispensed with any interstitial page. Type “Ford Explorer” in Google Browser and it will navigate directly to the deep site Ford Explorer page. Similarly type “Mortgages” it will directly navigate to “UK Mortgages Online”. All these solutions are browser-based. Google moved away from monetising error traffic but maintained its direct navigation approach with it's Google Toolbar and through a commercial relationship with Mozilla to apply direct navigation to results and brand sites to browser bar search on Firefox. The only other major keyword driven system was deployed by J-Word in Japan, which is based on 3721 company's downloadable system. 3721, a Chinese company, was acquired by Yahoo! in 2003 for $120m.

Error correction as a result of mis-spelt domain names and broken links in search results, resulting from 400 and 500 class errors are relatively new phenomena. Initial approaches that have been adopted were limited to the analysis of non existent domains (NXDomain). The first to be introduced was SiteFinder from Verisign using “wild card” technology provided by Paxfire in the US. This technology identified when a non-existent domain was being sought and offered the user the opportunity to buy the erroneous input. However, the service responded to both the “A” record domain name request to the authoritative name servers and “Mx” record which is associated with email. Providing a positive answer to a incorrect email address resulted in opening ISPs SMTP mail systems to virus attack. The service was quickly abandoned after a formal request by ICANN , the Internet’s governing body. This experience has made ISPs very suspicious of any DNS “modifications”. However, the level of NXDomain traffic has been identified as being a significant source of error traffic. Total network error traffic is estimated by Verisign to account for approximately 15% of total network traffic of which NXDomain traffic makes up about 20%. A recent study by Google recently indicated that 404 HTTP errors alone account for 6.96% of published pages on average of the 11.5 billion web pages currently published on the web or some 800 million error pages!

A more sophisticated approach is being rolled out by a British company called Barefruit. Barefruit is aiming to provide a relevant result for nearly 100% of network error traffic by not only dealing with the mis-spelt URL error resulting in an NXDomain response, but also covering the 400 and 500 class errors.

The Barefruit Service
Barefruit has a service which is based on the evaluation of broken links in search results and the identification of DNS errors. It can resolve 400 and 500 errors in particular the eponymous 404 Error as well as isolate NXDomain errors. It does this by utilising a proxy server to store a temporary cache of the page prior to the display of the error page. It then detects the error response and then analyses the broken link data and provides a further set of links rather than an error page. Barefruit estimate that a typical broadband user will generate fifty errors every month.

Barefruit patented solution covers the broadest range of error traffic.

1 Mis-spelled urls (NXDomain)
2 Keywords
3 400 Errors
4 500 Errors

The addition, of direct navigation methodology is relatively simple to achieve by combining the NXDomain redirection patches currently in the public domain with an extended taxonomy.

It differs from the previous direct navigation approaches in that it works at the ISP level and not the browser level, although a browser download is available, only deploying when an NXDomain event occurs in the browser or when a 400 or 500 error occurs in a link.

The Barefruit solution provides the ISP with an opportunity to take back a degree of control over “their” user traffic rather than losing it to third parties and also creates a revenue stream from users clicking through on sponsored links. One significant fillup for Barefruit is the rapid adoption of DPI - Deep Packet Inspection, which allows broadband and cable operators to monitor their web traffic and so identify http errors relatively easily and allow Barefruit to analyse and identify user preferences and hence present highly relevant results instead of errors.

Other Developments

Other developments in ‘Direct Navigation and Error correction’ are being pursued by a number of players although none currently offer a solution as comprehensive as Barefruit.

Google has launched its own keyword system in ‘Beta’ which is a direct click through system running from the address bar of Google enabled browsers. They are also introducing spelling correction at the browser bar level for mis-spelled URLs of the URL data stored in their database. This operates in a predictive mode similar to the methodology used in mobile phone text services. It does not eliminate mis-spellings but does reduce MURL events. The approach is not DNS based but its deployment in Europe is limited by the need to comply with EU Directive EU 2002/53. It also tends to offer the ISP with limited income opportunities.

Yahoo! is also developing a direct navigation service similar to Google’s ‘I feel lucky’ approach, taking users directly to the sponsored website. This is likely to be deployed through Yahoo! enabled browsers and their ISP partners such as BT Yahoo! in the UK. BT Yahoo! enable browsers already use keyword navigation utilising direct navigation. Yahoo! will use a variation in their T&Cs to overcome the issues associated with the EU Directive.

Both Yahoo! and Google suffer from lack of independence in respect of ISPs given Overture’s ownership by Yahoo! and relationship with BT Yahoo! and Google’s recent link up with AOL.

Paxfire, a US based company, has been promoting its mis-spelled URL service to ISPs. The service is based on a proxy server solution and relies on a data update solution similar to the ‘black listing’ of sites deployed by website filtering solutions. It was this service which was the basis of the Verisign SiteFinder service and has been demonstrated to be difficult to configure successfully to avoid key issues. Paxfire's Look-Up Service allows ISPs to capture search terms and misspelled URLs entered into any Internet web browser address bar and deliver relevant content and advertising. Paxfire's technology works at the network level and provides a plug and play platform that requires installation of the Paxfire Look-Up Engine (PLE) in front of a DNS server in the ISP’s network.

Golog is limited to NXDomain events and the current business model in Europe and appears to rely on a “forced redirection” after 4 to 7 seconds. Golog installs a software patch into an ISPs DNS infrastructure which redirects the NXDomain to a Golog analysis server which identifies a key term which is then looked-up against a Golog database and where appropriate Golog monetises its mis-spelled URLs through partnership with Miva (formerly eSpotting)or a PPA partner such as Zanox. The Golog solution does not cache the request and cannot deal with any 400 or 500 error traffic. There are some risks associated with the “forced rediection” approach as it can result in the user feeling that they have received an unrelated result which is a poor user experience and can result in the advertiser being charged for a click that a user hasn’t selected.

Conclusion

The development and interest in direct navigation and error correction is growing at a significant rate. In the competitive on-line advertising market, its user ability and untapped potential, it has the potential to supersede the current listings and search approach as users become more familiar with the ability to use the address bar as a direct navigation tool combined with auto error correction both at the browser and in the search listings. One key aspect of any solution that an ISP may apply lies in offering its customers a route to "opt-out" of any error redirection.

To date only Barefruit has developed a combination of direct navigation and error correction which covers nearly 100% of error traffic and it also enables the ISPs to be offered a very powerful revenue generating service from what had previously been regarded as rubbish traffic.