Companies like Yahoo!, Microsoft, and (especially) Google depend on being able to track your browsing habits to better target you with advertisements.  In the past 2 years, the proportion of ads being served through ad tracking networks has climbed from 5% to more than 30%, so these tracking networks like Doubleclick (Google), BlueLithium (Yahoo!) and aQuantive (Microsoft) are the lifeblood of the top web properties.  DoubleClick touches more than 80% of the addressable Internet population.

Tracking people's behavior on the web is incredibly lucrative, so everyone wants a piece of the action:

• FireFox, the popular web browser, has announced plans to track your browsing habits directly from the browser, without needing to use tracking cookies.
• AT&T has announced that they can track your browsing behavior directly off of the wire, without needing to extend the browser or use tracking cookies.  They will sell this information to ad networks, or use it to target their own ads.
• Several other ISPs have already embarked on projects to track people's behavior for ad tracking, supported by startups like NebuAd and Phorm.

With all of these companies fighting over you like a piece of meat, you'd think someone would ask you what you think.  Historically, the companies profiting from tracking your behavior have pointed out that people often say that they want privacy, but then give it up easily.  Google has gone so far as to say that there is "no such thing as complete privacy" and they don't get a lot of credit from the critics for sincerity.

So, must we resign ourselves to this trend?  Five years from now, will your web browser, ISP, and everyone else spy on you with impunity?

The answer seems to be "no".  The first volley was the class-action suit last week against Facebook for the abortive "Beacon" feature.  Facebook's "Beacon" feature wasn't actually intended to invade privacy, and didn't go nearly as far as the examples cited above -- but this case demonstrates that people actually *do* care about privacy, and are willing to take action to protect it.  Likewise, people almost immediately raised the privacy red flag when Yahoo! announced Fire Eagle location tracking service.

Far more interesting to me is the news yesterday that Congress will be taking up legislation to protect privacy online.  This legislation is aimed directly at web tracking networks and software or network operators who track your behavior.

This is huge.  The legislation would require companies to get your permission before spying on you.  A stronger proposal would require that those spying on your behavior allow you to opt-out at any time, and establish the equivalent of a "do not call" list.  There are many details to be worked out, and some well-funded parties with strong incentive to weaken the legislation, so I'm tempering my optimism.  But this is a huge step in the right direction.

What do you think?  Do we need legislation, or will the industry self-regulate?  And do you think that this proposed legislation stands a prayer of making it through the process with teeth intact?

Reader JenST responds to my recent analysis of "smartphone carnage", saying:

"See BT purchased Ribbit.  This could be the first move of the carriers to take control from Google, Microsoft, Apple."

JP Rangaswami, the guy at BT responsible for this acquisition, is one of my favorite bloggers, so I have followed this story with interest.  Although he doesn't talk about work on his blog, he is an extremely lucid and enjoyable thinker who understands this new world deeply.  And the news stories provide enough additional details to start forming opinions.

In short, I think JenST is right.  While it's too early to paint this as a direct assault on Google (or other carriers, for that matter), this is a really strategic acquisition.

The key thing to realize here is that BT is trying to do for carriers (and particularly, voice) what Facebook did for social networks, and what Amazon did for hosting.  This was BT's strategy prior to the Ribbit acquisition, and Ribbit was pursuing this strategy independently.  The essence of this strategy is to open up your platform, and let your developers evolve the difficult parts and edge cases organically.  Rather than trying to design a complex and monolithic platform speculatively before signing up a single developer, you define just the core pieces, and let the platform grow and adapt to real-world needs.

I respectfully disagree with Om, when he speculates that it will be *harder* for Ribbit to attract developers under the BT mantle.  When pursuing a Facebook/Amazon strategy, Ribbit had to convince developers that their network had critical mass -- a key value proposition of Facebook development is the millions of people already on the network.  With the BT partnership, Ribbit is instantly associated with a massive global network.  Now that the backing network is more than a toy (and a carrier with a real business model), Ribbit is a lot more interesting to serious developers.

Sam Dean expresses a slightly different concern, worrying that BT will squash the "openness" of Ribbit.  To be honest, I can't make heads or tails of this.  What exactly was so "open" about Ribbit in the first place?  It certainly wasn't "open source", and if all of the IP were entirely unencumbered, it's hard to imagine what the $105 million acquired.  Furthermore, the SDK currently supports only Flash, which is rather proprietary last I checked.  I am reminded of the recent controversy on FriendFeed over the promiscuous use of the word "open".  Let's just agree that Ribbit has some characteristics of "openness", just like the Facebook API and Amazon EC2/S3; and that this quasi-openness can add a lot of value to BT's walled garden just as Facebook's social graph and Amazon's physical infrastructure derive value from their controlled openness.

So ... will this move be the gasoline that lights a fire in the heretofore controlled world of telco platforms?  I don't know.  BT is clearly hoping to force the hand of other telcos and VOIP providers, just as Facebook and Amazon forced their larger competitors to follow suit.  So long as they nurture their developer community and show some measurable gains, other telcos will certainly need to pay attention.

But it seems that BT wish more to pre-empt a Google, Microsoft, or Apple move as much as disrupt their carrier competitors.  The money quote is JP saying:

"What we expect the competition to do – and we think it will much more likely be the Apples and Googles of the world – is to start positioning around a device [such as the iPhone] or an approach to connectivity [such as Google’s ad-based Web model"

Now, that is strategic thinking.  I am pleased (or am I?) that JP cannot imagine Microsoft being a player in this space.

I can still remember meeting with people from RIMM 10 years ago, as they were a relative upstart compared to Palm, and were feeling out Microsoft’s intentions in the smart phone space.  Since then, a lot has happened.  Lots of new players and threats have emerged, and many have fizzled out after great hype.

All of the recent hype about iPhone 3G would make you think that the Smartphone market is Apple’s to win (from RIMM).  Heck, much of the coverage makes it sound as if the iPhone is the first phone to have a popular developer ecosystem.  CNET even went so far as to predict that the iPhone would replace Facebook and MySpace as the preferred social network targeted by widget vendors and advertisers!  As in the past, the popular consensus is bound to look myopic in hindsight.

In this industry, there are 4 types of players who compete and cooperate:

·         Carriers – companies like AT&T or Orange who sell you communications bandwidth for your phone.  They own spectrum, and lease that spectrum to you.  They want you to A) pay as much as possible for as long as possible, B) not cost them a lot in customer support.  Carriers have the control, since spectrum is a scarce physical resource.  This is why everyone else is interested in “network neutrality” legislation.

·         Handset makers – these are companies like Nokia who make phones and want to get a profit on the hardware sale.

·         Phone OS providers – companies like Microsoft who try to license their software to handset makers.

·         Service providers – companies like Google who make profit when people use their service from a mobile phone.

I won’t take the risk of making predictions that could come back to bite me, but it’s safe to predict that the carnage is far from over.  For the next 5 years at least, the following 5 companies will all be very relevant with significant share.  For each one, I talk a bit about the business model and strategy to help understand how things might play out:

Google

Google Android aims to be an open-source operating system for Smartphones; free to all handset makers.  Google like to say that they “do everything out in the open”, so it was fun to see them get caught being secretive.  But it’s all a bit unfair, since Google is more open than Apple, Microsoft, or RIMM – and in fact that openness may be their downfall.

You see, Android is basically a strategic hedge by Google, to ensure that nobody can tie up the mobile platform with a closed, proprietary system.  Google benefits when everyone develops using web standards and targeting Google services.  Google’s business model is to profit on the services (like search) and use that profit to subsidize the phone user experience.  This is a really strong strategy; and Google’s absolutely dominant ability to monetize their services will make them relevant on all mobile phones, regardless of how many Android units have shipped in 5 years.

Google’s weakness is that their strategy is fundamentally opposed to the strategy of the carriers.  Phone Carriers want you to keep paying your bill, and to use data plans that are more profitable.  They don’t want you calling them with an expensive support call about some 3^rd party application that they didn’t even write.  Carriers may not mind open source, but they want a controlled developer ecosystem.

Symbian (Nokia)

Symbian is the dominant smartphone platform outside of the USA.  The big news recently was that Nokia has purchased Symbian and will open-source the platform. 

Unlike Google, who makes money from the services, the handset makers get their profit from selling the hardware.  An open-source Symbian means that they don’t see a sustainable business model in licensing the handset OS.  While open-source Symbian and Android are a blow to Microsoft’s Windows Mobile strategy (at a minimum, creating some pricing pressure), open-source Symbian is also blow to Google’s Android plans.  The handset makers are wary of Google and want to keep their options open.

RIMM

RIMM owns a proprietary handset, operating system, and services.  Because they control everything except the carrier, they can offer seamless end-to-end experience.  This is why BlackBerries are so strong in the USA.  RIMM knows how to work with carriers, too.

Apple

Apple is essentially copying RIMM’s strategy, controlling a proprietary handset, operating system, and services.  Just like RIMM, they provide a seamless end-to-end experience.  Just like RIMM, they make carriers happy by providing a sexy device that makes it easier for the carrier to sell expensive contracts.

Much has been made about how “innovative” the iPhone SDK and store is, but people apparently forget that Windows Mobile, Palm, and even RIMM have had additional applications available for a long time.  The sort of applications, and the download trends, look a lot like other platforms.  When people mention that Facebook is the #3 download from the App Store, they forget that Facebook released an app (web-based) for iPhone long before the SDK was released, and it was immensely popular.  Windows Mobile recently got two Facebook apps, and installs of the Facebook app for BlackBerry still outnumber iPhone.

What Apple is doing differently is important, though.  By centrally controlling the application store, they give an improved user experience.  And more importantly, they provide a visible brand where people wanting support can call *instead* of calling the carrier.  Apple’s app store will certainly increase expense for the carriers, but less so than the more open strategies of Google or Microsoft.

Apple business model is to profit up-front on the hardware, and break even on the services.  They take a 30% cut of app store revenues and charge a subscription for mobile me, but their primary strategy is to profit on hardware.  This gives them the free cash flow up front.

Microsoft

Microsoft’s business model historically has been to make money licensing our proprietary operating system.  As a platform, we offer C++, .NET, or Silverlight, as well as AJAX.

As the entire company moves toward a software plus services strategy, our mobile strategy combines operating system with services.  This is what the Danger acquisition was about, and it is no mistake that the Live Search app is one of the most popular applications for Windows Mobile.

Search is the Lever

Common wisdom says that this is about Apple and RIMM, but I think this is actually about Google and Microsoft.  It’s true that Google hasn’t shipped a single unit yet, and Microsoft’s primary revenue stream (licensing the OS) seems threatened by open-source Symbian and Android.  And neither company sells a sexy handset to drive cash flow-positive revenue.

But both companies control search engines, and search service is far more monetizable than any of the other services.  The end-to-end experience using the WLS app on Windows Mobile is the sort of experience Google would love to have on all mobile handsets, and you can bet that they will.  Apple getting $0.30 every time someone installs the “Flash Light” application is cool, but the revenues and margins of app store and iTunes store won’t be able to compete with search.  Like iTunes and app store, the Mobile Me service is an attempt by Apple to protect their high hardware margins.

Search will be a critical component of RIMM or Apple experience anyway.  Search is a really hard market to enter, and none of the other contenders will be able to afford the infrastructure necessary.

The New York Times and Times of London this week took two very different views on the issue of online privacy.

The New York Times opines that people (especially people in "terrorist" countries) need to get accustomed to having their activities recorded and judged by concerned fellow citizens.  Their thesis is that privacy is dead, and that this is a "good thing" (tm) because we can all spy on each other and stop bad guys.  This is the same argument against privacy that is made every time a stunned neighborhood in a privacy-loving culture discovers that a predator has been doing bad things in his house next door.

Conversely, the Times of London argues that too little privacy and too much spying by "fellow citizens" leads to mob justice.  They cite the recent example in China of a girl who impulsively recorded herself saying some disrespectful things about the Sichuan quake victims, and was tracked down and harrassed by angry citizens.

As more details of our lives become public and instantly indexed in powerful search engines, such questions are sure to arise again and again.  But I think that both the NYT and Times of London are missing the point.  They both presume that cultural norms and expectations about privacy can be swayed through a process of discourse and debate, or that negative outcomes can be avoided by prescribing policy correctly.

In reality, different cultures have different attitudes toward privacy, and these professed attitudes remain remarkably constant over time.  NYT lecturing Arabs or Austrians to be more like Chinese, or Times of London lecturing Chinese to be more like Austrians, are pointless wastes of ink.

Furthermore, we've learned that a group's cultural attitudes toward privacy are often a poor predictor of how they will behave when presented with specific new technological challenges to privacy.  Austrians may love privacy in principle, but they still give Doubleclick massive amounts of data about their personal browsing habits.  One could argue that this is because they are unaware of the level of tracking that's done, but I suspect that it's in large part because they don't really care as much as they say they do.

Study after study has shown that the attitudes toward privacy which people adopt and profess, do not necessarily translate to action in given situations.  People know how they *should* feel about privacy, and will happily parrot those beliefs -- but they all too often will give up their privacy at a moments whim and ignore warnings when their privacy has been compromised.  One partiularly sobering study showed many New Yorkers giving away their social security number and password to a stranger on the street after being told the information was for an "I Love New York" survey.  The participants' desire to contribute to the "New York Love" led them to eagerly give away very sensitive information.  We find that people readily give away personal information for many similarly impulsive payoffs, such as free access to download some tool or try a hot web site, or to mail a humorous video to a friend.

So, when it comes to privacy, why do so many people profess one thing and do another?  And what can be done about it?  I would love to hear your thoughts.

OK, after being initially skeptical, I’ve become convinced that Yahoo! SearchMonkey has the potential to really change the game in search.  The evidence is mounting that they have really thought this through, and that they intend to disrupt the existing order.  The plan is somewhat crazy, but this just might work.

When SearchMonkey launched, about 6 weeks ago, it seemed that the news was primarily lauded by proponents of RDF who believed Yahoo!’s endorsement of RDF would resurrect their beloved but anemic Semantic Web (with a big “S”) standard and give it legs to finally dethrone the small-“s” semantic web technologies like tags and microformats.  To understand why they were so excited, you need to understand that it is the search engines who strangled RDF in the first place.

7 years ago (when Google was still a serious underdog), I argued  that the search engines completely control the fate of “semantic web” standards, and explained that the major search engines have very little business incentive to support such standards.  You can read the whole whitepaper, but the summary is simple:

1)      Search engines are the dominant way that people query for entities on the web, and it’s nearly impossible to get authors to add the semantics and bootstrap the semantic web if search engines ignore the semantics or promote competing semantics.

2)      When it is difficult to extract semantics from documents, it gives advantage to incumbents with massive scale data centers who can extract semantics from natural language.  It creates barriers to entry for new competitors.

3)      As a top search engine, you want the most useful semantic information stored in a format that your competitors cannot utilize.

For 7 years, my thesis held.  At the time, I lobbied both Google and Microsoft to start indexing RDF (and later microformats).  My hope was that their desire to disrupt the (then) dominant Yahoo! search position would lead to a more open web.  But for 7 years, no search engine was crazy enough to truly adopt open standards for semantics.  In fact, Google even dropped support for meta tag’s rudimentary semantics during that time period.

Then came SearchMonkey.  For the reasons outlined above, indexing RDF and microformats is a pretty crazy underdog disruptive strategy, so I was skeptical.  At first, my skepticism seemed to be justified:

1)      At first, they supported only a handful of partners.  See point #3 above.

2)      The functionality was totally opt-in by consumers, and Yahoo! was doing nothing to evangelize it to average users.  It looked like a silly PR stunt to curry favor with the RDF and microformats camps, and clearly Yahoo! was not putting any wood behind it.

3)      Semantics can only be added by document owners, on their own subdomain.  This immediately favors large incumbents.  See the whitepaper for a description of why author-created metadata is a very weak form of semantics.

In the past 2 weeks, however, the first two reasons for my initial skepticism have been obliterated.  The SearchMonkey gallery has expanded, and there are a number of interesting services already available.  It appears that Yahoo! is promoting services which are not necessarily created by the site authors, which is huge.  Check out the Wikipedia Topics entry, for example.  And the PHP API entry is a perfect example of why opt-in by default was a good choice – I may want my search results to show PHP API entries, but most people do not.  In addition, Yahoo! has started to promote the gallery from the home page of search.yahoo.com, under the customize button.

This isn’t a PR stunt.  These guys are serious.  Yahoo! took the single thing that drives publisher behavior (search engine exposure) and tied it squarely to open semantic standards.

Now, let’s contrast this with the Google approach.  Google were the very first to offer “blended” search results, and much was made of the fact that Google Maps returns microformats on search results page.  But spitting up microformats from your proprietary index is the opposite of consuming microformats to enrich your index.  And the mechanism by which Google attaches semantics to the “plus box” is notoriously opaque.  Watching people beg Matt Cutts for information, insinuate that blended results on SERP amounts to paid placement, or speculate about the algorithm as it changes under their feet (did Google “plus box” really just start scraping hCard?) makes you appreciate the way that Yahoo! does it out in the open.

Google did pay lip service to “out in the open”, when they launched Google Base to much fanfare and started integrating Google Base results into the main search results page.  But Google Base still required publishers to store their content in Google’s servers, and the prominent listing on the search results page quickly became a distant memory and Google Base a black hole with little influence on the main search page.

I think people were a bit confused when Yahoo! claimed that SearchMonkey is a “long tail” strategy.  But the discussion of Google’s contrasts should have made it clear by now that they are right.  Yahoo!’s model of user opt-in makes room for both the default mass-appeal plugins (like Flickr) and the more niche plugins like PHP APIs.

Overall, this is very strong progress in just 6 weeks.  To keep up the momentum, Yahoo! needs to continue promoting to end-users, and should be more aggressive about influencing search results ordering when SearchMonkey plugins are installed.  For example, I have opted-in to the Yelp plugin, but perfectly good Yelp tresults often get pushed off of the page by CitySearch and others.  Random samplings of users who haven’t tried any customizations should be shown enhanced search results pages and offered the chance to customize.

In addition, Yahoo! should allow SearchMonkey plugins to customize results for other pages.  For example, I should be able to see the IMDB information next to a search result for a blog page that reviews a movie.  This would truly bootstrap the use of microformats, since adding a microformat to your page would automatically make it more useful to anyone using Yahoo!’s search engine.  Google tried something similar, with less than stellar results, when they started using scraped addresses from around the web to enhance their map “plus box”.  When Google scraped restaurant addresses from old and outdated sites, the search results page “plus box” started directing diners to the wrong location, leaving restaurant owners bewildered as they tried to figure out where the wrong data was coming from.  Yahoo!’s approach mitigates against this, since people opt-in to the provider, and they know where the data is coming from.  In Google’s approach, you get whatever plugins Google gives you, and you have no idea where they are getting the data.

If Yahoo! share stabilizes or increases, I would expect Google to respond by being more aggressive with their “plus box”, and perhaps embracing and extending, with an eye to extinguishing SearchMonkey.  SearchMonkey will encourage the greatest proliferation of microformats yet seen on the Internet, and as more microformats are available, Google will certainly start to leverage this information more in building their index.

Nik Cubrilovic at TechCrunch just posted a review of Google Gears, predicting Armageddon with the alarming headline, “Get Ready For A New Platform War. Google Gears Drives Straight At Microsoft’s Profits.”  He probably first read Dare Obasanjo’s post, “Google Gears as the Next Flash”, and then let his imagination run wild on scenarios.  Before this hype train goes completely out of control, it’s a good idea to lay out the timeline and facts and let people make more sober judgments about Gears going forward.

Both were reacting to the demo of a gears-enabled MySpace mail client prototype, shown at Google I/O.  The mail client wasn’t substantially different from the blazing fast Silverlight mail client AOL showed at MIX08, but it made news because it was the first time that a large web property has talked about possibly distributing Gears to millions of users.

Google I/O is where two former Microsoft executives keynoted in Googleskin clothing, saying “We grew up on the web, it’s in our DNA”.  One was the guy who battled for SOAP vs. REST at MSFT, and the other the former architect of Win32 and the “ahead-of-its-time” patent the web project.  On the flip side, the former architect of Gears at Google (Danny Thorpe) last week joined MSFT for the second time since shedding his Googleskin and leaving Gears.  All I can say about the DNA in this industry is, “it’s complicated”.

Anyway, these are some smart and visionary guys, so I will admit that I was inspired when they preached that Gears was the best way to move the web forward and fight proprietary runtimes.  Rather than sticking to simple offline storage, they were going to throw in the kitchen sink and any quasi-standards-inspired utility that might be useful to web site authors.  This isn’t a new vision for gears, in fact; nor was Google I/O the first that the strategy leaked into public.

Alex Russell, creator of the Dojo Toolkit, presented at MIX08 and had a lot to say about moving the web forward.  In this Channel 9 interview taken at MIX, Alex lays out the case for Gears as the basis of new bleeding edge browser innovations.  A few weeks later, he elaborated on his thoughts in his “Progress is N+1” post, neatly teeing up the next 3 months of Gears evangelism from people repeating his points.

Now that we’ve traced the influences behind Nik’s post, let’s refute a few of the claims and analyze the situation:

·         Targets Microsoft’s Profits:   This claim is very puzzling.  If anything, Gears would compete with proprietary features of Internet Explorer.  However, 100% of IE developer innovations are now squarely web standards (and creative commons where applicable),  and we don’t make any revenue from the browser anyway (let alone, profit).   Anyone who speculates that IE strategy is to drive proprietary web APIs or to drive revenue, is operating from zero evidence.  Google’s goal seems to be to ensure that the browser platform remains commoditized – a strategy we’ve already been betting on for years.

·         “Based on a True Standard”:  Fans will tell you that Gears will eventually be better than Flash or Silverlight because “Parts of Gears are based on standards and might be standardized in the future”.  When asked why they don’t operate strictly through the proven standards processes that created HTML, XML, JavaScript, and the like, they respond with slurs about how W3C is ineffective and standards bodies produced abominations like EJB.  The fact is, it is way too early to tell if Gears will help web standards or cause fragmentation of web standards efforts.  When someone argues that Gears is completely innocuous to W3C or Mozilla, you need to take that with a healthy dose of skepticism.

Overall, I think Gears is an interesting approach.  When developer innovations spread uniformly across all of the web browser clients, everyone benefits.  This is why Mozilla would talk about porting a scripting engine to IE, why Microsoft would release Activities spec under creative commons or contribute CardSpace code to other browsers.  And there are very few organization who can build cross-platform, cross-browser extensions of this complexity and deploy securely and stably to millions of machines.  The Gears engineers are world-class.  It’s interesting that this effort is being created separate from Mozilla, since there is nothing technically preventing Mozilla from porting arbitrary functionality to other browsers and devices.

But it’s also an added distraction.  Even if (and it’s a big “if”) Gears succeeds in driving forward a few innovations that make it into all browsers and provoke a few choruses of kumbaya, that won’t absolve the browser vendors from the extensive cooperation we are already doing.  Mozilla, Microsoft, Opera, Safari etc. will still have to continue working together to determine which innovations we take on behalf of the web developers.  W3C and ECMA will still need to decide which things become standards (or “recommendations” in W3C parlance).  Things already got pretty confusing with the initial ambiguity between WHAT-WG and W3C HTML5, and again when the political lines between ECMAScript and ActionScript were being tested.  As well-meaning as they may be, any new group of people creating new stuff and claiming that they represent the standards, just makes life more complicated for all of the browser vendors and potentially slows innovation as people are forced to sort through the mess.  The way this turns out is far from certain, but I’ll keep an open mind.