SearchMonkey is Disruptive

Joshua Allen — Tue, 24 Jun 2008 18:07:00 GMT

OK, after being initially skeptical, I’ve become convinced that Yahoo! SearchMonkey has the potential to really change the game in search. The evidence is mounting that they have really thought this through, and that they intend to disrupt the existing order. The plan is somewhat crazy, but this just might work.

When SearchMonkey launched, about 6 weeks ago, it seemed that the news was primarily lauded by proponents of RDF who believed Yahoo!’s endorsement of RDF would resurrect their beloved but anemic Semantic Web (with a big “S”) standard and give it legs to finally dethrone the small-“s” semantic web technologies like tags and microformats. To understand why they were so excited, you need to understand that it is the search engines who strangled RDF in the first place.

7 years ago (when Google was still a serious underdog), I argued that the search engines completely control the fate of “semantic web” standards, and explained that the major search engines have very little business incentive to support such standards. You can read the whole whitepaper, but the summary is simple:

1) Search engines are the dominant way that people query for entities on the web, and it’s nearly impossible to get authors to add the semantics and bootstrap the semantic web if search engines ignore the semantics or promote competing semantics.

2) When it is difficult to extract semantics from documents, it gives advantage to incumbents with massive scale data centers who can extract semantics from natural language. It creates barriers to entry for new competitors.

3) As a top search engine, you want the most useful semantic information stored in a format that your competitors cannot utilize.

For 7 years, my thesis held. At the time, I lobbied both Google and Microsoft to start indexing RDF (and later microformats). My hope was that their desire to disrupt the (then) dominant Yahoo! search position would lead to a more open web. But for 7 years, no search engine was crazy enough to truly adopt open standards for semantics. In fact, Google even dropped support for meta tag’s rudimentary semantics during that time period.

Then came SearchMonkey. For the reasons outlined above, indexing RDF and microformats is a pretty crazy underdog disruptive strategy, so I was skeptical. At first, my skepticism seemed to be justified:

1) At first, they supported only a handful of partners. See point #3 above.

2) The functionality was totally opt-in by consumers, and Yahoo! was doing nothing to evangelize it to average users. It looked like a silly PR stunt to curry favor with the RDF and microformats camps, and clearly Yahoo! was not putting any wood behind it.

3) Semantics can only be added by document owners, on their own subdomain. This immediately favors large incumbents. See the whitepaper for a description of why author-created metadata is a very weak form of semantics.

In the past 2 weeks, however, the first two reasons for my initial skepticism have been obliterated. The SearchMonkey gallery has expanded, and there are a number of interesting services already available. It appears that Yahoo! is promoting services which are not necessarily created by the site authors, which is huge. Check out the Wikipedia Topics entry, for example. And the PHP API entry is a perfect example of why opt-in by default was a good choice – I may want my search results to show PHP API entries, but most people do not. In addition, Yahoo! has started to promote the gallery from the home page of search.yahoo.com, under the customize button.

This isn’t a PR stunt. These guys are serious. Yahoo! took the single thing that drives publisher behavior (search engine exposure) and tied it squarely to open semantic standards.

Now, let’s contrast this with the Google approach. Google were the very first to offer “blended” search results, and much was made of the fact that Google Maps returns microformats on search results page. But spitting up microformats from your proprietary index is the opposite of consuming microformats to enrich your index. And the mechanism by which Google attaches semantics to the “plus box” is notoriously opaque. Watching people beg Matt Cutts for information, insinuate that blended results on SERP amounts to paid placement, or speculate about the algorithm as it changes under their feet (did Google “plus box” really just start scraping hCard?) makes you appreciate the way that Yahoo! does it out in the open.

Google did pay lip service to “out in the open”, when they launched Google Base to much fanfare and started integrating Google Base results into the main search results page. But Google Base still required publishers to store their content in Google’s servers, and the prominent listing on the search results page quickly became a distant memory and Google Base a black hole with little influence on the main search page.

I think people were a bit confused when Yahoo! claimed that SearchMonkey is a “long tail” strategy. But the discussion of Google’s contrasts should have made it clear by now that they are right. Yahoo!’s model of user opt-in makes room for both the default mass-appeal plugins (like Flickr) and the more niche plugins like PHP APIs.

Overall, this is very strong progress in just 6 weeks. To keep up the momentum, Yahoo! needs to continue promoting to end-users, and should be more aggressive about influencing search results ordering when SearchMonkey plugins are installed. For example, I have opted-in to the Yelp plugin, but perfectly good Yelp tresults often get pushed off of the page by CitySearch and others. Random samplings of users who haven’t tried any customizations should be shown enhanced search results pages and offered the chance to customize.

In addition, Yahoo! should allow SearchMonkey plugins to customize results for other pages. For example, I should be able to see the IMDB information next to a search result for a blog page that reviews a movie. This would truly bootstrap the use of microformats, since adding a microformat to your page would automatically make it more useful to anyone using Yahoo!’s search engine. Google tried something similar, with less than stellar results, when they started using scraped addresses from around the web to enhance their map “plus box”. When Google scraped restaurant addresses from old and outdated sites, the search results page “plus box” started directing diners to the wrong location, leaving restaurant owners bewildered as they tried to figure out where the wrong data was coming from. Yahoo!’s approach mitigates against this, since people opt-in to the provider, and they know where the data is coming from. In Google’s approach, you get whatever plugins Google gives you, and you have no idea where they are getting the data.

If Yahoo! share stabilizes or increases, I would expect Google to respond by being more aggressive with their “plus box”, and perhaps embracing and extending, with an eye to extinguishing SearchMonkey. SearchMonkey will encourage the greatest proliferation of microformats yet seen on the Internet, and as more microformats are available, Google will certainly start to leverage this information more in building their index.

A Data Center Built With Shipping Containers

Joshua Allen — Wed, 02 Apr 2008 20:06:00 GMT

Yesterday, Microsoft became the first in the industry to announce a data center designed entirely with shipping container-based server modules. It's a sizable data center, supporting somewhere between 150,000 an 440,000 servers. Mike Manos revealed some specifics about the Chicago data center at a talk at Data Center World.

James Hamilton has some commentary on his blog. James has been evangelizing the advantages of shipping container-based data centers for years, inside the company and within the industry. As he points out, several of the largest equioment companies are now providing containers, and you can be sure that we are not the only company buying them. But this is the first time anyone has talked publicly about a real industry-scale data center designed from the ground up for containers. I'm particularly interested by the fact that the containers use angle parking instead of being arranged at right angles.

This is a big accomplishment, and no doubt a sign of where the rest of the industry will be moving in coming years. Congratulations to Mike, James, and the rest of the team!

Greg Linden: Trends in Collective Intelligence and Centralization

Joshua Allen — Tue, 18 Mar 2008 22:02:00 GMT

Rich user experiences are going mainstream on a wider variety of devices and putting pressure on web standards. This trend is changing the face of the web as we know it, and we've covered this trend extensively here and at the MIX conferences. But there is another trend that is also changing the web forever, and Ray Ozzie's keynote at MIX08 was a shot across the bow regarding Microsoft's response to this seismic shift.

In his keynote, Ray talked about the "mesh", and the inexorable shift of services into large centralized data centers. We announced some cloud storage services, and discussed our philosophy of keeping control at the edges. This is really just the beginning, and we'll be having a deep conversation with the industry over the next year.

To kick off the conversation here on MIX Online, I asked Greg Linden to share his broad industry perspective about some of these topics. Greg led development of Amazon's ground-breaking recommender systems, created Findory, and recently joined Microsoft to work on some top-secret incubation projects. He continues to run the popular "Geeking with Greg" blog, where he riffs on large-scale centralized computing, data mining, and "collective intelligence".

Just a few of the topics we talked about:

Is MapReduce/Hadoop really as good as SQL?
What are the limits of social search?
What good is collective intelligence, anyway?
Will all of the world get sucked into one or two datacenters?

Entries tagged with search - MIX Online

SearchMonkey is Disruptive

A Data Center Built With Shipping Containers

Greg Linden: Trends in Collective Intelligence and Centralization