Many years ago, I dreamt of what computer networking could bring to the world and how it would be fantastic if we had unlimited access to all the published works and information in the world. At that time it was still necessary to access Bulletin Board Services through dialup, and the internet was difficult to access, even for computer literate. I had to shelve my dream since there were so many fundamental problems that needed to be solved before something like this could be realized.

There have been so many phenomenal advances since that time. A single example that stood out in my mind was Amazon.com. Amazon was a giant step in opening the door to enlightenment for the masses. It was an instant hit, and remains the most popular way to order books online to this day, and the have a rich API for accessing its information and services.

You can imagine my excitement when I first set my eyes on Wikipedia. I find myself using Wikipedia extensively as a resource for discovering information, and augmenting applications that I write with Wikipedia content. Incidentally all Wikipedia content is now available as XML, see wikimedia.

Even though Amazon, Wikipedia and a plethora of other such data sources exist, the dream of getting to any and all information has not yet been realized and data is trapped in silos. Why? In my mind the internet was invented for sharing information, but there has always been the push-pull relationship of free content vs. pay content. The fear naturally was that that providing information with no immediate reimbursement would be the demise of online businesses. Gradually we are seeing business models emerging that allow for businesses to provide information freely without fear of the kinds of financial ruin we saw in the dot.com bust. The walls blocking access to data are slowly, but steadily coming down.

With the advent of more and more data coming online, it's becoming easier and easier to create rich applications that consume information from disparate sources; that enable creating relationships between the various data sources, and in so doing, allowing creation of applications that are greater than the sum of the parts. Imagine: a forensic expert that needs to find a perpetrator by accessing information from an array of sources to piece together a story. To get an accurate picture of what happened really takes data from many, many sources.

I wrote a Pocket PC application many years ago that took Terraserver map image data and overlaid it with road route information, much like online mapping apps do today. At the request of an airplane pilot, I took it one step further by overlaying doppler radar data, to show weather conditions in a particular area. The pilot had had a specific need in mind that did not yet exist in an application. The point here is that a developer can never predict all the needs that application users will have. To make an application even more useful, apps need to be written with extensibility in mind.

My vision is to have typical applications be able to easily overlay and augment themselves with data previously unknown to the application. Where the application would expose extensibility mechanisms to allow for a host of user-driven interactions. This would enable users to discover things on their own in new and never before seen ways. A simple example of something like this is to allow third-party developers to add functionality to a right click menu. I'm thinking, would it not be interesting for the end-user to pick and choose individual items that can be added to a right click menu?

Trying to make disparate sources of data to work together inevitably leads to the homogenous data problem; it always needs to be massaged to integrate with other information, and generally requires manipulation through code. This makes it cumbersome to accomplish tasks in a timely manner for the developer. Incidentally Dapper is a very cool tool that allows parsing unstructured data from websites.

There needs to exist standards and interfaces for representing and sharing data in a uniform manner, though without complexity. Microformats are a good example (see Oomph) of a simple and clever way of sharing information within an existing framework (HTML/CSS). Today, though, sharing structured and unstructured information is largely an uncharted frontier, and to my knowledge there aren't many accepted standards for accomplishing this today. There is a growing need for data exchange. If you feel strongly about any particular standards that address this need, please comment.

In closing, PopFly is a simple example that enables composing information from a user perspective. PopFly is capable of treating data sources as building blocks that take input and have outputs that can be linked together in a "visual" manner through a designer. I argue that it is the beginnings of a much bigger idea: putting together data in new and unforeseen ways.