Articles, Opinions & Lab - MIX Online

Architecture Of The Archivist

Karsten Januszewski — Mon, 16 Aug 2010 17:46:07 GMT

Building The Archivist introduced several architectural difficulties, which were solved over a number of iterations and trials and error.

Implementing the core three features of The Archivist (archiving, analyzing and exporting) in a scalable and responsive way proved to be a challenge. While I’m tempted not to write this article as some will anticipate my mistakes, I’m going to suck it up for the sake of maybe helping someone else out there.

With this in mind, I’d like to go over the journey of The Archivist architecture with the hope that others may benefit from my experience.

If you don’t know what The Archivist is, you can read about here.

Architecture I: Prototype

My first architecture was a prototype meant to prove the feasibility of the application. Like many prototypes, I wrote this one quickly to prove the basic idea.

I started out with an instance of SQL Express and created four tables:

A User table. Because the archivist allowed different users to create archives of tweets, I needed a user table.
A Tweets table. Of course I needed a table to store tweets.
A Search table. In this table, I stored the searches: search term, last updated, is active, etc. It was joined to the tweets table.
An Archive table. This table was basically a look up table between searches and users. Because two users might start the same search, there was no point in duplicating that search, thus the creation of the archive table.

In fact, here’s exactly what it looked like:

I whipped up some data access objects using Linq To Sql and I was off and running. Tweets were pulled from Twitter, serialized to my data access objects and inserted into the database. To do the analysis, I pulled tweets out of SQL back into CLR objects and ran LINQ queries on demand at runtime to create the various aggregate graphs of data. To export the tweets to Excel, the graph of CLR objects was transformed to a tab delimited text file on the fly.

It was working great in my development environment with one user, one search and about 100 tweets.

I got this basic architecture working with the Azure development environment and then pushed it up to Azure. I had a Web Role for the rudimentary UI and a worker role for polling Twitter.

I then started doing some very basic performance and load tests. Guess what happened? First, performance crawled to a halt. Who can guess where the bottlenecks were? Well, first, the on demand LINQ queries were painfully slow. So was the export to Excel. Also, the performance of inserting tweets into SQL was a surprise bottleneck I hadn’t seen in my development environment.

Performance was a problem, but my database was filling up too… and fast. I had created the default SQL Azure instance of 1 gig. But three archives of 500,000 tweets could fill that up. I was going to fill up databases quickly.

Back to the drawing boards.

Architecture II: SQL-centric

I had always been concerned about the on-demand LINQ aggregate queries and expected they wouldn’t scale. As a result, I looked into moving them to SQL Server as stored procedures. While that helped performance, it still didn’t meet the speed benchmarks required to show all six visualizations in the dashboard. I then got turned on to indexed views. Ah ha, I thought! The solution to my problems! By making the stored procs indexed views in SQL, I thought I had the solution.

But I had another problem on my hands: the size of the SQL databases. My first thought was to have multiple databases and come up with a sharding structure. But the more I looked at the scalability needs growth of The Archivist, the more skeptical I became of this architecture. It seemed complex and error-prone. And I could end up having to manage 100 SQL Servers? In the cloud? Hmm…

I also had performance problems with doing inserts into SQL Azure. In part, this had to do with the SQL Azure replication model, which does three inserts for each insert. Not an issue with a single insert, but it wasn’t looking good for bulk inserts of tweets.

So, SQL could solve my analysis problems but didn’t seem like a good fit for my storage problems.

That’s when I called a meeting with Joshua Allen and David Aiken. Between the three of us, we came up with a new architecture.

Architecture III: No SQL (Well, Less SQL)

The more astute among you probably saw this coming: We dumped SQL for storing tweets and switched to blob storage. So, instead of serializing tweets to CLR objects and then persisting them via LINQ to SQL, I simply wrote the JSON result that came back from the Twitter Search API right to blob storage. The tweet table was no more.

At first, I considered moving the rest of the tables to table storage, but that code was working and not causing any problems. In addition, the data in the other three tables is minimal and the joins are all on primary keys. So I didn’t move them, and never have. So it turns out that The Archivist has a hybrid storage solution: half blob storage and half SQL server.

You might be wondering how I correlate the two stores. It’s pretty simple: Each search has a unique search id in SQL. I use that unique search id as a container name in Blob Storage. Then, I store all the tweets related to that search inside that container.

Moving from SQL to Blob Storage immediately solved my two big archival problems:

First, I no longer had to worry about a sharding structure or managing all those SQL Servers. Second, my insert performance problems disappeared.

But I still had problems with both analysis and export. Both of these operations were simply too taxing on large datasets to do at runtime with LINQ.

Time for another meeting with Joshua and David. Enter additional Azure worker roles and Azure queues.

Architecture IV: Queues

To solve both of these problems, I introduced additional Azure worker roles that were linked by queues. I then moved the processing of each archive (running aggregate queries as well as generating the file to export) to worker roles.

So in addition to my search worker role, I introduced two new roles, one for appending and one for aggregating. It worked as follows:

The Search role queried the search table for any searches that hadn’t been updated in the last half hour. Upon completion of a search that returned new tweets, it put a message in the queue for the Appender.

The Appender role picked up the message and appended the new tweets to the master archive. The master archive was stored as a tab delimited text file. Perfect! This solved my performance problem getting the tweets into that format, as it was now stored natively as that format and available for download from storage whenever necessary.

Upon completing appending, a message was added to the queue for the Aggregator role. That role deserialized the archive to objects via my own deserialization engine.

Once I had the graph of objects, the LINQ queries were run. The result of each aggregate query was then persisted to blob storage as JSON. Again, perfect! Now all six aggregate queries, which sometimes could take up to five minutes to process, were available instantaneously to the UI. (This is why performance is so snappy on the charts for archives of massive size.) In addition, I could also provide these JSON objects to developers as an API.

This architecture worked for quite awhile and seemed to be our candidate architecture.

Until a new problem arose.

The issue was that the aggregate queues were getting backed up. With archives growing to 500,000 tweets, running the aggregate queries was bottlenecking. A single archive could take up to five minutes to process. With archives being updated every half hour and the number of archives growing, the time it took to run the aggregate queries on each archive was adding up. Throwing more instances at it was a possible solution, but I stepped back from the application and really looked at what we were doing. I began to question the need for all that aggregate processing, which was taking up not only time but also CPU cycles. With Azure charging by the CPU hour, that was a consideration as well.

As it was, I was re-running the aggregate queries every half hour on every archive that had new tweets. But really, the whole point of the aggregate queries was to get a sense of what was happening to the data over time. Did the results of those queries change that much between an archive of 300,000 tweets and 301,000 tweets? No. In addition, I was doing all that processing for an archive that may only be viewed once a day – if that. The whole point of The Archivist is to spot trends over time. The Archivist is about analysis. It is not about real time search.

It dawned on me: do I really need to be running those queries every half-hour?

Architecture V: No Queues

We made the call (pretty late in the cycle) to run the aggregate queries only once every 24 hours. Rather than using queues, I changed the Aggregator role to run on a timer similar to how the Searcher role worked. I then partitioned work done by the Appender and moved its logic to the Searcher or the Aggregator.

One thing I like about this architecture is that it gets rid of the queues entirely. In the end, no queues means the whole application is simpler and, in software, the fewer moving parts, the better. So, the final architecture ends up looking like this:

Both the Searcher and the Aggregator are encapsulated in dlls, such that they can be called directly by the web role (when the user enters an initial query into the database) or by a worker role (when an archive is updated). So each worker role runs on a timer.

We had to make another change to how the Searcher worked pretty late in the game. The Searcher was polling Twitter every half hour for every single archive. This seemed like overkill, especially for archives that didn’t get a lot of tweets—it was a lot of unnecessarily polling of Twitter. With a finite amount of queries allowed per hour to the Search API, this needed revising.

The solution was an elastic degrading polling algorithm. Basically, it means that we try to determine how ‘hot’ an archive is depending on how many tweets it returns. This algorithm is discussed in detail here if you’d like to read more.

If you’d like to learn more about exactly how the architecture works, check out this Channel9 interview at about 10:00 in, where I walk through the whole thing.

Final Thoughts

The Archivist has been an exciting and challenging project to work on. This post didn’t even get into other issues we faced during the development, which included getting the ASP.NET Charts to work in Azure; dealing with deduplication and the Twitter Search API; writing and reading from blob storage; nastiness with JSON deserialization; differences in default date time between SQL Server and .NET; struggles trying to debug with the Azure development; and lots more. But we shipped!

And, perhaps more interestingly, we have made all the source code available for anyone to run his or her own instance of The Archivist.

Article vs. Section: We've Finally Gone Mad

Nishant Kothary — Thu, 12 Aug 2010 22:35:47 GMT

I'm not saying I have conclusive evidence that the <article> tag accelerates melanin depletion. But when coupled with other symptoms I exhibited—increased use of abusive language, heightened temptation to dress my cat up as Santa, and an obsessive need to watch Bachelorette (I do think we should cut Frank some slack, though)— one needn't stretch their imagination much to find a correlation.

Anyhow, I digress.

I know I'm coming to the <article> vs. <section> party a little late. I know the discussion is, for practical purposes, closed. But that says nothing about the resolution of ambiguity clouding the topic. And despite the numerous articles and sections (sorry, I had to) dedicated to debunking the topic, I didn't find many real-world examples.

So, here's one. How would you mark up the middle two "sections" in this wireframe?

A little more context:

Elsewhere. This is a listing of publications (mine, in this case) you can find "elsewhere". In other words, the individual items link off to other domains. I've been increasingly writing for online publications, so I figure I should catalog the writings.
Journal. The items in this listing lead to recent blog posts on my own domain, rainypixels.com.

It's probably worth noting that each of the "items" are comprised of the title of the post, a publish date, an excerpt and a link to read the complete piece, which leads to a permalink on the same domain or another domain.

This is a very common scenario; most blog templates rely on it (heck, you needn't look farther than the home page of this very site). It's also the scenario that unearths the insanity of the spec and signals to a world where smart, mind-reading machines exist.

The specific hairy issue in answering the aforementioned question deals with the choice of nesting sections within an article, articles within a section, articles within an article or sections within sections. Also, should a div be invited to the party? Trick question! You blinked. You're going to hell. Bwahahaha!

OK, pick your brains up off the floor. I'm going to give you an answer that offers, if not a cure to itchiness, then at least some relief:

<section id="elsewhere">
	<h1> Elsewhere </h1> 
	<article> Item 1 </article>
	<article> Item 2 </article>
	<article> Item 3 </article>
	<article> Item 4 </article>
</section>

When confronted with the choice between section and article (assuming you've eliminated all other possible tag choices for your task), this heuristic seems to work:

If you could see the content as an item or entry in an RSS feed, wrap it in an article. Otherwise, use section.

I am Jack's Raging Bile Duct.

One of my problems with this whole topic is that it's done little to semantically improve the structure of my document—and for practical purposes, I'm still going to markup articles and sections with semantic id's and classes to write manageable stylesheets, anyway. It seems that retraining my muscle memory is the only outcome.

The other issue pointed out by my colleague, Joshua Allen, is that if it's truly about syndication, it should have a permalink similar to an entry or item in an RSS feed. It's the established way of Web syndication, after all. But he's clearly smoking crack, because his solution requires that we first plead insanity, then beg for forgiveness for having spent uncountable man hours in almost-Kantian article vs. section debates, apologize to victims of melanin depletion and finally replace the culprit element with, oh, I don't know, <entry> or <item>! We shall do no such thing, Mr. Allen, so go back to your silly world where things make sense.

My hyperbolic negativity aside (I promise I'll be done soon. To quote Billy Crystal from Analyze That, "I am grieving. It's a process."), we may find solace in the fact that smarter folks like Jeremy Keith have arrived at similar conclusions. Read this, this and if you're really hungry for pain, this, for background. Or look no further than Mr. Keith's succinct (slightly resigned, and understandably so) conclusion to the "article" section of his absolutely wonderful book, HTML5 for Web Designers:

What's more problematic is that article and section are so very similar. All that separates them is the word "self-contained". Deciding which element to use would be easy if there were some hard and fast rules. Instead, it's a matter of interpretation. You can have multiple articles within a section, you can have multiple sections within and article, you can nest sections within sections and articles within sections. It's up to you to decide which element is the most semantically appropriate in any given situation.

I'm done complaining.

Time to look at the silver lining and give in to the opportunistic urges coded into my genetics; after all, I descend from a long line of entrepreneurs. It seems clear to me that the milieu is ripe for our most creative markup proposals, and I shall be making a case for my own pet interest—the <weimaraner> element—defined as follows:

The weimaraner element represents a section of content on a page that may be characterized as "cute", albeit a little clingy or passive-aggressive.

I further propose that browser vendors implement some sort of standard UI that overlays a picture of a Weimaraner, to defuse readers' mounting anger. It's a tried and tested theory that pictures of cute dogs reduce stress levels. Here's a quick 'n dirty example that uses this very article (notice its impact on your anxiety):

What do you think? Feel free to add your own spectacular markup accouterments in the comments below.

Using jQuery to animate a Hero

Tim Aidlin — Thu, 12 Aug 2010 18:28:35 GMT

For a long time designers were forced to use Macromedia / Adobe Flash for a lot of animations in their designs. With the rise of JavaScript, however, designer/developers are now empowered to create a vast array of animations and functionality that often used to require plug-ins. Now that the wider community has started to coalesce behind web standards, the use of JavaScript for progressive enhancement of websites has become de rigeur, providing wonderful opportunities for designers and developers to collaborate.

MIX Online’s Glimmer

I started using JavaScript with a bit of hesitation, coming from a background steeped in front-end web design and development, using Flash and then Silverlight for animation and interactivity. I found the JavaScript syntax a little difficult to understand at first. I felt I was lacking ways to visually design interactions with JavaScript.

We found other designers were having this problem too. To fill this need, last year we at MIX Online designed, developed and published Glimmer, which allows designers and developers to create interactive elements and animations on web pages using jQuery. Glimmer also helps you learn how to write good jQuery by allowing you to open your HTML in a program that interacts with your code and generates the correct files.

For help getting started coding your own jQuery, check out this great tutorial by Matt Doyle. http://www.elated.com/articles/super-easy-animated-effects-with-jquery/

Not long ago we released another Lab called The Archivist. We used Glimmer on the home page http://archivist.visitmix.com to add jQuery animations to the main Hero graphic. jQuery allows for some fun animation styles and shows off our new mascot, Friendly Green Robot.

The first section of the following tutorial demonstrates how I set up my HTML and CSS to take advantage of jQuery. The second half of the tutorial focuses on how to use Glimmer to produce that jQuery.

Download the files and dive in directly, or read on …

Watch The Screencast

To help you through the various steps below, I've created a short screencast that gives a brief overview of the process detailed in the tutorial. Check it out before you get into the details of building out the application on your own.

You need Silverlight to view this video. It’s fast, it’s free and it’s awesome. Click here to get it.

The File Structure

I almost always set up my file structure at the beginning of each project. I always copy the following from a folder named ‘_base’ when starting:

The HTML

To have your Hero banner animate correctly, you’ll need to think about the structure of your HTML and the CSS you’ll use to create it. I generally think of these animations as having ‘slides’ and ‘elements’ within those ‘slides’. In this example, we’re creating a graphic that will have two slides that animate in and out. The first slide consists of two elements. The second consists of three elements.

Below is the HTML I used to create the animated header. It has a very basic structure consisting of:

Main hero container

Slide 1
- Text
- Big Robot
Slide 2
- Little robot
- Screen 1
- Screen 2

<div id="hero">
 <div id="horizontalLine"><img alt="" src="images/robotHero/horizontalLine.png" width="940" height="19" /> </div>
  <div id="slide1">
   <div id="slide1text"><img alt="Start an archive" 
   src="images/robotHero/text1.png" width="323" height="100" /> </div>
   <div id="bigGuyPointing"><img alt="The Archivist" 
   src="images/robotHero/bigRobot.png" /> </div>
  </div>

  <div id="slide2">
   <div id="slide2text"><img alt="Start an archive" 
   src="images/robotHero/text2.png" width="209" height="100" /> </div>
   <div id="standingRobot"><img alt="The Archivist" 
   src="images/robotHero/standingRobot.png" /> </div>
   <div id="vis1"></div>
   <div id="vis2"></div>
  </div>
</div>

The CSS There are a few things to note in the CSS that will make the Hero animate correctly. First, ensure that the main container—in this case, #hero—has ‘overflow:hidden’. This will act as a mask that will hide the individual parts of the animation when they flow outside the boundaries of the hero container.

Secondly, all items that you want to animate should have their position set (‘position:absolute’, for example). It doesn’t necessarily matter to jQuery if the position is set to absolute or relative, as long as it’s set.

The third item to note in this example is that the second slide of the animation is set to ‘display:none;’. In the jQuery, we’ll be toggling the visibility of the different slides. We display the first slide by default, which makes for a good experience in the rare case that someone has his or her JavaScript turned off. Those users would just see a static image. We’ll be toggling the visibility of the different panels by using jQuery to modify the CSS ‘display’ function.

#hero{
width:940px;
height:180px;
background-image:url('../images/333background.jpg');
background-repeat:repeat;
overflow:hidden;
position:relative;
}

#horizontalLine{
display:none;
width:940px;
height:19px;
margin-top:150px;
position:absolute;
}

#slide1{
display:block;
}

#slide1text{
width:323px;
height:100px;
position:relative;
margin:40px;
}

#bigGuyPointing{
width:535px;
height:551px;
position:relative;
margin:-200px 0 0 400px;
}

#slide2{
display:none;
}

#slide2text{
width:209px;
height:105px;
position:absolute;
margin:40px 40px 40px 140px;
}

#standingRobot{
width:535px;
height:551px;
position:absolute;
margin:26px 0 0 30px;
}

#vis1{
position:absolute;
width:249px;
height:148px;
background-image:url('../../images/robotHero/line.png');
background-repeat:no-repeat;
margin:32px 20px 20px 380px;
}

#vis2{
position:absolute;
width:249px;
height:148px;
background-image:url('../../images/robotHero/pie.png');
background-repeat:no-repeat;
margin:32px 20px 20px 650px;
}

Glimmer

The next step is to fire up Glimmer and get started on the animation. Let’s go.

Download Glimmer

http://visitmix.com/labs/glimmer/

Getting Started

In the top menu, go to File > Open and find your html hero. Open that file.

Click Add New Action
Click Action Name 1 to expand the field

Step 1: Clear the field

Action Name—This will be the first step of the animation. Here, I’m naming the action ‘clear_canvas’. I want this action to happen directly when the page loads, so under Action Type I choose ‘load’. I leave the Trigger field blank because I want the animation to start without a trigger (say, a mouse-click, or hover-over).

The first Target I want to animate is the text “Start an archive …” I choose ‘#slide1Text’ from the dropdown. In that Target dropdown you should see all the elements of your html page. From here you can choose what you’re animating and what you’re doing to it.

Next I choose the effect I want to have on the text. In this case, I want to move it from the left of the screen to where it currently is. To do this I choose ‘X Position Animation’ from the Add Effects dropdown. Click the checkbox next to Start and in the Start field, enter -600. This means that we want the target—#slide1text—to start 600 pixels to the left of where it currently is. We leave the ‘End’ value at 0 because 0 is where the target originally was, and it’s where we want it to be at the end of this slide.

Underneath the Target dropdown you’ll see Options. Click the toggle and set the Action Duration to 600. This field gives you control over how long the animation takes. In this case, we’re setting the animation to 600 milliseconds. We also have the option to add some effects such as bouncing, easing, swaying, etc. In this case, we’ll add some elasticity to the animation and choose ‘easeOutElastic’. You’ll note you have a few options (such as ‘easeInElastic). The difference here is easeOutElastic makes the elastic happen at the end of the animation, while easeInElastic makes it happen at the beginning.

Let's Save!

As noted earlier, I like to keep all my files in their own file-type folders. I’m being asked to save JavaScript files, so I choose my js folder for the destination I’d like files written to. Glimmer will automatically save the .js file that’s particular to your site, as well as other .js files that enable effects and functionality.

Saving also inserts references to these JavaScript files into your HTML. Be aware: If you have your HTML open in another program, you should reload it before further editing since Glimmer inserts the references to the JavaScript files into the <head> of your HTML.

<!--Start Glimmer Insertion-->
<script type='text/javascript' src='js/jquery-1.3.2.min.js'></script>
<script type='text/javascript' src='js/effects.core.js'></script>
<!--glimmer generated file-->
<script type='text/javascript' src='js/hero-stage2.html.glimmer.js'></script>
<!--End Glimmer Insertion-->

Back to Step 1

In addition to bringing in the text, we also want to bring in the Big Green Friendly Robot. We want to do this a second or so later than the text. To accomplish this we first remove him from the stage by clicking ‘Add a target’, which spawns a new panel in the clear_canvas action.

We then set ‘#bigGuyPointing’ as the Target and again use X Position Animation. This time, though, we move The Big Green Friendly Robot to the right 600 pixels and leave him there by setting both the Start and End fields to 600.

We also need to ensure that this panel is visible when the animation loop begins again. To do this, we toggle the CSS:

Target: #slide1
Add Effects: Modify CSS Effect
Name: display
Value: block

Step 2: Bring On the Big Green Friendly Robot

Click on Add New Action and name this action ‘big_guy_in’. In this case, we’re going to have the clear_canvas action trigger the big_guy_in action, so we can leave the Action type and Trigger fields empty.

Click Add a target and choose #bigGuyPointing. Under Options, set the time to 600 milliseconds again and choose easeOutElastic. Since the Robot is already off the canvas, all we need to do is bring him in. That means that he should end at 0. Under Add Effects, we’ll choose X Position Animation and leave the End field at 0.

Now we have to trigger the big_guy_in action. We do this by setting a timer on the clear_canvas action. In the clear_canvas action, find the Add Effects dropdown and add the Timer Effect to either of the targets. In the Chained Action field enter big_guy_in and set the duration to 600. This effect will trigger the big_guy_in action in 600 milliseconds (.6 second).

Step 3: Waiting and leaving

Next we’ll build the exit of these pieces by doing what we did above, only backwards. First, choose Add New Action and name it ‘slide1_out’. Again, since this action is going to be triggered by the stage before it, we need not assign an Action Type or Trigger. We do, however, have to assign Targets, which are the text and Robot.

Choose #slide1text from the Target dropdown.
Click the Options panel and set the Action Duration to 600 and Action Easing to easeInBounce.
Choose X Position Animation and set the End field to 600
Save
Click Add a target
Choose #bigGuyPointing from the Target menu
Under Options, set the time to 600 and easing to easeInBounce
Choose X Position Animation from the Add Effects dropdown.
In the End field, type -600
Save
In big_guy_in action, choose Timer Effect from the Add Effects dropdown.
Enter slide1_out in the Chained Action field, and change the duration to 6000 (6 seconds). This will let the slide stay static for 6 seconds; then the two pieces will slide away.

Step 4: Bring in Slide 2

In this step, we’ll basically repeat what we did with the first part of the animation, but this time we’ll have to make the slide visible by changing a CSS setting. We’ll also have the parts of the slide come in vertically and set the opacity on the little Friendly Green Robot.

Choose Add New Action
Expand and name slide2_in
Target: #slide2
Add Effects: Modify CSS Effect
Name: Display
Value: block
Save
Action: slide1out
Add Effects (to either of the existing targets): Timer Effects
Chained Action: slide2_in
Duration 2000

You’ll see that this pops the second slide in, but we want to animate it nicely. We’ll do basically what we did to slide1 with the clear_canvas action. Let’s move the little guy off the stage and bring him on nicely.

In slide2_in, add a new Target of #standingRobot
Add effect: Y Position Animation
Check Start and give that a value of 500. This will push the robot down 500 pixels.
Leave End at 0, which is the endpoint of the animation
Add Effects: Opacity Animation
Check Start and set to 0 (100% transparent)
Set the End value to 1 (100% opacity)
Open the Options panel and set Action Duration to 600 and Action Easing to swing
We still have to get the text and panels off of the stage in order to animate them in during the next phase
Add a new Target: #slide2text
Add effect: Y Position Animation
Check Start and give both the start *and end* a value of -200. Do the same for the panels by choosing #vis1 and #vis2 as new Targets and repeating. If you’d like to have one of the pieces come from the bottom, change the value from -200 to 200.

Okay, now we have just the little standing robot on the second slide. Let’s save and move on to animating the text and panels.

Add New Action
Action Name: slide2_textpanels
Target: #slide2text
Y Position Animation
End value should remain at 0
In the Options panel, set the Action Duration to 600 and the Action Easing to ‘Swing’
Repeat this for the Targets #vis1 and #vis2

We still need to trigger this part of the animation. In the slide2_in action, add the Timer effect to the #slide2 Target. We’ll point the Chained Action to slide2_textpanels and set the duration to 600, which is how long it takes for the little Friendly Green Robot to slide up.

The last thing we need to do is clear off the second slide and go back to slide1. To do this, simply create a final action, move all the pieces off the canvas using Y Position Animation, and set a Timer Effect to trigger the clear canvas action. That will set the loop in motion.

Your final jQuery (.js) file should look like the following, with one exception:

jQuery(function($) {
var timer;
function clear_canvas(event)
{
clearTimeout(timer);
timer = setTimeout(eval("big_guy_in"),"600");$("#slide1text").css("left","-800px");$("#slide1text").animate({"left":0},600, "easeOutElastic", null);$("#bigGuyPointing").css("left","600px");
$("#bigGuyPointing").animate({"left":600},1000, "linear", null);
$("#slide1").css("display","block");
$("#bigGuyPointing").animate({"left":600},1000, "linear", null);
}

function big_guy_in(event)
{
clearTimeout(timer);
timer = setTimeout(eval("slide1_out"),"6000");
$("#bigGuyPointing").animate({"left":0},600, "easeOutElastic", null);
}

function slide1_out(event)
{
clearTimeout(timer);
timer = setTimeout(eval("slide2_in"),"2000");
$("#slide1text").animate({"left":900},600, "easeInBack", null);
$("#bigGuyPointing").animate({"left":-900},600, "easeInBack", null);
}

function slide2_in(event)
{
clearTimeout(timer);
timer = setTimeout(eval("slide2_textpanels"),"600");
$("#slide2").css("display","block");
$("#slide1").css("display","none");
$("#standingRobot").css("opacity","0");
$("#standingRobot").css("top","500px");
$("#standingRobot").animate({"opacity":1,"top":0},600, "swing", null);
$("#slide2text").css("top","-200px");
$("#slide2text").animate({"top":-200},1000, "linear", null);
$("#vis1").css("top","200px");
$("#vis1").animate({"top":200},1000, "linear", null);
$("#vis2").css("top","-200px");
$("#vis2").animate({"top":-200},1000, "linear", null);
}

function slide2_textpanels(event)
{
clearTimeout(timer);
timer = setTimeout(eval("slide2_out"),"6000");
$("#slide2text").animate({"top":0},300, "swing", null);
$("#vis1").animate({"top":0},300, "swing", null);
$("#vis2").animate({"top":0},300, "swing", null);
}

function slide2_out(event)
{
clearTimeout(timer);
timer = setTimeout(eval("clear_canvas"),"2000");
$("#standingRobot").animate({"top":-200},300, "swing", null);
$("#slide2text").animate({"top":200},300, "swing", null);
$("#vis1").animate({"top":-200},300, "swing", null);
$("#vis2").animate({"top":200},300, "swing", null);
}

clear_canvas();
});

I’ve left out one particular string of code that’s necessary to load the .js file into Glimmer correctly. You’ll note that there is some commented-out code at the very top of the .js file generated by Glimmer:

//Microsoft.Glimmer.Oneway
//<AnimationCollection ...

This code is a duplication of the .js code written by Glimmer, but it’s is only used to enable users to re-open the file in Glimmer. It’s perfectly safe to remove that code, but be aware that you will not be able to reopen the .js file in Glimmer moving forward.

Get To It

For some of you, the best way to learn is to jump right in and look at the code. Again, you can download the full source and use your favorite text-editor to view the code. For others, I hope the above step-by-step tutorial was helpful. As noted earlier, the markup presented above is not the only way to structure this animation, but it’s the way I found easiest to understand.

Have you made the switch to animating with JavaScript? Do you still find yourself using great plugins like Flash or Silverlight? Where do you find you have to choose and what are the deciding factors? Leave a comment below, and be sure to follow us on Twitter @mixonline.

What Comes After Sketching? Advice for Developers.

Jonah Sterling — Thu, 05 Aug 2010 23:34:08 GMT

In 1999, I overheard something that changed my career. I was the sole Designer at what later became Razorfish, where I sat in a space with a bunch of developers. Standing in the hallway one day, I picked up on something I hadn’t heard since art school: the judgmental, authoritative, biased tones of a Critique in progress.

Critique: A traditional art school activity consisting of ritualized verbal assault disguised as an enlightening series of observations regarding one’s artistic skills, and which is intended to ‘toughen one up’ for the real world

Like Goodall with the apes, I couldn’t be sure that what I was seeing was normal behavior or not. The developers were taking turns tearing into a sad block of code that had been hung on the wall with voracity and a certain glee. “Sloppy”, “bad formatting”, “I could have written this in five lines!”, “What’s with this semicolon use?” These were just the warm ups, and all I can convey without profanity.

After a few minutes, I approached the semi-circle of developers, as I felt like this was my kind of thing. When they saw that they were being witnessed (like something from the DaVinci Code) they quickly put their heads down, went back to their desks and put on headphones.

And so it hit me: Developers are artists too.

Developer as Artist

Over the following decade, I used that perspective to rise in the ranks, eventually becoming Group Creative Director and running a full UX team of Designers, Information Architects, Researchers and Integrators (Blend specialists). All of whom I view and treat as artists.

When I was asked to write a blog post for Mix Online, I wondered what a GCD could possibly say that would interest developers. After some deliberation, my team and developers instructed me to talk about what we do AFTER we sketch—where does a User Interface COME from?

That’s an important question that requires a serious answer, because when I thought about it, I was led to more questions. Such as, “Why don’t they already know this? They’ve been working along side designers for years and we have no inexperienced staff—hasn’t anything rubbed off?”

The Sketch, The Silver Bullet

As an industry, we have spent the last few years beating down the rigid role boundaries of the late 90’s and early 00’s. We’ve largely done this with the sketch, the best battering ram we could find and the great leveler of the field. Now you don’t need a design background to play a critical part in creating the project vision; you just needed a pen and ideas. Suddenly everyone can play.

Sketching: so easy a CEO could tell you to do it…

Books were written, presentations given, software built to embrace it. Clients got hip to it, methodologies adjusted to accommodate it. And once clients started respecting our loose and sketchy ideas as ‘final deliverables’, developers quickly got involved at the beginning of projects too. The impact to product and software design has been profound.

Largely as a result of the sketch, the developer is now a welcomed participant in all conversations with a client. A side effect of this fraternization is what I call skill-bleed: Our roles are more integrated and friendly, and more and more people are dabbling in tasks that have long been the well-defended territory of another group. We suddenly have jacks-of-all-trades popping up all over, and I’m all for it.

The one barrier that I haven’t seen break down, though, is the separation between developers and UI designers. As far as cultural memes go, this one is really, really deeply rooted. There’s a perception that a ‘professional’ developer is precise, analytical, has terrible fashion sense and under no circumstances has the ability to design—the same way that ‘professional’ designers are supposed to be wild, unpredictable, overly hip and elitist. (I’ll also point out that both frequently wear tight clothes, but for different reasons.) These are damaging and dangerous stereotypes that don’t do anyone any good.

From where I sit, I think this all goes back to the question of why a developer would ask me what happens after sketching. After a sketch, UX is usually in control and the developer is once again segregated to specialist tasks. They might conduct a technical audit or produce architectural diagrams, but that’s as visual as they feel they are allowed to get. This way of working supports the professional developer persona I mentioned above, and keeps developers from dabbling in design.

If a developer takes initiative, he or she may start creating prototypes, which involves making design decisions—something they are ‘never’ supposed to do, or do only by taking a personal risk. I don’t ever want to hear the caveat “I’m not a designer, so this UI I sketched sucks” from a developer again.

I’d really like to see a world where this prejudice stops. Developers are artists in their code, and with the right coaching and exercises that mindset can translate to UI design. Naturally, this kind of skill growth will take time, but I hope you’ll join me in believing that the old rules need to go.

Step 1: Permission To Be ‘Emotional’ Has Been Granted.

I think developers would benefit if we busted the myth that there is a ‘right’ answer when dealing with users. Humans tend to believe that the frontal lobe is what makes us different from animals, since it‘s where our rational thought is supposed to come from. Therefore, most (if not all) all the decisions we make must be ‘rational’—right?

This moment NOT brought to you by ‘rational’…

Well… maybe. What behavioral science has been uncovering is that the mid-brain, the emotional center, may have a lot more to do with our decision-making process than we would like to think. I’m paraphrasing a lot here, but the gist is that it’s really the emotional center that generates feelings like want, hate, love, etc. The ‘rational’ part of our minds merely rationalizes those feelings, giving us permission to perform the acts that achieve whatever ‘emotional’ outcomes are desired. What? Yes.

What’s my point? As a developer, the first step to post-sketch success is letting yourself off the hook and dropping the idea that right = rational. It doesn’t, and it never will as long as humans are involved. We can hedge our bets with research and studies, but trying to design as people were driven by reason and logic alone will lead you to disastrously lame User Experiences.

Look at it this way: If you can give yourself permission to make strategic emotional design decisions (make it cool, elegant or magical—whatever), you’ll actually come closer to being scientifically RIGHT in satisfying your users. Trippy, yeah?

Step 2: Up Your UI IQ

You can’t create a passable UI overnight, and doing so involves much more than good technique. There are hundreds of unsuccessful designers out there who don’t seem to understand this. To begin with, your UI is an extension of your client’s brand, one of the most critical resources you have in creating a successful design.

A lot of strategy goes into creating a good brand or a logo. Frequently, the more simple a brand, the more complex the strategy that went into it. Your client will likely have some key documents from their branding exercises. See if you can get a hold of them. If you can, great. If not, you can still move ahead.

Pay special attention to the logo; it’s the first clue to the language a client’s brand communicates in. The logo is how the brand says ‘Hello’.

“pssst. Your brand is a language, pass it on…”

From ‘Hello’, you can often make a number of safe guesses about your UI direction, like colors or button styles that could work or even what container styles may fit. Clients love it when even the initial prototypes are on-brand, so don’t discount this process.

Once you have an idea of what language you’re speaking, you need to know what to say. See if you can let the ‘emotional’ part of your brain drive for a bit and try to capture your visceral impressions. Look at logos, collateral (brochures), business cards, websites, posters or even the physical location (if applicable) for clues.

Deciphering a brand is a lot like wine tasting—at first the vocabulary is strange and fairly aggravating and you have to guess and fail a lot at the start, but it starts to make sense eventually. Once you’re fluent, a lot of doors may suddenly open for you.

The point of step 2 is really to underscore that brand is the cornerstone of a User Interface, but it should also impact your interaction models as well. For example, a client whose logo consists solely of a metallic gray calligraphic font is pretty much 100% not going to be happy with fast, snappy UI where things quickly appear and disappear—but a music site may love that model. The difference is the brand.

Step 3: Get Lots of Exercise

When I took on managing the Integration team (a front-end development team, more or less), my goal was to grow them as designers. So, I gave them a series of assignments to help them learn what to do post-sketch and teach them what design really consists of. I’ll detail these exercises below.

Getting enough exercise is important for growing designers…

Task 1: Find a User Experience that you think is very effective and be prepared to say why you think it works.

This was a loaded one, because I didn’t say what kind of an experience you should get. A website? An application? A toaster? When we did this at IdentityMine, I got a good variety of User Experiences back (no toasters, though ), and was pleased to find that the team could articulate what they liked in a user experience: clear navigation, simple presentation, dynamic content, etc. Things that, coincidentally, most users are into as well.

Good User Experiences popping up everywhere…

The important thing is that we talked about what a “good” experience was, so that we had the grounds for a meaningful conversation. The UX team also participated in the meeting, letting us know what they look for in a good design and essentially bird-dogging what’s involved in successful UI without it becoming personal (as in, not about their work or the developers’).

Task 2: The Attributes & Associations

The next exercise was to take the user experience in question and start divining the strategy and thought that went designing it. Yes, this is a fairly abstract exercise, but it’s really, really relevant. In analyzing the experience, a person who has his emotional receptors turned on should start to get ‘feelings’ from the visuals, the copy, the images and the flow of the experience.

Note: This may require a drink or two the first few times you try it.

So… inspired…

What my team was looking for were the brand attributes, the critical foundational components of the user experience. Here’s an exercise to help you recognize brand attributes:

While relaxing, focus on the brand experience and listen to your subconscious for words that describe what you are looking at. Is it fast? Responsive? Secure? Modern? Capture these emotional descriptors that the experience is conveying, and be ok with being off-the-mark. You can always come back and refine.

Collecting 5-7 of those brand attributes will get you ready for the next task.

Task 3: Get Personal

To teach my team to ‘get personal’, I instructed them to spend time trying to envision the intended target user (or persona) of the experience. See if you can do this:

Think of a couple potential target users of the experience and describe them in any way you find useful. Then, link features and functionality to the story of each user, and see if you can determine cause and effect. There should be a clear relationship between features and aspects of the persona.

There are a lot of ways to do a persona. Some are narrative style (day-in-the-life), some are focused on vital statistics, and some are focused on ‘needs statements’. All are valid, so you should try different versions to see what feels natural.

At first I was amazed by how detailed the team got with their personas, and then I realized I shouldn’t have been surprised—after all, this was all about analytics and detail, something they already excelled at—they had just never been asked to do this.

Task 4: Be Moody

Up until now, the team hadn’t really been asked to do anything that felt like ‘design’ to them. We were slowly demystifying the process and exploring how the different thought-based tasks worked to support the final design direction. We were starting to understand that visual design isn’t ALL visual. This next task, however, does require some design:

Using whatever tool you feel comfortable with (the usual Adobe products, PowerPoint, Keynote on your iPad), make a moodboard/stylepage. The goal is to assemble a collage of photos, colors, typography and UI elements into a single view that the sample experience could blend into. By creating an environment that the experience could ‘live in’, you are beginning to extend the brand associated with the experience. This is a very valuable skill to have.

My team’s moodboards varied in technical sophistication, but each had been able to capture parts of the essence of the experience, the critical lesson.

To keep this from being a totally subjective exercise, I asked the team to turn to the attributes and personas from the earlier tasks to validate their design choices. All of a sudden, it was much easier to pick photography that was right for the experience, because we had built a ‘case’ for making certain choices and could start to logically and rationally ‘prove’ that they had the right images, since they fit the persona, attributes, or style of the experience.

When I said this is a ‘retro’ exercise, I meant retro ‘fit’…

The UX team was on the whole pretty impressed. The Integration team had rapidly gone through a number of the key UX tasks and come out the other side proving that some myths need to be busted. They weren’t tightly coached or and there was nothing to copy (they had no help and did this on downtime)—they pulled this off because they were willing to break their mold, leave their comfort zone and be ‘artists’. They had opened themselves up to being emotionally empathetic at the right time, which caused them to be more rational and strategic than before.

Everyone’s An Artist

These tasks are far from new to UX teams, but they may be new to you or your team. If you haven’t seen them before, the reason may be that your designers don’t show their work. Meaning that often, designers get to a point where we don’t bother to capture all these steps as we go. We end up doing all of these things, but just in our heads and in a few seconds.

So don’t be fooled—designers/creatives/UX’ers (or whatever you call them) don’t wield any powers that are beyond your ability to command. While you were learning to program, we were being forced to cry in front of our classmates in critiques that burned the paint from the walls (maybe that’s why some of us come off as total jerks). Designers just have a head start designing. So don’t sweat it.

At this point you should have a light bulb going off about these exercises and how they can help you expand your skill set. If it isn’t a full-fledged light bulb, I’m hoping it’s at least a candle. What I’ve seen and what I believe is that everyone, including developers, are artists inside. How art expresses itself is different for everyone—some people sculpt, some act, others play music and some people (you know who you are) are coders.

By finding that part of yourself, and giving yourself permission to step out of the stereotype, you can grab a hold of your artistic passion for code and focus it on UI design.

… and that’s what you do after you sketch.

Is Your HTML5 10K Apart?

Nishant Kothary — Thu, 29 Jul 2010 18:27:57 GMT

Ten years ago, a little community for people who make web sites introduced the 5k Awards in an article titled Much Ado About 5K:

It was unassuming little web page that posed a singular challenge: could you build a complete website using less than 5 kilobytes?

5K? Was contest creator Stewart Butterfield nuts? The rules of the 5K Awards, in their entirety, read as follows: "All HTML, script, image, style, and any other associated files must collectively total less than 5 kilobytes in size and be entirely self-contained (employing no server-side processing)." Great. All he left out was the part about not using a monitor or keyboard. And the flagellation, of course.

Flagellation, indeed, Mr. Zeldman. But, as he knows all too well, it's hardly far from reality. Frugality of file size has always been a core tenet of great web development, and still is. That's why the 5K spirit matters as much today as it did a decade ago.

So, what better time than now—the year of HTML5—to resurrect the 5K Awards? Albeit with a slight facelift and adjusted for inflation.

Without further ado, we are excited to introduce the next chapter in the 5K Awards—10K Apart—in association with our friends at An Event Apart.

The Skinny

The premise is intact: Could you build a complete web application using less than 10 kilobytes? But, as mentioned earlier, we've taken inflation into account. We're letting you use jQuery, Prototype and Typekit without it counting toward the 10K (sorry, no back-end, though; client-side only). And we're giving you 5K more to play with, for those of you who already did the math. A whole 5K. 10240 bytes. Twice as much as your forefathers got!

10K in Prizes

As if the glory and pleasure of optimizing every byte like your life depended on it weren't enough, we're also emptying our pockets on this one. We're giving away prizes that amount to 10,000 bones to 13 winners in all. The grand prize winner will walk away with a $3000 Visa gift card, registration to his or her choice of an AEA event within the next year and a copy of the brilliant book by Jeremy Keith, HTML5 for Web Designers. OMFG is right.

Just One—OK, Five—Catches

Their names may you have heard. Jeffrey Zeldman. Eric Meyer. Nicole Sullivan. Whitney Hess. Jeremy Keith. These are the five people sitting between you and your prizes. Err... good luck?

4 Weeks & Counting

There you have it—one contest, thirteen prizes, five judges, four weeks. Surely you can do way better than the pandering Weimaraner sample application (we're looking at you Mr. Doob).

Well, what are you waiting for? Go forth and build an app (or two, or three) that stands 10K Apart from the rest.

How To Be Evil

Joshua Allen — Thu, 22 Jul 2010 20:24:40 GMT

Corporate evil is often in the news. Whether we’re lamenting BP’s senseless slaughter of Pelicans in the Gulf, or praising Google for slogans like “Don’t be Evil”, we clearly think that corporations, like people, need to be held accountable for evil.

Incentives are often at play when companies and people to do evil things, and we try to create government and corporate policies to remove bad incentives. For example, large fines for pollution make it less tempting for a company to cut corners on industrial plants just to increase profits. Not every company convinces itself it’s OK to commit evil in the face of temptation, though, so temptation can’t be the only problem.

If we had the book, “Recipes for Justifying Evil”, maybe we’d recognize when people are putting the nefarious ingredients together, and stop them before the foul dessert is finished. Maybe we could institute policies that make evil ingredients less likely to come together.

The Balance Sheet

Recent research, discussed in this article, shows that people subconsciously keep a “morality balance sheet”, which often leads them to follow up good deeds with bad ones:

People who bought “green” products were more likely to cheat and steal than those who bought conventional products.
When given an opportunity to endorse Barack Obama for president, voters were more likely to later favor white people for job openings.
After getting high-efficiency washers, consumers increased clothes washing by nearly 6 percent. Other studies show that people leave energy-efficient lights on longer.

One of the most surprising results shows that simply imagining or planning virtuous deeds makes people more evil: “Uzma Khan, a marketing professor at Stanford who studies the psychology of buying, once asked study participants to choose between buying a vacuum cleaner or designer jeans. Participants who were asked to imagine having committed a virtuous act before shopping were significantly more likely to choose jeans than those not thinking of themselves as virtuous.”

Even if you don’t intend to give your employees license to commit evil, evil is the likely result of asking them to imagine virtue, or of making virtue a part of your mission statement.

Accounting Errors

Besides the fact that “balance sheet morality” leads to hypocrisy, we don’t really have CPAs to audit the books and prevent accounting errors. Take Alex Tabarrok’s comparison of BP’s pelican slaughter with other bird massacres:

Number of birds killed by the BP oil spill: at least 2,188 and counting.
Number of birds killed by wind farms: 10,000-40,000 annually.
Number of birds killed by cars: 80 million annually.
Number of birds killed by cats: Hundreds of millions to 1 billion annually.

This doesn’t make BP less evil, but it points to another sort of error—people pay attention to moral issues that are starkly visible, only.

Dan Ariely, author of “Predictably Irrational”, talks about this exact effect in his new book, “The Upside of Irrationality”. Although the book was published before the BP oil spill, Ariely says, “Consider the disastrous oil spill from the wrecked Exxon Valdez. The estimates for cleaning and rehabilitating a single bird were about $32,000 and for each otter about $80,000.19. Of course, it’s very hard to see a suffering dog, bird, or otter. But does it really make sense to spend so much money on an animal when doing so takes away resources from other things such as immunization, education, and health care?”

Unfortunately, getting people to think about the billion birds per year who are killed by cats is counterproductive. Ariely reports the results of one experiment: “Unfortunately, those who thought in a more calculated way became equal-opportunity misers by giving a similarly small amount to both causes”. One of the commenters on Dan’s blog drives this point home, saying, “The BP oil spill was a tipping point for me. I sold my car, donated the money to gulf coast clean-up, and switched to mass transit/biking. It’s had a major impact on my life.”

Kant’s Recipe Book

Of course, keeping a balance sheet isn’t the only way we convince ourselves to do bad things. Philosopher Immanuel Kant studied the psychology of evil, and catalogued a sort of recipe collection of the ways that people convince themselves it’s OK to be bad. Robert Gressis, a philosopher at California State, has written an excellent paper that ties together Kant’s most important works on the subject. Kant’s recipe book is just one small part of the paper (mostly chapters 5 and 6), which I’ll excerpt here. In summary, Kant argues that we convince ourselves to do things we know to be wrong by indulging in “moral fantasies”. Gressis groups Kant’s explanations like this:

The Adequacy Fantasy – This happens when a company keeps a balance sheet of its moral actions and concludes that it has come out OK. As Gressis describes, “The adequacy fantasist adheres to a standard according to which she is no moral saint, but is also not morally deficient. By her own lights, she is doing well enough, although she could be doing better.” Sure, it’s wrong to cut corners on an undersea oil drill, but at least BP lobbied to free a suffering old cancer victim who was dying in prison. If the examples of hypocrisy from the previous section didn’t demonstrate why the balance sheet approach causes problems, Gressis gives Kant’s perspective in chapter 5.2.1 of his paper.

The Exceptionalist Fantasy – This happens when a company sees itself coming out well ahead of most others. “Compared to everyone else, we’re practically saints, so we can be excused for a few indulgences.” This exceptionalism need not be based on past deeds, but is often based on some intrinsic property of a company’s corporate structure or mission statement that it believes is exceptional.

The Despondency Fantasy – This is when a company, either through a false humility or fear of bankruptcy or competitors, concludes that it isn’t powerful enough to do what’s right. Kant also calls this timorousness or pusillanimity. You can sometimes see discussions of “ROI” and “cost-benefit analysis” veer off the rails in this way. I think this is probably similar to what Ariely calls the “drop in the bucket effect”.

Of course, Kant is talking about human beings and a universal moral law, so I have to butcher his ideas a bit to fit with a corporate mindset. However, I think his psychological insights apply to the way that companies and people rationalize things. Like people, companies can be held accountable to standards of right and wrong that go beyond the letter of the law and the pursuit of profit. Technically, it wasn’t illegal for BP to cut corners on the drill and kill thousands of pelicans, but we still judge them for it.

What To Do?

As we’ve seen, companies tend to do evil when they tell themselves stories: stories about how exceptional they are, stories about how much good they plan on doing, and so on. What can policy makers and companies do about this reality?

Robin Hanson concludes, “Be Stingy With Praise”. I would add, be conservative in describing your own company’s exceptionalism and mission, and don’t make a strategy out of convincing others that you’re philanthropic. Anytime you catch yourself paraphrasing one of Kant’s fantasies to justify a decision, stop yourself and question what you’re about to do.

Dave Winer’s old company slogan, “We Make Shitty Software” might go too far in self-deprecation, but company slogans can still be aspirational without donning a smarmy mantle of virtuosity.

What do you think? What are some corporate policies or slogans that hit the mark? Leave a comment below!

The Humor of Code Check-in Comments

Karsten Januszewski — Mon, 19 Jul 2010 21:21:06 GMT

I was recently reviewing over 500 code check-ins for The Archivist and in the course of my review, a narrative emerged. I could see the evolution of the project: sprint milestones, new features, etc.

Perhaps more interesting (and entertaining) than the narrative was the humor in some of the comments. Now, the vast majority of the comments were informative—and appropriate. But in the heat of the moment, well, some of the comments got more colorful.

Here’s a list of the ten more amusing comments:

1. “more futzing”

2. “don’t f*ck with my connection strings”

3. “fixed wonky export code”

4. “build for legal”

5. “fixed goofy routing logic in mvc controller”

6. “deploy for sprint” then “redeploy for sprint” then “another try” then “ok really the sprint build”

7. “fixed serialization, deserialization crap”

8. “who said dll hell ended with .net?”

9. “remmed out back door”

10. “fixed dirty logic”

Am I totally out of line? If you were my dev lead, would you come flog me? Care to share some of your more amusing check-in comments?

The Evolution of The Archivist

Tim Aidlin — Mon, 12 Jul 2010 16:14:22 GMT

In an earlier post I talked about working on The Archivist Desktop version, and how the software began its evolution to the web and the cloud. In this post, part II of a series, I'll take a step back and explain how we concepted and executed The Archivist Web (alpha)and talk about what came before the design and code.

Audience Research

Once we decided to take on the project, we turned to our established group of experts for guidance. We already had a prototype of a similar application in The Archivist Desktop, so we had a cadre of fellow-Microsoft employees, friends and colleagues in other companies, and current users we could tap for solid user-testing. We focused on two primary groups of users: our internal Microsoft team and users of The Archivist Desktop.

Getting Stakeholders and Internal Microsoft Team Members On Board

We started by discussing the project's intricacies and goals with our internal team members and stakeholders. We set up short individual meetings with people, gathered feedback on how they used The Archivist Desktop and asked what new features they wanted to see.

“Conversation” with current users

Again, because we had already released The Archivist Desktop, we had an already-established user-base. Over the years there had been a few “vocal” users – who either wrote in to thank us, or report problems and failures – who we felt could provide valuable feedback from real-world use-cases. At the beginning stages of building The Archivist Web we simply emailed a few of the users with whom we had been in contact and asked them a simple set of questions surrounding specific topics such as types of visualizations they would use, how often the tool needed to update, the way they were using the data and the like. By asking specific questions rather than asking for general feedback, users were able to focus on key issues and needs that we could act upon. Often, by asking a generalized question such as “What would you like to change,” you get very generalized answers that are hard to turn into actionable tasks.

We also communicated our plans openly, which helped us make contacts in other parts of the company and take advantage of Microsoft's existing technologies and knowledge-base. Discussing our plans with the Windows Azure team, for example, helped us overcome some serious hurdles with our data-storage methodology. Without their help, we might have been derailed at the beginning.

What Current Users Want

At the beginning stages of the project, we simply emailed a few of our most vocal users and asked them questions about what types of visualizations they might use, how often the tool needed to update, how they were using data, etc. Asking specific questions instead of relying on general feedback helped us identify key issues and needs we could easily act on.

User personas

From our sit-downs and email conversations, we were able to segment our audience into 'personas', defined as representations of real users, including their habits, goals and motivation.

There are many ways to approach user personas. Some designers spend a long time fleshing out the behaviors and market segmentations of a target audience. We, on the other hand, took a more rudimentary approach, following the example of Smashing Magazine great article on the same subject.

Drive-by vs. Return Users

We know that most users will come by the site once, do a quick search, check out some visualizations, and then leave. That’s just the nature of building websites, and especially one geared toward a niche audience.

We wanted to provide an engaging experience with few barriers to entry for the drive-by user, but it was more important to concentrate on bringing people back. So, we decided to focus on our return users, Marketing Managers and academics.

The Marketing Manager

Marketing Managers primarily care about tracking sentiment—who is tweeting about what, how often, and how people feel about a brand or site over time. But up until recently, there were only a few ways to gain access to targeted Twitter data, most of which was presented in a very raw format that was difficult to extrapolate from.

Our plan was to change all this with The Archivist, and make it easy to find, share and interpret large swaths of data suited to the Marketing Manager's needs.

The Academic

Because of our experience with The Archivist Desktop, we knew that there was a surprisingly large demand for the Archivist in academia. During the last year or so, we’ve talked with with numerous professors and students who have found The Archivist Desktop useful for their research.

Whereas the Marketing Manager wants to track 'sentiment', the academic is interested in raw data that's accurate, easy to access and appropriate for use in research.

Wireframes

I thought it might be useful to provide the full PowerPoint walk-through of our wireframe set, which gave our team an understanding of how different users might come to and interact the site.

What we couldn't do (Feature Scoping)

Coming to agreement on the wireframes was easy, but brought up questions about what data we could actually display to users. As I noted earlier, I spent a lot of time thinking about the frequent requests for filtering by date, secondary keywords and other items. Because of how Twitter works, some of this data wasn't consistently available. Other items were so gnarly we couldn't scope them into this release.

The Visualizations

Visualization explorations

As Karsten and I were wrapping up the early Archivist Desktop version, I explored additional visualizations I thought we might be able to incorporate into the downloadable WPF application. Since we were working in WPF at that point, I felt free to explore visualizations that might be useful to our audience. Some that I came up with:

For a few specific reasons, we decided to ship The Archivist using the ASP.NET charting contols, and made the decision to feature the following charts.

Tweets over time
Top Users
Top Words
Top URLs
Tweet vs. Retweet
Top Sources (software)
Use ASP.Net Charting Controls

In my next post, I'll talk about taking the site from wireframes and explorations to a functional HTML + CSS3 + jQuery site, and how I worked with Karsten, who made the beauty of the backend happen in the cloud.

Do you use the same process as us, moving from problem to persona to wireframes to design? Leave a comment and be sure to follow us on Twitter @Mixonline.

Less is More, More or Less

Nishant Kothary — Thu, 08 Jul 2010 21:54:27 GMT

I recently learned something extremely fascinating from a book called Gut Feelings: The Intelligence of the Unconscious. In Greg Gigerenzer's own words:

Intuitions based on only one good reason tend to be accurate when one has to predict the future, when the future is difficult to foresee, and when one has only limited information. They are also more effective in using time and information.

What Gigerenzer means is that people generally rely on one good reason to make decisions when the outcome is truly unknowable. And, that's OK! It's worked wonders for us for thousands of years.

But the more interesting insight hidden in there for us designers is that our best bet at predicting the future for something truly unknowable—say, how an audience will react to a new product, or what'll happen to the price of a certain stock—is often to follow your gut.

Mo Options Mo Problems

Now, this may seem preposterous to our spreadsheet generation, because it runs counter to the currently dominant microeconomic paradigm called Rational Choice Theory. According to it, human beings are rational—Homo Econonomicus—and given a set of alternatives, will weigh the costs and benefits of each to pick the most suitable one. Increase the alternatives, and you increase the chances of finding a more suitable pick, right?

The credo "more options are better" has become a staple in our generation. From supermarkets that carry hundreds of different types of cereals to our propensity to follow thousands of people on Twitter, we've given in to the notion that more choices lead to better outcomes. Sadly—and you know this in your gut—the exact opposite is true in most cases. More options lead to confusion. Not conversion, as many marketers would have you believe.

Gigerenzer's own take on that:

Complex analysis, by contrast, pays when one has to explain the past, when the future is highly predictable, or when there are large amounts of information.

Gigerenzer cites numerous studies in his book supporting this critical point, but there's a classic one we're all too familiar with—Apple.

An Apple a Day

When Jobs took over Apple in 1997, Apple sold over 40 different types of products, from printers to handhelds. Particularly overwhelming was Apple's lineup of computers, which hovered around 50 different models. Which Performa is right for you? The 5200CD or the 5215CD? Jobs later confessed that even he couldn't figure out the answer to that question. So how could customers?

Among the other unpopular restructuring decisions, Apple cut most of its product lines and reduced its computer line to four machines: novice and professional versions of a desktop and a laptop. History will remember it as a key step in facilitating Apple's comeback.

But, why is it that too many options paralyze us while fewer options spring us into action? Entire books have been written on the topic, but if you want a succinct explanation, you need look no further than Miller's Law.

Miller's Law: The Magic Number Seven

Princeton's celebrity cognitive psychologist, George Miller, published the seminal paper on the topic of human cognition in the Psychological Review in 1956. In it, he argued that the average human could hold 7±2 objects in working memory; this is now fondly known as Miller's Law. Miller's insight has had an immeasurable impact on the design of everyday things in the last 50 years.

The classic application of Miller's Law in the software industry is when a designer tries to convince stakeholders that the navigation menu of a web site needs to be limited to seven (give or take two) choices. But more generally, we can credit our subconscious pull towards "keeping things simple" to Miller; at least within the context of user experiences. That is no small feat.

Now if only someone can do the same for the benefits of white space.

KISS my Point

Your users are already overwhelmed. Jonathan Anderson of UXMag illustrated this in a very eye-opening post—These are your users... read and be horrified. But as Gigerenzer writes, even when they're not overwhelmed, they make their choices based on one or two criteria. Providing your users with more options is generally counter-productive. Of course, there are exceptions to this rule of thumb.

The bottom-line? Prune. Cut. Shorten. Simplify.

And once you're done?

Simplify some more.

How The Archivist Polls Twitter

Karsten Januszewski — Wed, 07 Jul 2010 17:58:19 GMT

You may be wondering how frequently The Archivist updates archives. Well, the answer to the question is more complicated that it may first appear. Let’s dig in.

The Archivist interacts with Twitter using the Twitter Search API, which it polls at variable intervals based on the frequency with which a particular archive is updated. We call this the elastic degrading polling function. This algorithm helps The Archivist be a good Twitter citizen, allowing us to poll Twitter conservatively while at the same time maintaining archives with the latest tweets.

Here’s how the algorithm works: When a user makes an archive ‘active’, the polling process begins. Every archive is inspected once an hour to determine how ‘hot’ it is. We determine how hot an archive is by recording how many results we get back each time we poll Twitter (the maximum we can pull at any one time is 1500). We use this number to determine how frequently to poll Twitter for that archive. Depending on the number, we either hold off on polling for a given interval or query again, based on the following buckets:

So, let’s look at an example. Say we have an archive going for the term ‘Wittgenstein’. When the Archivist checks on this archive at 10 AM, it discovers that the last query for Wittgenstein only returned 10 tweets. It also discovers that this archive was last updated at 9 AM. The Archivist won’t poll Twitter for this archive, because the tweet count isn’t high enough and the archive had been queried within 24 hours. Since the archive is in the 24 hour bucket, the same thing will happen when The Archivist checks on this archive each hour.

Once 9 AM rolls around on the next day, since 24 hours have passed, Twitter will be polled for the Wittgenstein archive .

Now, let’s say for some reason there’s a flurry of tweets about Wittgenstein—when that archive was updated at 9am, it pulled 600 tweets. In this case, the archive adjusts because it has become hot. It is now in the 1 hour bucket instead of the 24 hour bucket. So, when 10 AM rolls around, the Wittgenstein archive gets updated again.

But let’s say at 10 AM it pulls only 250 tweets. Well, now the archive moves to the 8 hour bucket. So,the Wittengenstein archive will not be polled again again until 6 PM. Let’s say it pulls 1000 tweets. Well, it goes back to the 1 hour bucket, since it appears to be hot. At 7 PM the term is checked again. This time, the response is only 10 tweets. It seems to have cooled off quickly, so we’ll move it back to the 24 hour bucket.

Some of you may notice that there’s a chance that The Archivist could possibly miss tweets when a term becomes hot. This is a reality of our architecture and is justified by the following: First, once a term gets hot, the amount of data can grow quickly. Ultimately, in that scenario,The Archivist becomes a statistical sample as opposed to a true historical record. Second, Twitter itself doesn’t guarantee that all tweets will be returned for a given search. See http://help.twitter.com/entries/66018-my-tweets-or-hashtags-are-missing-from-search and http://dev.twitter.com/doc/get/search for more on this. Consequently there is no way that The Archivist can ever claim to be a true historical record. Third, The Archivist is optimized for following non-trending topics over a long period of time, as opposed to trending topics over a short time. For a tool optimized for the latter scenario, see Archivist Desktop. Another option would be to run your own instance of The Archivist Web and tweak the polling algorithm, which would be trivial to do. Contact me if you are interested in doing so.

The Beginning of The Archivist

Tim Aidlin — Fri, 02 Jul 2010 20:35:04 GMT

A couple years ago, my friend irhetoric and I were chatting about Twitter, as we were at the end of building Flotzam, a social networking aggregator tool that collects and visualizes the ephemeral information floating around in the social sphere. In some ways, this ephemerality is beautiful: all this information swirls around us like leaves that appear and decay over time. On the other hand, if this information is valuable, capturing and keeping it could be useful.

The most striking—and easily accessible—of these potentially useful data streams is Twitter. Tweets, while often inane or funny or what-have-you, can impart important information, whether by themselves or in aggregate. Take, for instance, Twitter’s role in the Iranian Elections of 2009, in which Iranian protesters used Twitter to broadcast information that would have otherwise been suppressed, using the hashtag #iranelection. The election also provoked people from all nations to “show solidarity with the protesters” by changing their Twitter avatar to green. What good this did is unclear, but it’s an interesting way of disseminating information and sentiment.

The problem, though, is that you can only view tweets as far as two weeks in the past (or less for very active terms such as #iranelection). Fortunately, by 2009 Karsten and I had designed and developed The Archivist Desktop for MIX Online, and were able to save and capture this data. We ended up with 22,000+ individual tweets over a 2-week span.

The Archivist Desktop

The Archivist Desktop was originally released as a “mini-lab” and has been installed by users from such varied fields as marketing, software development, social sciences and research. It was a ‘small project’, but Karsten and I thought it was a worthwhile for a few reasons. With The Archivist, users are now able to:

Save tweets
Users can save the tweets they’re interested in.
Export the data to analyze
The Archivist provides an easy way to export to Excel for visualizations and data-mining.
Visualizations to interpret
There are two useful visualizations shipped with The Archivist, which can provide quick and easy understanding of the data the user has collected.
Distributed network
The Archivist is an installable program and works on a distributed network, which reduces the likelihood of one user being affected by another. This is especially important when dealing with Twitter rate-limiting.
Private, secure data
Sometimes users are sensitive about the data they’re collecting. By having an installable program, the data is saved to a user’s hard-disc and accessible only to those users who have direct access to that computer and file.

Challenges with The Archivist Desktop

While The Archivist Desktop is great in many ways, we needed to address a few problems we saw as the project matured. Over time, we were able to connect with users around the globe and encountered a few recurring requests:

“Always-on and connected?”
Because The Archivist Desktop is an application, it needs two things to work: it must be running and have an internet connection. If your computer goes to sleep, for instance, the application stops running. Or, if you’re running it on a laptop and you travel, you get no connectivity, no archives.
“Not Just Windows, please.”
Many of our users, including some of the MIX Online team at times, are Mac users. By making this project Windows-only and relying on various other technology dependencies, we were shutting out a large and potentially enthusiastic part of the community.
“I’d rather not install a program.”
We found that a large population of users were hesitant to install a program they were unsure of. The Archivist is a small file and, for Internet Explorer users, a one-click process. To many users, however, installing a program is a hurdle that’s very difficult to cross.
“I wish I could go further back in time.”
Suppose User A started an archive on the term “Microsoft.” If four months later User B started an archive on the same “Microsoft” keyword, all of the data User A collected would not be available to User B. That seems a waste. Data should be ultimately shareable. Free the data!
“More eye-candy, please.”
The users of The Archivist expressed they found great value in the data-visualizations The Archivist provided, but wanted more. The data visualizations people asked for were often the same, and included top keywords, top users, top links, and volume over time. There were also many other great suggestions on how we could parse the data from Twitter. Users had very specific requests that coincided with their goals, whether they were focused around brands, politics or pop-culture.
“I’d rather share a link or image than a bunch of data.”
While it’s easy to attach an Excel spreadsheet or .xml file to an email, that isn’t the best experience for sharing data or understanding around that data. How could we help people take full advantage of this new data and information?

The Archivist Web Alpha

We decided to move The Archivist to “the cloud” and create a web-based, Windows Azure-powered web tool. There are numerous benefits to creating the new version:

Always-on
By creating a web service, we were able to take advantage of the always-on nature of the internet and could poll Twitter for data whether the end user was connected or not. This enables users to set up archives without having to monitor their status more than once every 30 days to keep the archive active. We no longer needed to require users to actually run the program—because essentially, we’re running it for them.
Cross-platform
The Archivist Web runs on multiple browsers and platforms. This opens up The Archivist to essentially all internet-connected users. Some sort of authentication is required, however, to save archives. To make this as seamless as possible, we tapped into the Twitter authentication system and now have people simply use their Twitter credentials to authenticate themselves.
Data-sharing
By making The Archivist a cloud-based application and storing collected information in shared database, we can make archives that have already been created available to new users. For instance, if User A creates a “Microsoft” archive on January 2 and User B starts a “Microsoft” archive on March 2, User B’s archive should reach back to January 2. This data is unavailable through a normal Twitter search. This feature, in our opinion, is super valuable.

An additional type of sharing is the ability to tweet both The Archivist Web itself and your archive set—which will only display those archives you have set to “public” – individual archives, as well as individual visualizations within that archive. It’s as easy as sharing the url that appears in the address bar at the top of any browser.
No installation required
By no longer requiring installation, users can use the service without the commitment of trusting the source, waiting for the download, ensuring they have the right system requirements and the other barriers to entry.
More visualizations
Our users had asked for a lot of new visualizations. Of course, we had ideas on what *we* would like to see and found useful, but talking to our current and potential future users helped us determine which visualizations were the most useful to our audience.

In my next Opinion, I’ll be writing about the process of scoping the new project to our resources, time-constraints and technical limitations. From there I’ll try to explain the process of releasing a project and making it be successful.

Don't Talk Down to Me!

Joshua Allen — Thu, 01 Jul 2010 18:36:03 GMT

In school, we’re taught that we should write at a sixth to eighth grade level when writing for the general public. Newspapers have historically required their writers to write at a low grade level, and in his book "The Vanishing Newspaper", author Philip Meyer argues that newspapers are struggling, in part, because "Many newspaper stories are too hard to read". Meyer arrived at this conclusion by measuring thousands of newspaper articles with the Flesch-Kincaid readability test. He found that only a quarter of the articles were written at or below the eighth grade level. As a point of comparison, the Flesch-Kincaid grade level for this post so far is 10.4.

Of course, readability is just one element of simplicity. We praise students who are good at communicating complex topics in simple language, and Albert Einstein even said, “You do not really understand something unless you can explain it to your grandmother.” But I’m worried that all of this talk about simplicity may be missing the point.

Another Perspective

Tyler Cowen is an economist at GMU, and a founder of the hugely popular "Marginal Revolution" (MR), an economics blog I’ve enjoyed for several years. Cowen and blog co-author Alex Tabarrok consistently post on complex topics and never "talk down" to readers. Despite this, the readership has steadily grown, and MR boasts some of the most engaged and thoughtful commenters on the Web. In fact, Cowen sometimes writes posts he knows will be above most readers’ heads. Far from repelling or discouraging readers, most find this challenging and inspiring.

One could argue that MR has a unique audience, and Cowen certainly wouldn’t disagree, but I’ve noticed a similar pattern in other places. Our team runs a couple of large conferences each year: the Professional Developer’s Conference (PDC), for developers, and the MIX conference for Web developers and designers. Attendees can turn in session evaluations for each session, including verbatim comments, so we can evaluate speakers and topics. If we’ve learned one thing, it’s this: if you talk down to attendees or make your content too simple, you’ll be slaughtered in the evaluations. But if you challenge the attendees, you’ll make them happy, even if many find your content hard to follow.

Making Grandma Work for It

Does this mean that blog readers and conference attendees are vain posers who need to be treated like "big kids" to feel good about themselves? I don’t think so. Over the past few decades, research into a phenomenon called "contrafreeloading" has provided some clues that help explain why people don’t like being talked down to. In short, "contrafreeloading" means that animals, including humans, prefer to work for their food if given the choice between completely free food and identical food that requires work to obtain. There are only two exceptions to this rule: housecats and economists are both perfectly rational, and will happily freeload when given the chance.

The implications are clear. If it’s a topic she’s interested in, Grandma will prefer to work things out for herself rather than be spoon-fed. Don’t water things down, and don’t talk down to her!

What do you think? Do you have situations where spoon-feeding is definitely the right thing to do? Or have you had spoon-feeding backfire on you? Leave us a comment below.

Take Social Back: A Call To Action

Laura Porto Stockwell — Thu, 01 Jul 2010 16:40:32 GMT

I’m not quite sure how or why it happened, but at some point, the user experience community—the group of people who are truly experts in interaction design—lost their grip on the social media space.

In the void, a number of self-appointed social media experts (some knowledgeable, some completely faking it), have catapulted themselves into prominence. The success of these so-called experts has much to do with the fact that many of them are marketing inclined, and thus talk the talk. Proof positive can be found on Facebook, where the majority of corporate pages are simply channels for one-way messaging.

The thing is, social media does not play by traditional marketing rules. It plays by digital rules, where conversation is key and authenticity reigns. “Doing it right” means knowing how to develop and design interactive programs and environments.

Why Did UX Drop Out?

We, as user experience professionals, understand the rules of interaction, yet are reluctant to play. Why? If you’re a user experience pro, my guess is that you bailed out of the social scene for one or more of the following reasons:

You’re Bored.

You were probably one of the first users on Facebook. Now everyone, including our mother, is there and you’re ready to move on. How could you possibly help a client develop a Facebook strategy when you, yourself, can’t stand updating your status?

Remember that social media is more than Facebook and Twitter. There’s always a popular channel—that’s easy. But developing a strategy is much more complex, and way more interesting than you think.

Strategy starts with users and their needs and propensities. Are they creators or collectors? Do they blog or rate? Once we understand users’ needs, then we can develop a meaningful strategy, whether that be on Facebook or LinkedIn or a stand-alone experience. You should try it sometime.

You’re Extremely Hip.

Social media is very much en vogue, and you’re just too cool to be following trends. You’re a leader, after all (see point number one). If you have to read about someone else’s lunch again, you just might lose yours.

But consider this: throughout history there have always been certain rights attached to certain people. Some folks get to talk, some folks don’t. And that was true until very recently.

Somewhere in the mid 20th century, somewhere between William James and ARPANET, somewhere in the span between Hunter S. Thompson and fanzines, we (in the broadest possible sense) found our voices. In our post-aural, post-structural, post-modern society, communication is a right, not a privilege.

And that’s why you should rejoice to the high holy heavens every time that guy posts what he ate for lunch—because it means you live in a world where anyone can say anything at any time, when anything can be published to anyone in the world instantly.

This is bigger than you. Embrace it.

You Don’t Want to Be Associated With the Crap

You’ve witnessed a number of brands and marketers dabble in the social media space and you’re embarrassed by the display of low quality experiences. You should be. It’s pathetic. But that’s because marketers are trying to apply old marketing tactics to entirely new paradigms.

Marshall McLuhan said that all forms of new media take on the likeness of existing forms before they find themselves. That’s what happened to the Web (brochure-ware anyone?) and that’s what’s been happening to social media. The space is in dire need of experts who really understand it.

If you loathe the crap, stop whining and do something about it. Commit to bringing your “A” game to the table and treat social seriously.

You Cut Your Teeth At the Wrong Time

You entered the industry just after the Internet bubble bust, when budgets were very tight and there was no room for experimentation. During this time, community functionality (what we used to call social media) was often stripped out of projects, as was anything that wasn’t absolutely necessary.

It’s not your fault, but it’s time to learn. See the next point for a good reading list to get started.

You Just Don’t Get It.

No, it’s not that you don’t care, or that you’re too cool. You just seriously don’t get that the Web is social. But here’s an analogy that might help: separating social from digital projects is like separating plumbing from a kitchen redesign. It makes no sense.

For every project you strategize or wireframe, you should be thinking social. It’s not separate.

If you’re not sure where to start, there are numerous academics who have been writing about social media since The Well (Google it). For example, Howard Rheingold, or Barry Wellman at the University of Toronto. Henry Jenkins at USC and Jenny Preece at the University of Baltimore. There’s also former Microsoft researcher Marc Smith, and currently with Microsoft, Danah Boyd. Clay Shirky at NYU and Charlene Li with the Altimeter Group. Entire communities of people exist who study social media and their research and thoughts are easily assessable.

It’s Time To Step It Up

It’s time for the user experience community to step it up and realize that we have a real contribution to make in the social media space. Social media is about identity and reputation management, user needs and behaviors, motivations and interaction—all user experience specialties.

Don’t let the space go to so-called experts, some of whom weren’t even in digital three years ago. And please don’t make me hire a social media specialist for my team when I really just want to hire a user experience pro with a solid understanding of social media.

The time is now. More is asked of you. Let’s take it back.

The Archivist In Academia

G. R. Boynton — Mon, 28 Jun 2010 17:49:25 GMT

When the Nobel Committee announced that President Obama would receive the peace prize, how many tweets did that produce? 90,000. When Oprah announced that she would end her TV program when the contract was completed, were there more or fewer than the Obama tweets? Fewer. 73,000 fewer. How many tweets have there been about health care reform (#hcr)? Over 600,000.

Using The Archivist In Class

In the fall of 2009, I wanted to teach two courses about politically oriented messages on Twitter. My students were going to be the first in the country to have an entire course devoted to this subject. But there was a problem: how could we learn how many tweets there were about Obama receiving the Peace Prize? And anything else we might want to know?

It was difficult to find a solution. There are many systems for searching Twitter, including Twitter itself, which allows you to enter a search term and watch the tweets flow by. But we needed a system that would not only search for tweets, but also count, collect and export them. It turned out that The Archivist was the only system that could do the job.

Politics On Twitter

Many believe that Twitter messages are largely about the concerns of an individual. They interest a few friends, but have little wider interest. That may be true of most messages, but my students found that politically oriented tweets aren’t like that at all; People interested in politics circulate a lot of ideas, and those ideas are as likely to come from outside Twitter as in. URLs are much more frequently found in politically-oriented twitter messages than in other kinds of messages.

Even more convincing: Twitter was the tool of choice for celebrating President Obama's election in the fall of 2008. The number of tweets about his inaugural address in 2009 broke every record up to that point.

By studying Twitter, my students were able to explore a lot of politics—comparing members of congress tweeting with members of the British House of Commons, for example. What they did was only possible because of The Archivist. And the new web version is going to be even better.

note website on Bob’s computer :)

Introducing The Archivist

Karsten Januszewski — Mon, 28 Jun 2010 17:47:10 GMT

The Archivist is a new lab/website from Mix Online that lets people archive, analyze and export tweets. Here’s a little more about why we built The Archivist and who we built it for.

The Transitoriness of Twitter

Twitter isn’t going away anytime soon. From Twitter:

The tweets keep coming. And coming. Exponentially.

Then again, tweets are disappearing all the time. Tweets come and go. They’re ephemeral.

Tweets are transitory because that’s the way Twitter is designed. Twitter doesn’t allow access into its database for more than two weeks—and for a trending topic, you may not be able to access tweets older than 24 hours.

Recently, Twitter announced that it’s going to donate its entire archive of public tweets to the Library of Congress. How cool is that?

Bummer, though, that the database of tweets isn’t accessible yet.

Why a bummer? Because there are all kinds of things you can find when analyzing a dataset of tweets. Patterns emerge from the noise. Discoveries are made.

So how do you capture all those tweets before they disappear into the ether?

Enter The Archivist.

The Archivist: Archive, Analyze and Export Tweets

The Archivist sets out to accomplish three things:

Archive tweets
Analyze tweets
Export tweets

Now, to be clear, the Archivist is very different than what the Library of Congress will make available. The Archivist is not an archive of all tweets. The Archivist can’t go back in time.

But what The Archivist can do is allow individuals to begin archives right now on searches they care about.

Let’s take a look at features of The Archivist.

Creating Archives

The Archivist can create archives of tweets equivalent to any search done at http://search.twitter.com. But there’s a difference between searching directly on Twitter and searching from The Archivist: When you start a search from The Archivist, your search will continue to run—day and night, every day, every night. And The Archivist will create an archive based on that search.

Any tweet that matches your search gets saved and is available for you to peruse. You can come back later and see what’s been harvested in a day, a week or a month.

Again, it’s important to note that The Archivist can’t go back in time to get old tweets. But once you start an archive, you can be sure that, moving forward, The Archivist will monitor that term. So, using The Archivist takes some forethought: you need to know in advance that you’ll care about a given set of tweets later.

Doing Analysis

So now you’ve got a few giant archives of tweets based on your searches. What to do with an archive of 1,000 tweets? 10,000? 100,000? 1,000,000?

Well, this is where data mining comes into play. The Archivist data mines on your behalf, providing six visualizations that offer insight into a given archive of tweets.

The six visualizations are:

Volume Over Time
Top Users
Tweet vs. Retweet
Top Words
Top Urls
Source Of Tweet

These six visualizations are shown in a dashboard.

You can click each chart for a larger view with more data and interactivity. You can also immediately see a top user’s Twitter profile or go to the web page of a top url, for example.

You can read more about the specifics of these visualizations.

People often want to do deeper analysis than the six visualizations provided allow, or combine visualizations. (For example: “Who were the top users on a given date for a set of tweets that weren’t retweets?”)

Unfortunately, we can’t provide that level of analysis. But we know how you can do it, which gets to our next topic: Export.

Exporting Tweets

If you want a DIY analysis, you can download an archive and start slicing and dicing it yourself.

We allow all archives to be saved as tab delimited text files. These are easily loaded into Excel, imported into a database or parsed with a program. Here’s a screenshot of an archive that’s been pulled into Excel 2007 and then had pivot tables created:

And, with Excel 2010 and the PowerPivot plug-in, huge archives can be turned into pivot tables with lightening-fast analysis.

Who Is The Archivist For?

We designed The Archivist around three core scenarios. The first is what we called the “drive-by” scenario; the second is the “data miner”; the third is the “developer”.

The Drive-By User

The drive-by user is someone who comes to the site simply to get a real time snapshot on a given search. He/she doesn’t sign in, but remains anonymous on the site. The drive-by user can view anyone else’s public archives and do searches, see visualizations and even export searches. However, any searches started by the drive-by user are not persisted. That’s because you have to sign in if you want The Archivist to start archiving.

Searches by the Drive-By user create a non-persistent archive, including visualizations, of 500 tweets and will display the recent 100 tweets for that search:

So, for example, you could do a drive-by search on a term and quickly get a sense of what is happening with a given search. But, because you didn’t sign in, The Archivist won’t continue to monitor that search.

The Data Miner

The Data Miner is another scenario. The key difference between the drive-by user and data miner is that the data miner signs into the site. Authentication happens through Twitter. Once the data miner has signed into The Archivist, his searches are saved and persisted.

So, going back to the scenario, a data miner signs in and starts a search. The Archivist then continues to monitor that search term ad infinitum. If the archive gets to a certain size, The Archivist will close it and start a new one based on the same search.

Data miners have a homepage on The Archivist that coordinates with their Twitter username. So, for example, if I sign into The Archivist with the username of mixonline, I will have a homepage on The Archivist that looks like: http://archivist.visitmix.com/mixonline. I can see all the archives I created, how many tweets are in each archive, and whether the archive is active/closed and public/private on the homepage. Here’s an example of an Archivist homepage (signed in):

Note that you can see each archive’s size, activity and whether it’s public or private.

Data Miners can make an archive available to the public. By default, though, all archives are private. Here’s a screenshot of what my profile page looks like when someone else hits it:

Only archives that I’ve made public are available for others to see.

Archives created by Data Miners can grow to 500,000 tweets. Once an archive hits that number, it gets closed an a new archive gets started for that term.

The Developer

We’d be remiss if we didn’t provide a way for developers to access all this data and analysis. That’s why all public archives and data analysis we do to create our visualizations are available for download programmatically. So you could download the dataset of top users for a given archive and then create your own data visualization with that data, for example.

We’re also providing all the source code used to build The Archivist (You can download it here). With this, you can run your own instance of the Archivist. Read more about the developer documentation here.

In Summary

We are excited to release The Archivist and look forward to your feedback. Give it a whirl and let us know what you think!

Using Flotzam At A Conference

Karsten Januszewski — Tue, 15 Jun 2010 21:34:28 GMT

Flotzam at a conference is fun. It gives the audience the ability to immediately participate in the event—their photos, tweets, etc, show up on the big screen, giving them immediate satisfaction and making them feel like they are part of the event.

If you’d like to see Flotzam in action, you can try it out by installing it yourself. Or watch videos that shows off the application on YouTube.

Here’s a guide for using Flotzam at a conference.

Flotzam and Social Networks

Flotzam works with a variety of social networks. Here’s a brief intro:

Twitter integrations lets you see what people are saying about the conference in almost real-time.
Flickr and Facebook integration allows attendees to upload their photos of the event; Flotzam will pull and display the photos.
If there will be a lot of on-the-fly press about your conference, Flotzam’s Digg integration will be helpful.
Flotzam will display any RSS feed
Flotzam lets you see and display Youtube videos about your conference

The end result is that Flotzam aggregates all these networks and creates a compelling visualization.

A few things to note:

If you want to add another social network to Flotzam, you’ll have to write some code. Here’s how to do it: http://visitmix.com/labnotes/How-To-Add-a-New-Data-Source-to-Flotzam. If the social network you want to add supports RSS, it will be easier to integrate.
If you don’t like the skins we have, you can find a designer fluent in Expression Blend to reskin it, as discussed here: http://visitmix.com/labnotes/How-To-skin-Flotzam
Because Flotzam pulls data in real time, there’s a chance that inappropriate content could appear on the screen.

Getting Flotzam Up and Running

If you want to use Flotzam for your conference, you need to do the following:

Install Flotzam here: http://visitmix.com/labs/flotzam/install/flotzam.application or download the source here: https://code.msdn.microsoft.com/Release/ProjectReleases.aspx
You should have a machine with w/2GB RAM and a video card of 256MB. Be sure to test on your hardware to catch any issues in advance.
The machine must have network access. Flotzam does not do well with intermittent connectivity; if you can’t guarantee a stable connection to the internet, don’t use Flotzam.
Determine what feeds you want Flotzam to display.

Note that you’ll need to show attendees how to access the services you set up. If you don’t want to do this, one option is to simply have your conference folks update twitter and upload photos.
Here’s how to configure Flotzam:

All settings can be changed by clicking SETTINGS AND OPTIONS in the upper left corner.
If you want everyone at the event’s tweets to appear, you can create a Twitter account such as myconference, let people know about it, and then tell Flotzam to “Watch My Followers.” To do this, visit the Twitter settings panel and provide the username and password for the myconference Twitter account.
If you set up a Facebook event for your conference, you can enter the Event ID in the Facebook settings panel. If the URL for the event is http://www.facebook.com/event.php?eid=2367953648, the event id would be 2367953648. You'll have to provide your credentials to Facebook.
You can add a different Flickr tag in the Flickr settings Panel.
You can specify a DIGG setting and a YouTube tag.
You can manage which RSS feeds show up, but you’ll have to do so in Internet Explorer.
Lastly, select from the dropdown in the general settings tab if you want a different skin to show up.

How To Skin Flotzam

Karsten Januszewski — Tue, 15 Jun 2010 21:13:29 GMT

Here’s how to skin Flotzam:

1. Download the latest Flotzam source code and open Flotzam.sln in Expression Blend.
2. Hit F5 to see the application working. You should see different "flotzams" appear on the screen. By default, you will see public Twitters and Flickr images as well as videos posted to YouTube. You can change what Flotzams you get (personalize them, etc.) by tweaking the settings.
3. If you want to see prettier Flotzams, you can change the skin in the "General" tab of the settings. These are skins that Tim Aidlin made.
4. Now it’s your turn to make a skin. In Blend, go to the Resources tab and find the file new.xaml. Expand it and you should see seven templates. If the Resources tab is blank, go to the Project tab, open the file Window1.xaml and then go back to the Resources tab. You should see new.xaml. It should look like this:

5. Open DIGG_TEMPLATE. There you see the fields you have to work with in the designer.

6. Expand the stack panel. Inside the StackPanel you can see the visual tree with the different fields you have to work with.

7. Double-click [TextBlock] "Username". Change to the Properties tab. Notice that the Text field (under Common Properties) is highlighted in yellow. This is because the Text field has been databound to the Username field. DON'T MESS WITH THE DATABINDING. But feel free to mess with anything and everything else. For example, you can copy the whole TextBlock into a different container; you can change any of its other properties, but don't mess with any field that is highlighted yellow.

8. A default animation has been applied to each of the templates. You can see it by clicking the >> in the Objects and Timeline pane. You will now see OnLoaded1. This animation is very simple: it just fades out the entire Canvas over 5 seconds. To remove this animation, just go to the Triggers pane and click the minus arrow.

9. By default, Flotzams are placed on the scene at random. This is done from code. If you want to turn this off (so that they always enter stage right in the middle of the screen), open Window1.xaml.cs. On line 27, change the randomizerFlag to false.
10. You’re also welcome to change the background of Window1.xaml.

How To Add a New Data Source to Flotzam

Karsten Januszewski — Tue, 15 Jun 2010 21:13:08 GMT

Here’s how to add a new data source to Flotzam.

1. Create a new data model class that derives from DataModel in data.cs – look at the other derivations of DataModel to see how the model works
2. Add this class as a new Object Data Provider to ~~datatemplates.xaml~~ storyboards.xaml
3. Add new member variable for data at the top of window1.xaml.cs
4. Load member variable and activate it at line 194 of window1.cs.xaml
5. Add to Settings
5a. Create new tab for settings
5b. Add settings to Settings Property in Visual Studio Properties
5c. Update save_click for when user changes settings (window1.cs.xaml line 82)
5d. Update settings form in window loaded
6. Make a new datatemplate (or else muck with the data model to trick the data binding into working by renaming properties) in datatemplates.xaml
6a. Do databinding
7. Update PickPanel() method in window1.xaml
7a. Increase random picker by one
7b. Add to giant if statement
7c. Add member variable to count the items
7d. Update line 299 and add that setting to the condition to prevent infinite loop

Using VaryByCustom With OutputCache in ASP.NET MVC To Support Caching For Logged-in Users

Karsten Januszewski — Wed, 09 Jun 2010 17:32:37 GMT

Several pages of an upcoming Mix Online lab are fundamentally static, except for a single hyperlink that toggles depending on whether a user is logged in or not. To optimize the site, I wanted to use caching on these static pages. But by adding the OutputCache attribute to these pages, a user could end up getting a cached page with the wrong logged-in hyperlink: users who weren’t logged in would see a page that stated they were logged in—or vice versa.

The solution? Using the VaryByCustom parameter of the OutputCache attribute, so that both a logged-in and a logged-out version of the page are cached and served up appropriately. Here’s how to do it:

First, add a method to the global.asax.cs file, which overrides the VaryByCustom call:

    public override string GetVaryByCustomString(HttpContext context,
    string arg)
    {
        if (arg == "IsLoggedIn")
        {
             if (context.Request.Cookies["anon"] != null)
             {
                  if (context.Request.Cookies["anon"].Value == "false")
                  {
                       return "auth";
                  }
                  else
                  {
                       return "anon";
                  }
              }
              else
              {
                 return "anon";
              }
        }
        else
        {
            return base.GetVaryByCustomString(context, arg);
        }

    }

Then, decorate the controller actions with the OutputCache attribute:

    [OutputCache(CacheProfile = "StaticPage")]
    public ActionResult Index()
    {
       return View();
    }

When using caching, I like to keep all the profile information in web.config. This lets me update things without recompiling. Here’s what the cache profile looks like:

   <caching> 

        <outputcachesettings>             
            <outputcacheprofiles> 

                <clear /> 

                <!-- 24 hours--> 

                <add varybycustom="IsLoggedIn" varybyparam="*" duration="86400" name="StaticPage" /> 

            </outputcacheprofiles> 

        </outputcachesettings> 

    </caching>

Notice how the varybyparam attribute is set with an asterisk, and the varybycustom attribute has the string IsLoggedIn, the term that gets passed to the VaryByCustom override.

I’m using my own kind of authentication (more on that in a future post), so checking a cookie is sufficient. If you’re using ASP.NET Membership Authentication, you can use slightly different logic such that pages are cached based on the username itself. See this post for a sample—and tip o’ the hat to Chris Mullins for setting me on this path!

Accountability On The Web

Hans Hugli — Fri, 04 Jun 2010 20:08:29 GMT

This is what a prosecutor had to say about the incidents:

"You hate to tell people to be wary of your fellow citizens, but the reality is you've got to be wary when you are doing something when interacting with strangers on Craigslist," prosecutor Lindquist said. (CNN, 2010)

This is Craigslist’s "answer to the problem". Seriously? Is simply "trusting your instincts" and "meeting in a public place like a café" enough? I don’t think so.

The Accountability Problem

Because most people who use sites like Craigslist do so anonymously, accountability is a challenge. You can never be sure that who you’re communicating with is who they say they are. And even if someone truthfully identifies him or herself, there can still be malicious intent.

Since we can't measure intent, there is one obvious way around the problem: Make people correctly identify themselves.

Anonymity and Privacy

To log into Craigslist, all that's required is a verifiable email account, which can be created in 5 minutes by anyone. This protects account owners from having others modify their information, but it doesn't remove anonymity. No one knows who the person on the other end of the email transaction is.

The Craigslist model itself is also a problem. Like "ads wanted" in newspapers, anyone can apply. And unlike newspaper ads, using Craigslist is completely free, so there’s virtually no barrier to entry.

Facebook faces similar challenges. Last year the company implemented OpenID on their site, making it possible for users to log in with a single identity. This is a big step in the right direction, but it doesn’t help verify that people are who they say they are.

At the heart of the anonymity problem is a concern for privacy. How do we force people to identify themselves without invading privacy? How do we protect privacy without neglecting accountability? Case in point: Many Facebook users recently boycotted the site because of its "overly complex" privacy controls and "like" button, which shares too much information with others.

The crux of the issue is that privacy is at odds with a site that's completely free to users. Facebook's business model, for example, requires information about users to survive financially. This is a huge problem that shows just how difficult it is to solve the identity and privacy issue.

A Few Possible Fixes

Amazon.com and eBay have a workaround for privacy: rating systems that allow users to determine how reliable a vendor or seller is. While there is some risk in using Amazon and eBay, it's far less risky than using Craigslist.

And Twitter has introduced "verified accounts", but it took a law suit to get them to do it. Of course, Twitter still deals with impersonation issues.

What Needs To Happen

Craigslist and sites like it need more accountability than just a verified email address. A person's identity needs to be verified. They don’t need to share that info with the world, but there needs to be some assurance.

What Will It Take?

There is very little accountability on the web today. I haven't yet heard of lawsuits brought against Craigslist after the home invasions, but I wonder: what will it take to fix the issue?

What measures would you take to counteract this problem? Can we solve the problem without invading innocent peoples' privacy? Should we just keep rolling the dice?

Leave a comment or if you tweet, follow us on Twitter to learn about new content, opinions and articles.

HTML5 FTW!

Nishant Kothary — Wed, 26 May 2010 22:12:26 GMT

Frequently, design students or even folks looking to make a career shift to the Web approach me with the following question—"What's the one technology or software package I must learn to make a career out of designing on the Web?" I usually answer with three: HTML, CSS & JavaScript.

Oh no, not another post about the free love and the Open Web! Actually, let me tell you right off the bat that I'm not about to pull a Jean-Jacques Rousseau and get all anti-corporate on you.

Rousseau and The Web

Rousseau was a Genevan philosopher who earned his early ticket to fame by publishing the essay, Discours sur les sciences et les arts ("A Discourse on the Moral Effects of the Arts & Sciences"), in which he argued that arts and sciences were the root cause for all moral corruption.

According to Andrew Potter, the author of The Authenticity Hoax, Rousseau was a key figure in popularizing Romanticism—in other words, he was one of the first guys to put "the cool" into dissing all that is modern and consequently, laid the groundwork for the entire doomsday movie genre, which features films like "The Day After Tomorrow".

While it's certainly true that the Web tends to favor Rousseauian ways, the reality is that corporate blood, sweat and tears play a huge role in fueling its progress. Even the widespread adoption of open standards relies on the collective corporate nod.

And therein lies the simple reasoning for my answer.

Getting In The HTML5 Game

Google, Apple, Opera, Mozilla and Microsoft are all in the HTML5 game now. And you better believe—from Apple's heavy-handed stance on Flash to Google's recent Chrome App Store announcement—that the commitment goes deeper than just the browser. Many of them are betting their business models on positioning HTML, CSS and JavaScript as the front of the future Web.

Make no mistake; I'm not saying that HTML5 has won against Flash and Silverlight. Joshua's argument remains true today. The scenario you're building for will and should determine whether you pick standards, Silverlight, Flash, Java, choose-your-own-flavor or a combination of these.

But, from the perspective of a student investing in your future, I'd choose HTML, CSS and JavaScript in most cases. Between those three, you're going to learn the fundamentals of markup—allowing for migration to Flash's MXML or Silverlight's XAML. You'll also learn programming fundamentals using ECMAScript, which provides a great bridge to both Flash's ActionScript (which is actually based on ECMAScript) as well as, I daresay, C#.

I'm no expert, though. What do you think?

On Being a Bad Client

Thomas Lewis — Fri, 21 May 2010 21:43:04 GMT

I love these posts. But the thing is, they always come from the agency or designer's side. Here, I intend to give you a little insight from the bad client's side. Because even though I think most of my agency partners find me pleasurable to work with, I sometimes see myself in these "bad clients" posts. I thought maybe I could help bring some perspective to the relationship.

Bad Client Behavior #1: They Want a Ton of Changes At the Last Minute

You, the designer, probably did exactly what you should do. You went through a business requirements gathering exercise, then talked to some actual users (yes, you were that lucky) and delivered an amazing set of wireframes (on time!). And what did you hear from your client? Crickets.

So you pressed ahead. You sent over the visual comp of the homepage expecting everything to go smoothly, when BAM! Suddenly Mr. Bad Client's feedback is flowing and he's telling you, "We're way off track here!" You get more and more frustrated as the delivery date looms heavily and you have to cancel your plans to see the Human Centipede movie this weekend.

Here's the problem: Most of the time, your bad clients are super busy and just can't wrap their heads around a wireframe. In fact, they probably scanned it for about two minutes and declared it good. The visual comp is the closest thing to a real site in their heads, so that's when they begin providing the feedback you needed earlier in the process.

The fix? I recommend getting to the visual comp phase as fast as possible, and allowing room in the schedule to deal with feedback on the design.

Bad Client Behavior #2: Ugh! They Want You to Build a Site Without Content!

Ok, I'll admit it. I'm guilty of this. I've actually read Kristina Halvorson's book, Content Strategy for the Web. But I've been in situations where I was the primary contact with the agency, and my stakeholders just didn't have the content ready before putting together the site. Sorry Kristina!

After I apologized profusely, the great agency I worked with did the next best thing: They asked what kind of content would go in the various content blocks, and tried to use existing content instead of lorem ipsum. It actually worked out pretty well.

I was able to give them some guidance and examples of content we used in the past. For example, Microsoft is notorious for using long product names, titles of posts and titles of session for our events. So the typical "Lorem Ipsum" filler text wouldn't work for us. We were able to plan for that.

I believe that you should have your content prior to building a site. But sometimes reality sets in, and you have to adjust to the situation. This agency has a special place in my heart for working with me vs. lecturing me.

Bad Client Behavior #3: They Want To See 10 Different Comps

As a client, I have to admit I like the idea of having choices. And yes, I will always want a Frankenstein of a web site that combines the best of the three you have shown me. Sorry!

But you have to remember: Most clients are not keeping up-to-date on the latest design trends. Helping give them exposure is a good thing.

Also, remember there are a ton of stakeholders (especially in a large company) that have a say-so in the design (or at least who sign my paycheck). So although you may have created one concept to rule them all, it may be more fruitful in the end to let me choose from different concepts.

Bad Client Behavior #4: They're Obsessed With "Above The Fold"

First of all, let me just say: I'm with you on this one. Also in this bucket is:

Can you make the logo bigger?
It just doesn't "pop" for me.
Can you make it more clean?
Hey! You left some empty space! Can we put a blog roll or a puppy picture there to fill it up?

These are critical moments for educating the client. Explain the "why". Another great agency I worked with was really good about explaining what was not on the page, and why. This made it very easy to go back to the stakeholders and explain why something was done (or NOT done).

Bad Client Behavior #5: They're Just Idiots

First, your clients are not idiots. They probably know their knowledge space just as much as you know yours.

That said, I do believe in firing really bad clients. If they don't pay up, if they don't respect you, if they burn you—by all means, you should never do anything for them again. I recommend Zeldman's 20 Signs You Don't Want That Web Design Project.

But take time when you are frustrated to look at it from the client's side.

Now Say You…

Do you agree? Do you disagree? Please let me know what you think in the comments below. I would be particularly interested in bad clients you have dealt with. Did you "fire" them? Did you have a Jedi-trick you used to get over a rough patch? Please tell me about it below!

Also, follow me or MIX Online on Twitter if you like.

Using the ASP.NET Chart Control With Multiple Web Roles In Windows Azure

Karsten Januszewski — Thu, 20 May 2010 18:46:07 GMT

I use the ASP.NET Chart Control in Azure in an ASP.NET page with both markup and code-behind. I do this because the chart control overlays hrefs and tooltips on the chart. (Check out this post for a little more about how I use the charts: http://visitmix.com/LabNotes/ASPNET-Charts-and-ASPNET-MVC--Controller-vs-View.)

Things work fine if I configure the chart control to generate images in memory. But, with more than one web role, things break. This is expected, as the documentation states: “Do not use this [the memory storage] option in a server cluster or a multiple-process site.”

Somehow, I need to store the image to disk instead of memory, and then let the chart control know where to do it. Azure supports writing to disk through what they call a ‘local resource’, so I started using a LocalResource. But I kept hitting the following problem: I don't know the path to the LocalResource until runtime, but the Chart Control expects this information to be in the Web.Config file. And, if I changed the Web.Config at runtime, Azure got very unhappy.

I looked into using XDrive, which is similar to using a LocalResource, but the same problem ensued. Each web role ended up mounting its own drive, and the drive letter became dynamic. I need to know it to pass that drive letter to Web.Config.

But I found a solution: I just implement my own instance of IChartStorageHandler and have it point to blob storage. Here’s what the code looks like:

    public class ChartImageHandler : IChartStorageHandler
    {
        CloudStorageAccount account;
        CloudBlobClient client;
        CloudBlobContainer tweetContainer;
        public ChartImageHandler()
        {
            //wire up azure
            account = CloudStorageAccount.FromConfigurationSetting("DataConnectionString");
            client = account.CreateCloudBlobClient();
            tweetContainer = client.GetContainerReference(Urls.archiveContainer);

        }
        #region IChartStorageHandler Members

        public void Delete(string key)
        {
            CloudBlob image = tweetContainer.GetBlobReference(Urls.imageURL + key);
            image.Delete();
        }

        public bool Exists(string key)
        {
            bool exists = true;
            WebClient webClient = new WebClient();
            try
            {
                using (Stream stream = webClient.OpenRead(Urls.imageURL + key))
                { }
            }
            catch (WebException)
            {
                exists = false;
            }
            return exists;
        }

        public byte[] Load(string key)
        {
            //sometimes load gets called before save is done
            
            CloudBlob image = tweetContainer.GetBlobReference(Urls.imageURL + key);
            byte[] imageArray;
            try
            {
                imageArray = image.DownloadByteArray();
            }
            catch (Exception)
            {
                System.Threading.Thread.Sleep(1000);
                imageArray = image.DownloadByteArray();

            }
            return imageArray;
        }

        public void Save(string key, byte[] data)
        {
            CloudBlob image = tweetContainer.GetBlobReference(Urls.imageURL + key);
            image.UploadByteArray(data);
        }

        #endregion
    }

Implementing this interface turns out to be pretty easy, especially because the Azure Storage Client CloudBlob class supports DownloadByteArray and UploadByteArray. Because the signature of the interface uses byte[] all over the place, this comes in real handy. Thanks Azure Storage Client!

You’ll note that I’m using this static Urls.imageURL for pointing to blob storage. This string is simply a path to a container in Azure.

The only tricky bit is that sometimes the Load method gets called before the Save function is finished. This is probably because of the latency with blob storage, which is going to be a little slower than writing to disk. I deal with this by adding try/catch. This simply puts a 1 second sleep delay in the call before trying to download the image. This solves the problem, and I’ve never seen the Save function take any longer than that.

I then update the web.config to specify that the ChartImageHandler key passed the fully qualified namespace and location of my handler, instead of the default handler.

And now I can use the ASP.NET Chart Control in Windows Azure with multiple web roles. Rockin.

Usability Testing On The Cheap

Tim Aidlin — Mon, 17 May 2010 21:26:15 GMT

Recently, I've been doing a lot of usability testing. The great part about it is, each new person I sit down with reveals a wealth of knowledge that would otherwise have gone untapped. The bad thing is, user testing takes time out of a busy production schedule, adding to an already stretched timeline and minimal budget.

But it’s worth it.

Why Do it?

Clients always want their deliverables last week. We do our best to deliver quickly— sometimes within ridiculous timelines. Because of this, we often don't have time to put the design, product or idea in front of potential customers. Unfortunately, this can lead to myopia , and sometimes to a displeased audience and client.

If the product you deliver doesn’t resonate with your client's audience, the displeasure frequently reflects back on *you*: You didn’t understand the “greatness” of the product. You didn’t convey that greatness to the customer. You just didn’t get the message, and the final product doesn't perform.

It’s rare for clients to admit that their idea needs improving. When presented with actual data, however, they have an empirical foundation on which they can judge their ideas. This is where usability testing comes in.

Usability testing doesn't take long, and it helps give your clients confidence in the choices you make. It's also an excellent opportunity to discover major issues and address them before releasing a product to the public, and before the client begins receiving feedback.

Doing It Inexpensively

User-testing is fairly easy to do, and inexpensive. The best way I’ve found is to just sit down with a user,watch him or her, and take notes. It's best to find someone who's unfamiliar with your product or service, because the goal is to make your product as easy to use for first-time users as it is for veterans.

Generally, all this testing requires is some time and patience. However, if you're testing a user-interface, screen-capturing your session can help you evaluate data after-the-fact. If possible, your video should be accompanied by audio, which you can usually capture via the software you’re using, or even a mobile device with a microphone. Watching and listening to the user can lead to great insights.

Doing It Wisely

I mention listening to the user because I’ve found this is where they generally express frustration. It’s hard to tell when a user is really having problems just by watching where they click or move their mouse.

Ask your test subjects to narrate their actions as much as possible. Encourage them to voice their intentions, expectations and frustrations . Have them tell you what they’re doing with the mouse: What they’re looking at and what they’re looking for. Stream-of-consciousness is what you want here.

And you, you shut up. Of course, give your subjects guidance and answer questions when they ask—but do your best to stay out of the way. If you over-direct, your test subjects won't be as likely to explore, and you might get a skewed result.

Also, keep testing informal. Meet in a comfortable spot. Relax. Talk a bit. Keeping it loose will enable people to relax, talk to you more freely and use the product like they would if you weren't looking over their shoulders.

Telling ‘em What To Do

Users seem to fall into one of two categories: those who like specific direction, and those who like to explore.

Both of these types of users are valuable, so it’s helpful to come to your session with two scenarios in mind: one that's very directed, in which you ask the user to perform a certain set of tasks, or a specific workflow. The other is to simply let the users “go crazy” and use the product how they see fit. This is an interesting way to see what a user would do coming to your product “blind,” and can provide valuable insights on making your product as accessible as possible.

Other Methods

Although the most valuable data is often collected after a design has been developed and is usable, doing usability testing prior to production is a good idea. It’s important to create solid wireframes, if not fully-realized composites, and go through the same process as discussed above.

As well, if you have a serious budget and giant project, it may be worthwhile to invest in large-scale, “professional” usability testing. I’ve been fortunate to have been part of many sessions where there was a group of us sitting in a little room, watching 8 or so users interact with various user-interfaces. The sessions were moderated by a professional usability tester, and were highly focused, video-recorded and analyzed later. Of course, the data from these sessions was very useful.

I think, however, the small shop or individual designer can do quite well and get similar results from doing usability testing on the cheap.

What About You?

What’s your experience with usability-testing? Do you find it useful? Are your clients reluctant to pay for it? What sort of tips could you give other small shops and freelancers to do it on the cheap? Leave a comment below, and be sure to follow us on Twitter at @Mixonline.

Designers, Be Happy You’re Not Dolphins

Nishant Kothary — Thu, 13 May 2010 19:14:36 GMT

The film follows protagonist Ric O'Barry, a reformed dolphin activist from Flipper fame, who's joined forces with Louie Psyhoyo, leader of the Ocean Preservation Society, as they lead a team of vigilantes to covertly film the entire massacre in The Cove—the alleged scene of an annual dolphin massacre that's been deemed off-limits to all but a select group of officials and fisherman by the Japanese authorities.

Despite the one-sidedness, it's a great piece of filmmaking. The documentary positions the cause masterfully by gripping its audience with one of the seven basic plots.

Having said that, I'm completely astonished (but maybe I shouldn't be) at how this film has catapulted dolphins to the new status of the sacred cow. Don't get me wrong, I'm happy for Ric and dolphins worldwide.

But, let's be honest about the double standard here.

Flipper-flopping

I watched the film sitting next to two vegetarians (full disclosure—I like my filet mignon medium-well), one of whom couldn't see the difference between the dolphin massacre and industrial livestock farming in most developed nations, including the US. In essence, my vegetarian friend noted, both are systems in which humans kill animals for monetization and sustenance.

Ric argues that dolphins have Theory of Mind: They're self-aware, extremely intelligent and close to humans in the evolutionary chain, if you will. Hence, their captivity and murder are inexcusable.

As Sam Harris argues in his TED talk, Science Can Answer Moral Questions, our tendency to protect certain forms of life such as elephants, primates and now, dolphins, is rooted in our belief that they can experience a broad range of happiness and suffering. Conversely, we don't show such empathy for rocks, ants or, say, Cattle.

But this reasoning just doesn't sit well with many smart vegetarians I know. Some are infuriated that we bestow upon ourselves the right to make judgments about what species we may kill or hunt. Or that we presume to know which species are off-limits, since our understanding about their consciousness is incomprehensive.

Make no mistake—this isn't just an isolated opinion of a few vegetarians. It's even a core tenet of many religions—Ahimsa (non-violence), a core concept in Jainism, asserts that all living beings have a jiva (a soul) that is equal and thus, must be treated with equal respect. Practicing Jains go to extreme lengths to preserve life of all forms, from ants to root vegetables; many Jains have never tasted a potato because that's considered a sin!

Beliefs about what level of respect different forms of life deserve traverse a broad range of complex and subjective arguments.

The keyword here is subjective.

The Curse of Subjectivity

The word "subjective" is the arch nemesis of user experience. That's because "subjective" is usually an illusion when it comes to design.

Everyday, brilliant creations are adulterated with toxic accoutrements borne from the bowels of a belief that design is subjective. This distortion of reality seems to have become a core tenet in our organizational value system; it often leads to metaphorical Cove massacres of our products.

The challenge with design is that it's not as primitive as logic, but we tend to silently measure it logically. Design is a craft that approaches the complexity of natural language with nuances, contradictions, and emotions. Much like good writing, good design is something that doesn't just evolve, but is intelligently created by a craftsman. And it always grips us in an unexplainable way.

Because it is, often, unexplainable.

Explaining the Unexplainable

The unexplainable part of the practice design is founded on evolved problem-solving capacities that are often hidden from our consciousness. These are the same evolved capacities that allow ball players to catch seemingly impossible fly balls with awe-inspiring accuracy—a task that when solved mathematically relies on solving differential equations in four dimensions—which even the most sophisticated robots can't replicate.

The problem is that our reductionist methods always mislead us into mistaking the unexplainable for incorrect or illogical. The fact is that reductionist approaches generally do a piss-poor job of reverse-engineering the unexplainable when it comes to design—because they neglect the unquantifiable pieces of real equation.

We need to wrap our minds around a new definition of design that builds upon better evidence. Taking talent for granted, the definition lies somewhere between three key points: Edwards' assertion that artists have a trained ability to represent the world as audiences desire to see them, Gigerenzer's assertion that intuition and gut feelings are better suited for certain types of problem-solving and Gladwell's assertion that expertise in a trade is achieved at the magical 10,000 hour mark.

In simpler words, the practice of design occurs at the confluence of talent, training, intuition and experience. The rest is just background noise.

Thank God We're Not Dolphins

My point is that the practice of design isn't really subjective like the ethics of the dolphin trade or personal musical tastes, and while much of it is unexplainable, it's hardly illogical or incorrect.

But, I suppose the real moral is that designers should be thankful that they're not dolphins. I mean, can you imagine being a dolphin swimming up to humans, introducing yourself with a friendly high-pitched language that rivals the sophistication of their own, and impressing them with tasks they can't achieve, including mastering their own language, only to find yourself being held captive to perform circus tricks or simply being marginalized, massacred and consumed without consequence?

Wait a minute...

On Being an Intelligent Designer

Joshua Allen — Tue, 11 May 2010 22:48:18 GMT

I once heard an automotive designer remark on the design quality of a competitor, saying, “They have a factory in Dusseldorf that chews up steel and craps out Porsches.” He was conveying his opinion that the cars were created in a blindly mechanical manner, with no soul or intelligence behind the design. Indirectly, he was communicating his belief that design should be intelligent and purposeful—not mechanical.

I think about this anecdote when I hear myself saying things like “This project has evolved over time.” And it’s not just me: I’ve noticed people conflating the word “evolution” with “progress” in at least two print magazine articles just this month. People do it all the time—if you pay attention over the next week, I bet you’ll notice at least a few occurrences.

So here’s the thing: evolution is exactly the opposite of intelligent design! Evolutionary theory was developed to describe how a “Blind Watchmaker”, acting purely through chance variation with no intelligence whatsoever, could produce the various species of plants and animals. So when I say that my project has “evolved over time”, it literally means, “We randomly threw crap at the wall, and this is what stuck.”

This might be true for some projects, but we normally want people to think we applied some personal intelligence to our work. Try to think of other examples where we commonly use the word “evolve” and none of them really hold up. Did AJAX “evolve”, for example? Not really. AJAX grew through widespread experimentation that was intelligent and purposeful, not through chance variation.

OK, so maybe I’m being extreme. Everyone knows you mean “progress” when you say “evolution”, right? But I can’t help thinking of a gigantic machine, blindly chewing up random piles of steel and crapping out Porsches, every time I catch myself using “evolve” this way.

Horses and Porsches

Let’s invert things, just for the sake of argument. If evolution through blind chance variation has no place in our design projects, why does it have a place in the speciation of plants and animals? Whether you believe the original animal species were created by a blind watchmaker or by an intelligent Designer, it should have little bearing on animals going forward.

Our ancestors were able to achieve amazing results with very little technology, turning wolves and foxes into the crazy variety of domesticated dog breeds we see today. Stand up close to a horse, and try to imagine the centuries of intelligent human design that led to the domestication of this powerful beast whose greatest happiness in life is to go exactly where his human masters want him to go.

All of this animal engineering stopped hundreds of years ago, before the industrial revolution. Why? Today, we have immensely better tools. Not only could we do a much better job at the selection process needed to create new breeds, we could even create entirely new species synthetically from DNA! We talk a lot of smack about how much smarter we are than our ancestors, but I haven’t seen anything as spectacular as a Clydesdale or Weimaraner come out of the post-Darwin age. Because of our ancestors' courage, we had horses to usher in the industrial age and Chihuahuas to provide comic relief in so many movies. What new innovations will we bequeath upon our descendants?

Doing What You Do

You probably aren't a position to be creating new species or breeds of animals. But you are in charge of your own creations. Don’t simply go through the motions and then passively expect “evolution” to carry the ball forward. Be purposeful and intelligent, and let your imagination go wild. Whether it’s more like a Clydesdale, Chimera, or the fabled Labradoodle, make your project a reflection of your best ideas.

The Evils of URL Shorteners

Hans Hugli — Thu, 06 May 2010 23:27:04 GMT

How do we overcome the dilemma of opaque shortened links? And what other dangers might URL shorteners involve?

Some Solutions

URL Expanders

One solution is to use one of the many URL expanders out there. For instance, TinyURL has a URL preview feature for its service. And developers have written third-party Firefox and Chrome URL expanders for bit.ly.

The problem is that there are so many URL shorteners out there, you’d need to install a mass of utilities. In 2008 there were 90+ listed. This number has declined since then due to the unsustainable business models of some services, but there are quite a few still around. Most of these utilities are written for a specific shortener service.

A Single Service

The other approach is to have a single service resolve all shortened URLs in a single utility. There’s a Chrome extension that works with some of the more popular service providers. After installing the extension and visiting Twitter, though, I found that it even resolved some bit.ly URL’s incorrectly.

Still, I think it would be nice to have a ubiquitous way to get this same functionality to work across all browsers and all services.

What About You?

I’m interested in feedback. Does it bother you that shortened links are opaque? Would you like to find out more about a shortened link before you click on it? Would a popup be too obtrusive? What would be the right level of interaction? What wouldn’t drive you nuts every time you saw it?

Let’s say for the moment that you’re willing to put up with a popup, perhaps with a toggle. What information would you like to see displayed on it?

The full URL?
The document title (sometimes available)?
The number of click-throughs (sometimes available)?
Thumbnails?
Other info?

If you’d like to see a thumbnail, why do you think it would be useful? In the past, the Bit.ly API provided a thumbnail URL, but they’ve now deprecated that. My guess is that users didn’t think it added enough value.

Some Dangers

URL Shortening and Privacy

Here’s a second and possibly more concerning issue to think about. URL shortening services have successfully inserted themselves between the link and the link destination, something that Steve Gillmor first warned about. This means that they’re able to watch traffic and collect information about users. Most users probably don’t care, but to me this seems like a potential privacy issue.

If you arrive at a shortened link, would you prefer to bypass the shortening service and go directly to the final destination?

Link Rot

One last danger with URL shortener services is “Link Rot”, which goes completely against the grain of the “Permalink”. As the URL shortener services go out of business, links will no longer resolve to their final destinations. This can do some serious damage to websites that depend on shortened URLs. So it’s probably prudent to pick one of the more popular URL shortener services.

My prediction is that eventually people will gravitate toward just a few shortener services to avoid this issue. The remaining services will eventually fade away, leaving archived tweets with meaningless links. Sigh.

As I’ve pointed out, there are a few of issues to consider with shortened URLs. In the interim I guess I’ll have to click through some of the opaque links and be at the behest of the link poster. I’ve done a bit of looking around, and was unable to find a URL expander that worked consistently across all browsers. If you happen to know of one please let me know.

Leave a comment or if you tweet, follow us on Twitter to learn about new content, opinions and articles.

Save yourself some heartache

Tim Aidlin — Tue, 04 May 2010 20:34:01 GMT

When it comes to web design projects, I’ve learned how to screw up royally the hard way. Luckily, there are easy ways to avoid basic mistakes. Or ensure that if they do happen, they cost you minimal sorrow.

Here are just a few simple things that make workflow easier and help avoid disasters:

Set Up Your File Structure Early

As soon as I start a project, I create a simple directory for it. The project could be a fully functional website, an advertising campaign, or the various files required for putting on a conference like MIX. Each project gets the same basic treatment:

I start with the name of the project, and add subfolders for the various “media types” I'll likely use. Working with a common file structure from the outset lets me find my files easily. There’s little worse than working late into the night and *knowing* you made an update to a file, only to be stumped about where you put it the next day.

Within each of these directories, I might want to further branch my file structure. Given a fairly media type-based original structure, I generally start sorting by use type, leaving flat, site-related files in the root. For instance, images might be branched further, like this:

By setting up a consistent file structure, I avoid having to refactor image paths or file-references in the sites I build. For instance, if I decide later in the project that I need to pull all my images from the root and put them in an "images" folder, I would have to change all references in all documents that refer to images. What a pain. It's best to start out with a file-struture that I know will allow for conistency -- as well as room for growth. A final bonus is having consistency across project means I can dive into any project and know how to find the assets, files, or documents I need.

Save The Original

I can't count the times I’ve gone into a file, made changes that caused problems or looked bad, and then realized I couldn’t go back to where I started originally.

Layered files

When working with an image in Photoshop, the first thing I do is create a copy of the background layer and hit save. Now I have a layered file, which Photoshop prompts me to save as a Photoshop (.psd) file. I'm no longer touching the original image; I have two copies of it in my current document. I can always roll back and start from scratch.

When creating complicated graphics, I always keep each discreet element on a layer of its own. This often means larger files (sometimes with hundreds of layers), but it’s important to do this.

Having everything as its own layer provides the ultimate flexibility for revisions to the original design. It’s rare that we design an interface, animation or advertisement that will appear in only one format, size or medium. We have to anticipate going back into the original image to make revisions as projects organically evolve.

Masks

Masks are another important tool for making images malleable. For instance, let’s say you have an image that you want to fade to white at the bottom. Rather than using the gradient tool directly on the image, I've achieved the same effect by creating a white layer underneath it, and applying a gradient to a mask on the image layer.

Putting a mask on the image, rather than just putting the gradient on the image itself, makes it much easier to work with and change down the road.

Dealing With Markup

Make a copy!

When working with standard HTML, CSS or JavaScript files (I do this for images too), I’ll make a copy of any file I plan on working with and simply add an underscore to the beginning of the filename. For instance, the new copy of “index.html” would be “_index.html.” When I sort alphabetically, all the copies rise to the top of the list. This is super-handy when the changes I’ve made are committed and I no longer need the backup.

Make your code readble

Now that I’ve covered my tail and made sure I haven’t broken anything, I think about how to help someone else use my markup. Often, someone else is responsible for implementing the markup I write. Making my code accessible to these folks is really important. Commenting, a tool that lets you add markers and reminders to code, is a simple and useful way of doing this.

<!--  HEADER START --> <!--  HEADER START -->
Markup
<!--  HEADER END --><!--  HEADER END -->

Well, now I know I’m looking at the header. Thanks, Comments.

As well, I try to make sure my code is well-formatted and easy to work with. It’s hard to do sometimes, but I try to spend some time, take out inline styles, indent my nested <div>s, and try to be consistent in my naming conventions and how I write my styles. In short, I try to think of what would help me if I cracked open this code for the first time and needed to make changes. In three months, when I end up revisiting the project, I’m glad I spent some time making it easy to get into.

Archiving

At this point, rather than simply deleting the copies I’ve made, I move them to an “_archive” folder. The underscore, again, makes for easy sorting later. By keeping these copies, I now have a great visual history of the project's progress.

For Working With Lots of People: TFS

Microsoft Team Foundation System is the best thing I’ve found for helping me work with other developers and ensure my files are up-to-date and safely archived. TFS provides a central repository for project files, a way to check files in and out, and assurance that you’re not copying over someone else’s changes. The integration of TFS into Microsoft Expression Web has allowed me to work on a web project without having to fire-up Visual Studio, too, which is a great benefit.

What About You?

What's the worst way you've messed up a project and how did you recover? What hard-lessons have you learned that have lead to new ways of working? Are there any “standard operating procedures” you go through during different stages of a project? Let us know by leaving a tip or comment below, and be sure to follow us on Twitter @MIX Online.

Pagination, Deduplication and The Twitter Search API

Karsten Januszewski — Fri, 30 Apr 2010 17:43:03 GMT

A few changes to the Twitter Search API have caused me to change some of my code for an upcoming Mix Online lab.

I used to pass a since_id as well as a page number to get a sequence of search results back from Twitter.

It looked like this:

http://search.twitter.com/search.json?q={0}&since_id={1}&rpp=100&page={2}

Here {0} is the query, like “#mix10”; {1} is the since_id that I’ve stored in my database from the last search; and {2} is the page number. The rpp parameter is the result set size, max 100, which is what I pass. My code cycles through and asks for 15 pages, which is the maximum number of pages the Twitter API allows. Everything used to work great.

Then, things stopped working. Twitter didn’t seem to be respecting the since_id, and kept returning me results older than the since_id I passed.

For example, if I passed a since_id of 10—aka give me all tweets that meet my criteria greater than 10—I would end up getting results that could have an id of less than 10. For example, it might have returned an id of 9. But 9 was already in my database from when I passed an earlier since_id. I was getting duplicate results.

I did a little digging and discovered that Twitter was also passing back a warning in the result set metadata: "since_id removed for pagination." Hmm, I thought. What does that mean?”

I then came across a thread on twitter-development-talk in which a Twitter engineer, Taylor Singletary, discussed changing the semantic meaning of the Twitter id:

“Twitter has adapted to the increasing number of tweets in ways that have affected you in the past: We moved from 32 bit unsigned integers to 64-bit unsigned integers for status IDs some time ago. You all weathered that storm with ease. The tweetapoclypse was averted, and the tweets kept flowing.

“Now we're reaching the scalability limit of our current tweet ID generation scheme. Unlike the previous tweet ID migrations, the solution to the current issue is significantly different. However, in most cases the new approach we will take will not result in any noticeable differences to you the developer or your users.

“We are planning to replace our current sequential tweet ID generation routine with a simple, more scalable solution. IDs will still be 64-bit unsigned integers. However, this new solution is no longer guaranteed to generate sequential IDs. Instead IDs will be derived based on time: the most significant bits being sourced from a timestamp and the least significant bits will be effectively random. “

(My italics.)

Ah ha! If status ids are no longer sequential, we can’t count on them for pagination. Later in the thread, another Twitter engineer, John Kalucki, chimed in:

“In the existing continuously increasing id generation scheme on the Twitter.com API, I'd subtract about 5000 ids from since_id to ensure sufficient overlap in nearly all cases, but even this could be lossy in the face of severe operational issues -- issues of a type that we haven't seen in many many months. The search API has a different K in its rough ordering, so you might need more like 10,000 ids. In the new Snowflake scheme, I'd overlap by about 5000 milliseconds for twitter.com APIs and 10,000 ms for search APIs.

“Despite all this, things still could go wrong. An engineer here is known for pointing out that even things that almost never ever happen, happen all the time on the Twitter system. Now, just because they are happening, to someone, all the time, doesn't mean that they'll ever ever happen to you or your users in a thousand years -- but someone's getting hit with it, somewhere, a few times a day.

“The above schemes no longer treat the id as an opaque unique ordered identifier. And woe lies in wait for you as changes are made to these ids. Woe. You also need to deduplicate. Be very careful and understand fully what you summon by breaking this semantic contract. “

(Again, italics mine.)

Deduplication makes perfect sense, given that we can’t count on Twitter IDs for semantic meaning as far as sequential order. So, my code now has to ensure I’m not storing a duplicate tweet that could get returned when using pagination with the search api.

The thread continues and gets pretty interesting as people start asking to John what he means by K:

“A sequence can be on a continuum from unsorted to partially sorted to roughly sorted to totally sorted. Totally sorted is what we mean when we say "sorted". Partially sorted could mean anything, I suppose, but roughly sorted is a stricter definition than partially sorted. Informally it means that each item is no more than K items out of position. So, to totally sort the sequence, you need only consider K items.

“This is useful stuff for dealing with infinite sequences of events -- like, picking a random example, the insertion of new tweets into a materialized timeline (a cache of the timeline vector). The events get slightly jumbled as they go through the Twitter system and this causes confusion for developers who don't understand how we apply the CAP theorem. It's Brewer's world, we just live in it. And we haven't done a good job at explaining our QoS as we've made the CAP trade-offs, or how we've evolved them, etc. etc.

“To make things one step more complicated, at Twitter, K is a function of a number of factors, including the timeline, the user tweeting, the phase of the moon, and the general state of the Twitter system So, we have to think of the distribution of K over time as well.

“Crazy. We should just move this all into a single instance of Oracle and go home.

“http://twitter.com/jkalucki/statuses/10503736367

“A sequence α is k-sorted IFF ∀ i, r, 1 ≤ i ≤ r ≤ n, i ≤ r - k implies aᵢ ≤

aᵣ. “

(And again, italics mine.)

Hilarious. And intense.

The Long and Short of It

The long and short of this is that if you’re using since_id and pagination, you need to switch to passing the max_id you get back from the first search you do. My code now looks like this:

if (i == 1)
    stream = webClient.OpenRead(String.Format("http://search.twitter.com/search.json?q={0}&since_id={1}&rpp=100&result_type=recent", HttpUtility.UrlEncode(searchTerm.term), searchTerm.since_id));
else
    stream = webClient.OpenRead(String.Format("http://search.twitter.com/search.json?q={0}&max_id={1}&rpp=100&page={2}&result_type=recent", HttpUtility.UrlEncode(searchTerm.term), since_id_for_later, i.ToString()));

Where i is the page and since_id_for_later is the max_id I capture from the first search:

if (i == 1) 
{

    //here I extract the sinceID out of the JSON -- sure hope it doesn't get longer than 11 characters!

    //since_id_for_later = content.Substring(content.IndexOf("\"id\":") + 5, 11);

    since_id_for_later = twitterSearchResult.MaxId.ToString();

}

Above you’ll notice that at first, I was just parsing the string to get the id of the first tweet. But then I switched to deserializing the json because I had to parse the result to handle deduplication before storing the tweets in my database. You’ll also notice that I’m now passing the result_type=recent parameter in my searches. This is another change to the Search API, which is also new; you can read about in the docs.

TwitterBadge 2.1

Hans Hugli — Thu, 22 Apr 2010 20:35:14 GMT

The other day, I noticed that the TwitterBadge object on the Gestalt page stopped working. The original was based off the Gestalt 0.6 codebase and was never updated to the new bits since the System.Json namespace was not available in the new version of the DLR. We did this to reduce the download size of the codebase.

In this LabNote, I'll explain how I updated the TwitterBadge IronPython app to make a call to classes in an external DLL. My method employs the same technique as Ashish Ghoda’s IronRuby sample. I modified the sample to use external Python and XAML files and made it easy to change the twitter keyword you want to search for.

Here's the content of the HTML page:

<script src="http://gestalt.ironpython.net/dlr-latest.js" type="text/javascript"></script>

<script type="application/x-zip-compressed" src="json.zip"></script>

<script type="text/python" class="twitterbadge" src="twitterbadge.py" defer="true"></script>

<script id="twitterbadge" type="application/xml+xaml" Width="580" Height="180" src="twitterbadge.xaml"></script>

<script type="text/python" class="twitterbadge">
import System
import twitterbadge
twitterbadge.load_keyword("MIX10")
</script>

First, I added a reference to the Dynamic Language Runtime script and a reference to the System.Json.dll that's zipped up in the file json.zip.

Next, I added a reference to the Python script and XAML that defines the UI, and an inline Python script that bootstraps the application. The final script must be set to the same class name as the class name of the external Python file. The keyword used in this sample is "MIX10", but it can be modified directly in the HTML page to avoid making any edits in the external Python file.

Let's take a look at the external Python file to see what needed to be done to load the System.Json into scope.

from System import *
from System.Windows import *
from System.Windows.Threading import *
from System.Windows.Media import *
from System.Net import *
from System.Text import *
from System.Collections import *
from System.Collections.Specialized import *
from System.Windows.Media.Imaging import *
from System.Text.RegularExpressions import *

me = Application.Current.RootVisual # This localizes the XAML UI elements

First, I imported the namespaces I needed for the application. Next, I localized the RootVisual. Normally, a Python script is aware of the XAML UI since the DLR automatically localizes the "me" variable with the RootVisual of the XAML. But in this case, I was loading the script up manually, so I had to set it manually.

Next, I imported clr, which has a method that allows adding a reference to a file. Note that the path to the file contains the name of the zip file "json". Once I added the reference, I could import the System.Json namespace.

import clr
clr.AddReferenceToFile("json/System.Json.dll") # This loads the JSON dll that does not ship with the DLR
from System.Json import *

I also needed to make a call to the System.Windows.Browser.HtmlPage namespace to launch the browser when a user clicks on a tweet. Again, normally HtmlPage is localized if the Python is loaded automatically, but in this case I needed to import the System namespace and manually call the method.

import System
System.Windows.Browser.HtmlPage.Window.Navigate(Uri(url), "_blank")

The last modification is to add a callable method that accepts a keyword term.

def load_keyword(kw):
	global twitter
        global keyword
        keyword = kw
	twitter = Twitter()
	twitter.add_presenter(BadgePresenter())
	twitter.search(None, "?tag=" + kw)
        ...

load_keyword is called from the HTML page inline Python script. Click on the image below to try it. (You'll need to have Silverlight installed.)

You can grab the source code and copy the contents into a directory on your web server or embed the HTML script above to existing web pages—as long as the xaml, py and zip files are local to your web page.

Because it's not necessary to copy the DLR bits locally, this app is convenient. Note that on some web servers you may need to add mime-types to get the application to work. If this is the case, ensure you add the following:

.xap -> application/x-silverlight-app
.slvx -> application/octet-stream
.py -> text/plain (for Python)
.rb -> text/plain (for Ruby)
.xaml - application/xaml+xml

Design for Action

Tim Aidlin — Mon, 19 Apr 2010 20:12:00 GMT

Until recently, I used hover-states, flyout panels, rollovers and mouse overs when designing user-interfaces. I'm beginning to rethink these methods, though, especially when designing for touch devices, cell phones, etc.

In my opinion, interactions like these often hinder us because although they may work for the desktop or laptop scenario, they impede interactivity and frustrate users in other contexts.

In doing research for this Opinion, I found a great article by Bill Scott, “Musings on Mouse Hover” about the benefits of rollover/hover-states. For interfaces that require a mouse, I generally agree with Scott. He writes:

“Use hover to create a more lightweight but engaging way to cue the user to hidden functionality. Use it as a way to provide just-in-time details. Use it to keep your page visually simpler providing what is needed when they are most curious.”

But what happens when there's no cursor wandering around the user-interface? What happens when buttons are activated on-click, gesture, or similar interaction? Answer: nothing is lost if the hover-states are *enhancements*, but if they’re an integral part of your UI, users will get lost and frustrated.

Context & Clarity

We have to make user-interfaces clear. Functionality needs to be visually apparent and indicative of what the user should expect. We can no longer rely on images lighting up, buttons or sections gaining focus, alt-text giving instruction or dropdown menus appearing when a cursor triggers them.

Again, in the touch-scenario, the developer/designer cannot rely on an omnipresent cursor.

All of this thinking comes from being a Dad and, specifically, from experiences I've had with the Flash games at NickJr.com. A game like “Diego’s Underwater Adventure,” for example, is perfectly suited to a touch-device because it doesn't rely on keyboard interaction, right-clicking or any of the standards we often expect. This makes it easy to guide the character through the story and choose items for kids, to which these games are geared.

However, “The Great Roadrunner Race” (another Go, Diego, Go! game), which uses mouse-position to dictate the movement of the characters and keyboard interactions for things like jumping, leads to frustration ... why?

Stepping Outside the Laptop/Desktop Dichotomy

As user experience professionals, it’s our job to step outside the laptop/desktop dichotomy and consider the myriad devices on which our interfaces might now be available. I’m picking on hover-states here, but the problems include screen-sizes, required plugins and download times, too.

How do you accommodate for not only screen resolution and different browsers, but also different form factors, input mechanisms and platforms such as mobile and various providers?

Drop us a line below, and be sure to follow us on Twitter @mixonline.

Tips & Tricks for Accessing and Writing to Azure Blob Storage

Karsten Januszewski — Fri, 16 Apr 2010 15:58:56 GMT

Lately I’ve been doing a lot with Azure Blob Storage for an upcoming Twitter-related Mix Online lab, and wanted to share what I’ve learned.

Creating An In Memory .xls File And Uploading It To Blob Storage

I needed to create a .xls file on the fly and get it into Azure Storage. Here’s how I did it:

private void SaveAsCSV()
{
    CloudStorageAccount account = CloudStorageAccount.FromConfigurationSetting("DataConnectionString");
    CloudBlobClient client = account.CreateCloudBlobClient();
    CloudBlobContainer tweetContainer = client.GetContainerReference("archive");
    CloudBlob csv = tweetContainer.GetBlobReference(searchTermId.ToString() + "/data/master.xls");
    string header = "id\tusername\tdate\ttime\tstatus\tlang\timage\tsource";
    MemoryStream stream = new MemoryStream();
    using (StreamWriter streamWriter = new StreamWriter(stream))
    {
        streamWriter.WriteLine(header);
        foreach (TwitterSearchStatus t in masterStatuses)
        {
            string line = t.Id + "\t" + t.FromUserScreenName + "\t" + t.CreatedDate.ToShortDateString() + 
             "\t" + t.CreatedDate.ToShortTimeString() + "\t" + t.Text.Replace("\r\n", "").Replace("\n", "").Replace("\r", "").Replace("\t", "") + "\t" + t.IsoLanguageCode + "\t" + 
             t.ProfileImageUrl + "\t" + t.Source + "\t";
            streamWriter.WriteLine(line);
        }
        streamWriter.Flush();
        stream.Seek(0, SeekOrigin.Begin);
        csv.UploadFromStream(stream);

    }

A few things to note here: First, I wired up all the various cloud accounts, clients, containers and blobs I needed. Pretty straightforward.

Then, I created a MemoryStream and used a StreamWriter to populate my XLS file. Note that I had to flush the StreamWriter and seek back to the beginning of the memory stream before uploading it to Azure. I also uploaded the stream inside the using statement with the UploadFromStream method.

Azure handles create or update automatically, so you don’t have to worry if that blob is being created for the first time, or being overwritten. Nifty.

You also might notice all the string replace that I did on the text of the tweet itself. This gets rid of line breaks in the tweet, so the .xls file doesn’t get screwed up.

If you’re wondering where the TwitterSearchStatus class came from, it’s from the most excellent TweetSharp library, which I use for serializaing and deserializing tweets. I also use the TweetSharp library to do OAuth with Twitter, but that’s a topic for a different lab note. :)

Creating and Unzipping a .Zip File From A MemoryStream In Azure

My .xls files started getting pretty big so I decided to zip them up to help with IO. At first I was using GZipStream from the System.IO.Compression library, but I switched to the DotNetZip library so users could more easily unzip the files. I’m real happy with this library; it’s fast and easy to use. Hats off to the folks who wrote it.

Here’s the code I used to seek back in the MemoryStream (already in the memory of the .xls file), write it to a .zip file and upload it to Azure:

memStream.Seek(0, SeekOrigin.Begin);
//have to wrap GZip in using 
//because it does some writes on dispose
using (MemoryStream compressedStream = new MemoryStream())
using (ZipFile zip = new ZipFile())
//using (GZipStream gZipStream = new GZipStream(compressedStream, CompressionMode.Compress))
{
    ZipEntry entry = zip.AddEntry("master.xls", memStream);
    zip.Save(compressedStream);
    compressedStream.Seek(0, SeekOrigin.Begin);
    var csvCompressed = tweetContainer.GetBlobReference(searchTermId.ToString() + "/data/master.zip");
    csvCompressed.UploadFromStream(compressedStream);
}

Nothing fancy here. The DotNetZip library handles streams well.

Downloading and unzipping the file within Azure is also straightforward:

MemoryStream stream = new MemoryStream();
WebClient webClient = new WebClient();
using (Stream compressedStream = webClient.OpenRead(Urls.blobURL + searchTermId.ToString() + "/data/master.zip"))
{
    using (MemoryStream holdingStream = new MemoryStream())
    {
        compressedStream.CopyTo(holdingStream);
        holdingStream.Seek(0, SeekOrigin.Begin);
        using (ZipFile zipfile = ZipFile.Read(holdingStream))
        {
            ZipEntry entry = zipfile["master.xls"];
            entry.Extract(stream);
        }
    }
}
stream.Seek(0, SeekOrigin.Begin);

Again, nothing fancy here, although you might be wondering where the CopyTomethod came from. That’s an extension to the Stream class, which looks like this:

public static class StreamExtensions
{
    public static void CopyTo(this System.IO.Stream src, System.IO.Stream dest)
    {
        if (src == null)
            throw new System.ArgumentNullException("src");
        if (dest == null)
            throw new System.ArgumentNullException("dest");

        System.Diagnostics.Debug.Assert(src.CanRead, "src.CanRead");
        System.Diagnostics.Debug.Assert(dest.CanWrite, "dest.CanWrite");

        int readCount;
        var buffer = new byte[8192];
        while ((readCount = src.Read(buffer, 0, buffer.Length)) != 0)
            dest.Write(buffer, 0, readCount);
    }
}

Using WebClient To Access Blobs vs. CloudBlobClient– Benchmarking Time

In the above samples, you may have noticed that I used just a raw WebClient request to download blobs, whereas I used the CloudBlobClient for uploading. At first, I was using the CloudBlobClient to do gets on blobs. But I found that using WebClient was slightly faster. In fact, I did some benchmarking to compare:

DateTime startDownload = DateTime.Now;
Stream stream = null;
WebClient webClient = new WebClient();
stream = webClient.OpenRead(blobURL + currentArchive.SearchTerm.search_term_id.ToString() + "/data/master");
string content = null;
using (StreamReader sr = new StreamReader(stream))
{
    content = sr.ReadToEnd();
}
DateTime endDownload = DateTime.Now;
ViewData["DownloadTime"] = endDownload - startDownload;

startDownload = DateTime.Now;
CloudStorageAccount account = CloudStorageAccount.FromConfigurationSetting("DataConnectionString");
CloudBlobClient client = account.CreateCloudBlobClient();
CloudBlobContainer tweetContainer = client.GetContainerReference("archive");
CloudBlob csv = tweetContainer.GetBlobReference(currentArchive.SearchTerm.search_term_id.ToString() + "/data/master");
content = csv.DownloadText();
endDownload = DateTime.Now;
ViewData["DownloadTimeAzureClient"] = endDownload - startDownload;

Now, if you require permissions to do gets on your blobs, this won’t work because you have to pass authentication parameters in the HTTP header. Using the CloudBlobClient works great for this. That’s why I use CloudBlobClient for doing writes to blob storage.

The Blinding Light of Experts

Nishant Kothary — Wed, 14 Apr 2010 19:04:37 GMT

In the 30's, many doctors believed that mental illnesses like schizophrenia could be cured using procedures like transorbital lobotomy. The procedure, in a nutshell, involved a shock to the patient's head that rendered him unconscious, followed by the hammering of an icepick-like device into the patient's brain from below the eyelid. The device was then swiped back and forth in an effort to "cure" the patient.

In the 60's, some heart surgeons packed the chest cavity around the heart with asbestos fibers. Yes, that's the same asbestos that's been banned worldwide because of its fatal effects on humans.

Most recently, the entire US financial system caved in. As we are now learning, this was in no small part due to financial industry pundits' questionable motives and undying faith in the virtues of rational economics.

These are just a few examples of what happens when the wisdom of "experts" goes unquestioned.

We Did Start the Fire

The prevalence of "experts" involves a supply-demand problem. Expertise arises out of our demanding it. More often than we'd like to admit, it arises out of our desperation. We need to believe that there's a way to predict the stock market. We need to believe that homosexuality is a disease that can be cured with electric shock. We need to believe that the process of building software can be captured by a diagram with a few labeled boxes connected by arrows.

It's not the only reason, but it's one we can't discount. We reap what we sow. We have to bear some of the blame.

The Business of Being Right

Once you're an expert, you're in the business of being right. If you're right, business is good. If you're right a lot, business is better. Business is best when you're never wrong. As an expert, "being right" is the core tenet of your brand.

Industry experts aren't motivated—beyond the evolutionary pull of guilt—to admit their faults when they're wrong. This in no way implies that all experts claim to be right at all costs, even in the face of evidence to the contrary. But it's something to think about before you give an expert your undying trust.

"The Media"

The widespread acceptance of lobotomies can be attributed in part to a multi-decade long trend of its positive portrayal in popular press. The media—and I'm using the term loosely here to cover both popular press and social media—plays a tremendous role in evangelizing or demonizing expertise.

We trust brands. And we systematically conform in groups, even when we don't truly agree. At the junction of those two human characteristics lies an irrational need to believe. When one reputable "expert" endorses the advice of another, it tends to plug us into the matrix. It's difficult to unplug ourselves from it.

It's not because we're stupid; it's because we're human and imperfect by design. But remember, so is "the media".

Context is King

Someone smart once said, "Context is key—from that comes the understanding of everything." Without context we understand nothing and turn to the comfort of naïve black & white analyses.

Here's a clear and more personal example from just last week: MIX Online sponsored the 2010 IA Summit. A part of our sponsorship was to hook each attendee up with a beautiful printed copy of our A Website Named Desire poster. One of the featured speakers, an expert in visual thinking, threw it up on a slide as an example of how not to visualize a problem. It was too overwhelming, he suggested; it could be simplified. "It's not how I would have done it." Ironically, the poster is busy by design; building web sites is a hairy, hairy field and we wanted to depict that with honesty.

A sneak peek at a screen clipping of the upcoming store for the poster clarifies the context that he missed:

Experts often dole out advice without grasping the full context of a situation. It's just the nature of the business.

The Point

We live in the weird world of social media. A world where "everyone is a designer", we're all focused on our "whuffie" and the distance between average and expert seems to have become approximately 10,000 followers on Twitter.

History and studies have shown that experts are accurate only 50% of the time. It has also shown that the utility of expert advice increases in direct correlation with our effort of investment, ongoing involvement and ability to remain rational as we seek solutions to our unique problems.

The point is that we've got to step up our game now, more than ever: in how we identify, evaluate and utilize our experts.

Then again, I'm no expert on the topic. :-)

Recycled Creativity: A Note from Bill Buxton

Bill Buxton — Mon, 12 Apr 2010 18:50:11 GMT

This piece was originally an email written by Bill Buxton the week after MIX10. It was too good to not share, so we got Bill's blessing and are publishing it with some minor edits.—the MIX Online Team.

Through the magic of Bing and while searching for the terms "Buxton Natural User Interface," I rediscovered the following article that I forgot I'd written:

The "Natural" Language of Interaction: A Perspective on Non-Verbal Dialogues

The article, which I wrote in 1988, is one of those examples where what I thought was a fresh perspective on a topic (see my recent MIX10 keynote) turns out to have a lot in common with language and examples I used over 20 years ago.

In the article, I addressed the language of human-computer interaction and argued that:

The concept of 'natural language understanding systems' should be extended to include non-verbal dialogues (gestural interactions), because the latter are in many ways more natural than those based on words
Non-verbal natural language systems are more feasible, beneficial and likely to be delivered by available technology than verbal language systems

In so doing, I hoped that designers would adopt a mentality wherein non-verbal gestural modes of interaction would fall within the domain of natural language. My opinion was that this broadened notion of natural language would improve human-computer interaction.

Creativity: Forget What You Said, Then Say It Again

Rereading the article, I noticed a striking overlap with my MIX10 keynote—right down to the example of German and English, and ending with a paint program with similar attributes. Even more surprising was the 20 year number (which fits right in with the Long Nose) and the focus on the need for design and the challenge of moving from concept to practice.

As a person who tries to be creative and have new ideas, it's interesting to be reminded of how few ideas I actually have. Could it be that the two main skills and acts of creativity are (a) forget what you said before, and (b) have sufficient creativity to say that same thing repeatedly and dress it up in a sufficiently different way that even I think it's a different idea?

On a related note, the article reminded me just how long it takes for these concepts to take hold. When I wrote this in 1988, many of the concepts were 20 years old. That was 22 years ago. Sigh. On the other hand, if Natural UI is 'Natural', why would the basic concepts change?

Perhaps the continuity of the threads is not me being pathetic, but a sign that we're on the right path.

Gestalt Xbox Gamercard Widget

Hans Hugli — Wed, 07 Apr 2010 19:23:42 GMT

At MIX10, I presented a session on the Gestalt Widgets we built to showcase Gestalt's ability to run robust Rich Internet Applications. One widget we implemented was the Gestalt Xbox Gamercard Widget, which was inspired by Adam Kinney's Gamercard Silverlight application.

While we were releasing the Gestalt Widget pack, Duncan MacKenzie's web service was temporarily down, so we were unable to ship all the widgets together. Now that the service is back up, I wanted to get the Gestalt Xbox Gamercard Widget out into the wild—since it’s a great example of a widget that consumes a web service.

This particular widget was reverse engineered from the original sample Adam created using the tool Reflector. Reflector lets you examine .NET code in applications that aren’t obfuscated. This app made porting the app a snap.

This widget shows a few new concepts that differ from the other widgets:

Calling a REST-ful web service
Bringing a namespace into scope
Interaction between the DOM and Python code
A simple method for theming the application

WebService Calls

The widget uses the WebClient class to make calls to the web service and an XmlReader to parse the XML that’s returned. The reader is implemented in an order-agnostic way, so the implementation won't break when the order of the XML nodes and attributes change. When created, the XboxInfo class consumes the XML stream and creates an object graph created from the XML information.

Importing a namespace

In order to be able to call into methods or change properties of classes defined in external Python (or Ruby) files, just import that namespace by adding an import statement with the name of the Python file you wish to import (in our case, "gamercard"). Importing that namespace automatically executes code and makes any created classes available for manipulation. You can call the LoadTheme method, for example:

<script type="application/python" class="gamercard">
import System
import gamercard
gamercard.LoadTheme("classic")
</script>

DOM/Python Interaction

The widget accepts input from a HTML textbox and a dropdown. In order to make this possible, it was necessary to hook the onclick event from the "Go" button in the Python script.

The following Python fragment is located inside a script tag directly in the HTML page:

def OnClick(s,e):
   # do something
document.refresh.AttachEvent("onclick", System.EventHandler[System.Windows.Browser.HtmlEventArgs](OnClick))

Since DOM access is available from Ruby (or Python in this case) we can access the values of the textbox and dropdown and pass in those values to the LoadTheme and LoadUser methods. It’s also possible to call Python or Ruby methods directly from JavaScript, but I’ll save that for another post.

gamercard.LoadTheme(document.gamertag_theme.value)
gamercard.LoadUser(document.gamertag_name.value)

Theming the application

This application uses a simple but effective method for updating look and feel. It simply loads up a text file that contains color information, URLs and size information in a comma-delimited file that’s applied when loaded. We could have loaded a ResourceDictionary instead, but for this application it was much quicker to use a brute force method for applying style information.

The full source for this sample is located here. This is just another example of the ease and power of Gestalt. The best part about it is that you don’t need to have a development environment to build Gestalt applications—the Dynamic Language Runtime reports exceptions directly in the web page, which guides developers as they work with the code.

A Circuitous Route

Tim Aidlin — Tue, 06 Apr 2010 05:00:30 GMT

At industry conferences like MIX10, one of the most important things you can do is attend outside events—the ones hosted ad hoc or set up from people outside “the inner circle.”

So often, these events give you otherwise unavailable opportunities to meet Creative Directors, Designers, Developers, Evangelists and just regular guys and gals that have the same questions you do. It was during one such event that someone asked a great question, which spawned a discussion: “If you don’t have the 5 years’ experience requested by the job description, how do you break into the industry?”

Professor or Fast Food Super-Nerd?

Well, let's be honest: I have a degree in poetry.

Back in college, I thought my life would result in one of two career paths: Either, “Good morning class. My name is Dr. Aidlin and I’m here to teach you about Beat generation poetry and its relation to jazz and mid-twentieth century culture” or, “Would you like to hear what I think about Dostoyevsky while you wait for your French fries?”

Up until my senior year of college I got by using a Brother SX-4000 word-processor, which showed three lines of text at a time and required you to edit in the most ridiculous way. But you could edit, which was (kinda) better than a typewriter and carbon copies. But then I got into visual arts, while creating the layout and design for a literary journal I was producing at the time. This was 1995, and the graphical-user-interface was finally a standard—no more stupid DOS screens, crappy printouts, or no way to transfer your information. I could visually grab and move things where I wanted and use different fonts. Using rulers felt like using art-boards.

This was amazing.

So, job led to job which led to school and conferences and a million late nights searching and trying to meet people who knew more than me.

I did everything I could to learn what I'd missed in school and went out on a limb to either succeed to totally fail. I spent time building up my portfolio, learning and working super late on projects I thought were interesting, even if they had no commercial value or weren't even “appropriate” for the job I wanted.

What did this behavior show potential employers? Drive, intelligence, and the ability to learn on the fly. As Tim Brown from the amazing design company IDEO notes in his book “Change by Design,” (I'm paraphrasing) it’s absolutely important for companies to invest in a varied and diverse workforce to ensure a wide array of viewpoints, disciplines, and expertise solving important problems.

The best companies will look not only at your resume, but also at your skills as a whole, as well as your personality and interests.

Getting There

For me, the answer to the MIX10 attendee's question is simple: build your portfolio. Make it shine. Read books. Meet people. Make connections. And above all, don’t be afraid of failure. That’s how I got here, anyway.

How did you get where you are? What are you doing now to get further along? What advice could you give to people who are just starting out, looking for jobs or hoping to push their career?

Leave a comment below or tweet us at @MIXOnline.

Thought for the day: Nothing's Perfect

Nishant Kothary — Wed, 31 Mar 2010 19:15:04 GMT

Solutions—especially design solutions—come with some strange baggage: we (often subconsciously) expect them to be perfect.

But how could they be?

Design solutions are approximations, and meant to meet real life needs. Real life. Yes, I'm referring that excruciatingly imperfect, unpredictable and human type of real life. Whether you're designing a living space or architecting the information on a web site, your solution is not going to work in all instances.

It's just not.

But don't sulk about it. Accept it. Relish it. Find ecstasy in the idea that you'll discover the shortcomings of your solution and improve it over time.

As Winston Churchill once said, "To improve is to change; to be perfect is to change often."

It's true. Perfectionism is overrated. Striving for it is what's truly amazing.

Gestalt Consuming External Assemblies

Hans Hugli — Wed, 31 Mar 2010 16:16:48 GMT

Gestalt, the union of the Dynamic Language Runtime and Silverlight, enables writing Rich Internet Applications directly in HTML with Ruby/Python + XAML. Gestalt can call out to external assemblies that exist as loose files or are contained in zip files, which are treated as mini file systems by the DLR. We’ve had some inquires about this feature from our readers, so I thought it was worth calling out.

Ashish Ghoda has put together a fantastic Ruby sample (requires Silverlight to be installed) and an accompanying MSDN article that illustrates just this feature.

What does this mean for you?

What does being able to call out to external assemblies enable? It gives Gestalt the ability to consume third party written controls and libraries contained in assemblies in an easy and elegant way. XAML fragments can now include the namespaces of controls that are defined in those external assemblies. Additionally, code defined in libraries can be called directly from Ruby or Python, which gives Gestalt access to code that executes with the high performance of static code.

I recommend you take a look at Jimmy Schementi's session (27 minutes in), in which he talks about this in some detail.

Examining the Sample

I want to take a 10,000 ft. look at Ashish's sample from the perspective of calling out to external DLL's. Bingmaps.zip contains Microsoft.Maps.MapControl.dll and Microsoft.Maps.MapControl.Common.dll. By adding a reference to the bingmaps.zip file, as in the following, the DLR becomes aware of the files that are contained within that ZIP file:

<script src="dlr-20100305.js" type="text/javascript"></script>

<script type="application/x-zip-compressed" src="bingmaps.zip"></script>

<script type="application/xml+xaml" src="SilverlightMap.xaml" id="sl_map" Width="1350" Height="575" defer="true"></script>

<script type="application/ruby" src="SilverlightMap.rb" class="sl_map"></script>

Please also note that the "defer" attribute on the XAML file is set to true to prevent the DLR from loading up the XAML immediately. We'll see why this is necessary in a minute.

We now add a reference to the MapControl in the XAML file by adding a namespace, exactly as you would normally do in XAML.

<UserControl x:Name="silverlight_map"
  ...
xmlns:m="clr-namespace:Microsoft.Maps.MapControl;assembly=Microsoft.Maps.MapControl">
  <m:Map x:Name="map">
     <m:Map.Projection>
     <PlaneProjection RotationX="-0"
       RotationY="0"
       RotationZ="0"/>
     </m:Map.Projection>
  </m:Map>
</UserControl>

To add a reference to DLL's in Ruby, we add the following "require" statements to the Ruby file. Notice that the path includes the name of the ZIP file:

require "bingmaps/Microsoft.Maps.MapControl"
require "bingmaps/Microsoft.Maps.MapControl.Common"
...
include Microsoft::Scripting::Silverlight
DynamicApplication.current.load_root_visual_from_string File.read("SilverlightMap.xaml")
...

The XAML is loaded when the "load_root_visual_from_string" method is executed. We wanted to defer the loading of the XAML so that the namespace contained in the XAML would be recognized. If the DLR had loaded the XAML immediately, we would have gotten an error, since at that point the DLR was not aware of the MapControl dependencies. Adding the "require" statements allowed Silverlight to successfully load the XAML file and resolve the namespace references. The source and binaries for Ashish's sample are located here.

What’s next?

With its ability to call out to external DLL's, Gestalt opens up a wide spectrum of capabilities that were difficult, if not impossible, to achieve before.

The only downside of this feature is that utilizing external DLL's goes against the primary intent of Gestalt — to make all the source viewable to the end user. It's good to know, however, that there’s an option available for doing so... when it's your only option.

I've been wanting to add smooth streaming support to the Video Widget. The source code for Silverlight smooth streaming support is not available, but there is a DLL that ships with Blend that provides this capability. Being able to call out to an external DLL is exactly what's needed to be able to add this support!

Geolocation: Finding Your Happy (or Brutal) Place

Joshua Allen — Mon, 29 Mar 2010 18:28:22 GMT

An old adage says, "People may not remember what you said, but they'll remember how you made them feel." For technical folks, it's a good reminder that feelings are just as important as data. Nowhere is this truer than in dealing with location.

Good designs have interactions that behave like analogous interactions in the real world, using skills and behaviors that come naturally for people. With "place" more than anything else, our natural interactions tend to be primal and feelings-based. Sadly, none of the currently popular location-based or augmented reality services fully take advantage of this fact.

Data-Oriented Interactions: Shopping and Sports

Before talking about feelings and location, though, let's talk about some natural interactions that are data-oriented.

If you're building a system that augments people's experience of sporting events, you need to keep the data front and center. Data permeates everything in sports, starting with the scoring system. You probably know a huge sports buff who can quote copious statistics about all of the players in the league. Watching a game with this friend is probably a very data-intensive experience.

A game can inspire feelings of exultation or dejection, but these feelings are subordinate to the data. If you don't even know that the Canadians were heavily favored to win, or that the Canadians outshot the U.S. 45-23, or any of the other remarkable statistics about the hockey matchup, the U.S. victory leading up to the semifinals and subsequent defeat in the finals will not seem nearly as significant to you.

Shopping is another example where it makes sense to focus on data. When you're shopping in the real world, you focus on prices, colors, sizes, and other data. You remember what type of material the shirt was made of, what cut the pants had, whether they were on the clearance rack, and so on. An experienced shopper can memorize a tremendous amount of this information without even trying, and computers can naturally augment this human capacity by providing additional ways to store and share shopping data.

Place: Not So Data-Oriented

In contrast to shopping, people normally don't remember long lists of data about physical locations. Real estate agents and people with Asperger's are the exceptions that prove the rule. However, most location-aware tools overlay your experience with lists of data: the current traffic flow, temperature, dew point, phone book for nearby businesses, which of your friends have visited recently, and so on. All useful information, but completely unnatural. This isn't how our bodies and souls are primed to interact with location.

How We Interact With A Place

One clue to how we interact with our location lies in the classic architectural concept of "prospect-refuge". In his influential book, "The Experience of Landscape," Jay Appleton articulated this concept, which is now familiar to most architects and designers. When you design a landscape or a building, you need to provide lots of areas which offer both prospect and refuge. People instinctively feel uncomfortable when they cannot survey their surroundings, or if they're too far from a suitable shelter. Peering out from a cozy cave at the top of a hill feels good; being trapped at the bottom of a ravine with no shelter in sight feels naked and exposed. If you're not familiar with this idea, take time to observe people in a mall, club, or other large public space. Watch how they gravitate toward places that offer prospect and refuge, and avoid places that leave them exposed. Few will even be able to explain why they prefer one spot over another; the impulse and action are subconscious. It's an incredibly powerful innate drive.

What's so revealing about this example? Most architectural or design guidelines focus on things like aesthetics, balance, consistency, and cognitive continuity. But here we're talking literally about refuge— shelter from danger. It's a rather savage and brutish concept that doesn't fit well with our normal design guidelines, and all because we're talking about location.

Navigating a Place By Memory

Memories about location follow the same pattern. A person can easily forget the location of the vehicle licensing office during the two years between renewals, but can navigate by memory to a great restaurant in a foreign country visited only once, years ago. Someone can accidentally drive to the wrong city for a meeting or doctor's appointment, yet be assailed with a flood of memories about a long-forgotten relationship, triggered by the sight of particular park or intersection. Food and lovers are two examples that most people can easily identify with, but there are plenty of others. A heroin or meth addict who's been sober for 10 years can relapse at the mere sight of a building or alleyway from the old neighborhood. Soldiers with PTSD can experience severe panic attacks when placed in an environment that resembles the place where they were injured. Food, love, pleasure, pain, fear—these are all savage things. We're not talking about refined emotional experiences such as surprise, laughter, or annoyance. You remember where you got your first kiss, and many after that, but you don't remember where you laughed or were annoyed. This is what "natural interaction" with location looks like.

Sharing Our Experiences

Even when communicating our location-based experiences to others, we tend to rely on savage feelings instead of data. Communities preserve and pass these on as community lore. Ghost stories are one good example – ghosts linger in a place because of some unresolved emotional trauma or unrequited love that occurred there. Think of the memorials placed at the roadside where some teen was killed by a drunk driver. Long before we had written history, we were memorializing our communities’ greatest victories and deepest tragedies through landmarks and memorials. The Statue of Liberty and the Arc de Triomphe memorialize the emotional experiences of millions of people. Creating a landmark to commemorate the overwhelming emotions of the visitors to Ellis Island is very natural, and so is your tendency to “catch” some of that emotion when visiting the Statue of Liberty. Sending a tweet that says, “unlocked the Statue of Liberty badge”, not so much.

Only music and certain forms of poetry come close to enabling us to share our emotional memories in this way, and location is by far the strongest. A basic understanding of the brain's anatomy reveals why.

Place and the Brain

When most people imagine the brain, they first think of the rounded mass of wrinkled gray matter, the cerebral cortex. The cerebral cortex is the part of the brain that gives us thought, language, and consciousness. Memorizing sports scores or shopping lists is a cerebral activity.

Location is thoroughly non-cerebral, though. To get to location, we need to look at a smaller, more savage part of the brain: the limbic system. The limbic system is an ancient part of the brain, shared by all mammals, which is responsible for emotions, memory formation, and spatial recognition. Poetry and music also activate the limbic system. The dolphin's limbic system is larger and more extensive than a human's, but even a starving squirrel's tiny limbic system is enough to navigate him to the exact location where he buried his nut 6 months ago.

Brutal Emotions

Two components of the limbic system collaborate to help that squirrel find his happy place. First is the amygdala, which gives us “emotions”. These aren't the civilized emotions that we're talking about. It's hard to even talk about them at all, because they operate at a more ancient level than language. Next time you find yourself experiencing a flood of emotions upon visiting a certain place, try describing your sensations to a friend. Words and pictures are sadly deficient. What are the words to describe the dull inner aching you feel when standing in a field where a dear friend lost his life? How can words precisely describe the remembered orgy of gustatory pleasure that draws you back to a restaurant you visited years ago? Pleasure, pain, fear, joy: these reside in the amygdala.

Sense of Space

The other important part of the limbic system is the hippocampus, which is responsible for memory formation and location recognition. When the amygdala experiences a particularly strong emotion, the hippocampus goes into action, storing a rough three-dimensional spatial map of the location and all of the relevant landmarks leading up to the location. The "landmarks" that the hippocampus stores are not the sort of landmarks that you might use when giving directions ("turn right at the second stop sign"). Instead, the hippocampus stores rough spatial maps of the terrain along the route that got you to that location. Most cruise missiles use a similar navigation technique called TERCOM (Terrain Contour Matching), where cameras on the missile use the contours of the terrain to guide the missile to its destination. GPS satellites can be shot out of the sky, and you can't always paint a target with a laser (analogous to saying, "the second stop sign on the right"), so mimicking the hippocampus is often the best way to guide a cruise missile.

The hippocampus is far more efficient than the cruise missile, though. Not only does it remember the route you used to get to your happy place, it's very good at finding new routes when you need to get back to happy. Even if you begin at a completely different starting point, looking at the landmarks from an angle that you've never seen before, the hippocampus can normally get you to your destination.

Augmented Reality Can Cause Brain Damage

Every time I see someone navigating around town with his eyes fixed on a GPS-enabled phone, it makes me sad. Instead of taking in the sights and smells and allowing his amygdala to imprint his hippocampus with the shape of the city, he has become a mindless drone, taking instructions from an algorithm. We don't even expect our cruise missiles to navigate this way—why are we asking humans to do it? The brain is like a muscle which shrinks if you don't use it. We have reams of experimental evidence showing that the hippocampus will get bigger if you spend a lot of time navigating new routes by landmark. Since the hippocampus is such an old part of the brain, a weak hippocampus impacts many other areas of the brain. Likewise, when you memorize turn-by-turn directions for a new route through a familiar city, you're cheating your hippocampus.

You Make Me Feel Like a Natural Location

Now that we've described what a "natural interaction" with location looks like, some implications for location-oriented services should be obvious. Of course, there are many useful location-based scenarios that needn't be "natural", such as finding a post office while on a business trip (savages didn't have post offices). We have phone books and yellow pages for a reason, and they work fine for what they're for. But we shouldn't treat every location-based system as if it's a virtual phone book.

Here are a couple of example scenarios, just to illustrate the point:

Restaurant Guides

There are several popular location services that help you find food. At first glance, this is a perfect scenario for a "virtual yellow pages" application. A person just checks his GPS and chooses from a list of the 20 closest restaurants. But this is a huge step backwards. In the real world, nobody uses the yellow pages to find a restaurant. Starbucks or McDonalds, maybe, but when you're lusting for a gustatory treat in a new neighborhood, you don't want to risk a bad trip. Finding stashes of food when you’re hungry is exactly what your hippocampus is for, which might explain why we’d often rather wander than use the yellow pages.

Most of the services out there have integrated reviews, but these are just another database of information: you filter by type of cuisine, price range, average rating and distance; then you follow the step-by-step directions to get there—exactly like ordering a new camera from Amazon.com. But does anyone really want to find food that way? Not really. When you want a great dining experience, you rely on the experiences of people you trust. Several solutions come to mind. First, if I've eaten somewhere and had a great experience, that place should show up on my map with more prominence. Likewise, friends whose tastes I trust should show up more vividly than strangers’. Second, I shouldn’t need one application to find restaurants, and a different one to find taco trucks or street vendors. That's just stupid. Your stomach doesn't differentiate, and it's just unnatural for location-based service to differentiate.

When you're finding food, the data is important, but only in a supporting background role. As each location lures you in, you should be able to filter out places that don't meet your criteria ("eating a six course meal would take too long"). This is exactly how you do it in the real world. When you're choosing a restaurant, the first level of filtering is the amount of passion that each person (and each person's stomach) exhibits. When someone campaigns for a restaurant, they say, "The place was AWESOME!" They don't prattle on about the average wait time or how clean the restrooms are.

For me, a great food guide might show some sort of glowing indicator for a limited number of locations, with the intensity or "pull" of the locations determined by past experiences. The location indicators would be ranked on a single hedonistic scale, factoring the rankings from you and your friends much higher than strangers’ rankings. The locations could be oriented on a map or situated around you, with distance factoring into the "pull" of each. Imagine yourself being pulled in multiple directions simultaneously by delicious memories, and you’ll be approximating the real-world experience of standing in a neighborhood with great food.

Finally, when you're at some location experiencing the best meal of your life, you should be able to capture that fact without having to fill out a review and enter in a bunch of irrelevant data. All you need to capture is that it was awesome; you'll remember the rest of the information next time you're in the neighborhood.

Navigation Systems

Another scenario is navigation. Despite rapid improvements in recent years, navigation systems still fail to engage the brain in a natural manner. We know that cab drivers who routinely navigate by landmarks have larger hippocampi than the normal person. Not so with cab drivers who have switched primarily to GPS-based navigation systems. As more drivers and pedestrians rely on navigation systems for routine navigation tasks, we cheat our limbic systems.

The current mapping products from Bing and Google have taken some tenuous steps. First is the inclusion of 3D maps of major cities. Bing's offering is particularly cool, offering a sort of "Sim City" view of major cities. However, people catalogue terrain contours from ground level. The 3D maps might be useful for engaging a helicopter pilot's brain, but not so much for the rest of us.

More promising is the inclusion of street-level photography. With this feature, you can see a fairly accurate view of any point along your route. This would theoretically allow you to traverse a few routes virtually to prime your brain with a spatial map. However, this promise is currently unrealized. It's almost impossible to use these navigation systems in a way that activates your hippocampus. To have a proper sense of space, you need a panoramic view that includes far away landmarks like mountains and up-close landmarks like large trees, placed in a three-dimensional context. Street-level photos flatten out faraway landmarks, and keep you pointed in the direction that's least useful for building a mental spatial map. With smooth animation, you can get a sense of 3D, and almost get a feel for the contours of the road that you'll be travelling. But if you try this with a familiar route, you can easily feel what's missing. For example, you can click here to drive eastbound on NE 8th street (use the up arrow to drive forward). Try it in one of your local neighborhoods, then try driving the same route in your car, paying special attention to the cues that give you a sense of location.

We know that good video games can strongly activate 3D spatial encoding in the hippocampus, and good navigation systems should strive to do the same. At a minimum, this would require smoother animation and a two-axis control system like first-person shooters use, so that you could look around in all directions while navigating. A high level of detail is unnecessary; nobody remembers what the name on the "lost cat" poster says, anyway. My ideal system would blur the unnecessary details, and emphasize those details most likely to serve as landmarks. You would use music, time challenges, or games to arouse the limbic system while training the route.

One could imagine a system that is not designed to direct you turn-by-turn to a destination, but instead trains your mind to embed a proper spatial map of the area, so that you'd be able to navigate for yourself and enjoy the scenery on the way. You would enter a starting point and ending point. Then, rather than taking you on a virtual tour down the computed route, the system would take you on multiple virtual tours centered around the handful of most relevant landmarks, enabling your mind to subconsciously build a 3D spatial map of the important pathways around each landmark. Finally, it would stitch the landmarks together in a route, which would be easy to remember by this point.

You'd still need turn-by-turn assistance once in a while on-route, and it would make sense to give a very subtle nudge of some sort when important turns were coming up, but the navigation system should augment and complement you, not control and command you. Learning complex routes this way would certainly take more effort than following turn-by-turn directions like a zombie. But the payoff is large, especially in places you expect to be driving a lot. You're probably good at finding new routes or even shortcuts through your own local neighborhood, without the use of a GPS. A navigation system should help you expand this capacity to much larger slices of the landscape. Cabbies for decades have been able to mentally maintain efficient 3D spatial maps of London or Tokyo, two of the most complicated cities in the world, so your own city should be a breeze.

Conclusion

A few guidelines to keep in mind when evaluating any scenario:

A place is not a point.
Movement is kinesthetic through contours, not a list of instructions. People aren’t robots.
The locations we remember are glowing embers of past experiences, not lists of names.
People want to share your experiences more than your data. Start with what's most important.
Some navigational cues are better than others. Use introspection and experimentation to discover them.

The above two scenarios aren't meant to be fully-developed ideas, of course. They're just two scenarios I concocted to illustrate the point that the vast majority of location-oriented interfaces are terribly unnatural. The current state of affairst isn't because "natural interactions" for location are particularly difficult to implement. Rather, I suspect it's because most people haven't thought about location much, and don't realize how different location is. Once the differences become apparent, finding scenarios is like shooting fish in a barrel. What are some that come to mind for you? Do you have other examples of current location-aware services that do unnatural things? Join the conversation below.

Geeking Out On JavaScript: Douglas Crockford, John Resig and Erik Meijer at MIX 10

Karsten Januszewski — Fri, 26 Mar 2010 18:16:29 GMT

If you haven’t heard, all the sessions are now available for viewing from Mix 10 up at http://live.visitmix.com. There’s a ton of great sessions up there.

This year, we were thrilled to have two of the giants of Javascript at the conference: Douglas Crockford and John Resig. JavaScript designer and historian Douglas Crockford had a session, The Tale of JavaScript. I Mean ECMAScript. This is a must see if you want to understand the history and evolution of the world’s most popular computer language:

John Resig, inventor of jQuery, did a session at Mix entitled How jQuery Makes Hard Things Simple. Even if you are a jQuery expert, I’d recommend this session as he dives into some of the new features of jQuery 1.4.2 that are incredibly cool:

Perhaps most excitingly, we were able to put John and Douglas on a panel as part of the Channel 9 Live series with Microsoft’s Erik Meijer (programming languages guru and creator of LINQ, among other amazing accomplishments) to answer questions from the audience about JavaScript. This is the first time these three titans have shared the stage together. Check it out:

And, to really geek out on Javascript, check out this recently posted Channel9 video Research Perspectives on JavaScript with Erik Meijer, Ben Livshits and Ben Zorn. These are some amazing research scientists from Microsoft Research who have been doing analysis on JavaScript execution on large-scale, JavaScript heavy web sites. Along the way, they discuss a variety of topics about the programmable web:

Lastly, if you want to keep geeking out on killer jQuery sessions that were delivered at MIX 10, check out Six Things Every jQuery Developer Must Know by Elijah Manor and Building a Next-Generation Web Application with Microsoft ASP.NET MVC 2 and jQuery by Nate Kohari.

The 7 ½ Steps to Successful Infographics

Sarah Slobin — Fri, 26 Mar 2010 02:30:31 GMT

Take a look: Here's me when I started at the New York Times, where I was a graphics editor. I worked there for 15 years, on all the news desks, with the investigative team, and ran the biz section graphics desk.

The middle i.d. is my stint at Fortune Magazine, where I was the Infographics director and got to work with the awesome folks at CNNMoney.com (I'm fading!). And here's me now, working for Mr. Rupert Murdoch at the Wall St. Journal. Guess graphics make you gray. :)

Anyway, when Nishant from MIX asked me to write about what I do, it was kind of arresting, like the time that Montana patrolman intercepted my husband doing 94 on a long stretch of highway. Forced me to pause.

So here I am, pulling over. I'm going to deconstruct some of what I know and share my 7 ½ Secrets to Successful Infographics. Get comfortable. Get a cup of coffee. (Get me one while you're at it?) Feel free to read this in any order you like. Or if you're lazy, I mean busy, just read some of it. But keep this link around, because you never know...

1. Have an idea

Where does one procure an idea?

Usually they're found in the shower. Or at 3 a.m. in the middle of a deep sleep. So if you don't have a clue what you want to make a graphic about, I'd suggest having a good scrub and going to bed, though not necessarily in that order.

Don’t wake up with an idea? Here are some thoughts from the media outlets I've worked for:

Our stories are driven by the news, so our graphics are framed by 'what’s new' and sometimes 'what's different' and hopefully, 'what's relevant.' Editors have put their glasses on the tops of their heads and said things to me like, "Our job is to inform and delight our readers." That's a nice beginning.

I've also heard, "What we're doing here is trying to increase The Traffic," my response to which is, that pretty much limits us to focusing on a) sex b) violence c) cute animals d) Steve Jobs or e) all of the above in some twisted combination. I'm not even going to go there.

Here's a list I kept to amuse myself when I first started working for Time Inc/TimeWarner, CNN's parent company:

Anyway, new, relevant, different, informative and even entertaining are good places to start. If you're seized by an idea or have a story busting out of you, even better. But if you're not sure what to pursue, I suggest you tell me something I don't know or reveal something I couldn't otherwise find. Teach me or wow me, but don't cover ground that’s well-tread.

The best way to figure out what your story (aka your infographic) is? Get really honest and ask yourself, "What's the sexiest part of the task ahead of me?" If you don’t want to spend time with your graphic, no one else will, either.

Ok, so you've got your idea. Or maybe you’re stuck executing someone else's. Proceed to step two.

2. Get data

How is this done? I just dial the phone, say, "Hi, this is Sarah Slobin from (insert news organization here.) I need data for this story we're working on and if I could have that in the next hour or so, that would be really great because I'm on deadline..."

It totally works.

Not helpful, is it?

Let's try again.

Where do you find data?

Lucky for you, big data has hit critical mass and data viz is in. These days, you can't swing a cat without hitting a web evangelist who says datageeks are ninjas!!! And because transparency is the new black, everyone is giving away numbers.

Here's the chicken and egg part: before you can find data, you have to know a little about what you're looking for. And no, this doesn't mean just google your topic and read the wikipedia entry. Buck up. You have to approach your subject matter academically and do some research. If you can't face that, approach an academic.

Track down a professor type and tell her you’re looking for an expert who's super-smart about your topic. People love to be experts and if they're not, they're jealous of people who are and will know where to find them. Be patient. These folks keep wacky office hours.

Can't find an academic? Look for a scientist. Scientists are used to explaining things, since most of us slept through AP chemistry (or elected history of film instead.)

Can't find a scientist? Try a government agency like data.gov, census.gov or bea.gov. Here's a hint: if the name is 'Bureau of Labor Statistics', someone is sitting on metrics.

Government got you down? Try non-governmental organizations. Examples of NGOs are the U.N., the IMF, OECD or EUROFRTTPLF. (Kidding on the last one, acronyms get torturous.)

The truth is, digging up data takes focus. You have to chase it, just like my dog Sketch when she sees a pigeon.

Do, however, be mindful of your source. I'm told there are these people with "agendas" and, like crossing the street, you can find them to the "left" and to the "right." I hear these folks take data and twist it like balloon animals, so beware—those things always pop before you get them home.

3. Tools, and brandishing them

Say you've got a bunch of numbers, or a bevy of statistics, or a gaggle of geese. First: nix the geese; they poop everywhere. Now what? It’s time to fall in love (or at least get real intimate) with your material. Maestro, cue the romantic music.

This is where you get to choose your tools. I know, I know: our socks outlast our operating systems these days, so you want to know what the latest cutting edge program for data viz is, right? Ok, I'll tell you.

Some people work in Excel. Some people work in R. Some of you love your Illustrator to death. Processing, Manyeyes, Swivel, Tableau—it's all good. None of them are perfect, of course. But which is the best? I'll let you in on a trade secret: there's only one program you need to know if you're serious about infographics. This is crucial, so listen up and you too can be a data viz star:

Not!

It's not the paintbrush that produces a work of genius; it's the genius, right?

And ok, I know the Harvard attitude in the cubicle next to you is always waving some elegant little snippet of code in your face and throwing around 6-syllable words, implying you'll never catch up. Don't play that game. No one wins. I'm letting you off the hook for being an expert on every new program.

(Yeah, it's Britney the day she discovered her ponytail looks like a little fountain.)

My friend Dona Wong, who wrote the book on infographics, (ok a book, but still) told me about an intern of hers who was fresh out of college and worrying his skills would atrophy while the kids back at school were learning new technology. Look, it's lovely to have a Swiss army knife, but how many of you have actually used the little scissors? Learn one thing and do that well. If you've got a set of tools you can wield like Jackie Chan, go for it. If not, choose something that fits you comfortably and stick with it until you get good. Wax on. Wax off.

4. Scrub your data

Because working with data is truly unpredictable, I've developed a form of selective amnesia that allows me to cope. Every time I start a graphic, I'm optimistic that my material will show up perfectly packaged with a lovely bow and some chocolate covered pretzels in a cellophane bag. Instead, it usually arrives like the black sheep cousin at a family wedding—late, disheveled and smelling like something stuck to the bottom of a cat's feet.

Before you can visualize anything, you've got to make sure your material is clean, clean, clean and super-organized. If you don't have an obsessive-compulsive personality now is a good time to develop one. My friend Archie Tse at the Times spends a huge amount of time massaging his data, making spreadsheets that are works of art in themselves. Then he pops out heart-achingly beautiful graphics in like thirty seconds. It makes me crazy.

I'm pretty sure Archie stole this secret from this guy Abraham Lincoln. Lincoln said something like, "Give me six hours to chop down a tree and I will spend the first four sharpening the axe." (Full disclosure: I missed that conference call so I can't confirm this exact quote.)

There are myriad payoffs to having clean data. You can expect that at some point during your project, you'll run out of bandwidth, or someone will look over your shoulder and say, "Hey, what's that spike there, is that a mistake?" When that happens, you'll be grateful your numbers are in solid shape.

Here's where I remind you to always practice safe charting. Don't work on the original data. Instead, keep a copy of it. That way you get a do-over if you have an uh-oh moment and insert a mistake.

5. Get Freudian

Now you're ready to make charts. What's next? You need to deeply understand your data. Here's how I do it:

At the beginning of each graphic, I spend some time just staring without comprehending anything. I used to feel bad about this, like I was wasting time, until I realized its part of what they call "process."

So after I finish wasting time...

No, actually. If I stare at my spreadsheet (or table or daunting stack of white papers) for a while, I start to get it. I read it in small bits and go forward and backward randomly until something clicks. Did you ever look for the Ninas hidden in a Hirschfeld drawing? Ok, how about Where’s Waldo? Better reference? Once you find what you're looking for, you can't not see it.

You may want your data to tell a certain story, or think that it tells a certain story, but you must respect what the numbers actually say. Statistics are a little like anarchists: if you force them to stay in line, you're begging for trouble. It's crucial that you understand what the information in front of you says. This will guide your process.

When I was at the NYT, there was this reporter who drove a thousand miles across country chasing this thesis that population growth was sparked near off-ramps on the interstate. It was a lovely road-trip story; he gathered amazing anecdotes and the editors loved it. Except that when we mapped the census data it didn't support the thesis. Imagine how much gas he could have saved had he started by looking at the data.

If you're new to data viz, go slow. Start small. Look at the superlatives first. Find the largest number and compare it to the smallest. See how nicely your charts curve in the middle when you plot them in a fever line or a bar chart. Or look for the jagged rise and deep fall of the information, which you've probably seen if you've got your hands on something like, oh say, recent market data from anywhere on the planet.

Once you've got your head wrapped around what you're sitting on (I know, that's not physically possible), you can choose a charting form.

6. Let's play

And now, the fun part.

Classic western: Desperado, Antonio Banderas circa 1995. A guitar, a ponytail and a gun. He's standing on the dusty road flanked by his compadres. The bad guys pull up in a limo. He's calm, he's un-phased, he cracks his neck, stares deadpan into the camera and says (read with Spanish accent), "Let's play."

This is you! You're Antonio Banderas! The data are the bad guys! No, I don't mean you should shoot your statistics with a guitar case/rocket launcher, but at the beginning of every process there's always a chance to play it straight or play. I choose play.

"With charts?" you say, rolling one eye and keeping the other on your tweeting thumb. Yes, my friend. With charts.

Look around. The growing world of data viz means there's new and inspiring work being posted every day. From Facebook to Flickr to Feltron to Fry to Flowing Data, expression-by-infographic is everywhere. And guess what? If you gather up work you find compelling, you can keep it and use it as "reference." "Reference" is that thing that helps you think about what charting form to use. It changed my life.

I used to think ideas were supposed to just pop out of your head, or that good artists should be able to draw anything. ("A wheat thresher? Why of course, Bob, it looks something like this...") Then I learned that even my husband, a gifted illustrator (www.ericpalma.com), uses reference to spark ideas. (I'm not exactly sure what he was browsing when he made this Cheney drawing....)

It never occurred to me that you don't have to make it up all by your lonesome. Reference! You can use it for...inspiration!

Look, this rise of infoviz—we're not inventing something new. Sure, the web means we can make an interactive with big data we've collected on how often we e-mail gramps. But did you know that gramps was browsing Fortune magazine infographics back during World War II?

Florence Nightingale? She died a century ago. She made charts. Look it up.

Need inspiration? Go old school. Check out Chris Mullen's archives of visual storytelling. See what moves you.

7. Edit thyself

Ambitious?

Fabulous.

Got a lot to say?

Cool.

Really excited about your data?

Excellent.

Frame your idea clearly. Take a disciplined approach to visual storytelling. Your primary point should be clear and supported by context and detail. The main art should draw us in. Sidebars should be well-focused. Don't spam us with too much information; nobody has time these days.

Sure, you can go deep on the web, but let's be realistic—according to my last Omniture search, users spend only 4 ½ minutes with online calculators on average. And with calculators they have an agenda, like seeing how much is left in checking after they buy a new Xbox! Graphics are meant to be fast, visceral and easy to digest. Here's one I did on the day of a big candy company merger:

(Be kind; I did this in 6 hours on deadline and it was b&w originally.)

½. Respect the asterisk

When you were in middle school, did you ever do that 10-step exercise where step 1 is "read all the instructions before you start" and step 10 is "ignore instructions 2 through 9?" If you were like me, you ignored step 1, picked up your pencil and knocked-out 2 through 9, thinking you were a smarty-pants who was going to beat all the goody-two-shoes rule-followers. (I'm Spanky in this picture.)

Except the exercise was allll about attention to detail. Or tormenting students.

Making an infographic is one time when you must read the small print. If you get all the way through making the chart, pitching it to your boss, showing it off to your colleagues and then discover in the footnotes of the metrics that the methodology has changed and the numbers aren't comparable... Well, you're screwed.

Working with data requires a certain degree of rigor. Just does.* Make sure you know what the asterisk is connected to and suffer through the small type. There's gold in there. Companies don't disclose the jet they gave the CEO in the executive summary at the front of the report; they stick it in the footnotes four pages from the end.

Speaking of the end, you made it! Congratulations.

To recap, nap a lot. Have more coffee. Or if you'd like, friend me and send me what you’ve found and we can chat. It's been kind of nice pulling over and talking about infographics. I thank you for sticking around.

(Looking for the footnote? Good for you. I've left it out. My gift to you for paying attention. Now you know what the ½ in the title means.)

It’s The Little Things

Hans Hugli — Wed, 24 Mar 2010 19:14:42 GMT

There are many problems with Brass' argument, one of which is that his definition of "innovation" is unclear. Although he concedes that Microsoft moneymakers Office and Windows are innovative (because they make money), he questions why we can't be more like Google, Facebook and Twitter. The problem? The latter may be innovative, but they’re also free services based on a completely different model. Brass conflates the two.

As is the case in many big companies, some of Microsoft's product groups may be having trouble getting innovative features into their products. In other cases they may try to get too many features into their products. Still, Microsoft boasts phenomenal amounts of innovation. One only needs to spend a little time looking around to find it:

The videos on Channel9, for example, show how we are innovating and listening.

We've also shipped a variety of projects that raise the bar for innovation. The MIX10 Conference, which brings together hundreds of people from across disciplines, is yet another example of this. Here are just a few of this year's sessions:

Each one of these talks, and many more, introduced innovations on existing products, new technologies and techniques for bettering workflow and increasing productivity—this is the very essence of innovation.

The Little Things

Dick Brass' argument implies that innovation requires big sweeping changes, but this is not necessarily the case. Rather, innovation generally comes in the form of small things that make life easier or more productive. (Auto-completion is one simple example of this.) When we combine little changes with what already exists, as Microsoft is doing, we gain even bigger innovations.

The whole becomes greater than the sum of its parts.

Bill Gates articulates the notion of innovation way better than I ever could:

Every stupid thing we did first. –Bill Gates

Leave a comment or if you tweet, follow us on Twitter to learn about new content, opinions and articles.

Rethinking the Traditional Convention Booth at MIX10

Thomas Lewis — Wed, 17 Mar 2010 18:15:23 GMT

At our MIX10 conference for web developers and designers, the MIX Online community team (us) was given an opportunity to display a booth in The Commons. At first, we politely declined.

The Unbearable Lameness of Convention Booths

There were a few reasons why I was uninspired about having a booth.

I find the traditional pipe and drape booth layout rather boring.
We will have teams from Microsoft and sponsors who attendees will be super excited about and will eclipse what would be our little, humble booth.
We are not resourced on my team to “work” the booth and find “rent-a-booth-babes” insulting.
I would want a place that was a unique experience and I could hang out in with our amazing attendees vs. promote, promote, promote.

Then I realized…that’s exactly what I wanted: a unique experience where attendees and the team could hang out together.

Introducing the MIX Online Lounge

Instead of having a booth, why not have a lounge instead? But what about all the issues I had just alluded to having a problem with?

Well, I began putting on the designer hat and tackled the constraints one-by-one. By creating a lounge, I had just solved the “manning of the booth problem” since a lounge does not require constant feeding. We created an area that folks can sit down (more on this later) with plenty of seating space, power strips to plug in those hungry laptops and tables to put their feet up or show off something they are working on (a.k.a. networking). Now this does not mean that we won’t be there, in fact when someone on the team is not involved in other activities (i.e. presenting sessions, keynote dry runs, live streaming setup and maintenance, customer meetings, firedrills, etc.) we will be hanging out talking to folks.

How a Mushroom gets Attention in a Field of Flowers

Now our little lounge has some stiff competition for attention since we will be announcing some amazing thinks in the keynote. So how do you get attention when there are so many choices?

Do the unusual.

First, we decided to hand out commemorative posters for the event, not us. That’s right; we didn’t market our own community site, but instead decided to provide something to attendees that they would appreciate. Our URL is on them, but very understated. We also made sure that attendees didn’t have to worry about figuring out how to keep from damaging their poster (the paper stock is awesome that we used) by figuring out which tube roll to use (sort of like prototyping). We found a tube that was small enough for a suitcase but large enough not to crease them.

Second, our banner is not the traditional Microsoft banner. We even decided to deviate a bit from our MIX Online style guide and really vie for the eyeballs. This is not your father’s Microsoft banner. It plays off of our mascots, the mushroom man and lavender frog and is very noticeable at 25’x15’.

Finally, we decided to really go out of our way to create a unique experience. The amazing folks at ThinkAlt who worked with us to design this experience mentioned in one of our meetings:

“Hey, we know a designer who creates mythical creatures out of common furniture. Would you be interested?”

ABSOLUTELY.

So we decided instead of the traditional convention center standard-issue chairs and couches, wouldn’t you prefer to sit on a caterpillar couch or a 7-foot alien chair? Yes, yes we did.

Stop…mushroom men, lavender frogs, 7 foot aliens? Are you insane? Maybe; but we think that it will create excitement, buzz and adhere to the MIX conference ethos of “Unexpected”. Also, how would you like to hang out talking about web design while sitting on mythical creatures? We are looking forward to attendees hanging out, doing podcast and video casts from our lounge (note: if you want to interview someone from the MIX Online community site team, please e-mail us), and most importantly engaging in great conversation and fun with us.

To Be Continued…

As you can imagine, we are excited. Come back during the MIX10 conference for pictures of the lounge and some of the fun stories that came from it. If you are attending, be sure to come by and pick up your commemorative posters and stickers. Also, check your swag bag because the MIX Online community team put a little something lo-fi in there for you. Be sure to follow @mixonline and me on Twitter to keep up on what is happening at the lounge in real-time.

Update: The MIX10 conference is now over and I guess we could say that the MIX Online Lounge was a success. My worries that folks would see it as an “art piece” was unfounded. Attendees would catch up on e-mail, have power meetings and take pictures of the lounge and of themselves enjoying the lounge too.

Discovering Trustworthiness

Matt Brown — Wed, 10 Mar 2010 22:57:51 GMT

Want a subtle, rounded-corner effect that's compatible in modern browsers? You'll find a dozen tutorials to show you the way. Need a flexible CSS framework that supports hundreds of layouts? There are at least three, all with strong developer communities. IE float issues? Google 'em. In many ways, the internet itself now provides answers to nearly all our implementation questions. We'll never be scratching our heads wondering how? anytime soon.

And yet even today, it remains enormously difficult to design and launch a successful, well-designed website. Why? Because design is about much more than tactics and implementation. It's about answering the what question-aka, what’s the problem we’re trying to solve? It’s about finding the core question that frames a project.

In this article, I’ll detail the design process I used for the MIX Online re-design and show how open communication, flexibility in your design approach and a thoughtful discovery phase can help you solve the what question—even when the process gets stuck.

This article is part IV in a series covering the topic of Web Design. Check out Part I—The Anatomy of Web Design, part II—The Future of Wireframes, and part III—A Common Sense Content Strategy. Evan Sharp will contribute the final article. Follow us on Twitter or subscribe to our RSS feed to be notified of the publication of the remaining articles in this series.

What is the What: Understanding Strategy

Most design projects can be split into two basic parts: strategy and execution. Defined broadly, strategy is everything you do (the questions you ask, discussions you have, and research you undertake) before the 'real work' (hours in Photoshop, lonely nights in Textmate) begin. Designers obsess over execution, but it’s really the mysterious, conceptual beginnings of a design that determine its success or failure.

Before I go any further, I should note that neither phase is more important than the other. Answering what and how are both essential to producing a complete work. As any disciplined designer will tell you, a great idea or novel concept is nothing without proper execution. In short, the details still count.

However, framing the strategic what is perhaps the more difficult of the two questions. For one, it must be answered first, before the design production begins. You can’t begin designing a site without a cohesive plan and understanding of the problem you’re trying to solve. And even before the strategy can be devised, the problem itself has to be framed: "What is it we’re doing?"

So then, how do we figure out where to start? The answer is as simple as it is maddening: discuss the project to death. This discussion is often called the client discovery phase and as you may have guessed, it's a vital part of the design process.

Client Discovery: Finding The Truth

The early meetings and conversations with your client are the most instructive. Ideally, everyone is open-minded and prepared to help you isolate and frame the what question based on business goals, technical requirements and constraints.

And yet, it's quite difficult to find the core of the problem. Often, clients rigorously promote a solution they want to explore, even if it doesn't address the true goal. Other times, they're unable to express what makes their business or organization special. Even the best clients can be 'noisy' in early meetings—eager to give you all the information, even when you don't need it.

So how do you find the truth?

The short answer is: ask good questions. Unfortunately, this can be tricky. While it’s easy to compile as much information as possible and take your client's answers at face value, it's far more difficult to identify a what question that generates useful, concrete answers—and leads to a design strategy.

A good discovery process goes far beyond the 'easy' questions. In fact, you'll probably have to design the dialogue itself and constantly come up with fresh, nuanced ways of shaping the conversation in order to get good answers.

Here are a few questioning techniques that have helped me get good answers.

Give constraints. Instead of asking for free-wheeling, copy-and-paste answers to your questions, ask clients to limit their response to a tweet-length paragraph. This forces them to cut to the chase and reframe their ideas.
Fill in the blanks. Sometimes just answering a question isn't enough. Framing questions as 'fill-in-the-blank sentences' (e.g. "We started 'oursite.com' because we wanted to _____") may generate more detailed and nuanced answers.
Adjective dump. Many clients have a hard time expressing style, tone and visual concepts in complete paragraphs. I usually ask them to list adjectives that describe how they want the final site to feel.
Ask again. Sometimes, asking the same question multiple times is the only way to get the answer you need. If you're unclear what your client meant or don't know what to do with a particular answer, ask again.

All of these questioning methods force the client to think and act differently. Instead of eliciting a regurgitation of the RFP, they manipulate in a good way, forcing the client pay attention to the 'goals behind the goals.' This extra mental effort can help reveal deeper problems and clearer answers.

Once you’ve settled on a key question or problem to frame your design process, you can move into a collaborative, visual discovery phase. In visual discovery, you link the the problem (what) with potential solutions (hows).

Design Discovery: Try Before You Buy

Transitioning from discovery straight into design can be a dangerous undertaking, because it's difficult to directly translate the what question into full solutions (layout, type, illustration). The result is often that your client commits to a design direction before the time is right. And when that happens, things get rigid and costly to change in a hurry.

That's why it's best to work in an organic visual discovery phase that addresses the how, but keeps your process open and flexible.

Visual Research: Go Outside to Get Creative

Trolling a well-stocked RSS reader packed with CSS design galleries will only get you so far. To find real inspiration, you'll need to get out of your comfort zone and away from your computer. The web is a great place, but so is the rest of the world—you just need to get outside to see it.

My most successful designs have been inspired by non-web items and concepts: a concert poster, a vintage storefront, a found photograph or a friend’s photo exhibit, for example. Immersing yourself in art—visiting a museum, exploring a city’s architecture, or researching design trends in a local bookstore or library—is also helpful.

Of course, there are also great online resources you can explore as part of the discovery process. The excellent mocoloco.com, butdoesitfloat.com, grainedit.com, ffffound.com, and thedieline.com are all wonderful places to find design inspiration and research visual concepts and styles.

The important part is that you derive inspiration from multiple sources, both on and offline—not just other awesome websites. At the end of this phase, you should have a collection of styles and visual ideas to work with.

Presenting Visual Research: Moodboards

Once you've collected visual research, you'll need to present it to your client. But you can't just regurgitate a bunch of random pictures. Instead, you'll have to present your findings in an organized way, so they make sense.

There are a number of ways to do this, but I've found that moodboards—easy-to-make, lightweight process documents that group styles together into sets—are beneficial.

A moodboard is really nothing more than a collage or paste-up of design material that reflects an art direction or mood. It's a high-level summary of the design discovery work, tied to a style (modernist, constructivist, etc.) and tone (warm, cold, bold, playful).

Moodboards are a low-commitment, open-ended way for you and your client to find what combination of design elements answer the what question best, without worrying about layout and functional concerns or committing to a tactical solution. They let you ease into design slowly.

Another benefit of moodboards is that they anchor your discussions in concrete examples. It's much easier to talk about mid-century modern tables or Bauhaus inspired architecture when you have a few representative images in front of you.

Once your client has agreed to on a style and tone they like, it's time to tackle the how in full. In other words, now you can start the 'real work.'

As mentioned, there are a million designer resources out there that answer the how question, so I won't go into tactical tips and tricks here. Instead, I'll take you through a specific case study, the MIX Online redesign, that shows how answering what well in the client and visual discovery phases can produce a successful website—even when the process momentarily falls apart.

MIX Online Case Study:

"I want you to take this project personally." Gulp.

This was one of the first things Nishant told me during our kick-off meeting for the MIX Online redesign. It was exciting to hear, but also a little intimidating—given our tight timeline and limited resources, how would I find the freedom and mental space to "take things personally" and to successfully address both the what and the how?

It wasn't easy, but here's how I managed to do it.

Early discovery

Our initial client discovery sessions were short and sweet because we needed to move the project along quickly. Despite this, our talks produced some clear and actionable goals.

The new site would have to be:

Bold, distinctive, and unique. The space for design-focused, community-based web journals is crowded, and the new site needed to be strong and iconic enough to stand out.
Repositioned towards a larger audience. The previous design catered mainly to a technical crowd, not to designers. To appeal to this group, the design needed to be high-quality and trustworthy, and the content needed to be front-and-center, easier to read and presented in an engaging way.

With these two goals in mind, I proceeded with visual design. I was immediately drawn into the "bold, distinctive, unique" problem, and sought out ways to push the design forward.

Finding non-internet inspiration was helpful in these early stages. At the time, I'd recently picked up a turntable and had been making trips to Jive Time Records in Fremont. They had a mini-exhibition of a mid-60's era jazz album covers up, with playful, inspiring dot and circle patterns that were unlike anything I'd seen on the web. I explored this motif, as well as the 80's fashion resurgence, which featured similar bold bold colors, shapes and textures.

Although I didn't present formal moodboards in this project because of our time constraints, Nishant and I reviewed these stylistic samples on Basecamp. Everything was right up his alley—the new site would be bold, colorful, and exciting. We agreed on a general art direction.

Armed with what I thought was a successful discovery process, I dove straight into design.

Getting Lost

As discovery transitions into design, it's easy to become obsessed with a specific idea or question at the expense of the larger, overarching what—the big picture. Unfortunately, this is exactly what I did with my first designs.

My early design comps were a too-literal rendering of the "bold, distinctive, unique" concept we'd talked about in visual discovery. Sure, there were dots and circles. Yep, there were the angled shapes hinting at Vignelli's NYC Subway map from the early 70's. All the designs had plenty of functional whitespace, and the layout brought everything together.

But none of these directions were working. The problem, I soon find out, was simple. The work didn't address the larger goal of the project: repositioning MIX as a premier design journal. Instead, these designs merely refreshed the existing visual style.

Hitting Reboot

While our project was slipping into shaky territory, I started trolling back through our early discovery talks, looking for goals I might have missed.

In one of our earliest threads, Nishant and I had discussed that the new site needed to feel "trustworthy." Now that was an awesome concept, and one I'd glossed over in my first design. Right away I was hooked on figuring out how to make MIX a trusted publication.

I backtracked into discovery again, and started researching a handful of sites that I personally trusted—A List Apart, GOOD Magazine, The Morning News, and McSweeney's. Of course, good content is the core foundation of these websites, but I started to see common design patterns emerge too—great art and illustration, warm and inviting color palettes, serif type, and a sense of detail and the unexpected (like Kevin Cornell's always-awesome illustrations).

More than anything, though, these sites were proud of and focused on their content. The interface was built for the content; all other design elements played a supporting role.

That night, I sat down and created a fresh design based on the idea of showcasing MIX's wonderful content and gaining designers' trust. What I came up with was this:

Notice the clear and literary (yet playful) typography, the bold color circles that expose the brand concept of mixing, and the attention to detail in the microcopy illustrations: this is how I rendered the concept of trustworthiness.

Whereas my initial failed attempt had taken days, I created this design in a single sitting. I knew it was the right direction for the site.

Although we ran into snags getting sign off (these were related to how I'd pitched the earlier failed attempt, which you can read about in Nishant's article), the trustworthy design ended up being the direction MIX chose, because it addressed the important questions.

Be Open to Answering "What?"

If you've read to here, you'll notice that I haven't touched on many technical details of my design. I intentionally left out what 16-column grid I used, the rationale for my type choices, what applications I used to layout the elements, and how I crafted my drop shadows for a reason: I don’t think tactical choices are the most interesting part of the story.

The real story behind the MIX Online redesign was how answering the right core question—the what—through a discovery-focused process made the whole thing come together. In addition to this, continuing to search for the what, even when I got off-course, is what lead me to a successful design.

In all my projects, I've noticed that more discussion, careful questions and true exploration have made the process more fun and successful.

In the excellent book "Art Direction Explained, At Last!," designer Marco Fielder had this to say on the subject of strategy:

The what of a thing is very important to us. We try to find the clearest answer to this simple question. We never stop short of it. We push and push until we know for sure. Once we've got it, our design work becomes easy.

When you're in the right mindset and focused on the core problems, the details of a design are far more likely to just "fall into place." I hope you enjoy the new visitmix.com as much as I enjoyed designing it.

Stay tuned for Evan Sharp’s writeup of the absolute rock-solid HTML/css templates he created to bring my designs to life.

Bass-ackwards

Tim Aidlin — Tue, 09 Mar 2010 20:50:24 GMT

I recently had a conversation with a random developer I hadn’t met before. Within a minute of our introduction, he had launched into an enthusiastic, but very misguided, demo of a new technology he was excited about.

Why misguided? The technology in question was meant to solve a problem no one had.

When he asked me what kind of application or site I thought could use this technology, my first thought was, “Again, finding a problem for a solution. Exactly the opposite way I generally try to work on projects.” But instead of saying that, I asked:

“What problem does it solve?”

“What do you mean? You could move around data, put it here or there and connect it—all sorts of things.”

I said to him, “I guess I work a little differently. I try to think about the problem I’m trying to solve, and don’t think about the technology we’ll use until much further in the process of designing the spec.”

“But this technology could really be used in all sorts of applications,” he said.

“Totally. It’s super-cool, and I’ll think about it when I’m specing out my next project.”

“Maybe you can build a project around this technology.”

“Building a product around a particular technology sometimes works, but we find our most successful projects start from trying to solve a problem first, and applying technology to it second. It’s kinda like showing engineer a carburetor and telling him to build something around it."

This conversation struck me as a fairly extreme rendering of a common design snag: building for technology, not to solve problems. You see this all the time, and not just in software or Web site design. I’ve seen it in landscape design, city hall, and the day care, too. What are your favorite examples of solutions without problems? Leave a message in the comments below.

ASP.NET Charts and ASP.NET MVC – Controller vs. View

Karsten Januszewski — Fri, 05 Mar 2010 15:41:56 GMT

Like Hans’ most recent lab note, this one is about charts. However, my investigations center on a different charting platform: the ASP.NET charting controls. These are shipping as part of .NET 4.0 and are available for download today in .NET 3.5 for free. You can also download VS 2008 tooling support and the ASP.NET and Windows Forms sample projects, which provide more than 200 samples.

It appears that there are two architecture options with the charts and ASP.NET MVC, each with pros and cons.

Option 1 – Render The Chart As A FileContentResult Through A Controller

The first option is to send the chart to the client as a file using the FileContentResult. This methodology is well explained with sample code in the following blog post: http://www.myblogon.net/archive/2009/02/15/21.aspx. It is very clean architecturally and maintains the separations of concerns that is core to the MVC design pattern. Once you have your controller set up, adding the chart is as simple as adding the route to the src attribute of an img tag:

<img src="/Chart/SimpleChart" alt="Sample Chart" />

Of course, that src attribute doesn’t have to be static, but could contain parameters generated dynamically, through MVC routing and/or through querystrings, as in the following:

<img src="<%= Url.RouteUrl("ImageShow",new                 
                {  
                    controller = "Image",                     
                    action = "Index",

                    archive = Model.archive_id,  
                    username = Model.User.screen_name,

                    visualization = ViewData["Visualization"]  
                } )%>?value=<%=ViewData["param"]%>" alt="Chart" />

What’s particularly cool about this methodology is that the page is rendered separately from the images, so you get a kind of asynchronous behavior for free.

However, there are two drawbacks to this methodology.

The first is a biggie: you lose interactivity with the chart as far as tooltips, hotspots, hrefs in the charts, etc. If you only care about serving up an image, option 1 is the way to go. But if you want to take advantage of the interactivity hooks provided by the charts, you’ll need to look into option 2.

The second is that the chart has to be entirely created via code, as opposed to using markup to instantiate the chart. Why is this an issue? Well, it gets into designer/developer workflow. I’d much rather let my designer tweak the settings of the chart layout and style. He’s comfortable with tweaking HTML, CSS and XAML, so working the asp:chart xml is totally within his scope. It means he can mess with settings, hit save, refresh his browser and see his results. But, if the chart is entirely generated in code, he can’t do that, which means a much less fluid workflow.

Option 2 – Render The Chart As An ASP.NET Control In the View

The second option is to render the chart in the view, either inline in the page or via CodeBehind. This is well documented in a sample shown in this article: http://www.codeproject.com/KB/aspnet/MvcChartControlFileResult.aspx. By rendering the chart in the view, both of the drawbacks above are mitigated. You can add interactivity to the chart. Consider the following Page_Loaded event in CodeBehind of a view:

        protected void Page_Load(object sender, EventArgs e)       
        {             
            foreach (DataItem dataItem in ((ChartModel)this.Model).Data)

            {                 
                string x = dataItem.Name;                 
                int y = dataItem.Count;                 
                int ptIdx = this.Chart1.Series[0].Points.AddXY(x, y);                 
                DataPoint pt = this.Chart1.Series[0].Points[ptIdx];                 
                pt.LegendText = x + " (" + y + " )";                 
                pt.LegendUrl = "http://example.com/" + x;                 
                pt.LegendToolTip = "Click to view " + x + "'s page";                 
                pt.ToolTip = "#VALX: #PERCENT";                 
                pt.Label = "";             
             }     
        }

You can see how the controller passes the data to the chart through the model, but the actual databinding happens in the CodeBehind. I found that walking the data and setting each x and y coordinate gives you access to the DataPoint object, which is where you set things like LegendUrl, LegendToolTip and Tooltip – the very interactivity I was referring to. If you look at the HTML generated after the control renders (via IE8’s Developer Tools for example), you’ll see that the ASP.NET map control will create a <map> element withmultiple child <area> elements that correlate with the image.

The styling of the chart all happens in the .aspx page:

<asp:Chart id="Chart1" runat="server" Height="480" Width="640" Palette="BrightPastel" imagetype="Png" BorderlineDashStyle="Solid" BackSecondaryColor="White" BackGradientStyle="TopBottom" BorderWidth="2" backcolor="#D3DFF0" BorderColor="26, 59, 105">   
    <Titles>   
        <asp:Title Text="Markup and code behind" />   
    </Titles>   
    <legends>   
        <asp:Legend IsTextAutoFit="False" Name="Default" BackColor="Transparent" Font="Trebuchet MS, 8.25pt, style=Bold"></asp:Legend>   
    </legends>   
    <borderskin skinstyle="Emboss"></borderskin>   
    <series>   
        <asp:Series Name="Column" BorderColor="180, 26, 59, 105" Legend="Default" ChartType="Pie" IsValueShownAsLabel="false" >   
        </asp:Series>   
    </series>   
    <chartareas>   
        <asp:ChartArea Name="ChartArea1" BorderColor="64, 64, 64, 64" BorderDashStyle="Solid" BackSecondaryColor="White" BackColor="64, 165, 191, 228" ShadowColor="Transparent" BackGradientStyle="TopBottom" >   
            <area3dstyle Rotation="10" perspective="10" Inclination="15" IsRightAngleAxes="False" wallwidth="0" IsClustered="False"></area3dstyle>   
            <axisy linecolor="64, 64, 64, 64">   
                <labelstyle font="Trebuchet MS, 8.25pt, style=Bold" />   
                <majorgrid linecolor="64, 64, 64, 64" />   
            </axisy>   
            <axisx linecolor="64, 64, 64, 64">   
                <labelstyle font="Trebuchet MS, 8.25pt, style=Bold" />   
                <majorgrid linecolor="64, 64, 64, 64" />   
            </axisx>   
        </asp:ChartArea>   
    </chartareas>   
</asp:Chart>

The big drawback? Well, it bends the rules of ASP.NET MVC. I’m not sure how “evil” this really is though, as both benefits seem to outweigh the bad practice of adding CodeBehind to a view.

A Common Sense Content Strategy

Tiffani Jones — Thu, 04 Mar 2010 23:12:15 GMT

At the same time, we're not quite sure what to make of the content mania just yet—because we're still figuring out what the word "content" really means, what all the written disciplines on the web are, and what their deliverables and processes look like.

In this article, Part III of a series of V addressing the anatomy of a web design, I'll talk a little about what content mania is all about. I'll also dive into the MIX Online redesign, a case study that illustrates how a common sense approach, solid collaboration between writers and designers, and an organic process make for sweet websites.

This article is part III in a series covering the topic of Web Design. If you haven't already, check out part I—The Anatomy of Web Design, part II—The Future of Wireframes, and part III—Discovering Trustworthiness. Evan Sharp will contribute the final article. Follow us on Twitter or subscribe to our RSS feed to be notified of its publication.

Ye Olde Content Mania

As nascent content practices grow into suit-and-tie disciplines, the variety of web writing services out there has split and multiplied. The result is a thick film that covers terms like "content strategy," with its multiple sub-disciplines that include but are not limited to:

Editorial strategy: Creating guidelines for governing your content. Determining your voice and tone, how you deal with user-generated content, when you publish and so forth.
Web writing: Creating useful copy that sends the right message & helps online readers GTD. In addition to being great writers, web writers should understand the basics of user experience design, information architecture, metadata, SEO, and marketing copy (aka "copywriting," which is not the same as web writing). If your site is big, you might need a web writer who can manage all your content, too.
Metadata strategy: Creating the lesser known content about the content.
Search engine optimization: Making your content easier for search engines to find using keywords, metadata, and other magic.
Content management strategy: Determining what kind of technology will be best for publishing your content, then customizing it to fit your needs.
Content channel distribution strategy: Figuring out when and where to distribute your content. Twitter? Facebook? Blog? Email? Daily? Monthly?

This list (courtesy Kristina Halvorson) is straightforward, but initially confusing to all but seasoned web professionals. After all, don't editorial strategy and web writing overlap? Where does content strategy end and information architecture begin? Don't coder people handle metadata and SEO? Why not just call web writing "copywriting"?

When Danger Comes Knocking

Because of these questions, it can be scary when someone calls needing content strategy work. Oftentimes, what client X calls "content strategy" is really some hybrid of its sub-disciplines—a mixture of web writing and content channel distribution strategy, for example. Other times, client X doesn't know what she needs—she's just heard how awesome content strategy is and wants to take a whack at it.

Unfortunately, there's not always enough room in the budget for a juicy discovery phase, which would allow the content strategist to figure out what the problems are and deliver a thorough, full-scale solution (Better voice and tone? New content management system?) that also coheres with work that's been paid for and done.

As you might expect, problems arise when confusion about what content strategy is meets high expectations, low budgets and an undefined process. Hence, content mania: the feeling that agencies and web pros alike really want better content, but don't know quite what it looks like or how to get it.

The good news is, not every project is content-manic. Every now and again, you stumble across a solid business model and the right expectations.

The MIX Online Case Study

The Ideal Client

The MIX Online team came to me talking like the perfect client. They knew exactly what their business model was, because they'd spent time polishing it. They already had fairly solid editorial, content channel distribution, metadata, SEO, and content management strategies. Their timeline, budget and expectations were reasonable.

Really, what MIX Online needed was simple: a basic review of its editorial strategy & information architecture and some powerful web writing.

Even though Nishant hired me as content strategist, he was open to the idea that I might not need to address every one of the sub-disciplines—he told me to use my best judgment and provide intelligent recommendations.

He was also insistent that Matt, my business partner at thingsthatarebrown and the project's visual designer, and I use our common sense on the project and avoid unnecessary deliverables. He wanted us to do what we'd do if we were creating a site for ourselves.

Eureka.

Discovery: the Right Thing, the Easy Way

Nishant had been filling us in about the project's goals over the months before kick-off, so we had a great sense of what our work would entail. Still, we spent time teasing out additional business and marketing goals, and getting to know MIX Online's business better, in the discovery phase.

We asked simple questions, such as:

Describe your business model
Tell us more about your business and marketing goals
What areas of your site do you feel are working? What areas aren't?
What action would you most like users to take on your site?
Let's work out some key themes and messages. Fill in the blanks. 1. MIX Online is better than any other site like it at _______. 2. We started MIX Online because _______. 3. My favorite thing about MIX Online is _______
Give me 4 or 5 adjectives that describe the personality of your site _______
Who are your main competitors? How are you different from and better than them?
Give me some specs about your audience (phrases such as "point and click liberals" are fine)
Do you have any additional branding guidelines or strategy documents I can review?
Show me some writing whose tone & style you think matches with MIX Online's
And so on and so forth

Even though Nishant's answers were in line with what we'd been discussing, and even though we didn't unearth any shocking information, this phase was invaluable—at the end of it, we had well-defined blueprint we could refer to throughout the project to keep our work on par with MIX Online's goals and vision.

In this case discovery was the right thing, the easy way.

A Minimalist Content Strategy

A common-sense rendering of content strategy best practices drove this part of the project.

Nishant had already performed a thorough audit of his content, which he then transformed into functional wireframes. Most content strategists will tell you that diving into IA before content strategy begins in full is against their religion, but it worked beautifully for this project, since much of the content strategy had been defined in advance.

However, since part of my role was to evaluate the wireframes, I did my own content audit and competitor analysis to ensure my recommendations made sense.

I kept my content audit short and sweet, and reviewed all the content on the existing site to get a better sense of MIX Online's personality and determine whether there were any messages, videos or bits of text I could keep. When I found something that matched the tone and strategy we'd defined in discovery, I copied and pasted it into a document I called "Messages To Keep."

I also reviewed competitors to learn more about how MIX was different and to understand what Nishant meant when he said he did or didn't like competitor X's copy. I captured all my thoughts in a simple analysis document that I referred to later, to remind me of MIX's positioning relative to its peers.

This phase was minimalist, but it gave me what I needed to create on-tone, on-brand messages.

The Functional Wireframes

While I strategized, Nishant finished his functional wireframes. My job was to ensure that the content and navigation on the wireframes reflected MIX's three core offerings—Writings, Labs and Events—and that the taxonomy was clear and consistent.

This turned out to be a straightforward exercise, because Nishant's wireframes were thorough and his instructions for evaluating them were clear. Here's how he put it:

We reviewed, recommended, agreed, high-fived and signed off. It was time to write.

Writing For Design. Design for Writing.

One advantage of working with your husband (in my case Matt, the designer) is that you don't always have to rely on a rigid process as a buffer against things going south.

Matt and I had already discussed our strategy for web writing and design early on, and had decided that an organic process where the writing and design informed one another would be best.

While Matt began drafting moodboards, I drew from my earlier "messages to keep" findings and sketched out key themes and ideas for each page. Then, once Matt had completed his design discovery, I made sure that the tone of my messages and his visuals harmonized.

After that it was a matter of creating text for all areas of the site: intro messages and paragraphs, calls to action, interactive text, product descriptions, bio and "ethos" messages, etc.

Since one of my goals was to make sure the writing and design worked together to tell an engaging, cohesive story about MIX, I didn't just write a bunch of copy based on the content requirements and hand it off to Matt. Instead, I'd create copy, he'd incorporate it into the design, and we'd evaluate. If the copy wasn't working, I'd go back and rethink it.

It sounds like a lot of back and forth, but this organic way of working proved very effective and efficient—we didn't have to freak out if the final copy deviated from the content requirements. We'd built up a solid enough foundation that being "fluid" and "organic" was not a problem.

Though Nishant's visualization of this project doesn't reflect the time Matt and I spent, it's a good indicator of just how "organic" the process was.

The Power of Micro-Copy.

Sometimes little things make the biggest difference. In this case, little bits of copy—which I'll dub micro-copy—joined forces with Matt's visuals to create a strong site narrative.

Part of our task for the redesign was to steer Microsoft in the right direction by placing MIX at the center of the web community. Accordingly, the writing and design needed to be current, modern, and community-oriented; and the tone of the writing needed to be professional, whimsical, trustworthy, and smart. In general, the site needed to arouse the curiosity of web savvy readers.

A secondary goal was to ensure that the copy, especially intro messages on each page, were not overbearing or overly "shouty." Putting effort into the micro-copy helped us create the narrative we wanted without overwhelming readers with too-bold, in-your-face marketing messages.

You're probably familiar with another business, Flickr, who uses micro-copy to its advantage. Notice how the sidebar callouts and playful asides make a big difference.

There's not a ton of micro-copy on MIX's site, but what's there makes a big difference in users' experience, by giving readers a sense that MIX is a warm, engaging place to be.

A Project-Sensitive Approach

Structured disciplines like content strategy, information architecture, and the like exist for a reason—most notably to produce good work on time and on budget.

But the reality of client work is that a tightly structured process is not necessarily a surefire way to produce excellent work. Sometimes it's better to adapt our processes to our clients' and to build on the great work they've already done.

For the MIX redesign, being flexible and organic worked beautifully. It allowed us transform Nishant's functional wireframes and our collaborative discovery into a writing and design that have worked well so far. Even better: it also allowed me to experiment with different ways of thinking about and producing content in collaboration with a designer.

The project-sensitive approach I'm talking about won't work for every single project, but it's perfect for projects where agency and client trust one another a great deal.

The Future of Wireframes

Nishant Kothary — Tue, 02 Mar 2010 19:33:31 GMT

This article is part II in a series covering the topic of Web Design. Check out Part I—The Anatomy of Web Design, Part III—A Common Sense Content Strategy and Part IV—Discovering Trustworthiness. Evan Sharp will contribute the remaining article. Follow us on Twitter or subscribe to our RSS feed to be notified of its publication.

Riddle me this: How do you piss of a UX professional? The answer: Call him a "designer".

These days, user experience professionals look down on the word "designer" because it implies that their primary role is to paint pretty pixels. UX is more than that, they clarify. Much more!

Just how much? Well, here's a diagram (that uses pretty pixels) to explain how much more—

Holy guacamole, Batman! Elliptical hotness!

But wait! Does this diagram mean there is only one successful UX professional in the world? Steve "DamnItNotHimAgain" Jobs? Because no so-called "designer" can possibly wear all these hats.

It's time we end this madness! While we're squabbling over why one type of designer is different or better than another (and falling prey to one of the oldest colonization tactics in the book: divide and conquer), our industry continues to build crappy software. And without us.

Convergence in the Simulacrum

IA, content strategy and visual design are quickly converging on the Web. As design becomes democratized and users demand more than the "craigslist experience", business owners are discovering that the way to keep users on a site is to differentiate themselves through intelligent content and innovative design that exist within the natural patterns available to the respective mediums (browsers, phones).

In 1999, Jakob Nielsen wrote an article that was undeniably ahead of its time: Differences between Print Design & Web Design. He argued that anything that is a great print design is likely to be a lousy web design due to the inherent differences between the two mediums. Most notably:

Print design is 2-dimensional while web design is 1-dimensional and n-dimensional simultaneously. The web experience is fundamentally a "scrolling experience" dominated by movement between pages through hyperlinks.
Print is about seeing, web is about doing. The "look and feel" of a web site is dominated by "feel", i.e. the user experience. This is because doing always makes a stronger emotional impact than seeing.
Print is immensely superior to the Web in terms of speed, type, image quality and canvas size. The Web has problems associated with low bandwidth, low screen resolution and limited screen sizes that affect the user experience significantly.

Fast-forward to 2010 and it's pretty easy to see how these differences have diminished drastically—or even disappeared. Users have become savvier and technology has improved. The leading edge of web design capitalizes on how minor these differences are today. Print-y web sites like Jason Santa Maria's personal blog or Carsonified's business site are just two examples of this. You can find hundreds more on web design showcases like Drawar.

I daresay to God that the Web is finally converging with Print. So, where does this leave our trusty wireframes?

The Problem with Wireframes Today

You can find hundreds of definitions of the word ‘wireframe' on the Web. Here's one that represents what's typically practiced today [courtesy webopedia]:

A wireframe is a visualization tool for presenting proposed functions, structure and content of a Web page or Web site.

Unfortunately, the most common interpretation of this definition practiced in workplaces leaves much to be desired:

A wireframe is a line drawing that dictates exactly what functionality and content is located where on a Web page or Web site.

This interpretation drastically limits the potential of a web design. It sets a glass ceiling for the visuals and copywriting, two supremely important aspects of great web design. It promotes the notion that visual designers and copywriters needn't bother themselves with size, location and functionality of the elements of a design and that their individual products—the UI and the copy—don't play much of a role in shaping the flow and interaction on a web site.

This may have been true in 1999, when we were all getting used to the new UX metaphors of the Internet. But it certainly isn't the case today, in a time where mediums and disciplines are converging and metaphors like the "back button" or "scrolling" have become muscle memory.

Bottom line? We need to update our interpretation of the word "wireframe".

Rethinking Wireframes

A little less than a year ago, Isaac Pinnock of Made by Many wrote an article titled, The Future of Wireframes? Our articles share titles and arrive at very similar conclusions, but they use some very contradictory terminology along the way.

Isaac wrote about terrible "functional wireframes" from the 90's being replaced with wonderful "visual wireframes" of today, for example:

10 years ago the first wireframes I used were about as functional as you can get – a long list of page elements: static text, dynamic text, input text, radio button and so on. They were universally awful. About the only concession to help people understand how the page worked was to group common functionality into individual tables.

The wireframes were functional rather than visual as they were used to describe how the page should be built. Certainly, when you consider the screen from a developer's perspective a list of different functional elements is probably quite logical.

However, from a user experience point of view this was a killer. Functional wireframes are incredibly difficult to read – the method of presentation gets in the way of being able to translate the information into a real screen, especially at the review stage.

This is going to seem very confusing, but unlike Isaac, I use the term "functional wireframe" in a very positive light and take for granted that wireframes are "visual". By attaching the label "functional" in front of "wireframe", I am asserting what is and isn't in scope for that particular wireframe. The first page of my wireframe deck explains this clearly:

Functional wireframes—call them whatever you want—strike a balance between functional specifications and traditional wireframes. They have some very powerful benefits, especially when the members of a design team have cross-disciplinary skills (this is usually the case today). Most notably:

They democratize layout decisions, allowing the natural synthesis of a more unified final design.
They encourage collaboration and allow designers (visual, IA, content, interaction, etc.) to arrive at a holistic vision.
They help manage client and stakeholder expectations by focusing the discussion on page-level functionality during reviews.

Of course, for functional wireframes to work, there is a higher tax on softer skills like communication, feedback loops, and trust. Functional wireframes are not for organizations that haven't grasped basic principles that allow us to predictably design great solutions that meet business needs.

But, as the MIX Online redesign case study demonstrates, functional wireframes can truly promote the behaviors necessary to create more cohesive designs.

The MIX Online Case Study

We created MIX Online's functional wireframes over the span of exactly one week (the IA phase). As promised on the introductory page of the wireframe deck, I attached a color code at the top of each wireframe to indicate my confidence in its layout. Here's an example of a profile page wireframe:

The process I used was pretty simple: I created a couple of wireframes a day, updated the deck and then posted it on both Basecamp projects. You may remember from Part I of this series—The Anatomy of Web Design—that the MIX Online stakeholders were restricted to one dedicated Basecamp project, while the design team was restricted to another. I served as the conduit.

On the stakeholder side, the MIX Online team was focused on finding gaps in my thinking and ensuring that I wasn't missing any needed functionality. On the design team side, Tiff, Matt and I were engaged in intensely iterative conversations—digging into the content strategy, formulating a strategy and tone for copy, brainstorming around visual design, developing a category/sub-category taxonomy, and so on.

What we didn't spend any time discussing was layout—in other words, the location of all the different boxes of information. Functional wireframes postpone layout discussions and ultimately rely on the Visual Designer to finalize layout of the various elements on the page. This is not to say that I didn't suggest page-level layout in my wireframes. In fact, I did—and in many cases, strongly so.

For the most part, though, I didn't get caught up in layout details. My primary interests were to ensure that I had created a sensible sitemap and captured all the right functionality to support our business. I was also focused on ensuring that I'd partitioned the functionality between pages to encourage the right behaviors among users.

You can download the final set of wireframes by clicking on the image below. I encourage you to compare the wireframes to the actual site. Notice that in many cases, even the layouts marked "green" didn't quite work out. For instance, the Labs page is one where Matt decided to go with a simplified layout similar to our prior lab page.

As a whole, this approach departs from traditional IA methods—which include providing an actual blueprint for the site, layout and all. But it's precisely this departure that is necessary to take designs to the next level.

In parts III & IV of this series—Tiffani's article on Content Strategy and Matt's on Visual Design—you'll see more examples and evidence relating to effectiveness of the organic functional wireframe approach.

Back to the Future

Daniel Pink argues in his book—A Whole New Mind: Why Right-Brainers Will Rule the Future—that software design is fairly unchartered territory, but will mature in this century.

Web design is one of the sub-genres of software design that's just started a slow crawl, characterized by constant retracing of steps, out of its infancy. Less than a decade ago, the introduction of WYSIWYG editors promised to deliver the holy grail of web design. Today, good ol' text editors like TextMate—the ancestors of WYSIWYG editors—are taking their place yet again.

The point is that the discipline of user experience could and probably will look completely different a few decades from now. IA as we understand it could cease to exist, and it could very well merge with visual design. The possibilities are endless.

Which is why we need to entertain the idea that we're doing it all wrong at the moment and embrace the notion of cross-disciplinary approaches whenever we get the opportunity. Functional wireframes are but one attempt at that.

Isaac Pinnock said it best in the conclusion of his article:

The best sites are those where there's a seamless divide between the look, the content and the experience. Being able to model an interaction and understand how someone moves through a site are crucial skills in this trilogy. It's time designers stepped up to the plate and claimed wireframes as their own.

Silverlight Toolkit Charting Controls

Hans Hugli — Mon, 01 Mar 2010 17:43:05 GMT

The Problem with Charting Control Styling

Recently, my team needed some basic charting controls for a project we were working on, so I investigated the Silverlight Toolkit charting controls available on CodePlex. We quickly discovered that the Charting control styling was cumbersome in Expression Blend. I ran into the following issues:

The Toolkit runtime DLLs don’t support editing styling templates
The Themes approach that the Silverlight Toolkit employs doesn’t have design time support

If you’re looking to use the Silverlight Toolkit Charting controls, here are a few options for working around these issues.

Getting Ramped Up

I was pointed to a good blog from December 2008 that describes how to style the charting controls. The blog is written well, but dated—I’ll clarify any inaccuracies below.

I installed the Silverlight 3 Toolkit with the installer located on the home page of the CodePlex site. I then fired up Blend 3, created a new Silverlight Web project and added references to the Silverlight toolkit binaries: System.Windows.Controls.Datavisualization.Toolkit.dll and System.Windows.Controls.Toolkit.dll. Then I opened up the Assets panel by clicking the chevron on the left hand menu and chose the Chart control by clicking on it, as shown here:

Once I clicked the Chart control, I then created the control on the designer surface by dragging an area on it. It created a "Column" Bar Chart that showed pre-populated data.

I wanted to create a Pie chart, so I deleted the ColumnSeries node in the “Objects and Timeline” view by selecting it and pressing delete. This left me with an empty chart. I added a PieSeries by dragging one from the Assets panel into the chart control on the design surface; nothing changed visually since the bindings to the data source have not been defined as they were in the default ColumnSeries. I switched to the XAML view in the design window and added bindings to the data source. In order for the PieSeries element to bind to the PointCollection that was created when we created the chart control, I needed to add the following attributes: IndependentValuePath="X" DependentValuePath="Y" ItemsSource="{Binding}"

This is how the XAML fragment for the chart control appears:

<charting:Chart Title="Chart Title">

	<charting:Chart.DataContext>

		<PointCollection>

			<Point>1,10</Point>

			<Point>2,20</Point>

			<Point>3,30</Point>

			<Point>4,40</Point>

		</PointCollection>

	</charting:Chart.DataContext>

	<charting:PieSeries Margin="0" IndependentValuePath="X"
		DependentValuePath="Y" ItemsSource="{Binding}"/>

</charting:Chart>

The chart control can be bound to any data source. In this sample, the PointCollection data is supplied so that the designer will show some basic data on the design surface.

Blocking Issue

The tutorial I was following assumed that the designer had the Toolkit project’s source code in his or her project, rather than just a reference to the runtime binaries. That is to say, if you simply install the Silverlight Toolkit with the MSI file, adding the necessary binary references to your project will not allow the designer to modify the chart styles.

Here’s an image (circled in red) that shows that editing a copy of a template of the current selected item is unavailable.

The Workaround

At the time of this writing I still don’t know why the designer behaves this way with runtime binaries, but the workaround is to have the design solution also contain the source code for the Toolkit charting dependencies. You can download the entire Toolkit source code from the homepage of the CodePlex Silverlight Toolkit project by clicking on the download button, but you don’t need to add all projects to your solution to accomplish the goal.

Begin by deleting the existing references to the Toolkit runtime assemblies. Then add the following projects to your solution: Controls.Data.Toolkit, Controls.DataVisualization.Toolkit, Controls.Input.Toolkit, Controls.Layout.Toolkit, Controls.Theming.Toolkit and Controls.Toolkit. Lastly, add references to all these projects to the project that contains the page you want to create the chart in. Once you’ve added the references, the “Template Editing” option becomes enabled. From there you can modify the templates in the designer. The design surface will update visually as modifications are made.

Now it’s also possible to modify the DataPointStyle templates and the LegendItemStyle templates, which allow fine-grained control over how the data points and legend appear.

Erratum to Pete’s Blog Post

Pete’s blog post makes reference to a type: StylePalette. However, the correct type is Palette. The type’s name was changed after the December 9^th, 2008 release of the Toolkit. At the time of this writing, the version from November 2009 is available.

Additionally, the path to the namespace reference in the XAML has changed from:

xmlns:datavis="clr-namespace:Microsoft.Windows.Controls.DataVisualization;
	assembly=Microsoft.Windows.Controls.DataVisualization"

to:

xmlns:datavis="clr-namespace:System.Windows.Controls.DataVisualization;
	assembly=System.Windows.Controls.DataVisualization.Toolkit"

Themes

In theory, the themes support is very nice, since it’s very easy swap out the themes. However, you can’t author the theme ResourceDictionaries in Blend, since Blend doesn’t allow editing the XAML for a ResourceDictionary directly. As a result, you’ll find yourself jumping into the styling for a chart and manually moving it to a ResourceDictionary. Not the best designer experience.

Summing Up

The Silverlight Toolkit Charting controls work well out of the box, but if you are a designer and would like to restyle them, it will take a bit of work. Here is a link to much more useful information about the Silverlight Toolkit. A developer will have an easier time modifying the styling, but developers are not designers and generally can only take the design so far. Be aware that if you decide to use these charting controls, they aren’t particularly Blend-friendly. Perhaps this will change in the future.

Do you have some designer-friendly charting controls that you can recommend? Silverlight-based or not? Let us know. Leave a comment or if you tweet, follow us on Twitter to learn about new content, opinions and articles.

Cheers to Saying No

Tim Aidlin — Mon, 22 Feb 2010 23:03:59 GMT

The other day I was out having a pint with a web-designer friend of mine, and she relayed a conversation with a client she'd had that day. I had to chuckle a little.

“So, Client X called me today, and she’d like to do a whole branding and website campaign,” she said.

I had heard about troubles working with Client X in the past. “Client X, huh? What’d you tell her?”

“Well, I gave her a standard ballpark figure, but told her I wouldn’t be able to get on it until the end of next month.”

“And of course she needs all of this done now.”

“Of course. So she suggests that she have one of her friends build the website while I work on the branding, and we might be able to get it out on schedule.”

“But you can’t start on this until the end of next month,” I said.

“Oh, but if someone else is building the website, then I should be able to do it now. According to her, of course.”

“Ah, yes. And this web designer is going to build the site without any branding or content, I’m assuming.”

“And Client X wants me to go ahead and ‘take the reigns’ on working with the web developer and just include her on the meetings as necessary.”

“Ah. So, project management, too. I see. And will you be writing content?” I asked.

“No, but she’d love me to help edit it.”

“Hmmm.”

“Oh, and if I could add a shopping cart, and some cool animation, that’d be great, too, but that might require more work with the web developer. Is that hard to implement?”

“Wow,” I said.

“Yeah,” she responded. “So, no, I’m not taking on the client.”

“Thank you,” I sighed. “I don’t know if either one of us would be able to stand having beers and talking about this for months and months and months. Because you *know* this would change 4 times.”

“Cheers to saying no.”

“Cheers.”

Where do *you* draw the line with projects? When do you push back on a feature, request, or even the project itself? Let us know by leaving a comment below and be sure to follow us @mixonline.

Articles, Opinions & Lab - MIX Online

Architecture Of The Archivist

Architecture I: Prototype

Architecture II: SQL-centric

Architecture III: No SQL (Well, Less SQL)

Architecture IV: Queues

Architecture V: No Queues

Final Thoughts

Article vs. Section: We've Finally Gone Mad

Using jQuery to animate a Hero

MIX Online’s Glimmer

Download the files and dive in directly, or read on …

Watch The Screencast

The File Structure

The HTML

Main hero container

Glimmer

Download Glimmer

Getting Started

Step 1: Clear the field

Let's Save!

Back to Step 1

Step 2: Bring On the Big Green Friendly Robot

Step 3: Waiting and leaving

Step 4: Bring in Slide 2

Get To It

What Comes After Sketching? Advice for Developers.

Developer as Artist

The Sketch, The Silver Bullet

Sketching: so easy a CEO could tell you to do it…

Step 1: Permission To Be ‘Emotional’ Has Been Granted.

This moment NOT brought to you by ‘rational’…

Step 2: Up Your UI IQ

“pssst. Your brand is a language, pass it on…”

Step 3: Get Lots of Exercise

Getting enough exercise is important for growing designers…

Task 1: Find a User Experience that you think is very effective and be prepared to say why you think it works.

Good User Experiences popping up everywhere…

Task 2: The Attributes & Associations

So… inspired…

Task 3: Get Personal

Task 4: Be Moody

When I said this is a ‘retro’ exercise, I meant retro ‘fit’…

Everyone’s An Artist

… and that’s what you do after you sketch.

Is Your HTML5 10K Apart?

The Skinny

10K in Prizes

Just One—OK, Five—Catches

4 Weeks & Counting

How To Be Evil

The Balance Sheet

Accounting Errors

Kant’s Recipe Book

What To Do?

The Humor of Code Check-in Comments

The Evolution of The Archivist

Audience Research

Getting Stakeholders and Internal Microsoft Team Members On Board

“Conversation” with current users

What Current Users Want

User personas

Drive-by vs. Return Users

The Marketing Manager

The Academic

Wireframes

What we couldn't do (Feature Scoping)

The Visualizations

Visualization explorations

Less is More, More or Less

Mo Options Mo Problems

An Apple a Day

Miller's Law: The Magic Number Seven

KISS my Point

How The Archivist Polls Twitter

The Beginning of The Archivist

The Archivist Desktop

Save tweets

Export the data to analyze

Visualizations to interpret