Tuesday, November 3, 2009

Simple is a Feature Too

Last week we ran a round of usability tests on Newsforwhatyoudo.com. Random folks in a coffee shop were invited to try out the app for the first time in exchange for a $10 coffee gift card. They were given a few common tasks to complete, but no other direction. We sat next to them watching what they did. When you see normal humans (i.e. people who don't live and breath social media and web development) use your app its like opening a door into another universe. A universe where all those bells, whistles, and knobs you wanted for your own usage becomes a source of confusion. Steve Krug, author of Don't Make Me Think! describes a typical user viewing a web page as "driving by a billboard at 60 mph". Its an apt description of how users used our application.

The impact on our roadmap was interesting. Some features are just going away. Others are being combined, automated, or put under an "advanced" tab to reduce visual distraction. The two most important things people wanted were simplicity and social features. Simple doesn't imply an absence of function - in many cases it will require more functionality to make the complex simple. When reading the news, users don't want to use an application, they want to read the news and interact with their colleagues and friends. A successful application is one that is nearly invisible, while bringing content and social interaction to the fore.

The experience reinforced the need to regularly look at the UI and say "What can we remove, combine, or automate or hide?" Maybe once every 4 sprints, plan a "simplicity" sprint where the only objective is simplifying the user experience. Testing regularly with users helps, but ultimately the drive towards ease of use must come from the developers themselves. Facebook provides a good example for this process. On one hand Facebook and its thousands of third party developers have added mountains of new functionality. At the same time they're constantly simplifying the experience to make sure that the site doesn't become the next MySpace. Considering the UI experience, there's no better model than Google, which reduces the infinite complexity of the Internet down to one list of links given a few keywords.

On the social front we're planning Facebook integration and a new take on the collaborative reading experience. I think the best feature idea in our brainstorming exercise came from pushing hard on simplicity and social interaction as goals. What if we really examined every assumption about what a feed reader is and assumed the user only cared about content and social interaction? Could we design an highly relevant news experience if the user did nothing but read content and interact with their friends?

Yes we can.

Friday, August 28, 2009

Kill Your Signup Form with Rails

Nobody likes Signup Forms

Even though the gradual engagement meme has been around for a while, and everyone just hates signup forms, they just seem to keep popping up like a bad habit. My site, Newsforwhatyoudo.com was one of the guilty parties. We saw users coming back to the site repeatedly, but not signing up. The percentage that looked at the signup form and then bolted was uncomfortably high. It was time to kill the signup form. This blog post documents how we implemented gradual engagement using Ruby on Rails and restful authentication.

Gradual Engagement Principles

Here are some principles I've gleaned from learning about gradual engagement.

  • Make sure the user sees a direct benefit to every sign up step. When you do ask for information, do so only after 1) the site has demonstrated user value, 2) the user has a vested interest in the site, demonstrated by having customized some aspect of its behavior, and 3) you provide a carrot that justifies providing signup information.

  • Set reasonable defaults automatically and let the user customize them later. For example, in our original join workflow, Newsforwhatyoudo asked users to add subscriptions and topics as part of the process of joining a group. Instead we now automatically give users the most popular subscriptions and let them change these later.

  • Security only when its needed. Forcing a user to establish a username and password up front may be necessary if you're developing a banking site where all the information being displayed to a user is confidential, or the user's actions have financial impact. Most web sites don't have this issue. Even E-commerce sites can let users browse, search, compare and price without any security - its not until the credit card information is required to make a purchase that formal signup is required. For most sites anti-spam is a bigger issue, and there too, anti-spam measures like captchas and email confirmations should be used only after the user does something that requires their use. I emphasize the word "after", because if you wait until after a user has filled out a comment form, then present the confirmation, you're more likely to have the user perform the confirmation because they've committed energy to filling out the comment.

Keeping out the spammers

The most time consuming part of implementing gradual engagement was figuring out how to keep the spam bots out. At newsforwhatyoudo.com our strategy is two tiered. Actions that change state - particularly those that submit forms - require the client to run javascript. This eliminates most spam bots. If we detect a client that's not running javascript, we throw up a captcha. Most users never see the captcha and can just submit the form. The captcha eliminates bots that can run javascript. None of this helps if someone designs a bot specifically to attack our site, but then again neither did our old process of asking for user credentials and showing a captcha, so we've gained usability without losing anything.

Here's an example with a "email support" form.

   <% form_for :ask_support, :url => {:controller => 'home', :action => 'email_support'} do |f| -%>

><%= label_tag 'Your Email *' %><br />
= f.text_field :email %></p>
<p><%= label_tag "Email message *" %><br />
= f.text_area :message %></p>

<%= render :partial => 'shared/check_for_bots' %>

><button type="submit">send</button></p>

<% end -%>

The check_for_bots partial is added to any form, link, or button that changes state. This partial looks like this:

   <% if @show_captcha && ENV['RAILS_ENV'] != 'test' %>
<p style="padding-top:10px"
<%= label_tag "Help us reduce spam by filling in the below" -%>
= recaptcha_tags %><br /><!-- @human =true by default if were showing captcha -->
<% elsif ENV[
'RAILS_ENV'] != 'test' %>
<input type="hidden" name="turing_a" id="turing_a" value="Please do not alter" />
<%= javascript_include_tag
'turing_a' %>
<% end %>

check_for_bots adds a hidden input field to post and put requests. The "turing_a" javascript file runs when the form loads and returns a value in the hidden input field based on how long the user viewed the form before hitting submit. If the length of time is below a threshold or incorrect, we reject the input and instead show a captcha. After that the standard captcha acceptance routine applies. The javascript we use is adapted from Stephen Hill's Javascript Captcha.

// stop bots that cant run js or are too quick to submit the form
// check hidden value for a reasonable interval: interval in seconds

var turingA = function() {
if (document.getElementById("turing_a")) {
a = document.getElementById("turing_a");
if (isNaN(a.value) == true) {
a.value = 0;
} else {
a.value = parseInt(a.value) + 1;
setTimeout("turingA()", 1000);


Back on the server side we need to validate the hidden input field. If validation fails, we render a captcha.

def email_support
@popular_groups = Group.most_popular
raise MissingFields unless params[:ask_support]
@ask_support = AskSupport.new params[:ask_support][:email], params[:ask_support][:message]
raise MightBeBot if !@human
raise MissingFields if @ask_support.email.blank? || @ask_support.message.blank?
"about page",
params[:ask_support][:message] )
flash[:success] = "Got your request, we'll be in touch"
rescue MissingFields
flash[:error] = "Couldn't process your request, make sure both fields are filled in."
render :action => 'about'
rescue MightBeBot
@show_captcha = true
flash[:error] = "Your browser must be running javascript, and/or you must enter the correct words below"
render :action => 'about'

The check_for_bots method is added to any method we need to protect. It sets @human which the caller can check as part of its input validation.

  # Used as before filter on POST and PUT actions to determine
# whether the requestor is likely to be human. Requestor needs to be able to execute .js
# and wait at least 4 seconds before submitting the form, or this fails and returns false.
# This method doesn't require the user to do anything, other than have a js capable browser
# Note when testing in rails this method will (and should) always fail.
def check_for_bots
if current_user && current_user.status == 'verified' # already showed captcha and user passed
@human = true
# verify_recaptcha is true if in test mode or success, false if fails.
# verify_recaptcha may only be called once per action and get a valid result!
if params[:recaptcha_challenge_field]
@human = verify_recaptcha
if current_user
self.current_user.status = 'verified' # for other actions set it here.
@verified = true # for NewsC#more there is no current user yet, so tell auto_create_user
# to set status
elsif params[:turing_a].to_i >= 3 # 3 second delay
@human = true
@human = false

Auto creating users

The next part of getting rid of the signup form is to automatically create user objects, sessions, and remember-me cookies. This is what our old signup process did. We encapsulate this process into a method and have it run whenever an un-authenticated user tries to perform an action that requires a user object/session. We are using restful-authentication, and pulling out the relevant lines of code from there and from sessions controller yields this handy method:

  # Create user object if doesn't already exist.  Set cookie.   
def auto_create_user!
user = User.auto_create_user_object # doesn't save to db as handle remember cookie does.
user.status = 'verified' if @verified # user passed captcha, prevents showing captcha again
User.current_user = self.current_user = user # !! now logged in (sets session)
handle_remember_cookie! true # sets cookie and saves user so they can get back after session is over
# cookie set to 5 years.., the above uses @current_user set in previous line.

Note that since the user doesn't have the ability to "log in", you'll want to set your remember-me cookie time span to an appropriately long time. This can be changed in vendor/plugins/restful-authentication/lib/authentication/by_cookie_token.rb Once the user object is auto-created, the user can return to their accounts page and add a login and password if they want to use their account on another computer. We auto-generate a random username for display purposes, letting the user customize that later if they want to.

That's it! The old signup form still works, but is no longer necessary to use all the site's functionality. We've been in production for about two weeks now with zero spam infiltration, but time will tell. If anyone has better ways to do any of this stuff, give some comment love below.



Monday, August 17, 2009

China & Brazil Stock Markets - Fading Fad or Growing Opportunity?

This morning I took a look at the relative market performance of the Chinese, Brazilian, and US stock markets over the last 5 years. (link to dynamic chart). Looking at their return over both boom and bust periods and correlating it to fundamentals was instructive.

In the 2001/2 recession these markets were either small or non-existent from an international investment perspective. This last year was the first recessionary period that these markets have weathered as "major" markets. If you invested in China or Brazil in 2008 or 2007 your investment is lying bleeding and tattered on the ground, with 60-80% declines. On the other hand, if you invested early in the cycle - say in early 2005, you're up around 150%.

Looking at fundamentals, the Chinese economy (GDP) grew at 6% Y/Y over the last 1 year recessionary period and closer to 9% Y/Y over the last 5 years. US GDP in contrast grew at around 2.4% annually over the last decade. So its no surprise that the Chinese stock market has dramatically outperformed the US market over the last 5 years. The Brazilian economy grew at around 4.7% over the last 5 years and its stock market similarly out performs the US market over that period.

The data supports the idea that these growth economies are better long-term investment opportunities than the US market. If your investment horizon is say 10 years and you don't like playing momentum, this might be a good time to invest since both China and Brazil are down dramatically from their highs. On the other hand, these markets have not decoupled from the US market - if the US market moves down 10%, these markets will probably move down 30 or 40%. If you believe the US market will take a second dip in the near term, the best time to invest may be 6-12 months in the future.

Tuesday, July 28, 2009

Tokyo Cabinet and Me

There's been lots of talk about Tokyo Cabinet (TC) lately so I had to get my hands on this shiny new toy. The appeal is that key value stores like TC offer better performance than traditional RDBMs, and are also dramatically easier to setup. They're not right for every project, but I had a project come up that seemed like a good fit. The project was a background task that interacted with the Twitter API and had to keep track of a large (millions) number of key-value combinations. Fast existence tests for keys and basic get/set/iterate methods was all that was needed. Essentially what I wanted was a persistent Ruby Hash.

I looked at Rufus Tokyo and the Ruby bindings written by Mikio Hirabayashi. I ended up using Hirabayashi's Ruby bindings, because they were a little closer to the metal, and the less code that can break the better. While there is no gem, installing from source was easy, both on Centos and Leopard. If you're on Leopard, don't use the macports versions - they're broken. I'm not using Tokyo Tyrant, as only one process needs to access this data I'm using Tokyo Cabinet directly.

The only things the Ruby bindings didn't give me were a simple open method and a way to serialize values so I could put any Ruby data structure into the cabinet.

Hirabayashi's Ruby bindings already provide Ruby Hash like access, including hsh['foo'] = "bar", and hsh.each. Nor did it make sense to create my own interface class when really all I needed was to extend a tiny subset of TokyoCabinet::HDB's existing methods. What I came up with is this:

require 'tokyocabinet'
include TokyoCabinet

# Duck Punch of TokyoCabinet Hash database that uses YAML to serialize Ruby objects. Serialization
# necessary since TC expects values as strings or integers so won't handle other data types. Also provides
# a consistent, simpler open method.
class TokyoCabinet::HDB

# initialize db and return handle to db. There is one db file per data structure, e.g.
# new hash means new database and database file so call init again. Creates db file
# if it doesn't already exist.

alias original_open open
def open(path_to_db)
# open the database
if !self.original_open(path_to_db, HDB::OWRITER | HDB::OCREAT)
ecode = self.ecode
STDERR.printf("open error: %s\ntry to open file #{path_to_db} - ", self.errmsg(ecode))

alias original_get_brackets []
def [](key)
result = self.original_get_brackets(key)
result ? YAML.load(result) : nil

alias original_set_brackets []=
def []=(key,value)
self.original_set_brackets(key, YAML.dump(value) )

alias original_each each
def each
self.original_each { |k, v| yield( k, YAML.load(v) ) }


Dear MySQL, I think we should see other people.

Saturday, May 16, 2009

If you're doing a startup, don't miss this

If you're in or thinking of starting a new company, or even a new product category in a large company, these are essential to maximizing the chances of success. I've been in many companies big and small and the correlation between success and following the principles discussed in these classes has been 100%. Steve Blank is an experienced entrepreneur that teaches at Berkeley.



Thursday, May 14, 2009

Tagging the real world

Maybe I just missed it, but this seems to have gotten much less coverage than it deserves. The demo opens up a huge set of possibilities around making the physical world searchable and interactive, just like the web. I know little about what technologies tochidot is using, but I've been exposed to several companies, two of which I worked for, that have developed components of making such a system possible.

Basically you need
  • text recognition on low-rez camerphones: snaptell.com
  • image recognition and comparison to like images: searchartgalleries.com
  • geolocation data
  • a smart phone with a big screen (Apple, Google Phone, ..)
  • a backend that matches incoming images and geolocation data to historical data such as tags, reviews, user comments, etc.
My head is swimming with the possibilities.

Wednesday, May 6, 2009

Announcing use_uuid and acts_as_distributed

I've open sourced two Rails plugins that facilitate building distributed Rails applications. Use_uuid integrates UUIDs and schema less attributes into rails models. Acts_as_distributed replicates state changes to model objects to a queue so they can be distributed to other databases and applications. Both of these plugins are currently in production in newsforwhatyoudo, and are tested with Rails 2.2.2

Rails is a great framework, but its not simple building a distributed application. These are two small steps towards making it easier. I'll be talking at the SF Ruby Meetup about how we use these plugins on May 13 - see the meetup link here. It would be great to collaborate with other developers on building out a collection of plugins that enabled all the key portions of distributing applications in an integrated way.

Tuesday, April 28, 2009

Residential Real Estate Investor Group on newsforwhatyoudo

Newsforwhatyoudo.com has launched a social news / feed reader site for professional news. One of the professional groups is focused on residential real estate investors, particularly distressed real estate. Its much more automated and collaborative than the news groups on linked in.

Give it a try here. Its in private beta, so you'll need to use this ticket when you sign up:


This is good for the first 50 users.

Friday, January 23, 2009

Wingsuit basediving

Ah, to be young and careless again. If I ever get diagnosed with a terminal disease, this is what I'm going to do.

wingsuit base jumping from Ali on Vimeo.