Friday, August 28, 2009

Kill Your Signup Form with Rails

Nobody likes Signup Forms

Even though the gradual engagement meme has been around for a while, and everyone just hates signup forms, they just seem to keep popping up like a bad habit. My site, was one of the guilty parties. We saw users coming back to the site repeatedly, but not signing up. The percentage that looked at the signup form and then bolted was uncomfortably high. It was time to kill the signup form. This blog post documents how we implemented gradual engagement using Ruby on Rails and restful authentication.

Gradual Engagement Principles

Here are some principles I've gleaned from learning about gradual engagement.

  • Make sure the user sees a direct benefit to every sign up step. When you do ask for information, do so only after 1) the site has demonstrated user value, 2) the user has a vested interest in the site, demonstrated by having customized some aspect of its behavior, and 3) you provide a carrot that justifies providing signup information.

  • Set reasonable defaults automatically and let the user customize them later. For example, in our original join workflow, Newsforwhatyoudo asked users to add subscriptions and topics as part of the process of joining a group. Instead we now automatically give users the most popular subscriptions and let them change these later.

  • Security only when its needed. Forcing a user to establish a username and password up front may be necessary if you're developing a banking site where all the information being displayed to a user is confidential, or the user's actions have financial impact. Most web sites don't have this issue. Even E-commerce sites can let users browse, search, compare and price without any security - its not until the credit card information is required to make a purchase that formal signup is required. For most sites anti-spam is a bigger issue, and there too, anti-spam measures like captchas and email confirmations should be used only after the user does something that requires their use. I emphasize the word "after", because if you wait until after a user has filled out a comment form, then present the confirmation, you're more likely to have the user perform the confirmation because they've committed energy to filling out the comment.

Keeping out the spammers

The most time consuming part of implementing gradual engagement was figuring out how to keep the spam bots out. At our strategy is two tiered. Actions that change state - particularly those that submit forms - require the client to run javascript. This eliminates most spam bots. If we detect a client that's not running javascript, we throw up a captcha. Most users never see the captcha and can just submit the form. The captcha eliminates bots that can run javascript. None of this helps if someone designs a bot specifically to attack our site, but then again neither did our old process of asking for user credentials and showing a captcha, so we've gained usability without losing anything.

Here's an example with a "email support" form.

   <% form_for :ask_support, :url => {:controller => 'home', :action => 'email_support'} do |f| -%>

><%= label_tag 'Your Email *' %><br />
= f.text_field :email %></p>
<p><%= label_tag "Email message *" %><br />
= f.text_area :message %></p>

<%= render :partial => 'shared/check_for_bots' %>

><button type="submit">send</button></p>

<% end -%>

The check_for_bots partial is added to any form, link, or button that changes state. This partial looks like this:

   <% if @show_captcha && ENV['RAILS_ENV'] != 'test' %>
<p style="padding-top:10px"
<%= label_tag "Help us reduce spam by filling in the below" -%>
= recaptcha_tags %><br /><!-- @human =true by default if were showing captcha -->
<% elsif ENV[
'RAILS_ENV'] != 'test' %>
<input type="hidden" name="turing_a" id="turing_a" value="Please do not alter" />
<%= javascript_include_tag
'turing_a' %>
<% end %>

check_for_bots adds a hidden input field to post and put requests. The "turing_a" javascript file runs when the form loads and returns a value in the hidden input field based on how long the user viewed the form before hitting submit. If the length of time is below a threshold or incorrect, we reject the input and instead show a captcha. After that the standard captcha acceptance routine applies. The javascript we use is adapted from Stephen Hill's Javascript Captcha.

// stop bots that cant run js or are too quick to submit the form
// check hidden value for a reasonable interval: interval in seconds

var turingA = function() {
if (document.getElementById("turing_a")) {
a = document.getElementById("turing_a");
if (isNaN(a.value) == true) {
a.value = 0;
} else {
a.value = parseInt(a.value) + 1;
setTimeout("turingA()", 1000);


Back on the server side we need to validate the hidden input field. If validation fails, we render a captcha.

def email_support
@popular_groups = Group.most_popular
raise MissingFields unless params[:ask_support]
@ask_support = params[:ask_support][:email], params[:ask_support][:message]
raise MightBeBot if !@human
raise MissingFields if || @ask_support.message.blank?
"about page",
params[:ask_support][:message] )
flash[:success] = "Got your request, we'll be in touch"
rescue MissingFields
flash[:error] = "Couldn't process your request, make sure both fields are filled in."
render :action => 'about'
rescue MightBeBot
@show_captcha = true
flash[:error] = "Your browser must be running javascript, and/or you must enter the correct words below"
render :action => 'about'

The check_for_bots method is added to any method we need to protect. It sets @human which the caller can check as part of its input validation.

  # Used as before filter on POST and PUT actions to determine
# whether the requestor is likely to be human. Requestor needs to be able to execute .js
# and wait at least 4 seconds before submitting the form, or this fails and returns false.
# This method doesn't require the user to do anything, other than have a js capable browser
# Note when testing in rails this method will (and should) always fail.
def check_for_bots
if current_user && current_user.status == 'verified' # already showed captcha and user passed
@human = true
# verify_recaptcha is true if in test mode or success, false if fails.
# verify_recaptcha may only be called once per action and get a valid result!
if params[:recaptcha_challenge_field]
@human = verify_recaptcha
if current_user
self.current_user.status = 'verified' # for other actions set it here.
@verified = true # for NewsC#more there is no current user yet, so tell auto_create_user
# to set status
elsif params[:turing_a].to_i >= 3 # 3 second delay
@human = true
@human = false

Auto creating users

The next part of getting rid of the signup form is to automatically create user objects, sessions, and remember-me cookies. This is what our old signup process did. We encapsulate this process into a method and have it run whenever an un-authenticated user tries to perform an action that requires a user object/session. We are using restful-authentication, and pulling out the relevant lines of code from there and from sessions controller yields this handy method:

  # Create user object if doesn't already exist.  Set cookie.   
def auto_create_user!
user = User.auto_create_user_object # doesn't save to db as handle remember cookie does.
user.status = 'verified' if @verified # user passed captcha, prevents showing captcha again
User.current_user = self.current_user = user # !! now logged in (sets session)
handle_remember_cookie! true # sets cookie and saves user so they can get back after session is over
# cookie set to 5 years.., the above uses @current_user set in previous line.

Note that since the user doesn't have the ability to "log in", you'll want to set your remember-me cookie time span to an appropriately long time. This can be changed in vendor/plugins/restful-authentication/lib/authentication/by_cookie_token.rb Once the user object is auto-created, the user can return to their accounts page and add a login and password if they want to use their account on another computer. We auto-generate a random username for display purposes, letting the user customize that later if they want to.

That's it! The old signup form still works, but is no longer necessary to use all the site's functionality. We've been in production for about two weeks now with zero spam infiltration, but time will tell. If anyone has better ways to do any of this stuff, give some comment love below.



Monday, August 17, 2009

China & Brazil Stock Markets - Fading Fad or Growing Opportunity?

This morning I took a look at the relative market performance of the Chinese, Brazilian, and US stock markets over the last 5 years. (link to dynamic chart). Looking at their return over both boom and bust periods and correlating it to fundamentals was instructive.

In the 2001/2 recession these markets were either small or non-existent from an international investment perspective. This last year was the first recessionary period that these markets have weathered as "major" markets. If you invested in China or Brazil in 2008 or 2007 your investment is lying bleeding and tattered on the ground, with 60-80% declines. On the other hand, if you invested early in the cycle - say in early 2005, you're up around 150%.

Looking at fundamentals, the Chinese economy (GDP) grew at 6% Y/Y over the last 1 year recessionary period and closer to 9% Y/Y over the last 5 years. US GDP in contrast grew at around 2.4% annually over the last decade. So its no surprise that the Chinese stock market has dramatically outperformed the US market over the last 5 years. The Brazilian economy grew at around 4.7% over the last 5 years and its stock market similarly out performs the US market over that period.

The data supports the idea that these growth economies are better long-term investment opportunities than the US market. If your investment horizon is say 10 years and you don't like playing momentum, this might be a good time to invest since both China and Brazil are down dramatically from their highs. On the other hand, these markets have not decoupled from the US market - if the US market moves down 10%, these markets will probably move down 30 or 40%. If you believe the US market will take a second dip in the near term, the best time to invest may be 6-12 months in the future.