Bolder Boulder: Electronic timing meltdown

Fifty thousand of my friends and I ran the 29th Bolder Boulder yesterday.  At the starting line and at every mile along the way, there were two blue lines on the ground about a foot wide running across the course.  And there was an incessant beeping (more like a continuous tone since I was in a fairly dense crowd) every time I crossed one of these lines.  This was the "detection" end of the a new system in which we all wore little plastic tags, zip-tied to our shoes in the hopes of getting precise timing for every section of the course.

It is Tuesday morning and the results don’t seem to be available anywhere yet.  In fact, the Bolder-Boulder Web site is only occasionally accessible.  This reminded me of some customer service guidelines that can help relieve the load when something like this happens (as it always does):

1. Transparency–give your users up-to-the-minute information about what is going on. If you have 50,000 people hitting your web site multiple times in a matter of hours, you start to have additional load problems that you didn’t have before the failure.  This slows the site, increased frustration and decreases satisfaction.  Instead, tell people what is going on, preferably with a temporary blog where additional traffic isn’t going to multiply your problems on your main site.  Put prominent links to this timely information on a temporary, lightweight (minimal server load) homepage so your users get to the information they need to make a good web site use decision. Consider releasing the story in a way that a Google search will result in the information being communicated without users hitting your site.
2. Give an estimate date or time for the race results to be available.   Keep the projected date/time updated. Plan on, a huge traffic surge at this time and get the resources together to accommodate it in the meantime.
3. Syndicate.  Publish the content to some trusted partners to take the load off your site. Users want the information; getting it from your site is secondary.  You may have a business model around the traffic, but poor customer satisfaction may make any short term losses do to lost traffic irrelevant.
4. Run parallel systems.  Until you have verified that your system can handle the load, run the old system in parallel.  "If you can’t run it on papers, you can’t automate it" is a good rule for all business process modeling.  In this case, run the old and new system in parallel so you have a fallback.

All of these suggestions boil down to two aspects of responsibly made commitments:

• Clearly set expectations as the what?, when?, where how?
• Avoid denial! during a crisis, commit only to things that your organization can do (You can dream big on your own dime, not while you have customers lined up waiting for you basic services to be deliverred.)
• Competently meet or exceed expectations

Making responsible committments doesn’t stop after the first glitch. In fact, that is just the time when excellent organizations get diligent about it.