We’re keeping track of an ongoing power outage at the Wikispaces.com hosting facility at our offsite status page:

http://status.wikispaces.com/

What Happened: This Saturday, April 14, Wikispaces.com was offline from 7:53 PM PDT to 11:23 PM PDT. The cause was a power outage at our hosting facility in Dallas, Texas.

We’re sorry for any inconvenience this outage caused you. We pride ourselves in keeping Wikispaces running quickly and reliably day in, day out. (In March, the site was offline for a total of 3 minutes — 99.994% uptime.) We look at every time the site is slow or down as a chance to improve our operations going forward. Saturday’s outage is no exception.

Our server hosting facility was running on generator power on Saturday due to a series of power outages caused by storms in the Dallas area. During this time, Wikispaces was up and running normally. When utility power came back on Saturday night, a 2500 amp circuit breaker catastrophically failed during the switch away from generator power. (Homes in the US commonly have 20 amp circuit breakers handling lights, appliances, and wall sockets. Imagine that kind of circuit breaker, only 125 times larger - a big piece of industrial electrical equipment.) For around 30 minutes, our servers stayed running on battery power, which finally ran out just before 8:00 PM PDT. Our hosting company immediately put electricians to work replacing this breaker with a spare stored onsite, and started bringing our servers back online around 11:00 PM PDT.

We go to great lengths to make sure all the moving parts of Wikispaces are redundant - network connections, firewalls, web servers, database servers, storage systems, etc. Our hosting company does too, and both their generators and battery systems worked as expected. Unfortunately, Murphy’s Law also applies to industrial circuit breakers, the kind of component you expect to fail outright very, very rarely. Thankfully they had a spare and got us back up and running quickly given the circumstances.