You may remember the power outage that took down our servers a few weeks ago.

The folks at IPHouse have come to the conclusion that one of the 480V circuit breakers in one of the power distribution units is faulty, and they are going to replace it. Unfortunately, it's the PDU that powers our servers.

I'm sorry to say that we went through another power failure at our co-location facility this morning.  The power apparently went out to our servers at approximately 5 AM, and is currently in the process of being restored.  Some of our services are now up, and the rest are in the process.  At this time we have no idea what happened, and will send out another note if we have more information later.

As I'm sure many of you noticed - the whole Cruiskeen Consulting web infrastructure went down this morning. We're still checking through things, but at the moment we believe just about everything is now functioning.

We colocate our servers at IPHouse in Minneapolis. This morning at about 8:45 power to our rack went out, taking down all of our equipment. We're still waiting to find out what the exact cause of this power failure was.

I'm going to be out of town at DrupalCon 2008 in Boston. This will mean that upper-level support issues may be a little slower to be resolved while I am out of town, though of course we'll make a best effort. I'll be leaving Menomonie on Feb. 29, and will return on March 11.

As some of you probably noticed, we had a few issues today.  They were indirectly related to the server upgrades we're doing, in that we had a hardware failure at the worst possible time, while some of the normally redundant services weren't redundant due to a server replacement.

We've replaced the offending hardware and taken that server out of the configuration temporarily.  Life should be returning to normal once some of the backlog of work on the servers settles down.