Burst Pipe Cripples Dallas County: Would Your Company Survive?

Burst Pipe Cripples Dallas County: Would Your Company Survive?

Posted by Andy at *

Monday night, an 8″ water main burst in downtown Dallas, flooding the Dallas County Records Building. The basement of the Records building just happens to be where the county’s critical servers are housed – now under 6 feet of water. On Tuesday, county operations were reduced to old fashioned pencil and paper – including the court system and the jails. Needless to say, things have ground to a halt without an effective disaster recovery plan in place. One attorney, frustrated by the mess in the courts summed it up the best:  “We’re at a standstill. Things are happening, they’re just happening at a snail’s pace.”

But, this is government – these things are almost expected, right? There is already plenty of finger pointing going on, and the budget excuses are flying. No surprise. But, before we get too carried away in blaming Dallas County for their ineptitude, let’s look in the mirror. How would my business fare faced with a similar crisis?

It’s rarely a burst pipe.

Let’s call it really bad luck that a water main burst outside of the building and caused this catastrophe. That will probably never happen again in 100 years. Hopefully another tornado won’t hit downtown Ft. Worth again either. It may seem like a lot of fuss to implement disaster recovery plans  just in case one of these oddball events strike your business. But, it’s not these big events that keep me up at night – it’s the ordinary stuff. Stuff like spilled Cokes, power surges, accidental kicking, tripping over cables, and accidental bumping by the janitorial staff. We get to visit a lot of small businesses, and we’ve seen a lot of “server rooms”, or lack thereof. An astounding number of critical business systems are placed on top of filing cabinets, spare desks, or on the floor in plain sign in the office. It’s surprising how often the “server” is pulling double-duty as the receptionists workstation. In the event of a break-in, a thief will quickly grab the most important looking stuff under the assumption it is worth more. Worse still, imagine a disgruntled employee with easy access to your critical systems.

Check those tapes.

Dallas County is confident that no data has been lost due to their off-site backups. While I applaud the implementation of their off-site backup system, it will not surprise me at all to start hearing about faulty backups in the news next. This is especially true if they are still backing up to tape. Tape backups are notoriously fraught with complication during most restore attempts – if the tapes are still in good enough physical condition. We visit a lot of small business offices that have been dutifully changing tapes and taking them home at night only to find out they are blank when they really need them. Regardless of the method, an effective backup needs to be regularly verified.

Is your backup enough?

A good, verified backup, stored in an off-site location will likely protect you from data loss in almost every situation. But is protection from data loss enough? Off-site backups are good, but often times the data restoration times are measured in days, not hours. This also assumes that good hardware exists to restore to. The real question is how quickly your company can resume full operation. The Dallas County courts and criminal justice systems ground to a snail’s pace without their systems.  Oh well, so an already full court docket gets pushed for a few days – big deal. It’s a bigger deal if your call center is reduced to taking orders on legal pads. It’s a very big deal if your e-commerce website is down. Preventing data loss is important. Preventing downtime is more important.

Get Prepared

As small businesses, it’s tough to justify elaborate disaster recovery plans for somewhat unlikely scenarios. I understand. There’s a reason the receptionist is using the “server” to browse Facebook – it’s all you have. I get it. Here are some easy things you can do to be more prepared:

• Get your critical systems organized and secure. Don’t picture elaborate server rooms with special server racks. Often times this can effectively be done in a spare closet with some shelving from Home Depot. Make sure the door locks and the airflow is sufficient and you’ve got yourself an entrepreneur’s server room.
• Implement an effective tapeless, off-site backup system. Managed off-site backup has become extremely cost-effective and will make sure that your data is safe and sound no matter what happens to your building.
• Choose a backup solution that can use virtualization. Many tapeless backup methods will allow a backup image of your server to be restored as a virtual machine on alternative hardware. While this is not the ideal long-term solution, it will keep you productive while your new servers are being restored.
• Go over your security protocols with your IT professional. Who has access to your critical systems both in the office and remotely? When is the last time passwords were changed?
• Don’t forget your laptops. How much critical information is stored on the business owner’s laptop?

Common sense is still the best weapon in preparing against disasters. These simple tips will help get you thinking in the right direction. Of course, more elaborate and complete disaster recovery solutions are available. If your company can justify the expense, I recommend doing so. Can your business afford not to?

For the full story on the burst pipe, check out the Dallas Morning News.