How to handle weekend IT outages

Submitted by Patrick Grote on Sat, 06/07/2008 - 2:54pm.

When you manage IT teams, your customer's schedule becomes your schedule. This means if your customer's are working on the weekends you need to provide support, which can be problematic when your team is lean or new on the job. Here are a series of best practices we follow to ensure our customer's always have the support they need when we're not in the building:

Defining priority and service levels: This is the critical step; without it you're not going to succeed. You need to ask your customers what priority they place on certain types of issues and also what service they expect. For instance, a broken mouse or keyboard means someone could move to another workstation. An online system going down is going to require intervention from the local support team.

Extra equipment: This one goes hand in hand with the priority and service levels. Our customers decided that an extra supply of common peripherals such as mice, keyboards and monitors would allow them to repair small issues.

Automated phone tree: We dropped our pagers years ago, but we have cell phones. We worked with our telephony team to set up and automated phone tree for contacting us when our customers need us. The customer simply calls a number, leaves a message and then hangs on. The PBX then begins to call our team in a predetermined order.

System notification: Nagios is an open source system you need to use. Through Nagios we receive automated SMS messages letting us know when there are system issues.

Documented support processes: This is key for our customers, as they can simply look at what they need to do during an issue and work through it. The important thing is to make sure you review and update these processes on a quarterly basis.

VPN access: A no brainer for most support teams, you need a way to get into your company network. The access is needed for testing, researching and notifications.