Worst Case: Disaster Planning and Recovery

by | Mar 6, 2020 | Article, Blog, Business, Software For Business | 0 comments

Thinking about the Unthinkable

Most humans don’t like to think about worst-case scenarios. It’s stressful and, usually, unproductive to dwell on the bad things that might happen. Yet every business small owner, every CEO, every IT department head, knows that at least some time has to be spent thinking about the unthinkable. 

There are many facets to disaster planning — ensuring employee safety and making sure you have adequate insurance come to mind — but one critical aspect is access to your company’s data. How long can your business operate without knowing customer information, order status, or inventory availability? Chances are you can get by with some ancillary information — printed reports, people’s memories — for a short time, but it’s highly likely that you will reach the limits of these methods before long. 

Putting Together a Plan

Brainstorming

The hardest thing about disaster planning trying to predict the unpredictable. Every aspect of your company’s operations have to be addressed. You might start with tasking everyone in your company with making a list of the data resources they depend on to do their jobs. If they will be asked to use their home internet as part of your planning, you should also make sure everyone has an adequate connection with sufficient bandwidth. 

Prioritizing

It also helps to prioritize these elements — what data is most important? How can it be protected? What’s expendable? Once you know what’s important, you will have a lot more clarity on the procedures necessary to get back up and running after a disaster. Your planning will take your particular technology configuration into account, and the deficiencies in your current setup will become more apparent.

Action

It can take time to act on your plan. It will probably require acquiring new equipment and services, and much configuration before you have something workable. It’s also most likely going to be costly. Keep in mind that the cost of being completely unable to get any work done on a company-wide basis costs a lot more!

Drill!

The last step, once everything is in place, is to run drills. These can be planned — I can’t imagine the need to spring surprise drills on your team. What’s important is that you are proving that your methods work. Follow up with an open discussion among all involved parties. What worked? What didn’t? Were there things that could be streamlined, or weak links that, in a different set of events, could cause the plan to fail? This is where feedback from your team members becomes really critical. everyone is going to have different needs and concerns that must be met in order to stay operational just make sure everyone understands the goals and limitations built into the disaster plan.

Scenarios

As I discussed above, this kind of planning can be difficult. That said, here are the things we focus on with our clients and for our own company as well.

Loss of an on-premises server

Despite the growing interest in cloud-based hosting, as of 2019, 98% of businesses use on-premises servers*. What if yours crashes, and cannot quickly or completely be recovered? What if it’s stolen? Sure, you have backups. How quickly can they be recovered? Do you have a backup server? Some IT departments keep the previous generation server as a backup when they upgrade to a new one. If this is as far as the disaster goes, a backup on-premises server and regular (cloud) backups of the data contained on it are adequate. But what if…

Natural or Man-made Disaster

Different parts of the country have different kinds of events to guard against. Here on the west coast, we have earthquakes, which come completely out of the blue with no warning. Other parts of the US have tornadoes, hurricanes, blizzards, and other extreme weather events. On top of that, there are other, man-made disasters that can happen. Flooding from a burst pipe, fire, a chemical spill… this is where it gets creepy to think about, but the point is this: there is the possibility that something will happen that not only takes down your server, but also makes your building temporarily or even permanently uninhabitable. 

What then? At Alchemy, this is the first scenario we explored. We have two on-premises servers, containing all of the work we are doing for our clients. If something like the above happened, we would be dead in the water. Our answer to this scenario was to set up a secondary server in the cloud. It is not on durning the day, normally, but it spins up at night, long enough to accept a full set of backups, then goes back to sleep. Once we got this all set up, we ran a test on a Friday afternoon. We shut down our on-premises servers and pretended that we were evacuating the building. Everyone went home. Our IT manager logged in and spun up the server, we each told him what projects we needed to work on (using Slack, which is completely cloud-based) and we were able to get back to work in about 40 minutes. We considered this to be a successful drill. 

Loss of Internet Connection

Depending on your setup, the loss of internet connectivity can be very disruptive. If you use cloud-based services like Slack, Dropbox, Quickbooks Online, the problem becomes even more acute. If your server is in the Cloud, the problem has now grown to crisis proportions. If your internet provider can assure you of a relatively short downtime, you may be able to weather the storm. If you have customers or field personnel who log in to your server via the web, it may feel like more of an emergency. Your plan for these things will depend on these details. 
If you need zero downtime due to a loss of connectivity, you will have to spend some money and time planning for it. If you are in a relatively urban area, you may be able to get two different connections coming in to your office, from two competing services (e.g. Spectrum Cable and Frontier Fios). There are network routers that include automatic WAN failover when it senses a loss of connection, and also will handle the outward-facing IP configuration so incoming users are not affected by the inevitable change of IP address that occurs when you change networks. 
At the other end of the spectrum is the lowest-tech methodology: setting up cell phones to act as very low-bandwidth internet connections for the computers in your office. This works, but be aware of a few things: first, it’s probably not a good idea to ask anyone who does not have an unlimited data plan to do this, as it’s going to add up quickly. Second, a cell phone cannot provide anywhere near the bandwidth your users are used to getting, so everyone needs to cooperate on minimizing the amount of traffic. If cell phone tethering proves impractical, you could keep one or more hot-spot devices as a standby. These are supplied by your mobile phone provider, and create a WiFI network everyone can log onto. They, too, use a mobile cellular connection, so they are in theory not faster than tethering to a cell phone, but this is their one job, and there will be no inconvenience with incoming phone calls. 

Other Scenarios

As I stated earlier, disaster planning is only as good as your imagination. There are certainly going to be company-specific scenarios that have to be addressed. Most importantly, never put the needs of company operations ahead of the safety of your team. If your disaster plan involves someone being tasked with grabbing a hard drive on the way out the door, you’re doing it wrong. 

How We Can Help

At Alchemy Consulting Group, we deal in some of the most important data our clients access on a minute-by-minute basis. Needless to say, we have a strong interest in keeping it accessible. We can help you through the process of disaster planning and assist you in implementing it (usually with the help of your IT professional). Contact us for more.

*https://www.spiceworks.com/press/releases/spiceworks-study-shows-on-premises-server-infrastructure-remains-vital-to-the-workplace-despite-growing-iaas-adoption/

*https://www.ciodive.com/news/on-premise-servers-persist-in-98-of-businesses-even-with-cloud-hype/549643/

Submit a Comment

Your email address will not be published. Required fields are marked *