Every company will face High availability and Disaster recovery challenge eventually. Disaster recovery has been developed in the mid to late 1970s when managers running their computer systems recognized the dependence of their organizations to the computer systems. At that time, the computers were big mainframes.
Can you imagine a company without an IT system these days? ERP, documents, databases, business intelligence, CRM, etc…. that is a lot of data, which can eradicate a company if it gets lost in an incident or a disaster. For example, a server gets broken, data is lost. Unless you have a backup copy which you can restore. What happens if all the servers get broken? Where is the data kept, how long does a company need to recover, what happens if a building where data is stored gets demolished?
Will a company be able to continue with the operation and if not, how long does a company need to restore the operations?
There are many Disaster recovery vendors and products. Sometimes it may not be needed to invest in a third party solution if your vendor already provides its own solution. When you are hunting for a DR solution, please note that all the vendors claim they have the best solution which in turn is easy to use. Always try before you buy. There are many differences in price, features, and usability.
Disaster Recovery tips that ensure the best possible results
1. Involve management and gain support
When preparing for a disaster recovery scenario, always get a management onboard. Explain why DR is necessary and what resources are required to get the project started. Also, it’s easy to think that once DR is set up, there won’t be any downtime. You need to ask your management what is acceptable downtime and explain cost involved if the required downtime is lowered. For example, if the main location goes down, it may be acceptable that the secondary site goes up after 12 hours. If 12 hours is acceptable for some companies, banks may require instant switchover. It’s all possible, but the cost is much higher for the second scenario.
2. Provide detailed information about the test and determine the outcome
It’s always good to provide detailed information about the test, starting with the nature of the test, goals, objectives, procedures. Also, think about all possible procedures and don’t forget about employee notification. When preparing for the test, you need to know who will be on the team and you need to secure the availability of people and various companies that will be required (security, ERP support vendor, Network IT support, etc…). You can use this time for SLAs testing for the services you pay.
3. Document the outcome
It’s not enough to test a disaster recovery plan. Having a DR document without a proper test is almost like a disaster waiting to happen. The first time you will put a document through the test, you will find problems which you have to document. Before the main test, consider having a cold test, where you test as much as possible internally. A cold test may fix some of the issues with the first draft document. For example, you don’t need to shut down the main site just to test virtual servers in a secondary location. With Nakivo Backup and Replication, you can start a server without network support. That way, you will see that a server is operational and you won’t be responsible for any conflicts in the applications when the replica boots.
4. Consider all the IT elements in the plan
When doing a disaster recovery procedure, consider all IT assets like switches, routers, firewalls with networking, all the hardware that is required, like servers, storage, NAS boxes, all the applications that are needed like ERP, CRM, Sharepoint, all databases that are required for the applications. Don’t forget for archived databases which may still be required. Don’t assume, ask department which services are important for their existence.
5. Test your BC / DC plan thoroughly
When you will test your disaster recovery/business continuity, test it realistically. Restore from backup is not considered disaster recovery/business continuity. Consider that the primary site is down, test and restore in a secondary location. If you considered your plan realistically, think also about the main switch that may go down and check if you have a spare. Review all IT assets.
6. 3-2-1 rule
In order to avoid problems with the backup, you can easily follow the 3-2-1 rule which works for personal and corporate data types. The reason, why 3-2-1 rule gained such popularity, is simple, the 3-2-1 backup should:
- have at least three copies
- be stored on 2 different media
- Have one copy stored in a secondary location
Let’s take a look at the example. Having 3 copies is simple. With Nakivo Backup and Replication we can keep one copy on a backup repository, we can keep one copy on a NAS. We can use replication with WAN acceleration to copy the backup quickly to a secondary location. We can use the same infrastructure in the secondary location or we can use a NAS (WD, Synology, QNAP) if we have a lower budget. We can also use Amazon Cloud or any of the Nakivo Backup cloud partners to store the backup in the cloud.
7. When you are gone
Perhaps you won’t stay with the company forever. You have to write a plan for someone else. Also, consider that someone else won’t have the same technical skills as you do.
8. Have a plan which is always up to date and accessible if a disaster strikes.
What good is a disaster recovery plan if it includes procedures for VMware and HP equipment if you moved two years ago to Hyper-V for hypervisor and Dell for servers and Cisco for network equipment? The plan is not usable, is it? Also, think how would you access the plans if the main location is burned to the ground. That Sharepoint server that was hosting all the DR plans is not accessible anymore.
9. Archive the backup and verify backup integrity
You can have the most recent backup ready in case of a disaster, but keep in mind that even after a year a document may be needed. Think about that for a moment. I don’t think that all virtual servers backup is needed from one year ago. But data from the file server or database from the oracle server may be very useful. Also, it’s easy to set up a backup and forget about it. How do you know that the backup is recoverable? You test the backup. In order to save time, backup integrity checks can be automated.
12. Train your staff on disaster recovery procedures
One of the most important parts of disaster recovery is staff training. If you are woking in a retail, what happens when the ERP system goes down? Are the cash-desks offline? Can the store’s trade? If not, is staff trained to issue manual receipts? What happens if a telephony goes down? Who to call? How will the staff communicate with the warehouse? When a disaster strikes, it is not only about the servers, it’s a company-wide process problem which has to be contained with the written procedures.
Have you heard about Murphy’s law? “Things will go wrong in any given situation if you give them a chance,” or more commonly, “whatever can go wrong, will go wrong.”
Disaster is waiting to happen. It may not happen to your company now, but sooner rather than later it will strike. Somewhere. What we can do is to get prepared. Living in this world and in this time, things got much easier. For example, to restore a virtual server is a much easier task than what it was with physical servers. Also, we can use affordable technology which makes a backup size much smaller with deduplication, and faster across the WAN with WAN acceleration.
In addition, if your company has a secondary location like a store or the warehouse, you can use that location as an off-site backup destination to replicate backup to a Synology based NAS with Nakivo Backup and Replication installed.
If you don’t have a secondary location, you can use Amazon AWS as a secondary location and have your data ready in case of a disaster.