How to Test Your Disaster Recovery Plan Before You Need It

Every business has a backup plan. Almost none have tested it. When disaster strikes, untested plans fail in ways nobody anticipated. A quarterly recovery drill is the difference between a two-hour inconvenience and a two-week crisis. Here is how to run one.

What a Recovery Drill Actually Tests

A recovery drill answers one question: if the primary server disappeared right now, could you restore every service from backups within an acceptable timeframe? It tests more than the backup files themselves. It tests:

• Backup integrity: Are the backup files complete and uncorrupted?
• Restore procedure: Does the documented process actually work, or are there missing steps?
• Dependencies: Does the restore order matter? If the database must be restored before the application, does the process reflect that?
• Credentials and access: Can you access the backup server, the offsite storage, and the restore environment with the credentials you have?
• Time to recovery: How long does the full restore actually take? Does it meet your recovery time objective?

How to Run a Drill

1. Schedule it. Put it on the calendar. Quarterly. The first Friday of March, June, September, and December. Do not cancel it.
2. Spin up a temporary environment. Use a cheap VPS, an old server, or even a local machine. This environment must be completely separate from production. You are testing whether you can rebuild from nothing.
3. Follow the documented restore procedure exactly. No shortcuts. No we remember an extra step that is not in the documentation. If the documented procedure does not work, the procedure is wrong, not the person running it.
4. Time each step. How long does it take to download the backup? To restore the database? To start the application? To verify it works? This data informs your recovery time objective.
5. Verify the restore. Log into each restored application. Check that data is current. Run a few test transactions. Confirm that file uploads, email sending, and integrations work.
6. Document everything. What worked? What did not? What took longer than expected? Update the recovery plan with findings. If a step was missing, add it. If a dependency order was wrong, fix it.

Common Failures in First Drills

• Missing database dump. The backup script backs up Docker volumes but forgets to run pg_dump or mysqldump. Databases in running containers cannot be safely backed up by copying files. You must use the database's own dump tool.
• Expired credentials. The backup server password was changed three months ago and the backup script was never updated. The most recent backup is three months old.
• Wrong restore order. The application starts before the database, fails to connect, and enters an error state that requires manual intervention.
• Missing configuration. The backup contains application data but not the Docker Compose file, environment variables, or SSL certificates. You can restore the data but cannot start the application.
• Offsite backup is too slow. The offsite copy takes 12 hours to download. Your recovery time objective was four hours. You need faster offsite storage or a local cache.

The Minimum Viable Drill

If quarterly feels like too much, start with a minimum viable drill every six months: pick one critical service (email or CRM), restore it from the most recent backup, verify it works, and record how long it took. One service is better than none. After two cycles, expand to the full stack.

VPS1 runs recovery drills for managed clients as part of our service. You do not have to do this alone.