Both the master and slave are corrupted
Table of Contents
It is possible that the data corruption happens at the database contents level rather than being due to errors at a lower level (i.e, disk, filesystem, etc.). (If you think that only your Master-DB is corrupted, see the Master-DB is corrupted scenario.) Data corruption of all your database instances is commonly due to errors or bugs in the application, which might cause things like inserting incorrect records into the database, deleting valuable records, or disconnecting table relationships. Since these changes are valid from an SQL point of view, they will be propagated to the slave instances immediately. Therefore, both master and slave databases likely have the same data corruption.
The best way to recover from this situation is to restore the data from the most recent backup of snapshots for your EBS stripe that was taken before the data corruption took place. In this situation, the recovery is the same as when there are no running DB instances! (master or slave) with the exception that you need to be more careful when choosing which snapshot you use to restore the database, because the snapshot could have been taken after the database was already corrupted.
By default, the ServerTemplate is designed to take periodic snapshots of both the master and slave databases. Snapshot management includes daily, weekly, monthly, and yearly archives, which allows you to always have a clean set of snapshots from which to recover.
Required actions: What recipes should I apply to solve the situation?
Overall Strategy: Existing database instances should be discarded and a new master-slave cluster should be started from scratch. Basically, you need to disable the current MySQL instances and starting from scratch.
-
Disable or Enable Continuous Backups
- Disable existing databases from generating further application errors by turning off MySQL. SSH into your database instances and execute "service mysqld stop" from the command prompt.
- Launch a new Master-DB
- Restore from a previous DB backup
- Check out the restored database, and iterate backwards as necessary (restoring older snapshots) until the database is no longer corrupted.
- If necessary, re-point or restart your application to use the new master (same DNS name, but different IP).
- Launch a new Slave-DB
- Initialize Slave-DB from Master-DB
- Analyze the old master/slave database instances to determine the root cause of the failure/corruption.
- Finally, terminate the corrupted master/slave database instances.
- © Copyright 2015 RightScale Technical Support