There are two main data corruption cases that can occur in the master DB.
If you think that your DB is corrupted you probably have seen something strange that you understand. If it's caused by disk or filesystem corruption, you'll probably see error messages in the instance log...(i.e., syslogrelated or even mysql-server related). If the corruption is due to case 2, you'll probably see that there is data missing, or perhaps application errors complaining about missing relationships, records not found or failed transactions.
It is possible that the data corruption happens at the database contents level rather than being due to errors at a lower level (i.e, disk, filesystem...). This is commonly due to errors or bugs in the application, which might cause things like inserting incorrect records into the DB, deleting valuable ones or disconnecting table relationships. Since these changes are valid from an SQL point of view, they will be propagated to the slave instances immediately, therefore both master and slave databases will most likely be corrupted in exactly the same way.
The way to recover from this situation is to restore the latest backup taken before such corruption took place. The recovery mechanism for this case is somehow similar to the situation where There are no running DB instances (master or slave)! with the exception that we have to be more careful about which backup file to restore from...since restoring from the latest backup might not be a good solution if the backup was taken after the corruption already existed.
Overall Strategy: Existing database instances should be discarded and a new master-slave cluster should
be started. Basically we're disabling the current MySQL instances and starting from scratch.
NOTE: To locate the latest non-corrupted backup file, you must choose from the available binary backup files in your S3 directory. The bucket and path for these are specified by the BACKUPFILE_PREFIX and BACKUP_S3_BUCKET input variables in our MySQL templates (viewable by locating either master or slave MySQL templates, and then clicking on the 'show' link next to input variables). It's important to know when the corruption occurred in order to ensure that you are restoring the most recent non-corrupted backup.
© 2006-2014 RightScale, Inc. All rights reserved.
RightScale is a registered trademark of RightScale, Inc. All other products and services may be trademarks or servicemarks of their respective owners.