Note: Please go to docs.rightscale.com to access the current RightScale documentation set. Also, feel free to Chat with us!
Home > ServerTemplates > Archive > Pre-11H1 > MySQL-EBS v1 (No Stripe) > Manager for MySQL-EBS Runbook > MySQL-EBS Problem Scenarios > Both the master and slave are corrupted

Both the master and slave are corrupted

It is possible that the data corruption happens at the database contents level rather than being due to errors at a lower level (i.e, disk, filesystem, etc.).  (If you think that only your Master-DB is corrupted, see the Master-DB is corrupted scenario.)   Data corruption of all your database instances is commonly due to errors or bugs in the application, which might cause things like inserting incorrect records into the database, deleting valuable records, or disconnecting table relationships. Since these changes are valid from an SQL point of view, they will be propagated to the slave instances immediately.  Therefore, both master and slave databases likely have the same data corruption. 

The best way to recover from this situation is to restore the data from the most recent snapshot (backup) that was taken before the data corruption took place.   In this situation, the recovery is the same as when there are no running DB instances! (master or slave) with the exception that you need to be more careful when choosing which snapshot you use to restore the database, because the snapshot could have been taken of the database after it was already corrupted.

As a best practice, we recommend periodically archiving EBS Snapshots (Backups) so that you'll always have a clean snapshot from which to recover.

Required actions: What recipes should I apply to solve the situation?

Overall Strategy: Existing database instances should be discarded and a new master-slave cluster should be started from scratch.  Basically, you need to disable the current MySQL instances and starting from scratch.

  1. Disable or Enable Continuous Backups
  2. Archiving EBS Snapshots (Backups) 
  3. Disable existing databases from generating further application errors by turning off MySQL. SSH into your database instances and execute "service mysqld stop" from the command prompt.
  4. Launch a new Master-DB
  5. Restore a previous DB backup  (You will need to use the DB_RESTORE_PREFIX_OVERRIDE input variable to specify an uncorrupted snapshot.)
  6. Check out the restored database, and iterate (restoring older snapshot) until database is not corrupted.
  7. If necessary, re-point or restart your application to use the new master (same DNS name, but different IP).
  8. Launch a new Slave-DB
  9. Initialize Slave-DB from Master-DB
  10. Analyze the old master/slave database instances to determine the root cause of the failure/corruption. 
  11. Finally, terminate the corrupted master/slave database instances.

 

You must to post a comment.
Last modified
21:34, 16 May 2013

Tags

This page has no custom tags.

Classifications

This page has no classifications.

Announcements

None


© 2006-2014 RightScale, Inc. All rights reserved.
RightScale is a registered trademark of RightScale, Inc. All other products and services may be trademarks or servicemarks of their respective owners.