Note: Please go to to access the current RightScale documentation set. Also, feel free to Chat with us!
Home > ServerTemplates > Archive > 11H1 > Runbooks > Database Manager for MySQL Stripe Runbook > MySQL Stripe Problem Scenarios > Both the master and slave are corrupted

Both the master and slave are corrupted

It is possible that the data corruption happens at the database contents level rather than being due to errors at a lower level (i.e, disk, filesystem, etc.).  (If you think that only your Master-DB is corrupted, see the Master-DB is corrupted scenario.)   Data corruption of all your database instances is commonly due to errors or bugs in the application, which might cause things like inserting incorrect records into the database, deleting valuable records, or disconnecting table relationships. Since these changes are valid from an SQL point of view, they will be propagated to the slave instances immediately.  Therefore, both master and slave databases likely have the same data corruption. 

The best way to recover from this situation is to restore the data from the most recent backup of snapshots for your EBS stripe that was taken before the data corruption took place.   In this situation, the recovery is the same as when there are no running DB instances! (master or slave) with the exception that you need to be more careful when choosing which snapshot you use to restore the database, because the snapshot could have been taken after the database was already corrupted.

By default, the ServerTemplate is designed to take periodic snapshots of both the master and slave databases.  Snapshot management includes daily, weekly, monthly, and yearly archives, which allows you to always have a clean set of snapshots from which to recover. 

Required actions: What recipes should I apply to solve the situation?

Overall Strategy: Existing database instances should be discarded and a new master-slave cluster should be started from scratch.  Basically, you need to disable the current MySQL instances and starting from scratch.

  1. Disable or Enable Continuous Backups
  2. Disable existing databases from generating further application errors by turning off MySQL. SSH into your database instances and execute "service mysqld stop" from the command prompt.
  3. Launch a new Master-DB
  4. Restore from a previous DB backup 
  5. Check out the restored database, and iterate backwards as necessary (restoring older snapshots) until the database is no longer corrupted.
  6. If necessary, re-point or restart your application to use the new master (same DNS name, but different IP).
  7. Launch a new Slave-DB
  8. Initialize Slave-DB from Master-DB
  9. Analyze the old master/slave database instances to determine the root cause of the failure/corruption. 
  10. Finally, terminate the corrupted master/slave database instances.


You must to post a comment.
Last modified
21:33, 16 May 2013



This page has no classifications.



© 2006-2014 RightScale, Inc. All rights reserved.
RightScale is a registered trademark of RightScale, Inc. All other products and services may be trademarks or servicemarks of their respective owners.