From time to time, the need may arise to manually attach a MySQL server's EBS volume(s) to an instance and configure them for EBS backups.
In particular, this can be useful when you do not take regular snapshots of the EBS volumes and you wish to simply attach the volume to a server, since the volume will have the most up to date data available in the database. This article will explain the manual process of locating your Master or Slave MySQL EBS volumes, attaching them to a running instance, and configuring and mounting them so that you can proceed to take a new EBS snapshot or backup and initialize a new slave from that backup.
First, some important notes/disclaimers:
Step 1: Launch a fresh MySQL Master server.
Be sure that there are no automatically attaching volumes from a boot time attachment assignment! We will manually be attaching the volumes after we have located them, so we do NOT wish to automatically attach them!
Be sure that the "INIT_SLAVE_AT_BOOT" input is also set to False, since we do not want to restore snapshots and initialize a slave (a MySQL master should already always have this input set to False, but it's good to double check to be sure).
Step 2: Locate the volumes we want to attach and restore.
We must now search for the latest EBS volume(s) for our MySQL server that have the data that we want on them. To do so, it's usually easiest to filter through all of your EBS volumes by Lineage name.
Your lineage should be easily identifiable, as it was configured when you setup the server originally. It can be found on the server's Inputs, specifically the DB_LINEAGE_NAME input (for 11H1 based servers, the input name may vary on other versions of the template). Here is an example screenshot using the lineage 'GBLineage':
At this point we need to find our Master MySQL EBS volumes that we will manually attach and configure on the server. To do so, let's navigate to:
Clouds -> (Region) -> EBS Volumes
Insert your lineage name into the Filter box at the top, then click 'Apply.' Be sure to also click the 'Age' column after it filters if you wish to sort it by age (to locate the newest volume available, which will likely have the data that you want):
Once you have located the volumes you want, click on the volume link to open the volume info page.
NOTE: If you have a multi volume EBS stripe, then you will want to ensure that you open each volume in the stripe to restore, as we will need to attach all of them in the next few steps.
Step 3: Gather information about the volumes.
When you open up the volume info pages, you will see quite a bit of information about them that we will need when attaching them to the newly launched fresh MySQL master. Here is a screenshot to demonstrate:
Take note of the following:
Step 4: Attach our volumes.
Once we have located our MySQL volumes for our master server, we will now want to attach them with the appropriate device names.
To do so, navigate to your server's Info page by finding it under the Manage -> Servers menu (or by searching in the top right corner of the dashboard). Once on your server's info page, click the Volumes tab.
Once there, click Attach Volume. In the next page, be CERTAIN that you are selecting the correct volume nicknames and be CERTAIN that you are attaching them as the correct device name!
Once you've found it, click "OK" to attach it.
Repeat this process if you have multiple volumes with different device names to attach.
Step 5: SSH to the server and issue LVM commands.
Once all of the volume are mounted, they should be automatically detected by LVM, so we will want to open up an SSH terminal to the newly launched MySQL master. Once you have it opened and ready to go, we will want to use these commands:
The output of each of these commands should look similar to the following, albeit your volumes will be of different size and you may have more physical volumes (this test example only has a single EBS volume, so it's not a stripe).
pvscan PV /dev/sdk VG vg-ebs-rightscale lvm2 [4.00 GiB / 2.00 GiB free] PV /dev/sdb VG vg-data lvm2 [149.11 GiB / 9.64 GiB free] Total: 2 [153.10 GiB] / in use: 2 [153.10 GiB] / in no VG: 0 [0 ]
So pvscan reports a single 4GB volume in this case in the volume group 'vg-ebs-rightscale'
Now we want to run the vgscan and lvscan command.
IMPORTANT NOTE: If you run the 'lvscan' and 'vgscan' command and they do not return output similar to what is below, we may need to run the 'lvchange -ay' command first to activate the logical volume. To do so, run this command:
This will ensure that the logical volume and volume group can be scanned appropriately. After running that, retry the vgscan and lvscan commands below and it should work for you.
Next, output of vgscan command:
vgscan Reading all physical volumes. This may take a while... Found volume group "vg-ebs-rightscale" using metadata type lvm2 Found volume group "vg-data" using metadata type lvm2
This confirms that we have a volume group named "vg-ebs-rightscale." This is a VERY important group name, as it MUST be named 'vg-ebs-rightscale' because our EBS utilities/scripts are HARD CODED to look for this name!
and finally the lvscan command:
lvscan ACTIVE '/dev/vg-ebs-rightscale/lvol0' [2.00 GiB] inherit ACTIVE '/dev/vg-data/lvol0' [89.46 GiB] inherit ACTIVE '/dev/vg-data/swap' [50.00 GiB] inherit
So, as we can see, the logical volume name that we need to mount is named /dev/vg-ebs-rightscale/lvol0. We will need this when running the mount command next.
Step 6: Mount the logical volume on the server.
If all of this looks correct so far, we are now ready to mount the logical volume. By default, we place the mount point at /mnt/mysql/ so we will want to run this command:
mount /dev/vg-ebs-rightscale/lvol0 /mnt/mysql
This should mount the logical volume to the /mnt/mysql/ directory as needed.
Step 7: Run df and ensure data is available.
Next, we can run the 'df' command from the terminal to ensure that the logical volume is mounted properly. Sample output below:
df -h Filesystem Size Used Avail Use% Mounted on /dev/sda1 9.9G 1.8G 7.6G 19% / /dev/mapper/vg--ebs--rightscale-lvol0 2.0G 144M 1.9G 8% /mnt/mysql
As we can see, the /dev/mapper/vg--ebs--rightscale-lvol0 device is mounted on /mnt/mysql as it should be.
We can now issue a 'cd /mnt/mysql' to change to that directory and you can view the database files from your EBS volume(s).
Step 8: Register DNS for our new master server.
Next, we need to register our DNS A record for the Master MySQL server so that it resolves to the private IP.
To do so, you will want to run the 'DNS master DB register' script. The 11H1 script can be found here
Be sure to fill in the appropriate inputs:
Step 9: Run a fresh EBS backup on the master server's volume(s).
Now that we have the volumes attached, mounted and DNS A record configured, we should be able to run the DB EBS Backup script. This script should be located on the new master server's Scripts tab. The 11H1 variant of this script is here.
This script should complete successfully. Once it does, you will now have fresh EBS snapshots of each of your EBS volume(s) attached to the master server.
Step 10: Let the snapshots complete fully, then initialize a new slave if desired.
Now that we have fresh snapshots and a working master, the only thing left to do is to initialize a new slave server from these snapshots.
To do so, simply clone/launch a new server, then once it goes operational run the 'DB EBS Slave Init' script from the scripts tab. Be sure that it is using the same lineage as the master server so that it grabs the latest snapshots that we just took.
Alternatively, set the INIT_SLAVE_AT_BOOT input to 'true' and then launch the slave.
If you still have questions or you experience issues with this process
Feel free to call us at (866) 787-2253 option #2, or submit a support ticket from the dashboard using the Support -> Email link from the top right corner of the dashboard.
© 2006-2014 RightScale, Inc. All rights reserved.
RightScale is a registered trademark of RightScale, Inc. All other products and services may be trademarks or servicemarks of their respective owners.