How can I manually reattach and mount my Rightscale created database EBS volumes?

Background

From time to time, the need may arise to manually attach a MySQL server's EBS volume(s) to an instance and configure them for EBS backups.

In particular, this can be useful when you do not take regular snapshots of the EBS volumes and you wish to simply attach the volume to a server, since the volume will have the most up to date data available in the database. This article will explain the manual process of locating your Master or Slave MySQL EBS volumes, attaching them to a running instance, and configuring and mounting them so that you can proceed to take a new EBS snapshot or backup and initialize a new slave from that backup.

Answer

First, some important notes/disclaimers:

• This will ONLY work for database EBS volumes that were originally created using Rightscale tools, such as the 'DB create MySQL EBS stripe volume' script. The 11H1 version of this script is found here. If the EBS volumes were not originally created using our tools then STOP NOW, as this FAQ WILL NOT WORK FOR YOU. 
• This article is designed to explain how to manually attach and restore volumes to a MySQL Master server, after which you can take a new EBS backup/snapshot of each of the volumes and proceed with initializing a new slave server from that fresh backup.
• This article is designed around the 11H1 servertemplate release, however other versions will also likely work as the tools will be very similar.

Step 1: Launch a fresh MySQL Master server.

Be sure that there are no automatically attaching volumes from a boot time attachment assignment! We will manually be attaching the volumes after we have located them, so we do NOT wish to automatically attach them!

Be sure that the "INIT_SLAVE_AT_BOOT" input is also set to False, since we do not want to restore snapshots and initialize a slave (a MySQL master should already always have this input set to False, but it's good to double check to be sure).

Step 2: Locate the volumes we want to attach and restore.

We must now search for the latest EBS volume(s) for our MySQL server that have the data that we want on them. To do so, it's usually easiest to filter through all of your EBS volumes by Lineage name.

Your lineage should be easily identifiable, as it was configured when you setup the server originally. It can be found on the server's Inputs, specifically the DB_LINEAGE_NAME input (for 11H1 based servers, the input name may vary on other versions of the template). Here is an example screenshot using the lineage 'GBLineage':

At this point we need to find our Master MySQL EBS volumes that we will manually attach and configure on the server. To do so, let's navigate to:

Clouds -> (Region) -> EBS Volumes

Insert your lineage name into the Filter box at the top, then click 'Apply.' Be sure to also click the 'Age' column after it filters if you wish to sort it by age (to locate the newest volume available, which will likely have the data that you want):

Once you have located the volumes you want, click on the volume link to open the volume info page.

NOTE: If you have a multi volume EBS stripe, then you will want to ensure that you open each volume in the stripe to restore, as we will need to attach all of them in the next few steps.

Step 3: Gather information about the volumes.

When you open up the volume info pages, you will see quite a bit of information about them that we will need when attaching them to the newly launched fresh MySQL master. Here is a screenshot to demonstrate:

Take note of the following:

• Description - This will contain the volume device label, such as '/dev/sdk' or /dev/sdj.' We will need this when we manually attach the volumes to ensure they get attached with the appropriate device names (important for stripes with multiple volumes)
• Availability zone - Volumes can only be attached to servers in the same availability zone. Thus, we need to ensure that our newly launched MySQL master server and the volume's availability zone match. If they DON'T match, you can take a snapshot of the volume, then create a new volume from that snapshot in the appropriate availability zone. Alternatively you can re-launch your MySQL server in the same availability zone as your MySQL volumes.
• Created From - This will contain the Master or Slave designation, indicating if it's a master or slave volume. This article only explains how to attach a master volume, so we want to locate our 'master' designated volumes.

Step 4: Attach our volumes.

Once we have located our MySQL volumes for our master server, we will now want to attach them with the appropriate device names.

To do so, navigate to your server's Info page by finding it under the Manage -> Servers menu (or by searching in the top right corner of the dashboard). Once on your server's info page, click the Volumes tab.

Once there, click Attach Volume. In the next page, be CERTAIN that you are selecting the correct volume nicknames and be CERTAIN that you are attaching them as the correct device name!

Once you've found it, click "OK" to attach it.

Repeat this process if you have multiple volumes with different device names to attach.

Step 5: SSH to the server and issue LVM commands.

Once all of the volume are mounted, they should be automatically detected by LVM, so we will want to open up an SSH terminal to the newly launched MySQL master. Once you have it opened and ready to go, we will want to use these commands:

pvscan

lvchange -ay

vgscan

lvscan

The output of each of these commands should look similar to the following, albeit your volumes will be of different size and you may have more physical volumes (this test example only has a single EBS volume, so it's not a stripe).

pvscan
  PV /dev/sdk   VG vg-ebs-rightscale   lvm2 [4.00 GiB / 2.00 GiB free]
  PV /dev/sdb   VG vg-data             lvm2 [149.11 GiB / 9.64 GiB free]
  Total: 2 [153.10 GiB] / in use: 2 [153.10 GiB] / in no VG: 0 [0   ]

So pvscan reports a single 4GB volume in this case in the volume group 'vg-ebs-rightscale'

Now we want to run the vgscan and lvscan command.

IMPORTANT NOTE: If you run the 'lvscan' and 'vgscan' command and they do not return output similar to what is below, we may need to run the 'lvchange -ay' command first to activate the logical volume. To do so, run this command:

lvchange -ay

This will ensure that the logical volume and volume group can be scanned appropriately. After running that, retry the vgscan and lvscan commands below and it should work for you.

Next, output of vgscan command:

vgscan
  Reading all physical volumes.  This may take a while...
  Found volume group "vg-ebs-rightscale" using metadata type lvm2
  Found volume group "vg-data" using metadata type lvm2

This confirms that we have a volume group named "vg-ebs-rightscale." This is a VERY important group name, as it MUST be named 'vg-ebs-rightscale' because our EBS utilities/scripts are HARD CODED to look for this name!

and finally the lvscan command:

lvscan
  ACTIVE            '/dev/vg-ebs-rightscale/lvol0' [2.00 GiB] inherit
  ACTIVE            '/dev/vg-data/lvol0' [89.46 GiB] inherit
  ACTIVE            '/dev/vg-data/swap' [50.00 GiB] inherit

So, as we can see, the logical volume name that we need to mount is named /dev/vg-ebs-rightscale/lvol0. We will need this when running the mount command next.

Step 6: Mount the logical volume on the server.

If all of this looks correct so far, we are now ready to mount the logical volume. By default, we place the mount point at /mnt/mysql/ so we will want to run this command:

mount /dev/vg-ebs-rightscale/lvol0 /mnt/mysql

This should mount the logical volume to the /mnt/mysql/ directory as needed.

Step 7: Run df and ensure data is available.

Next, we can run the 'df' command from the terminal to ensure that the logical volume is mounted properly. Sample output below:

df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/sda1             9.9G  1.8G  7.6G  19% /
/dev/mapper/vg--ebs--rightscale-lvol0
                      2.0G  144M  1.9G   8% /mnt/mysql

As we can see, the /dev/mapper/vg--ebs--rightscale-lvol0 device is mounted on /mnt/mysql as it should be.

We can now issue a 'cd /mnt/mysql' to change to that directory and you can view the database files from your EBS volume(s).

Step 8: Register DNS for our new master server.

Next, we need to register our DNS A record for the Master MySQL server so that it resolves to the private IP.

To do so, you will want to run the 'DNS master DB register' script. The 11H1 script can be found here

Be sure to fill in the appropriate inputs:

MASTER_DB_DNSID

DNS_USER

DNS_PASSWORD

Step 9: Run a fresh EBS backup on the master server's volume(s).

Now that we have the volumes attached, mounted and DNS A record configured, we should be able to run the DB EBS Backup script. This script should be located on the new master server's Scripts tab. The 11H1 variant of this script is here.

This script should complete successfully. Once it does, you will now have fresh EBS snapshots of each of your EBS volume(s) attached to the master server.

Step 10: Let the snapshots complete fully, then initialize a new slave if desired.

Now that we have fresh snapshots and a working master, the only thing left to do is to initialize a new slave server from these snapshots.

To do so, simply clone/launch a new server, then once it goes operational run the 'DB EBS Slave Init' script from the scripts tab. Be sure that it is using the same lineage as the master server so that it grabs the latest snapshots that we just took.

Alternatively, set the INIT_SLAVE_AT_BOOT input to 'true' and then launch the slave.

If you still have questions or you experience issues with this process

Feel free to call us at (866) 787-2253 option #2, or submit a support ticket from the dashboard using the Support -> Email link from the top right corner of the dashboard.