Note: Please go to docs.rightscale.com to access the current RightScale documentation set. Also, feel free to Chat with us!
Home > Clouds > AWS > Tutorials > Grow the Size of an EBS Volume

Grow the Size of an EBS Volume

Prerequisites

An EBS volume that has been determined to be too small. The example below includes a deployment that uses an EBS volume to house its master/slave database setup.

Note: This article describes how to manually increase the size of a single EBS volume used to store your database contents. Use the "EBS Stripe volume grow and restore" or, in the case of a MySQL database, the "DB EBS slave init and grow stripe volume" RightScript, which handle both striped and single EBS volumes, to automate this task. Refer to the instructions at Change the Total Size of the Volume Stripe for more information.

Warning: It is highly recommended that you become familiar with this process and practice growing the size of an EBS volume in a staging environment before attempting to perform this procedure in your production environment.

Overview

An EBS volume cannot be resized once it is created.

However, when you create a new EBS volume, you can specify the size of the volume upon creation or when you restore an EBS snapshot. The default size of an EBS volume is equal to the size of your current database plus 5GB. 

Since the size of the volume does not resize "on-the-fly," at some point you may need to expand the EBS volume to compensate for a growing database. To grow an EBS volume with almost no site downtime in a production environment you need to perform the operations on the slave instance and then promote it to a master.

Steps

Lower your DNS TTL

Before you begin you should log into your dynamic DNS Provider (e.g. DNS Made Easy) that you're using to manage the DNS records for your Master/Slave database setup and make sure you have a low TTL for your database master's A-record.  If you are using RightScale's MySQL database ServerTemplates and following best practices, you should be using a TTL <= 120 seconds.  (You might want to use an even lower TTL like 60 seconds.)  So when you promote your slave to master, it will only take a few minutes for DNS to refresh.  After a couple minutes, nslookup should return the new master's IP address.  Note:  During this promotion your application servers will not be able to properly reach the new master. 

Example: DNS Made Easy

After logging into your account:

  • Managed DNS -> Records (for your domain name) -> Select your master DB A Record name
  • Change the TTL to 60  (Note: Make a note of what you change it from.  You will want to change it back later.)
  • Select the Continue action button


Also, if you are using Rails you will need to restart your Mongrels because it cache's the IP addresses of the database (see Step 8).

SSH into the Slave and stop database services

SSH into the slave and run the following command to stop MySQL and unmount the file system.

df                    # Check disk/mount paritions before and after the umount
service mysqld stop  
umount /dev/sdk
df

 

Detach the EBS Volume from the Slave

  • Go to Clouds -> CloudName -> EBS Volumes
  • Locate the Slave's EBS volume.  To confirm you have the correct one, make sure that:
    • The Server column = "Slave DB" (or similar nickname you used for your slave Server)
    • The Device column = /dev/sdk
    • The Status column = "in-use (attached)"
  • Click the correct eraser action icon to detach the EBS volume from the slave

Restore snapshot

Restore your most recent database snapshot and give it a much larger volume size.  The nickname will be something like: prefix-slave-20091281500, where prefix is the lineage name of your snapshots that you've already specified in your server input and 20091281500 is the timestamp of when the snapshot was taken.

  • Navigate to Clouds -> CloudName -> EBS Snapshots
  • Select the play action icon on your most recent snapshot in order to create a new EBS volume
    • Database size - You do not need to change anything except the database size.
    • Description - Although not mandatory, it is often useful to add some tracking information in the Description field.  (For example, Doubled EBS volume size from 20 to 40GB on 1/1/2010.) 
  • Select the Create action button when ready

Attach the new EBS Volume to the Slave

Once the volume is created, you are ready to attach it to your slave.  (Depending on the size of the database, it may take some time to become 100% complete.) 

  • Select the Attach action button and select the same slave instance that the volume was previously attached to.  You will be presented with a multi-pane window.  Select YourDeployment -> YourSlaveServer
  • Make sure the device is set to: /dev/sdk
  • Check the Attach now option  (Do not check the Attach at boot option since we only need to attach it this time only.)
  • Select the Ok action button when ready.  This operation could take a minute or two to complete.  Follow the progress via the Events and/or Audit Entries.  The "status" should progress from "attaching"  ->  "in-use (attached)"

Mount and Grow the Filesystem

Once the EBS volume is attached you can mount it to your slave and grow the filesystem and verify the new size. Then start mysqld and verify the slave is replicating and the database is intact.  Perform the following command from an SSH window on your slave:

mount /dev/sdk /mnt/mysql    # mount 
df                           # check the size
xfs_growfs -d /mnt/mysql     # grow the filesystem
df                           # Verify the increase in volume size
service mysqld start         # Start up database services
mysql
show databases;              # make sure you see your databases
show slave status \G         # Should show slave status, in sync with the master, waiting for new events 

 

Promote Slave to Master

The last entry of the "show slave status \G" mysql command (above) reveals the number of seconds the slave is replicating behind master. (For example:  Seconds_Behind_Master: 0)  Once the slave is fully caught up synchronizing with master you can promote it to master.

  • Go to your slave's Scripts tab
  • Run the "DB EBS Promote to Master" operational script (select its Run action icon)
  • Recall you can monitor the scripts progress via Events and/or Audit Entries.  You are looking for "completed" and "success" Events.
  • Once completed, verify the "old" slave is now your "new" master database Server.  From the "old" slave's SSH window, rerun the following mysql command:
    • show slave status \G   # Should be the empty set now
    • show master status \G # Should reveal your old slave is now the master

Important!  When satisified, remember to rename your "old" slave to your "new" master.  (For example:  NEW Master DB)  This is such a simple task, yet not naming your Servers clearly can be very confusing and produce incorrect user actions with devastating results.

Restart Mongrels and Web Services (optional)

Restart your mongrels and any other web services to make sure they hit the new master.  If your application does not require a restart, you can proceed to the next step.  (This is a application specific step that not all will need to perform.)

Terminate the Old Master and Launch a New Slave

  • Terminate the OLD Master database Server
  • Clone the running master database Server.  Rename it appropriately, it will become your new slave (For example:  Slave DB)
  • Launch the new slave Server
    • Verify that the Input INIT_SLAVE_AT_BOOT = true
  • Verify that both the master and slave have the larger volume size
    • Look at the Volumes tab on your running Master DB Server
    • Look at the Volumes tab on your running Master DB Server

Post Tutorial Steps 

  • Double check that all Servers in your Deployment are named correctly
  • When things have settled, you can change your DNS setting's TTL back to what it was at the start of this Tutorial (for your Master-DB).
You must to post a comment.
Last modified
16:45, 21 May 2013

Tags

Classifications

This page has no classifications.

Announcements

None


© 2006-2014 RightScale, Inc. All rights reserved.
RightScale is a registered trademark of RightScale, Inc. All other products and services may be trademarks or servicemarks of their respective owners.