Grow an EBS volume after it has already been created.
An EBS volume that has been determined to be too small. The example below includes a deployment that uses an EBS volume to house its master/slave database setup.
Note: This article describes how to manually increase the size of a single EBS volume used to store your database contents. Use the "EBS Stripe volume grow and restore" or, in the case of a MySQL database, the "DB EBS slave init and grow stripe volume" RightScript, which handle both striped and single EBS volumes, to automate this task. Refer to the instructions at Change the Total Size of the Volume Stripe for more information.
Warning: It is highly recommended that you become familiar with this process and practice growing the size of an EBS volume in a staging environment before attempting to perform this procedure in your production environment.
An EBS volume cannot be resized once it is created.
However, when you create a new EBS volume, you can specify the size of the volume upon creation or when you restore an EBS snapshot. The default size of an EBS volume is equal to the size of your current database plus 5GB.
Since the size of the volume does not resize "on-the-fly," at some point you may need to expand the EBS volume to compensate for a growing database. To grow an EBS volume with almost no site downtime in a production environment you need to perform the operations on the slave instance and then promote it to a master.
Before you begin you should log into your dynamic DNS Provider (e.g. DNS Made Easy) that you're using to manage the DNS records for your Master/Slave database setup and make sure you have a low TTL for your database master's A-record. If you are using RightScale's MySQL database ServerTemplates and following best practices, you should be using a TTL <= 120 seconds. (You might want to use an even lower TTL like 60 seconds.) So when you promote your slave to master, it will only take a few minutes for DNS to refresh. After a couple minutes, nslookup should return the new master's IP address. Note: During this promotion your application servers will not be able to properly reach the new master.
Example: DNS Made Easy
After logging into your account:
Also, if you are using Rails you will need to restart your Mongrels because it cache's the IP addresses of the database (see Step 8).
SSH into the slave and run the following command to stop MySQL and unmount the file system.
df # Check disk/mount paritions before and after the umount service mysqld stop umount /dev/sdk df
Restore your most recent database snapshot and give it a much larger volume size. The nickname will be something like: prefix-slave-20091281500, where prefix is the lineage name of your snapshots that you've already specified in your server input and 20091281500 is the timestamp of when the snapshot was taken.
Once the volume is created, you are ready to attach it to your slave. (Depending on the size of the database, it may take some time to become 100% complete.)
Once the EBS volume is attached you can mount it to your slave and grow the filesystem and verify the new size. Then start mysqld and verify the slave is replicating and the database is intact. Perform the following command from an SSH window on your slave:
mount /dev/sdk /mnt/mysql # mount df # check the size xfs_growfs -d /mnt/mysql # grow the filesystem df # Verify the increase in volume size service mysqld start # Start up database services mysql show databases; # make sure you see your databases show slave status \G # Should show slave status, in sync with the master, waiting for new events
The last entry of the "show slave status \G" mysql command (above) reveals the number of seconds the slave is replicating behind master. (For example: Seconds_Behind_Master: 0) Once the slave is fully caught up synchronizing with master you can promote it to master.
Important! When satisified, remember to rename your "old" slave to your "new" master. (For example: NEW Master DB) This is such a simple task, yet not naming your Servers clearly can be very confusing and produce incorrect user actions with devastating results.
Restart your mongrels and any other web services to make sure they hit the new master. If your application does not require a restart, you can proceed to the next step. (This is a application specific step that not all will need to perform.)
© 2006-2014 RightScale, Inc. All rights reserved.
RightScale is a registered trademark of RightScale, Inc. All other products and services may be trademarks or servicemarks of their respective owners.