Note: If you cannot find a topic, go to docs.rightscale.com where all RightScale documentation will soon be located. Also, feel free to Chat with us!
Home > Partners > Gluster > ST and Runbooks > Gluster ServerTemplate and Runbook

Gluster ServerTemplate and Runbook

The Gluster ServerTemplate allows for the creation of a Gluster Storage Appliance within the RightScale architecture.  The boot and operational scripts automate much of the initial set up and configuration of the cluster.

Table of Contents

Description and Inputs

Boot Scripts

The following Boot scripts are listed in order.

RightScript:  SYS Timezone set v1

Description:  Creates a Gluster volume on top of new EBS volumes.  Created volume spans across all EC2 instances in the domain.

Required Inputs and Default Settings:

  • SYS_TZINFO - choose the timezone from a pull-down menu.
RightScript:  SYS Log Remote Logging Client v7 [rev 6]

Description: Delta set for configuring syslog-ng to do remote logging

Required Inputs and Default Settings:

  • $SYSLOG_SERVER - server to send logs to
RightScript: SYS Monitoring install v9 [rev 12]

Description:  CInstalls and configures monitoring data collection for RightScale.  Uses collectd 4.*.

Required Inputs and Default Settings:

  • MON_PROCESSES - Space separated list of processes to monitor

 

RightScript: MISC ssh priv key install v2 [rev 4]

Description:  Installs SSH key for root.  On first launch, it creates ~/root/.ssh/ is_rsa and installs the private key. 

Required Inputs and Default Settings:

  • None
RightScript: Gluster Install Script

Description:  Converts base system image into Gluster storage server.  Setup  yum, install rpm, configure system settings.

Required Inputs and Default Settings:

  • None
RightScript: Gluster BootStrap

Description:  Prepares a Gluster storage server to become operational with the basic account configuration

Required Inputs and Default Settings:

  • AWS_EC2_CERT  - Drop down menu of EC2 certificates associated with your AWS account
  • AWS_EC2_KEY - Drop down menu of private keys associated with your AWS account
  • GLUSTER_DOMAIN - Alphanumeric word (no spaces) to name your cluster.  Used to tag all EC2 instances and EBS volumes.

 

RightScript: Gluster set support level

Description:  Requires uses to set the purchased support level.  Drop down box allowing "Standard" or "Platinum"

Required Inputs and Default Settings:

RightScript:  Probe Gluster

Description:  Finds an existing Gluster Server belonging to the same domain and 'peer probe' with it to join domain

Required Inputs and Default Settings:

  • None

 

Operational Scripts

RightScript:  Gluster Provision Volume

Description:  Creates a Gluster volume on top of new EBS volumes.  Created volume spans across all EC2 instances in the domain.

Required Inputs and Default Settings:

  • EBS_VOLUME _NAME  - name of Gluster volume to create
  • EBS_VOLUME_SIZE_GB - Size of EBS volume as a number (in GiB) to create on each EC2 instance.  **Currently the drop-down box only allows for a maximum of 1TB.  For any size greater than 1TB, please choose to "override dropdown".  The choice must be a multiple of 8.
  • EBS_VOLUME_TYPE -Drop down menu of two entries ("distribute" or "replicate"). If replicate is selected, configuration will be for 2 copies of all files in the volume. The number of EC2 instances in the domain must be a multiple of 2 for replicate reasons.
RightScript:  EBS Scavenge

Description:  Searches for dangling EBS volumes (due to either terminated instances or explicitly released) and attaches them to the current instance.  This script must be executed on new EC2 instances without any other EBS volumes attached

Required Inputs and Default Settings:

  • NUMBER_OF_VOLS _TO_SCAVENGE- the number of volumes - at most- to scavenge into the system.  Recommended value is 1.

 

RightScript:  Gluster Replace Brick

Description:  Script to update Glsuter volume configuration with new location of a scavenged volume.  This script is to be executed when  recovering / scavenging EBS volumes in a domain which has existing EC2 instances with volume configuration.  When all EC2 instances of the domain are terminated, use the "Gluster Recreate Scavenged Volume v1" script instead.

Required Inputs and Default Settings:

  • EBS_VOLUME _NAME- Name of the Gluster volume to recover.  Note that this will be the name of the subdirectory in the scavenged volume.
  • OLD_EC2_LOCAL_HOSTNAME - Hostname to which the EBS volume was previously attached.  This information can be obtained from the "Audit entry" log of the scavenge operation.

 

RightScript:  Gluster Provision and Replace Brick

Description:  This script will internally provision EBS storage and also replace the disconnected brick with the new provisioned volume.

Required Inputs and Default Settings:

  • 'Previous Volume name' - EBS_VOLUME_NAME - Name of the previous Volume to be replaced
  • 'Previous Volume size (per brick per server)' - EBS_VOLUME_SIZE - Size of the Previous volume to be replaced.

 

 

RightScript:  Gluster Recreate Scavenged Volume v1

Description:  script to recreate Gluster volume with scavenged volumes.  This is similar to "Gluster Volume Provision" except it uses scavenged volumes instead of new volumes.  Note that this script is applicable only when recovering a Gluster volume where all EC2 instances were lost.  New instances for each of the lost instances must be created and scavenged at the rate of 1 EBS volume each.  It is also important to note  that this script is safe only to run on distribute volumes.  To recover replicated volumes please contact technical support.

Required Inputs and Default Settings:

  • EBS_VOLUME_NAME - The  name of the volume to recreate.  The name will be the name of the subdirectory in the recovered volume
  • EBS_VOLUME_SIZE_GB - UNUSED.  Enter any number that is a multiple of 8
  • EBS_VOLUME_TYPE - Drop down menu of distribute or replicate.   Only distribute is supported in this version
RightScript:  Gluster Client Install and (re) mount

Description:  <Script to mount (or automatically remount if already mounted) a Gluster volume, using the native client from the specified server. 

Required Inputs and Default Settings:

  • EBS_VOLUME_NAME -The name of the volume to mount
  • GLUSTER_SERVER - Hostname of the mount server where the volume will be mounted.  This can be any server which is part of the volume.
  • GLUSTER_MOUNT_POINT - Path to the directory to mount volumes on.  The directory will be created if it does not exist
RightScript:  Gluster Volume Self Heal

Description:  Runs a full self heal of the volume.  Please note, the standard self-heal policy of healing files on the first access applies without the script.

Required Inputs and Default Settings:

  • GLUSTER_MOUNT_POINT - Path to the mount point of the volumes where self-heal will be run

 

RightScript:  Gluster Restore Geo -Replication

Description:  This Script helps when there has been a catastrophic failure in one of the whole region and might have lost of data or nodes in question. During the failure, all the traffic goes to slave nodes without Master knowing about the additional changes on files.

After a significant amount of downtime when the Master comes back up, this script helps in populating the new files which went into Slave volume back onto Master and starting geo-replication back again from master to slave.

Assumptions:

  • User knows about the private key for Master host
  • Master knows the hostname for the Master server
  • They each know about Slave volume and Master volume names respectively

 

Required Inputs and Default Settings:

  • MASTER_HOST -> the Host node which was previously the master for the geo-replicated data and was down due to some catastrophic incident

  • MASTER_PEM -> the Private key for accessing the node within the region, usually provided by your Rightscale account

  • MASTER_VOLUME -> GlusterFS volume name where the geo-replication was started before downtime

  • SLAVE_HOST -> the Host node which was on receiving end of the geo-replicated files, but currently was being used as the primary file server

  • SLAVE_VOLUME -> GlusterFS volume name on slave host.

Decommission Scripts

Common Runbook Operations

Preparing a New Deployment and Gluster Cluster

  • A new deployment is part of the RightScale architecture
  • To create a new Gluster deployment, under the "Deployment Tab" choose New
  • Complete the next screen, asking for such items as "nickname" and "Default Availability Zone", then Save
  • When your dashboard appears, you will have the ability to "Add" a new server
  • Choose the "Gluster" Gluster Storage Appliance
  • Nickname your server (make sure to determine a good naming convention
  • Save your work
  • When the dashboard screen appears again, choose to launch the server
  • you will now be asked to add your inputs such as:
    • Gluster_Domain: -this is your Global Name Space
    • MON_Process
    • PRIVATE_SSH_KEY: unless otherwise noted, choose the default for your region
  • Choose to "Save and Launch"
  • Once completed, you will see your server in a "pending" , then "Booting" status, and finally "Operational"
  • Once the first server is "Operational" the next server can be added to the cluster

Preparing a New Cluster

  • A cluster is defined by a domain name and is required input
  • Servers automatically associate themselves to this domain when they are started
  • The first server in a domain must be given time to create the domain before associating others (this is until the server goes "operational"
  • Once the first server is deployed subsequent servers can be added as needed

 

How to Duplicate an Environment within the Same RightScale account

  • Create a new deployment in your RightScale account

  • Create a new server with a new domain name using the Gluster Storage Appliance server template

  • Make sure the domain name is different if the new deployment is in the same EC2 region. Boot the first server and give it time to go to the 'operational' state

  • Create remaining number of servers as the first deployment, with the new domain name

  • Gather list of existing volumes with 'gluster volume info all' on the old deployment

  • Create volumes with same name and size in the new deployment

  • Copy over volume data from mount point to mount point using rsync for each volume

 

Restore Volume - 1 server crashes

  • Start a new RightScale server with the Gluster Storage Appliance template in the same deployment, with the same domain name as the server which crashed
  • Run 'Gluster EBS Scavenge' operational script with 'NUMBER_OF_VOLS_TO_SCAVENGE' set to 1
  • View the output of the script getting executed in 'Audit Log' tab of the server
  • On successful scavenge, the log contains the gluster replace-brick command example
  • Using the old server name from the previous step output, and the name of the volume, run the 'Gluster Replace Brick' script by passing them as the input

 

Restore Volume- multiple servers crash

  • Create a new server with the same domain as the crashed server set and give it time to go to the 'operational' state
  • Create remaining servers to match the total number of servers in the domain as the set of servers which crashed
  • Run 'Gluter EBS Scavenge' script on all servers, each scavenging 1 volume
  • If the volume type was either 1) simple distributed or 2) simple replicated (i.e not distributed replicated) run 'Gluster Recreate Scavenged Volume v1' with the same volume name and type (size does not matter)

 

NOTE: In the current version of the server template, if either you had more than one volume in the domain or had a distributed replicated type, the restoration process is not supported via operational scripts and must be done from the command line manually. It is highly recommended to have the restoration done by involving Gluster support/engineering.

 

Geo-Replication - Failing over to the SLAVE volume should the MASTER fail

(Please Note: Under the current 3.2 release, this is a ONE WAY process from MASTER -> SLAVE and cannot be undone or reverted.  Please do not test in a production environment.)

In the unlikely event that one region becomes unstable, and you are unable to access the MASTER VOLUME, the following steps will move storage traffic to the SLAVE VOLUME.

  • Unmount the volume on the clients
  • On the MASTER volume - STOP geo-replication
    • # gluster volume geo-replication <volume name> <slave URL used when geo-rep was started> stop
  • Stop the Gluster volume on the MASTER
    • # gluster volume stop <volume name>
  • Outside of the Gluster instances, point DNS to the backup location.  Set the TTL to a minimum to promote the change

 

In the backup region:

  • Bring up the clients and mount the volume from the secondary system
  • There is no further action necessary on the SLAVE system
You must to post a comment.
Last modified
23:41, 16 May 2013

Tags

This page has no custom tags.

Classifications

This page has no classifications.

Announcements

None


© 2006-2014 RightScale, Inc. All rights reserved.
RightScale is a registered trademark of RightScale, Inc. All other products and services may be trademarks or servicemarks of their respective owners.