Database Manager for Microsoft SQL Server (v13.5 LTS) - Runbook

Generate Principal and Mirror Certificates

1. Go to Design > Credentials > New and create credentials for the private key passwords that will be paired with the certificates you are about to create. Although you can use the same password for each certificate, it's more secure if each certificate has its own unique password.
   • SQL_PRINCIPAL_PRIVATE_KEY_PASSWORD
   • SQL_MIRROR_PRIVATE_KEY_PASSWORD
2. Go to the "current" server's Scripts tab.
3. Run the 'DB SQLS Generate and Save a Certificate' script two times because you want to generate unique certificates for both the principal and mirror servers. You can technically use one certificate for both servers, but that will be less secure. Select the appropriate password credential (that you just created) for the PRIVATE_KEY_PASSWORD input each time you run the script. First, create a certificate for the principal database server using the principal credential (SQL_PRINCIPAL_PRIVATE_KEY_PASSWORD), then run the script again to create a certificate for the mirror using the other credential (SQL_MIRROR_PRIVATE_KEY_PASSWORD).
    

DATABASE

4. Go to your deployments Servers tab and RDP into the server.
5. Go to Start > Computer > Local Disk (C:) and open up credential.txt, which should be the credential for the principal server. The second text file (credential1.txt) was created the second time you ran the script, which should be for the mirror server.
   Note: Each time you run the script, it creates a newly-incremented credential. For example, if you ran the script a third time, there would be a credential2.txt file. 
    

6. Copy the two lines of text from the "credential" file.
7. Go to the RightScale Dashboard and paste in the two lines into a new credential (Design > Credential > New) and name it appropriately. (e.g., SQL_PRINCIPAL_CERTIFICATE)

8. Click Save.
9. Repeat the same steps to create a credential for the mirror certificate. (e.g. SQL_MIRROR_CERTIFICATE) from the credential1.txt file

These certificates and credentials are then used to fill in the PRINCIPAL_CERTIFICATE, MIRROR_CERTIFICATE and WITNESS_CERTIFICATE inputs along with their respective *_PRIVATE_KEY_PASSWORD inputs for configuring a mirroring session. More details can be found in the 'Setup a Mirroring Session' section of the Database Manager for Microsoft SQL Server - Tutorial

Create a Microsoft SQL Backup of the Data and Log Volumes

Run the following script to create a backup of data and log volumes. This script uses Microsoft Volume Shadow Copy Service (VSS) to freeze disk activity on data files and then issues the RightScale API backup call to snapshot all the volumes. Once this backup call completes, VSS is disabled and the volume(s) are unfrozen. The SQL server will continue to operate normally during this entire process and does not even know that its volumes were backed up. Additionally, this script can create a differential or full SQL Server backup of all databases and upload the backup to a Remote Object Storage. 
 

Important! VSS does not guarantee consistency of log files.

To perform this operation, follow the steps below:

1. Go to the Scripts tab of a running Microsoft SQL Server.
2. Run the DB SQLS Backup Data and Log volumes Operational Script.
    

DATABASE

REMOTE OBJECT STORAGE

To verify that the script was successfully executed, go to the storage container you specified to view your file. If you used AWS to store your backups, you can go to Clouds > AWS Global > S3 Browser and navigate to your container name. Open up your container and you should see .bak files with the name specified for the DB_LINEAGE_NAME input.

Create an EBS Backup Volume

Use the SYS Create Backup volume operational script to create an EBS volume of a specified size and attach it to the instance under a specified drive letter. This is useful if you need additional disk space for backup operations.

Create a SQL Server User

Use the DB SQLS Create login operational script to create a new SQL Server user (login) and password for your application's use. Enter the following RightScript inputs before running this script.

Note: It's recommended that you set up credentials for password values and any other sensitive data as shown in the examples below.

Graceful failover

A graceful failover will switch one of the standby servers to the primary server in the event that a mirror instance type or storage is increased and roles need to be switched, or if maintenance is required on the principal server. To perform a graceful failover, you'll need synchronized principal and mirror servers that are in a healthy state. However, if there is a problem with the principal server and you are in a disaster recovery scenario, follow the Disaster recovery failover steps instead. 

1. Navigate to your mirror server and click on the Inputs tab.
2. Make sure FORCE_SERVICE_ALLOW_DATA_LOSS is set to "False."
3. Click on the Scripts tab and make sure all inputs required by the DB SQLS Promote to Principal and DNS Register IP scripts are set.
4. Navigate to the principal server and click on the Scripts tab and run the DB SQLS Manual failover operational script.
5. Verify the script successfully executed on the principal and mirror servers. Once completed, the two servers should have switched roles and the DNS (sub)domain should now point to the new principal server.

Disaster recovery failover

In the event that a principal server becomes unreachable, a new mirror server can be set up to prevent data loss.

1. Navigate to the mirror server and go to the Inputs tab.
2. Make sure FORCE_SERVICE_ALLOW_DATA_LOSS is set to "True."
3. Click on the Scripts tab of the mirror server and run the DB SQLS Promote to Principal operational script.
4. Set the SERVER_MODE input to 'Mirror' on the old principal server, then re-launch it, making it a 'new' mirror server.
5. Once the new mirror server becomes operational, verify that the two nodes are synchronizing.
6. Change FORCE_SERVICE_ALLOW_DATA_LOSS input to "False" on the new principal server.

Replace mirror server

In the event that a mirror server becomes unreachable, needs to be moved to a different zone, or the instance type or data/log volumes for the mirror server has changed, the mirror server can be replaced. To replace a mirror server, an initialized mirroring session between principal and mirror server is required. 

1. Navigate to principal server and click on the Scripts tab.
2. Run the DB SQLS Switch mirroring off operational script.
3. (Re)launch the mirror server and set SERVER_MODE to "Mirror."
4. Navigate to the Scripts tab of the principal server and run the DB SQLS Init principal script.
5. Verify the scripts successfully executed on the principal and mirror servers by viewing each server's audit entries.

Verify and repair log files

Microsoft VSS is used for volume snapshot based backups to guarantee consistency of data files. However, Microsoft VSS does not guarantee consistency of log files. To verify consistency and repair log files after performing a database restoration, run the DB SQLS Repair log files operational script, which checks log files for all attached databases. No inputs are required to run the script.

Configure 'tempdb'

Use the following script to configure the 'tempdb' directory on the Microsoft SQL database volume (e.g., D:\ by default) according to Microsoft recommendations (http://msdn.microsoft.com/en-us/library/ms175527.aspx). The script accomplishes the following tasks:

• Sets recovery model to simple
• Discovers CPU core count on system, then creates the same number of data files as core count. All files are the same size by default. Default size for each file is 1 GB, but can be overridden using the OPT_TEMPDB_DATAFILE_SIZE input.
• Sets the growth mode of data files to 10%

To perform this operation, follow the steps below:

1. Go to the Scripts tab of a running Microsoft SQL Server.
2. Run the DB SQLS Configure tempdb Operational Script.

Increase instance type or data/log drives for principal and mirror servers

To gracefully change the instance type used by the principal and mirror servers, the space allocated for data and log volumes, the mirror will have to be relaunched using the new settings and synchronized with the existing principal server. After this, the Mirror will be gracefully promoted into a principal and a new mirror will be launched using the new settings.

1. Follow the Replace mirror server steps and ensure your changes to the new mirror are made prior to step 3 (re-launch the mirror server).
2. Follow the Graceful failover steps.
3. Follow the Replace mirror server steps.

Add a new database (schema)

To add a new database to the principal server and enable synchronization for all non-system databases, follow the steps below.

1. Add new database(schema) to the principal server.
2. Navigate to the Scripts tab of the principal server and execute the DB SQLS Switch mirroring off operational script.
3. Navigate to the Scripts tab of the mirror server and execute the DB SQLS Switch mirroring off operational script.
4. Navigate to the Scripts tab of the principal server and run the DB SQLS Init principal script.
5. Navigate to the Scripts tab of the principal server and run the DB SQLS Init mirror script.
6. Verify the scripts successfully executed on the principal and mirror servers by viewing each server's audit entries.

Restore from the latest backup

If you have previously completed a backup of your Microsoft SQL database, you can restore a standalone or synchronized pair of servers from the latest backup.

1. Navigate to the Inputs tab for the server you would like to restore and configure the OPT_FORCE_CREATE_VOLUMES input. Make sure the input is properly set:
   • If the OPT_FORCE_CREATE_VOLUMES input is using the default value of "False," then the boot script DB SQLS Setup volumes and services will restore from the latest volumes/ROS (Remote Object Storage) backup in the lineage (which is defined by the DB_LINEAGE_NAME input).
   • If the OPT_FORCE_CREATE_VOLUMES is set to "True," then the server will not attempt to restore at boot time, and fresh empty volumes will be created instead. 
2. Once configured, you can go to the Scripts tab and run one of the following scripts to restore from a backup:
   • DB SQLS Restore all databases from Remote Storage
   • DB SQLS Restore database from Local Disk / Remote Storage -This is primarily used for user database imports.
3. Once the restore is complete, the server can stay as a standalone or be initiated as a principal as long as there is a mirror server.

Recreate master key

Previously, if you created a SQL backup snapshot using a ServerTemplate before the v13.5 LTS version, a random password was used to create the master key for the database. If you restored the database from an old snapshot or ROS backup which already contains a master key encrypted with unknown password, you can use this script to delete the existing master key of the master database and create a new one encrypted by the provided password, as specified by the MASTER_KEY_PASSWORD input.

Warning! You should only run this script if you no longer want the existing master key to be used. Services that are currently using the master key to access the database will be disrupted when this script is executed.

As a safety precaution, you must turn off the safety mechanism by overriding the MASTER_KEY_SAFETY input and setting it to 'Text: off'.

Use this script to delete existing master key of master database and create new one encrypted by provided password. This could be useful after restoring from snapshot or ROS backup which already contains a master key encrypted with unknown password. Note that before deleting existing master key all dependent objects (e.g. mirroring endpoints and certificates) are also deleted. The script requires the MASTER_KEY_SAFETY parameter to be set to "off" to avoid accidental deletion of existing master key.

To perform this operation, follow the steps below:

1. Go to the Scripts tab of a running Microsoft SQL Server.
2. Run the DB SQLS Recreate master key Operational Script.

Restore from a specific backup

The steps used to restore a specific backup differs slightly from restoring from the latest backup.

For example, The OPT_DB_RESTORE_TIMESTAMP input will have to reference the exact backup to restore from, but it cannot be used here because it is being used by the DB SQLS Setup volumes and services boot script. Therefore, the value for OPT_DB_RESTORE_TIMESTAMP can be obtained from:

• Tags associated with the volume snapshot:
  • For example:

rs_backup:lineage=mydatabase
rs_backup:timestamp=1347359186

• Backup filenames uploaded to ROS (Remote Object Storage)
  • For example:

                    SQLS_BAK_mydatabase_full_20120726911410.ini

Important!  Use the values in bold for the DB_LINEAGE_NAME and OPT_DB_RESTORE_TIMESTAMP inputs. Do not include the full 'tag' in the input value!

Change the backup retention policy

There are several inputs that can be modified to control how many backups of the Microsoft SQL data volume (D:\ by default) should be retained for archiving purposes.  The backup and cleanup operations are performed by the “DB SQLS backup data volume” script which is automatically scheduled for execution every 4 hours on a running server (by default).  The script can also be executed manually at any time.

1. Go to the Deployment's Inputs tab.
2. Update the following Inputs to control the backup retention policy.  See Archiving Snapshots for more details. Hover over the input tooltip to view the default settings for each input.  If the input is set to '-ignore-' the default settings will be used.
   • DB_BACKUP_KEEP_DAILY
   • DB_BACKUP_KEEP_LAST
   • DB_BACKUP_KEEP_MONTHLY
   • DB_BACKUP_KEEP_WEEKLY
   • DB_BACKUP_KEEP_YEARLY

Graceful termination or shutdown

It's important to terminate the Windows server properly.  Before you can use the standard Terminate action button in the Dashboard to terminate the instance, you will first need to execute an Operational Script that will ensure a graceful shutdown of the Server.  If the script is not run prior to shutting down the server, a backup snapshot of the detached volume will not be taken and the volume will become orphaned (un-deleted).  In order to protect against data loss and orphaned (un-deleted) volumes, follow the steps below:

1. Click on the Inputs tab of the Server that you want to terminate/shutdown.
2. Make sure that the DISABLE_SAFETY input, which prevents the Server from being accidentally disabled, is overridden and set to "off" (e.g., text: off), otherwise you will not be able to run the following script.
3. Click on the Server's Scripts tab. 
4. Run the "DB SQLS DISABLE SERVER - backup, detach and delete volumes" Operational Script.  The only input that is required to run the script is DB_LINEAGE_NAME.  Other inputs can be set to ignore.   If you defined the inputs at the Deployment level, you will not be prompted to specify any missing inputs.  The script performs the following clean-up actions:
   • Stops SQL Server
   • Takes EBS Snapshot of the EBS data volume(s) (D:\ by default).  The Snapshot will receive a tag based on the DB_LINEAGE_NAME input
   • Un-mounts and detaches EBS data volume(s) (D:\ by default) from the Server
   • Deletes the EBS Volume(s)
5. Once the script has been successfully completed, it is now safe to terminate the Server.  Click the Server's Terminate action button.