Why are my backups failing when running the db:do primary backup recipe?

Overview

When running the db::do_primary_backup recipe using a v12.11 ServerTemplate, you may experience an error message that contains the following:

The error occurred line 184 of /opt/rightscale/sandbox/lib/ruby/gems/1.8/gems/rest-client-1.6.7/lib/restclient/request.rb in method 'transmit' while executing:

182       raise RestClient::ServerBrokeConnection
183     rescue Timeout::Error
***       raise RestClient::RequestTimeout
185     end
186

*ERROR> Subprocess exited with 1
*RS> failed: db::do_primary_backup

This is due to a timeout issue with the "rightscale_tools" gem.

Resolution

The timeout in the "rightscale_tools" gem is set to 15 seconds. The ServerTemplate uses this gem to manage volumes and backups. Each device takes about 1-2 seconds to complete, so depending on the number of devices, the timeout length may be too short.

To fix the issue, do the following:

• Log into the machine
• Open the file: /opt/rightscale/sandbox/lib/ruby/gems/1.8/gems/rightscale_tools-1.1.23/lib/rightscale_tools/api/10.rb
• Increase the timeout value for @snapshot_timeout:

 @snapshot_timeout = 15
 @platform = RightScale::Tools::Platform.factory options
end

If you are seeing the same issue for v13.4 ServerTemplates and newer, there is a RightScript available to patch this issue. You can import and add this RightScript to the end of the boot scripts section of your editable ServerTemplate which will apply the patch and automatically remove the timeout from the gem and allow the API to handle it.