Note: Please go to docs.rightscale.com to access the current RightScale documentation set. Also, feel free to Chat with us!
Home > Clouds > AWS > AWS FAQs > What do I do if my server fails to boot and reports issues with meta-data?

What do I do if my server fails to boot and reports issues with meta-data?

Table of Contents
  1. Background
  2. Answer

Background

AWS has recently introduced a few changes to their meta-data system which has inadvertently caused instances using older RightImages to fail at boot time. The following guide contains answers for commonly seen problems that have arisen since this change.

Answer

v 4.x Linux RightImages

During boot of a v4.x Linux RightImage, a script called fetchec2metadata.rb is executed which fetches meta-data from the EC2 meta-data server and parses for the values.  This recent change brought a new metric called EC2_METRICS_VHOSTMD. This key contains data that causes our older meta-data parser to fail resulting in the instances failure to boot properly.  The errors in the log files and in audit entries commonly resemble:
/var/spool/ec2/meta-data.rb:4:in `require': /var/spool/ec2/meta- data-cache.rb:16: syntax error (SyntaxError)             <metric context='host' type='string' unit=''>
We have patched all RightScale v4.x RightImages that are in use, however if you have re-bundled based on a previous image, you will need to update fetchec2metadata.rb with a patch and either restart the service or re-bundle the image and re-launch. We advise to migrate to v5.x RightImages as a base if at all possible. If you must remain on the previous image, the patch for fetchec2metadata.rb is here:
36c36
<         f.puts("ENV['#{key}'] = '#{val}'")
---
>         f.puts("ENV['#{key}'] = '#{val.gsub("'","\\\\'")}'")
55c55
<         f.puts("export #{key}='#{val}'")
---
>         f.puts("export #{key}=\"#{val.gsub(/"/,'\\\"')}\"")
After this has been applied, you should be able to restart the service on the instance using '/etc/init.d/rightscale start' or you can re-bundle the image and re-launch the server with the new image.
 

v 5.x Linux RightImages

Our current RightImages should be able to parse the meta-data properly, however we have noticed occasionally that the EC2 meta-server will fail to supply all the meta-data after multiple retries by our scripts.  The error will resemble this in the audits and logs:

RightScale: /opt/rightscale/right_link/agents/lib/instance/ec2_metadata_provider.rb:143:in `curl_get': uninitialized constant RightScale::Ec2MetadataProvider::SubprocessFormatting (NameError)
RightScale:  from /opt/rightscale/sandbox/lib/ruby/1.8/tmpdir.rb:129:in `mktmpdir'
RightScale:  from /opt/rightscale/right_link/agents/lib/instance/ec2_metadata_provider.rb:110:in `curl_get'
RightScale:  from /opt/rightscale/right_link/agents/lib/instance/ec2_metadata_provider.rb:91:in `recursive_fetch_metadata'
RightScale:  from /opt/rightscale/right_link/agents/lib/instance/ec2_metadata_provider.rb:74:in `each'
RightScale:  from /opt/rightscale/right_link/agents/lib/instance/ec2_metadata_provider.rb:74:in `recursive_fetch_metadata'
RightScale:  from /opt/rightscale/right_link/agents/lib/instance/ec2_metadata_provider.rb:89:in `recursive_fetch_metadata'
RightScale:  from /opt/rightscale/right_link/agents/lib/instance/ec2_metadata_provider.rb:74:in `each'
RightScale:  from /opt/rightscale/right_link/agents/lib/instance/ec2_metadata_provider.rb:74:in `recursive_fetch_metadata'
RightScale:  from /opt/rightscale/right_link/agents/lib/instance/ec2_metadata_provider.rb:53:in `metadata'
RightScale:  from /opt/rightscale/bin/ec2/fetch_metadata.rb:45:in `initialize'
RightScale:  from /opt/rightscale/bin/ec2/fetch_metadata.rb:120:in `new'
RightScale:  from /opt/rightscale/bin/ec2/fetch_metadata.rb:120
RightScale: uninitialized constant RightScale::Ec2MetadataProvider::SubprocessFormatting
RightScale: /opt/rightscale/bin/ec2/wait_for_eip.rb:44:in `require': no such file to load -- /var/spool/ec2/meta-data.rb (LoadError)
RightScale:  from /opt/rightscale/bin/ec2/wait_for_eip.rb:44

You may also see something similar to this as well:

Fetching EC2 Meta-Data
	curl: (22) The requested URL returned error: 404
	curl: (22) The requested URL returned error: 404
	curl: (22) The requested URL returned error: 404
	curl: (22) The requested URL returned error: 404
	curl: (22) The requested URL returned error: 404
	curl: (22) The requested URL returned error: 404
	curl: (22) The requested URL returned error: 404
	curl: (22) The requested URL returned error: 404
	curl: (22) The requested URL returned error: 404
	fetching Ec2 meta-data
	curl exited with status (22, retrying..
	curl exited with status (22, retrying..
	curl exited with status (22, retrying..
	curl exited with status (22, retrying..
	curl exited with status (22, retrying..
	curl exited with status (22, retrying..
	curl exited with status (22, retrying..
	curl exited with status (22, retrying..
	curl exited with status (22, retrying..
	Fetching EC2 User-Data

Usually a reboot or re-launch will suffice to fix this error. If this error keeps occurring, please notify RightScale support for further investigation.

There is also another unrelated issue that you might confuse with this, as the error is similar but the cause is different.  If you were to use the 11H1 RightScale App or Load Balancer ServerTemplate with a RightImage of a newer Epoch, you might see something along the lines of this:

22:23:23: : no such file to load -- /var/spool/ec2/user-data.rb (LoadError)
	from /opt/rightscale/lb/bin/apache_config_vhost.rb:19
22:23:23: Script exit status: 1

In the switch from RightLink 5.6 to 5.7 the location of the cached meta-data was moved from /var/spool/ec2 to /var/spool/cloud.  This change was required as we hardened support for Multi-cloud usage.  If you see this error, you will simply need to roll back to a compatible 11H1 RightImage or create a RightScript that makes a symlink on boot.  Make sure to add it to the top of the ServerTemplate so that it is run before depedent scripts. 

ln -s /var/spool/ec2 /var/spool/cloud

v 5.x Windows RightImages

All of our current publicly available Windows RightImages should be able to boot without issues, however you might need to update your ServerTemplate to use a current MultiCloud Image.  If you're still seeing errors such as:

Source        : RightScale Service
EntryType     : Error
TimeGenerated : 11/30/2011 7:11:31 PM
Message       : Error waiting for cloud data: An item with the same key has already been added.
                   at System.ThrowHelper.ThrowArgumentException(ExceptionResource resource)
                   at System.Collections.Generic.Dictionary`2.Insert(TKey key, TValue value, Boolean add)
                   at System.Collections.Generic.Dictionary`2.Add(TKey key, TValue value)
                   at RightScaleCommon.FileUtilities.ReadDictionary(String filePath, String pairDelimiter, String pairSeparator, Boolean caseSensitive, Boolean throwOnFailure)
                   at RightScaleCommon.Cloud.BaseCloud.WaitForScriptedData(String scriptPath, String dictionaryFilePath, WaitHandle cancelEvent)
                   at RightScaleCommon.Cloud.BaseCloud.WaitForCloudData(WaitHandle cancelEvent, IDictionary`2& userDataDictionary)
                   at RightScaleCommon.Services.ServiceMixin.WaitForCloudData(WaitHandle cancelEvent, IDictionary`2& userDataDictionary)

or

### Content of [C:\ProgramData\RightScale\RightScaleService\log\fetch_userdata_rb_20110731015139.log] is:
curl: (22) The requested URL returned error: 404
!!!!! User data empty

then you are probably using an older image or bundled RightScale RightImage.

Patch RightLink on a stuck instance

Open the following file in NotePad (the file path depends on i386 or x64 platform):
 
i386: "C:\Program Files\RightScale\SandBox\right_link\agents\lib\instance\ec2_metadata_provider.rb"
x64:  "C:\Program Files (x86)\RightScale\SandBox\right_link\agents\lib\instance\ec2_metadata_provider.rb"
 
Hit Ctrl+G and goto line 52 or search for '/latest/meta-data/' and you should see the following Ruby code:
 
# Fetches EC2 metadata for the current instance.
          #
          # === Returns
          # metadata(Hash):: tree of metadata
          def metadata
line 52>>>  url = RightScale::CloudInfo.metadata_server_url + '/latest/meta-data/'
          return recursive_fetch_metadata(url)
        end
 
Change this line to say '2011-01-01' instead of 'latest' meta-data, as follows:
 
url = RightScale::CloudInfo.metadata_server_url + '/2011-01-01/meta-data/'
 
Reboot the instance or issue the following commands at the command prompt:
 
net stop rightscale
net stop ec2config
net start ec2config
net start rightscale
 
Instance should now proceed as normal and boot.  Keep in mind this will need to be applied each time the instance is booted.  To permanently fix the issue you'll need to upgrade RightLink or move to the latest RightImage.
 

Upgrade RightLink

1) Launch a server or instance with the image that needs to be upgraded and wait for the initial password. (10 - 15 mins)
2) RDP into the server and go to Control Panel > Add/Remove Programs.
3) Uninstall RightScale RightLink
4) Download the latest Windows RightLink installer from the RightScale mirror and install RightLink.  Make sure to select the installer with the appropriate arch.  At the time of this writing, we suggest 5.17.14, however you might want to check the latest Windows installer first.
5) After the installation is finished you will want to complete the following guide to Bundle, Register, and Tag the image.

Tags recommended by the template: stage:draf

You must to post a comment.
Last modified
21:29, 16 May 2013

Tags

ec2

Classifications

This page has no classifications.

Announcements

None

Powered by MindTouch ®

© 2006-2014 RightScale, Inc. All rights reserved.
RightScale is a registered trademark of RightScale, Inc. All other products and services may be trademarks or servicemarks of their respective owners.