Objective

To add Splunk functionality to an existing deployment

Prerequisites

This tutorial assumes that you already have at least one deployment.  Currently, Splunk is only supported on Amazon EC2 instances.   

Overview

Splunk is powerful third party software that can index and search massive amounts of IT data efficiently.  RightScale has partnered with Splunk so that you can easily add Splunk monitoring to your RightScale deployment in the cloud.

If you want to use Splunk to monitor more than one deployment or account, you will need to create a Security Group with the appropriate open ports, so that the Splunk "indexer" instance will be able to collect log data from each "forwarding" instance, as well as provide public access to the Splunk Indexer's web port so that you can use an external browser for admin purposes and browsing the aggregated log data.   We recommend creating a separate Splunk deployment that contains a Splunk Indexer.  This server will collect log data from any server in any deployment that has been enabled to act as a Splunk forwarder.  

The following is a high level data flow diagram of the way Splunk works.

Steps

Step 1: Splunk Security Group

The first step is to create a "Splunk" security group.  The security group must allow any Splunk "forwarders" to send traffic to the Splunk "indexer" so that it can aggregate log data from all the "forwarder" instances. 

Go to Clouds -> Security Group.  Create a new security group called "Splunk" and open the following ports.  You must also open additional ports for any other servers/networks that your forwarders will be on.

• 22 - Port for SSH access
• 80 - Port for the Splunk Web Port
• 9997 - Port for the Splunk Indexer

By default, the forwarders use port 9997 to send data to the indexer.  If your Splunk indexer is not in the same deployment as your forwarders and has a separate security group, you must open port 9997 so that it can receive log data.  You must also provide public access to the Splunk Indexer's web port, SPLUNK_WEB_PORT = 80 (default), so that  external browsers can be used for browsing the log data.

Step 2: Splunk Credentials

Next, you will need to create the following Splunk Credentials.  Go to Design -> Credentials and select the New Credential action button.  Create the following credentials:

• SPLUNK_ADMIN_USER - Define a username for Splunk.  (ex: splunkadmin)  You will need to enter this value when you login to the Splunk Indexer.  You can define any username of your choice.
• SPLUNK_ADMIN_PASSWORD - Define a username for Splunk.  (ex: splunkpassword)  You will need to enter this value when you login to the Splunk Indexer.  You can define any username of your choice.
• SPLUNK_LICENSE - (Optional)  If you already have a license for an existing Splunk account, you can add it here.

Step 3: Deployment Setup

We recommend creating a separate Splunk deployment that will only contain your Splunk Indexer. 

Go to Manage -> Deployments.  Create a new deployment called, "Splunk" and select the desired availability zone.  To avoid additional cross-zone data transfer costs, you should select the same availability zone as the servers from which you'll be collecting log data.

Next, add a server to the Splunk deployment using the most recent version of the "SPLUNK Indexer" ServerTemplate.  Notice you will find this Server Template in the "Partners" drop down selection.

Be sure to select the "Splunk" security group. To minimize data transer costs, select the same availability zone as the other servers that will be forwarding their data. 

Click Add.

Step 4: Create and Attach an EBS Volume to the Splunk Indexer

Step 5: Create and Configure Splunk Certificates

You will need to launch the Splunk Indexer two times.  First, you must launch the Splunk Indexer so that you can obtain the values for the the SPLUNK_CERT and SPLUNK_ROOT_CERT credentials.  When the server becomes operational, log in to the server and copy the Splunk CERT files from the splunk directory.

Open an SSH console to your Splunk indexer.

Step 6: Configure Servers to become Splunk Forwarders

There are two ways to create Splunk forwarders.

• Add Splunk Forwarder RightScripts to an existing ServerTemplate (SPLUNK Forwarder Install and SPUNK (re)start forwarder)
• Clone the Splunk Forwarder ServerTemplate and add your own RightScripts

The Splunk Forwarder ServerTemplate under the Partners tab is provided as an example of how to properly configure a Splunk "forwarder" instance.  This template does not provide any real functionality other than to demonstrate how to set up a Splunk forwarder as it shows the required scripts for any Splunk Forwarder.  If you simply want to test Splunk functionality, you can launch this template as-is or clone it and add your own RightScripts to it.

The preferred method of creating a Splunk Forwarder is to add the Splunk RightScripts to an existing ServerTemplate.

To modify an existing server so that it acts as a Splunk Forwarder, simply add the SPLUNK Forwarder Install RightScript as a boot script to the ServerTemplate.  (You may need to clone the ServerTemplate to add the RightScript.)  The next time that the server boots, it will start sending its log data to the Splunk Indexer instance.  As a best practice, if your servers are already operational, we recommend terminating and relaunching a server instead of executing the script on a running server.

You should also add the SPLUNK (re)start forwarder RightScript as an operational script, which will allow you to restart Splunk on a running server via the Dashboard (if necessary).

Once you've added the "forwarder" scripts above, you will need to define the following inputs so that each forwarder instance will know the fully qualified domain name (DNS Hostname) of the Splunk Indexer.  We recommend setting this input at the deployment level of your forwarder instances.  You will also need to add the Splunk certifications to all deployments with forwarders.

Step 7: Login to Splunk and Test Forwarder/Indexer Communications

Now that you you've added a Splunk Indexer instance to your deployment and configured the other instances in the deployment to forward their data to the indexer, you can login to the indexer instance and start using the Splunk interface to analyze the data in your deployment.  To login, you must connect to the public DNS of the Splunk Indexer instance.  Use the username and password values that you defined earlier when you created the Splunk Credentials.  For example, the Splunk Indexer deployment's input values for SPLUNK_ADMIN_USER and SPLUNK_ADMIN_PASSWORD. 

The following list summarizes the steps to quickly test that your Splunk Forwarders are communicating with your Splunk Indexer:

• Navigate to the public DNS of your Splunk Indexer
• Login to your Splunk Indexer
• Select the Main page in the Dashboard drop-down menu
• You should see all configured Splunk forwarders along with specific indexed data in the 3-column display