Adding Httperf Load Tester

Import the ServerTemplate

• Navigate to: Design -> MultiCloud Marketplace -> ServerTemplates
• Search for and import the "Load Tester - httperf v2" ServerTemplate to your local collection

Add a load testing Server and launch it

• Browse to your deployment
• Pick the correct cloud and select the Add Server action button. Fill out the form appropriately, including the following fields:
  • ServerTemplate - Imported, Load Tester - httperf
  • Nickname - LoadTester
  • Select your SSH Key and Security Group
  • EIP - none
  • Associate IP at launch - Select (check) this option
  • When ready, select the Add action button
• Before launching your server, set the following Input at the deployment level
  • LB_HOSTNAME - Set this to the fully qualified domain name of your front end servers. For example, www.rightscaletraining.com
• Launch your server

Configure the Server Array

You need to make sure your server array is enabled, and configured properly. Specific settings for the array vary based on application, and what you want to accomplish with your load tests. If you are simply learning the mechanics of server arrays and our alert/monitoring subsystem, you may want to limit the number of servers you can scale to, and how quickly the scaling process works. In our example settings below we change the maximum count of servers from the default 20 to just 5. The amount of time it takes to scale is artificially shortened by changing the calm time.

• Select your server array
• Select the Edit action button. Change parameters as needed. The following changes are examples only!
  • Default max count: 5
  • Resize calm time: 2
  • If your server array is not already enabled, enable it.
    • See the "Status" setting. Select the enable action link if needed.
• Select the Save action button to save your settings

Initiate the load test from the shell

This step requires to SSH into the httperf server and run the httperf command manually. This command will send 100 requests per second to get "/index.php" hosted on server with IP "50.57.80.214", port 80. After 60000 requests (~10 minutes), httperf will stop and display the load statistics.
Hit Control+C at any time to abort the load and return to the shell.

httperf --hog --timeout=5 --client=0/1 --send-buffer=4096 --recv-buffer=16384 --rate=100 --num-conns=60000 \
--num-calls=1 --uri=/index.php --server=50.57.80.214 --port=80 -v

Initiate the load test using an operational script

• Optional: SSH into your front end servers. Get a sense for how busy they are. For example, run "uptime" and "top" to view various system resources before, during and after the load gets applied. You should observe httpd (apache) bubble up to the top, the CPU %id will drop way down (possibly to 0% idle time), free memory will drop as well. This will make the log file viewing as a Post Tutorial Step more informative.
• Navigate to your load test Server -> Scripts tab
• Run the "PERF httperf start" operational RightScript
• There are several missing Inputs you will have to set. Suggested values for a typical load test are included below. More complicated values are considered application specific and you will need to know your application and httperf well in order to stress test in accord with your requirements.
  • PERF_MAX_RATE
    • The "rate" is the number of connections that are created at a time. The min and max rates are used to change the rate each time the test is run by setting it somewhere between the values of min and max. (PERF_MIN_RATE and PERF_MAX_RATE respectively. See the "- -rate" option on the httperf manual page for more information.)
    • Set to 10000
  • PERF_MIN_RATE
    • Set to 5000
  • PERF_PERIOD
    • Value is tied to a sine wave (sinusoidal period). If not clear how this works, its is best to set the value to 1 (minute).
    • Set to 1
  • PERF_SERVER
    • Public DNS of your frontend server(s). This could also be a publicly routable IP address of your frontend. However, if you have set up DNS records for the load balancing tier, it is best to enter the public DNS name.
    • For example, www.yourApplicationFE.com or ec2-50-16-4-153.compute-1.amazonaws.com
      (Note: Do not include http:// when specifying the DNS name.)
  • PERF_TEST_DURATION
    • Total length of time in minutes you want httperf to run
    • Set to 20
    • Note: You can run the "PERF httperf stop" RightScript to shutdown the test prior to this length of time.
  • PERF_TEST_NAME
    • Name of the test you are about to run. Can use a descriptive name. It will be used for naming logs in the /home/httperf/ directory.
    • For example, perf_test
  • PERF_TEST_RUN
    • Time to run each test (in minutes). This value plays a factor in the number of connections that are established.
    • Set to 5
  • PERF_URI
    • URI used in repeated test requests to increase load. Typically the index.php for the default application (such as the e2e Gaming deployment or other deployment you just build out). For a PHP based application, this should be a PHP page able to respond to load test queries. The value is also used in construction of part of the log file name. (See Post Tutorial Steps.)
    • For example, set to /index.php
• Navigate to the Monitoring page for your deployment. Within about 45 seconds your front end server should be under significant load, voting to grow your array, etc.

Additional Documentation and Logging

This step is considered optional, considering time, interest and previous level of understanding of httperf.

• httperf is documented on line here: http://www.hpl.hp.com/research/linux/httperf/docs.php - The more you know about this load tester the more intelligently you can set the PERF_* Inputs, or add additional Inputs, etc.
• A log file for each run should be generated on your Httperf load test server. You should be able to determine the exact file and location based on what you set PERF_TEST_NAME.
  • Directory path: /home/httperf-<Version>/httperf_<PERF_TEST_NAME>_<Number>_<Date>-<Time>
    • Example: /home/httperf-0.9.0/httperf_perf_test_1_20101013-030340
  • Log file name: httperf_<PERF_TEST_NAME>_<PERF_URI>_<Date>-<Time>_<PERF_MAX_RATE>
    • Example: httperf_perf_test_1_index_20101013-030340_100000.log
  • The log file will display a wealth of information, including system load (uptime numbers), errors, and connection information (rate, time, length). It also conveys each httperf command line with actual parameter values. For example:

httperf --hog --timeout=5 --client=0/1 --server=www.rightscaletraining.com --port=80 --uri=/index.php \
--rate=200 --send-buffer=4096 --recv-buffer=16384 --num-conns=30000000 --num-calls=1
#Note:  Above command is a single line of text.  The \ followed by a <CR>/<LF> is for display purposes only.

Cleanup

Depending on whether you are going to keep your load tester running or not, you may need to perform some cleanup tasks. For example, if you lowered the resize calm time to speed up your testing, you can change it back. Similarly, you may need to change the min or max count, or disable the server array entirely. Remember, cloud charges apply to running servers.

Other Considerations

• Before you validate your array's scaling thresholds, it may help to segment array load testing from array functional testing. This allows you to isolate functional issues, for example, the joining and leaving of new application server instances to and from the load balancing pool, from load-related performance issues.
• Some social networking platforms have usage restrictions in their Terms and Conditions that may prohibit load testing through their platform. In this case, it is common to send load directly to a specific application server instance, for example a php index page, to validate your array's scaling thresholds.
• Your first test should be small. Watch the monitoring graphs on a single application server instance to determine which system resource becomes bound first (CPU,MEM,etc). Once determined, you will need to create an alert escalation based on a specific metric. This alert escalation will drive array grow and shrink voting. Then, you can throttle your testing volume and see the array shrink and grow.
• If you are using AWS' ELB, reported metrics will be limited as compared with what you would expect to see on a specific load balancer instance. In this case, to you can watch the number of operational application server instances to tune your array thresholds.