Why does the Zeus Traffic Manager connect and disconnect script fail?

Background

When using the Zeus/Riverbed Traffic Manager 7.x templates found here:

http://www.rightscale.com/library/server_templates/All?s%5Badvanced%5D=&s%5Bfilter_value%5D=zeus&x=0&y=0&s%5Bfilter_type%5D=title_and_description&s%5Bcategory%5D=All&s%5Bcloud_vendor%5D=&s%5Bcloud%5D=&s%5Border%5D=date_desc

You may notice that when running our "Zeus Disconnect v1" or "Zeus Connect v1" Rightscripts they will fail. There could be a couple of reasons behind this, and this article aims to explain those reasons as well as provide best practices straight from Riverbed (formerly Zeus) when using these load balancer templates within your environment.

The scripts may fail and may exhibit an audit entry that contains text similar to the following:

>>>>>>>Detaching application servers from host 1.2.3.4 <<<<<<<<<<<<<<
Adding server...
Pool.removeNodes example-pool-name [ "10.0.1.2:10030" ]
Failed to update Traffic Manager configuration
Error performing action
WARNING, only 1 out of 2 lb hosts could be disconnected

Answer

Essentially, when running the aforementioned Zues Connect or Disconnect scripts, they will attempt to connect into each of the Zeus traffic managers, however the traffic managers are already clustered and replicated amongst each other.

The primary reason behind this failure is DNS configuration. When dealing with a Zeus Traffic Manager setup, the DNS requirements are slightly different than our HAProxy load balancing setup.

As an example, you may have an HAProxy DNS setup with 2 HAProxy load balancers and multiple application servers in an autoscaling array. In this environment, the DNS may be setup like so:

HAProxy environment setup:

- HAProxy example nicknames: HAProxy-LB-1 and HAProxy-LB-2

- IPs of both load balancers: 1.2.3.4 and 5.6.7.8

- DNS A Record: www.domain.com

- A record resolves to: Both IPs

In this scenario, you have a single A record with both HAProxy Elastic IPs assigned to it. This works for HAProxy because there is no internal replication or clustering  between HAProxy nodes, and it utilizes DNS as a failover mechanism in that when a client connects to LB-1 (1.2.3.4) and it fails, it should proceed to connect to LB-2 (5.6.7.8) depending on the client's configuration.

On the application servers, there is an input named LB_HOSTNAME. This value would be set to 'www.domain.com' to match up with the A record for both of your load balancers.

Zeus Traffic Manager setup: This is a best practice directly from Riverbed/Zeus!!

- Zeus Traffic Manager example nicknames: Zeus-TM1 and Zeus-TM2

- IPs of both load balancers: 1.2.3.4 and 5.6.7.8

- DNS A records: You should have multiple A records.

- One A record should reflect all IPs of every traffic manager in the environment, just like the above HAProxy configuration. This is used for web traffic between the load balancers and app nodes. For example:

- www.domain.com = 1.2.3.4 AND 5.6.7.8

- One A record with the individual IP for each Traffic Manager. For example:

- zeustm1.domain.com = 1.2.3.4 ONLY

- zeustm2.domain.com = 5.6.7.8 ONLY

When defining the LB_HOSTNAME input on the application node, you will want to use an A record hostname for a single Traffic Manager (i.e. zeustm1.domain.com). If you use the DNS A record that is being used for traffic (i.e. www.domain.com) then you will experience the aforementioned connect and disconnect script failures. The reason you want to use a single IP for a single Traffic Manager is because the Traffic Managers are already clustered together. Thus, there is no need to connect or disconnect from each of them, and we only need to connect/disconnect from one of them, and the internal cluster replication will take care of the rest.

Additionally, if using the HAProxy setup (a single DNS record only with all IPs assigned to it), you may run into an issue whereby a single terminated application node gets left behind in the Traffic Manager app pool. This may not be adequate if you do not have a health check setup, as the traffic managers might attempt to send valid traffic to this terminated node, which obviously will not respond. Thus, it is best practice to use the Zeus specific setup if using their templates in your environment.

Further questions or comments?

Call us at (866) 787-2253 or feel free to open a ticket in the dashboard by navigating to Support -> Email in the Top Right Corner of the dashboard.