Attributes and Error Handling

Attributes and Error Handling Overview

Some statements can be adorned with attributes that affects their behavior. Attributes appear after the statement on the same line right before the do keyword for expressions that have one (e.g. define, sub, concurrent, map, foreach). There can be multiple attributes specified on a single statement in which case they are separated with commas. Attributes allow specifying and handling timeouts and handling errors and cancelations.

An attribute has a name and a value. The syntax is:

name: value

The acceptable types for attribute values depend on the attribute: they may be numbers (e.g. wait_task: 1), strings (e.g. on_timeout: skip), arrays (e.g. wait_task: [ "this_task", "that_task" ]), or definition names with arguments (e.g. on_timeout: handle_timeout()).

Some of the attributes define behavior that apply to tasks, while others define behavior for the whole process. Processes and tasks are described in detail in the Cloud Workflow Processes section. For the purpose of understanding the attributes behavior described below, it is enough to know that a single process consists of one or more tasks. A task is a single thread of execution.

Some attributes attach behavior to the expression they adorn and all their sub-expressions. The sub-expressions of an expression are expressions that belong to the block defined by the parent expression. Not all expressions define blocks so not all expressions have sub-expressions. Expressions that may have sub-expressions include define, sub, concurrent and the looping expressions (foreach, concurrent foreach etc.).

The exhaustive list of all attributes supported by the language are listed below in alphabetical order:

The task_name, task_prefix and wait_task attributes are described in Cloud Workflow Processes. This section describes in detail the other attributes dealing with errors and timeouts.

Errors and Error Handling

Defining the steps involved in handling error cases is an integral part of all cloud workflows. This is another area where workflows and traditional programs differ: a workflow needs to describe the steps taken when errors occur the same way it describes the steps taken in the normal flow. Error handlers are thus first class citizen in RCL and are implemented as definitions themselves. Handling an error could mean alerting someone, cleaning up resources, triggering another workflow, etc. RCL makes it possible to do any of these things through the on_error attribute. Only the define, sub and concurrent expressions may be adorned with that attribute.

The associated value is a string that can be any of the following:

• skip: aborts the execution of the statement and any sub-expression then proceeds to the next statement. No cancel handler is called in this case.

• retry: retries the execution of the statement and any sub-expression.

To illustrate the behavior associated with the different values consider the following snippet:

sub on_error: skip do
  raise "uh oh"
end
log_info("I'm here")

The engine generates an exception when the raise expression executes. This exception causes the parent expression on_error attribute to execute. The associated value is skip which means ignore the error and proceed to run the first expression after the block. The engine then proceeds to the next expression after the block (the log_info expression). If the attribute value associated with the on_error handler had been retry instead, then the engine would have proceeded to re-run the block (which in this case would result in an infinite loop).

As mentioned in the introduction, a cloud workflow may need to define additional steps that need to be executed in case of errors. The on_error attribute allows specifying a definition that gets run when an error occurs. The syntax allows for passing arguments to the definition so that the error handler can be provided with contextual information upon invocation. On top of arguments being passed explicitly, the error handler also has access to all the variables and references that were defined in the scope of the expression that raised the error.

The error handler can stipulate how the caller should behave once it completes by assigning one of the string values listed above (skip or retry) to the special $_error_behavior local variable. If the error definition does not define $_error_behavior, then the caller uses the default behavior (raise) after the error definition completes. This default behavior causes the error to be re-raised so that any error handler defined on a parent scope may handle it. If no error handler is defined or all error handlers end-up re-raising then the task terminates and its final status is failed.

The following example shows how to implement a limited number of retries using an error handler:

define handle_retries($attempts) do
 ​ if $attempts <= 3
    $_error_behavior = "retry"
  else
    $_error_behavior = "skip"
  end
end
$attempts = 0
sub on_error: handle_retries($attempts) do
 ​ $attempts = $attempts + 1
  ... # Statements that will get retried 3 times in case of errors
end

Errors can originate from evaluating expressions (e.g. division by 0) or from executing resource actions (e.g. trying to launch an already running server). A variation on the former are errors generated intentionally using the raise keyword. In all these cases the most inner error handler defined using the on_error attribute gets executed.

The raise keyword optionally followed with a message causes an error which can be caught by an error handler. All error handlers have access to a special variable that contains information about the error being raised. $_error is a hash that contains three keys:

• "type": A string that describe the error type. All errors raised using the raise keyword have the type set to user.
• "message": A string that contains information specific to this occurrence of the error. The string contains any message given to the raise keyword for user errors. 
• "origin": A [ line, column ] array pointing at where the error occurred in the RCL source.
   

define handle_error() do
 ​ log_error($_error["type"] + ": " + $_error["message"]) # Will log "user: ouch"
  $_error_behavior = "skip"
end
sub on_error: handle_error() do
 ​ raise "ouch"
end

define handle_terminate_error do
 ​ foreach $error in $_errors do
    @instance = rs.get($error["resource_href"]) # Retrieve the instance that failed to terminate
    if @instance.state != "stopped"          # Make sure it is still running
      log_error("Instance " + @instance.name + " failed to terminate, retrying...")
      sub on_error: skip do
        @instance.terminate() # If so try again to terminate but this time ignore any error
      end
    end
  end
  $_error_behavior = "skip" # Proceed with the next statement in caller
end
sub on_error: handle_terminate_error() do
  @instances.terminate()
end

define retry_launch ​do
 foreach $error in $_errors do
   @server = rs.get($error["resource_href"]) # Retrieve the server that failed to launch
   if @server.state == "stopped" # Make sure it is still stopped
     log_error("Server " + @server.name + " failed to launch, retrying...")
     sub on_error: skip do
       @instance = @server.launch() # If so try again to terminate but this time ignore any error
     end
     @_partial = @_partial + @instance # @instance may be empty in case the launch failed again
   end
  end
  $_error_behavior = "skip" # Proceed with the next statement in caller
end
sub on_error: retry_launch() do
@instances = @servers.launch()
end

define handle_errors do
  log_error("Process failed while handling " + inspect(@servers)) # Note: handler has access to @servers
  $_error_behavior = "skip"
end
@servers = rs.get(href: "/api/servers/123")
sub on_error: handle_errors() do
 ​ @servers.launch()
end

Timeouts

The timeout and on_timeout attributes allow setting time limits on the execution of expressions and specifying the behavior when a time limit is reached respectively:

sub timeout: 30m, on_timeout: handle_launch_timeout() do  
 ​ @server = rs.get(href: "/api/server/1234")
  @instance = @server.launch()
  sleep_until(@instance.state == "operational")
  @server = rs.get(href: "/api/servers/1235")
  @instance = @server.launch()
  sleep_until(@instance.state == "operational")
end

The block in the snippet above must execute in less than 30 minutes otherwise its execution is canceled and the handle_launch_timeout definition is executed. Timeout values can be suffixed with d, h, m or s (respectively day, hour, minute or second).

Note that there does not need to be a on_timeout associated with all timeout attributes. Instead the most inner expression that includes a on_timeout attribute gets triggered when a timeout occurs:

sub on_timeout: outer_handler() do
  ...
  sub timeout: 10m, on_timeout: inner_handler() do
    ...
    @instance = @server.launch()
    sleep_until(@instance.state == "operational")
    ...
  end
  ...
end

In the snippet above, inner_handler gets executed if the sleep_until function takes more than 10 minutes to return.

Similar to the on_error attribute, the on_timeout attribute can be followed by a definition name or one of the behaviors values (skip or retry).

Note: Using the raise behavior in an on_timeout attribute will cause the next on_timeout handler to be executed. Timeouts never cause error handlers to be executed and vice-versa.

On top of specifying the behavior directly in the on_timeout attribute as in:

sub timeout: 10m, on_timeout: skip do
 ​ @instance = @server.launch()
end

It's also possible for a definition handling the timeout to specify what the behavior should be by setting the $_timeout_behavior local variable:

define handle_timeout do
  ​$_timeout_behavior = "retry"
end

Finally, the timeout handler may accept arguments that can be specified with the on_timeout attribute. The values of the references and variables at the point when the timeout occurs are given to the handler:

define handle_timeout($retries) do
 ​ if $retries <  3
    $_timeout_behavior = "retry"
  else 
    $_timeout_behavior = "skip"
  end
end
$retries = 0
sub timeout: 10m, on_timeout: handle_timeout($retries) do
  $retries = $retries + 1
  sleep(10 * 60 + 1) # Force the timeout handler to trigger
end

The snippet above will cause the handle_timeout definition to execute three times. The third times $retries is equal to 3, the timeout handler definition sets $_timeout_handler to skip and the block is canceled.

Labels

The task_label attribute is used to report progress information to clients. It does not affect the execution of the process and is simply a way to report what it is currently doing. The label attribute can be used on sub and call:

define main
  sub task_label: "Initialization" do
    ...
  end
  ​sub task_label: "Launching servers" do
  ​  ...
  end
  ​call setup_app() task_label: "Setting up application"
end

Logging

As shown in the snippet above RCL has built-in support for logging which helps troubleshoot and develop cloud workflows. Each process is associated with a unique log that is automatically created on launch. Logging is done using the log_title(), log_info() and log_error() functions:

• log_title(): To append a section title to the log
• log_info(): To append informational message to the log
• log_error(): To append an error message to the log

Logs for a process can be retrieved using the RightScale API or through the RightScale dashboard by looking at the process audit entries.

Summary

We have seen how a cloud workflow may use attributes to annotate statements and defining additional behaviors. Attributes apply to the statement they adorned and some also apply to its sub-expressions. Definitions can be written to handle errors, timeouts and cancelation. Definitions handling errors that occur during resource action execution have access to all the underlying low level errors and can modify the return value of the action.