The implementation of a true cloud-based grid processing solution involves four basic components: a queuing system, a storage system, a computing platform, and a comprehensive framework to interconnect these components and ensure proper messaging and job flow. RightScale Grid, in concert with the Amazon Web Services solutions provided by SQS, S3, and EC2, provides the requisite components for the successful implementation of such a grid processing solution. This section will present a brief overview of each component, and subsequent sections will discuss the interrelations of these components and how they can be combined to implement dynamically elastic grid or batch computing applications.
Simple Queue Service offers a reliable, highly scalable, hosted queue for the storage and retrieval of messages as they are passed from one compute process to another. These messages can be up to 8 KB in size and can remain in a queue for up to four days. Messages are enqueued and dequeued via simple API (Application Programming Interface) calls, and access control mechanisms can be enacted to provide secure storage and retrieval of these messages. Additionally, messages are stored redundantly across AWS datacenters in order to enhance reliability and availability.
AWS’s Simple Storage Service provides a storage mechanism for any type of user or application data. Individual files are limited to 5 GB in size, but there is no hard limit on the total amount of data that a user can store in S3. While there is obviously a practical limit, for all intents and purposes, S3 can be thought of as a limitless storage bucket (for example, some RightScale customers currently store petabytes of data in S3). As with messages in SQS, files in S3 are stored in multiple datacenters within the AWS infrastructure to ensure reliable and efficient access to the data.
The Elastic Compute Cloud service provides virtually unlimited compute capacity in the cloud, providing users with more flexibility in performing computationally-intensive applications. While the elasticity this service provides is of great benefit in implementing scalable websites that can expand and contract capacity based on dynamic traffic patterns created by viral events or other unscheduled circumstances, it is also ideally suited to the grid computing paradigm.
The RightScale Grid framework coordinates the automated workflow of messages and jobs as they move through the computational, storage, and retrieval processes. RightScale Grid also provides the mechanism to implement the elasticity of the grid processing solution. The input queue(s) of the system are continually monitored, and when certain criteria are met, additional worker instances are launched to handle the increased processing load. When the number of items in the input queue decreases, idle servers are automatically terminated, thus taking full advantage of “utility computing” in that the user only pays for the computing resources they use (and more importantly, that they need).
© 2006-2014 RightScale, Inc. All rights reserved.
RightScale is a registered trademark of RightScale, Inc. All other products and services may be trademarks or servicemarks of their respective owners.