How can I download large attachments from an S3 bucket?

Background

Downloading large attachments from an S3 bucket can be problematic, due to issues of speed and ease of maintenance/access. Here are the ways that you currently have to manage attachments, with their associated pros and cons:

Attach the file to a boot RightScript
    Pros: Revisioning, ease of use in the script
    Cons: Servers in non US regions will take between 40-90 minutes to download the file. Additional data transfer costs for the servers outside of us-east-1.

Have a bucket for each supported region and replicate the file across.
    Pros: Fast download times and no data transfer as the file is located in the same region
    Cons: Users will have to rev the attachments and replicate them. Increased number of inputs and complexity for the RightScript. The script will receive a bucket name, bucket file and the aws creds to download.

Store the attachments in a CloudFront enabled bucket.
    Pros: Fast download times. Only one bucket to manage.
    Cons: Users will have to rev the attachments. Added data transfer cost. As far as I know only public URLs can be used to download the files.

Using one bucket(ex in us-east-1) and downloading the file using an http client(ex: aria2c) that support multiple connections.
    Pros: Fast download times. Public and private URLs can be generated. Only one bucket to manage.
    Cons: Users will have to rev the attachments. Added data transfer cost. Increased number of inputs and complexity for the RightScript. The script will receive a bucket name, bucket file and the aws creds to download.

So what should you do if you decide to host the file in an S3 bucket or as an attachment in a script?

Answer

We recommend using the aria2 downloader for anything that is located in an S3 bucket or is an attachment for a RightScript, that is over 1GB in size. The reason why we recommend this downloader, is that aria2c is able to make concurrent connections to download one file, often giving a download speed that is 2 to 10 times faster than s3cmd, curl, or wget.

Aria2 can be found from the following site, along with simple usage instructions: http://aria2.sourceforge.net/

Below are some simple graph comparisons, showing overall data speed, compared to s3cmd, curl and wget: