Using AWS S3 with s3cmd

Our goal is to use a simple command line utility to access our S3 resources from our laptop Linux.

First, you should create a IAM User with permissions just to access your S3 Buckets. There are several options to configure an IAM User but my suggestion is to create an admin user with only access to S3. The process is similar to the Read Only User we have created before for Newvem but this time selecting the Policy Template for "Amazon S3 Full Access". This way we are sure that anything that happens to this user will only affect our S3 Buckets.

In this example I will download all the log files that have been automatically created by AWS Cloudfront during the experiment of exploring the amount of CloudFront Edge Locations that exist today and delete those files afterwards.

- Install s3cmd

$ sudo apt-get install s3cmd

- Configure:

$ s3cmd --configure Enter new values or accept defaults in brackets with Enter. Refer to user manual for detailed description of all options. Access key and Secret key are your identifiers for Amazon S3 Access Key: (Access Key for the admin S3 User we have created before) Secret Key: (Seccret Key for the admin S3 User we have created before) Encryption password is used to protect your files from reading by unauthorized persons while in transfer to S3 Encryption password: (your-password) Path to GPG program [/usr/bin/gpg]: When using secure HTTPS protocol all communication with Amazon S3 servers is protected from 3rd party eavesdropping. This method is slower than plain HTTP and can't be used if you're behind a proxy Use HTTPS protocol [No]: On some networks all internet access must go through a HTTP proxy. Try setting it here if you can't conect to S3 directly HTTP Proxy server name: New settings:   Access Key: ***********************   Secret Key: ***************************   Encryption password: ***********   Path to GPG program: /usr/bin/gpg   Use HTTPS protocol: False   HTTP Proxy server name:   HTTP Proxy server port: 0 Test access with supplied credentials? [Y/n] y Please wait... Success. Your access key and secret key worked fine :-) Now verifying that encryption works... Success. Encryption and decryption worked fine :-) Save settings? [y/N] y Configuration saved to '/home/joan/.s3cfg'

- Listing Buckets:

$ s3cmd ls

- Listing Bucket contents (folders):

$ s3cmd ls s3://Bucket-Name

- Listing Bucket contents (files):

$ s3cmd ls s3://Bucket-Name/Folder-Name

- Download all folder content:

$ s3cmd get s3://Bucket-Name/Folder-Name/*

- Delete all folder content:

$ s3cmd del s3://Bucket-Name/Folder-Name/*

Note: This command last will delete all files AND the folder.

Note: Keep in mind that any access to AWS S3 becomes a GET, PUT, POST or LIST action and some cost may occur. Refer to Amazon Web Services S3 Pricing for details.