Amazon

There are 6 posts tagged Amazon (this is page 1 of 1).

Converting Amazon EC2 On Demand Instance to Reserved Instance

If you've created an Amazon EC2 On Demand instance and you later realize you'll be running the server indefinitely, you can convert the instance to a Reserved Instance to save some money.

However, there really isn't anything to "convert"; your existing On Demand instance stays running and you simply purchase a Reserved Instance inside the same Zone (this part is important, see below). It's entirely a billing thing and Amazon handles this seamlessly.

[...] the conversion of an on-demand instance to a reserved instance is purely a billing issue. Simply buy a reserved instance, and the instances will be billed as reserved instance. No changes need to be made to your AutoScaling group.

Please note that reserved instances currently need to be purchased for a specific AZ. In order to get the benefit of reserved instance pricing, your AutoScaling group should be configured to launch instances in those AZs.

For example, if you only buy a reserved instance in us-east-1a, you should setup your AutoScaling group to launch instances in us-east-1a. If your AutoScaling group is configured to launch instances in us-east-1a and us-east-1b, and AutoScaling launches an instance in us-east-1b, it will be billed as an on-demand instance unless you also buy a reserved instance in us-east-1b. (source)

Using CURL to Upload Files via POST to Amazon S3

A few months ago I wrote a post about creating Amazon S3 HMAC Signatures without PEAR or PHP5. One of the things I was using that PHP script for was to feed the necessary information to a bash script hosted on a remote machine. The bash script was to upload a file via POST to Amazon S3 using the information provided.

Since CURL was already installed on the remote machine, I wanted to use that to do the actual uploading. I found very little help on the net regarding how to do this with CURL so here you go:

eris:~ raam$ curl
-F "key=screenshots/current_screenshot.jpg"
-F "acl=public-read"
-F "AWSAccessKeyId=2EO6H8MX1X8YWEA0V432"
-F "Policy=eyAiZXhwaXshdGlvbpI6ICIyMDA4LTErLTAxVDtyOjAwOjAwLjAsMFoiLAogICJjb25kaXRpb25zPjogWwoJeyJidWNrZXQiOiAiczNwaG90b3MubW9hcHAubmV0IiB9LAogICAgWyJzdGFydHMtd2l0aCIsICIka2V5IiwgIkxpdmVTaG90cy8iXSwKICAgIHsiYWNsIjogInB1YmxpYy1yZWFkIiB9LAoJWyJlcSIsICIkQ29udGVudC1UeXBlIiwgImltYWdlL2pwZWciXSwKICBdCn0K"
-F "Signature=20uh08kU75ADHL49NyhYRgZW8BY="
-F "Content-Type=image/jpeg"
-F "file=@current_screenshot.jpg"
http://screenshots.ekarma.net

Keep in mind this assumes the current_screenshot.jpg file is in your current directory.

Amazon S3 HMAC Signatures without PEAR or PHP5

The Amazon S3 proposal for uploading via POST describes how to assemble a policy document that can be used to create a time-sensitive signature. The obvious advantage to this method is that you don't have to worry about someone stealing your secret AWS key or uploading random files without your permission.

Here is the example policy document from the proposal:

{ "expiration": "2007-12-01T12:00:00.000Z",
  "conditions": [
    {"acl": "public-read" },
    {"bucket": "johnsmith" },
    ["starts-with", "$key", "user/eric/"],
    ["content-length-range", 2048, 20971520]
  ]
}

This Policy document is Base64 encoded and the Signature is the HMAC of the Base64 encoding.

The application I am developing at work requires this signed policy method of uploading files to S3, however I needed to do it with PHP4 and preferably without any extra PEAR packages. This posed somewhat of a challenge, as all the tutorials I found on the web explained how to sign the policy using the PEAR Crypt_HMAC package or some feature of PHP5.

I eventually figured it out, and I'm here to show you how. The two functions used were found on the web (I don't remember exactly where) and worked perfectly for my situation.

(Note: I had a lot of trouble saving the contents of the following code in WordPress due to some Apache mod_security settings configured on my server.)

/*
 * Calculate HMAC-SHA1 according to RFC2104
 * See http://www.faqs.org/rfcs/rfc2104.html
 */
function hmacsha1($key,$data) {
    $blocksize=64;
    $hashfunc='sha1';
    if (strlen($key)>$blocksize)
        $key=pack('H*', $hashfunc($key));
    $key=str_pad($key,$blocksize,chr(0x00));
    $ipad=str_repeat(chr(0x36),$blocksize);
    $opad=str_repeat(chr(0x5c),$blocksize);
    $hmac = pack(
                'H*',$hashfunc(
                    ($key^$opad).pack(
                        'H*',$hashfunc(
                            ($key^$ipad).$data
                        )
                    )
                )
            );
    return bin2hex($hmac);
}

/*
 * Used to encode a field for Amazon Auth
 * (taken from the Amazon S3 PHP example library)
 */
function hex2b64($str)
{
    $raw = '';
    for ($i=0; $i < strlen($str); $i+=2)
    {
        $raw .= chr(hexdec(substr($str, $i, 2)));
    }
    return base64_encode($raw);
}

/* Create the Amazon S3 Policy that needs to be signed */
$policy = '{ "expiration": "2007-12-01T12:00:00.000Z",
  "conditions": [
    {"acl": "public-read" },
    {"bucket": "johnsmith" },
    ["starts-with", "$key", "user/eric/"],
    ["content-length-range", 2048, 20971520]
  ]';

/*
 * Base64 encode the Policy Document and then
 * create HMAC SHA-1 signature of the base64 encoded policy
 * using the secret key. Finally, encode it for Amazon Authentication.
 */
$base64_policy = base64_encode($policy);
$signature = hex2b64(hmacsha1($secretkey, $base64_policy));

That's it! This method doesn't require PHP5 and doesn't require any additional PEAR packages.

s3delmany.sh – Delete Many S3 Objects With One Command

Update: The Amazon S3 service API now allows for deleting multiple objects with one request (up to 1,000 objects per request). Please see the Amazon S3 Developer Guide for more information.

I've been doing some stuff at work using Amazon S3 to store files and during my testing I uploaded a ton of files that didn't need to be there. Unfortunately, the command line tool I'm using, s3cmd, does not allow me to delete multiple files at once. There is no way to do a wild-card delete. This means I would need to get the full path to each object and delete them one by one:

./s3cmd del s3://s3.ekarma.net/img/1205794432gosD.jpg
Object s3://s3.ekarma.net/img/1205794432gosD.jpg deleted
./s3cmd del s3://s3.ekarma.net/img/1205794432g34fjd.jpg
Object s3://s3.ekarma.net/img/1205794432g34fjd.jpg deleted

Yea, there's no way I'm doing that for over 200 objects. I mean come on, there are tools to automate this kind of stuff! So I created s3delmany.sh:

#!/bin/sh
# -------------------------
# s3delmany.sh
# Author: Raam Dev
#
# Accepts a list of S3 objects, strips everything
# except the column containing the objects,
# and runs the delete command on each object.
# -------------------------

# Redirect output to the screen
2&gt;&amp;1

# If not using s3cmd, change this to the delete command
DELCMD=&quot;./s3cmd del&quot;

# If not using s3cmd, change $4 to match the column number
# that contains the full URL to the file.
# This basically strips the rest of the junk out so
# we end up with a list of S3 objects.
DLIST=`awk 'BEGIN { print &quot;&quot; } { print $4, &quot;t&quot;} END { print &quot;&quot;}'`

# Now that we have a list of objects,
# we can delete each one by running the delete command.
for i in &quot;$DLIST&quot;; do $DELCMD $i
done

Download
s3delmany.zip

Installation
1. Extract s3delmany.zip (you can put it wherever, but I put it in the same directory as s3cmd).
2. Edit it with a text editor and make sure DELCMD is set correctly. If you're not using s3cmd, change it to match the delete object command for that tool.
3. Make it executable: chmod 755 s3delmany.sh

Usage
If you're using s3cmd and you placed s3delmany.sh in the /s3cmd/ directory, you should be able to use the script without modifying it. The script works by taking a list of objects and running the delete command on each one.

To pass s3delmany.sh a list of objects, you can run a command like this:

./s3cmd ls s3://s3.ekarma.net/img/ | ./s3delmany.sh

This will delete all objects under /img/. Make sure you know the output of your s3cmd ls command before you pass it to s3delmany.sh! There is no prompt asking if you're sure you want to delete the list, so get it right the first time!

Hint: s3cmd doesn't allow you do use wild-cards, but when you run the ls command, you can specify the beginning of an object name and it will only return objects starting with that. For example, s3cmd ls s3://s3.ekarma.net/img/DSC_, will return only those objects that begin with DSC_.

Alternate Usage
If you have a text file containing a list of S3 objects that you want to delete, you can simply change print $4 to print $1 and then do something like this:

cat list.txt | ./s3delmany.sh

By the way, print $4 simply tells s3delmany.sh that the S3 objects are in the 4th column of the data passed to it. The ./s3cmd ls command outputs a list and the object names are in the 4th column. The awk command expects the columns to be separated by tabs (t).

If you have any questions or comments, please don't hesitate to use the comment form below!