Posts Tagged: Projects

s3delmany.sh – Delete Many S3 Objects With One Command

Update: The Amazon S3 service API now allows for deleting multiple objects with one request (up to 1,000 objects per request). Please see the Amazon S3 Developer Guide for more information.

I’ve been doing some stuff at work using Amazon S3 to store files and during my testing I uploaded a ton of files that didn’t need to be there. Unfortunately, the command line tool I’m using, s3cmd, does not allow me to delete multiple files at once. There is no way to do a wild-card delete. This means I would need to get the full path to each object and delete them one by one:

./s3cmd del s3://s3.ekarma.net/img/1205794432gosD.jpg Object s3://s3.ekarma.net/img/1205794432gosD.jpg deleted ./s3cmd del s3://s3.ekarma.net/img/1205794432g34fjd.jpg Object s3://s3.ekarma.net/img/1205794432g34fjd.jpg deleted

Yea, there’s no way I’m doing that for over 200 objects. I mean come on, there are tools to automate this kind of stuff! So I created s3delmany.sh:

#!/bin/sh
# -------------------------
# s3delmany.sh
# Author: Raam Dev
#
# Accepts a list of S3 objects, strips everything
# except the column containing the objects,
# and runs the delete command on each object.
# -------------------------

# Redirect output to the screen
2&gt;&amp;1

# If not using s3cmd, change this to the delete command
DELCMD=&quot;./s3cmd del&quot;

# If not using s3cmd, change $4 to match the column number
# that contains the full URL to the file.
# This basically strips the rest of the junk out so
# we end up with a list of S3 objects.
DLIST=`awk 'BEGIN { print &quot;&quot; } { print $4, &quot;\t&quot;} END { print &quot;&quot;}'`

# Now that we have a list of objects,
# we can delete each one by running the delete command.
for i in &quot;$DLIST&quot;; do $DELCMD $i
done

Download
s3delmany.zip

Installation
1. Extract s3delmany.zip (you can put it wherever, but I put it in the same directory as s3cmd).
2. Edit it with a text editor and make sure DELCMD is set correctly. If you’re not using s3cmd, change it to match the delete object command for that tool.
3. Make it executable: chmod 755 s3delmany.sh

Usage
If you’re using s3cmd and you placed s3delmany.sh in the /s3cmd/ directory, you should be able to use the script without modifying it. The script works by taking a list of objects and running the delete command on each one.

To pass s3delmany.sh a list of objects, you can run a command like this:

./s3cmd ls s3://s3.ekarma.net/img/ | ./s3delmany.sh

This will delete all objects under /img/. Make sure you know the output of your s3cmd ls command before you pass it to s3delmany.sh! There is no prompt asking if you’re sure you want to delete the list, so get it right the first time!

Hint: s3cmd doesn’t allow you do use wild-cards, but when you run the ls command, you can specify the beginning of an object name and it will only return objects starting with that. For example, s3cmd ls s3://s3.ekarma.net/img/DSC_, will return only those objects that begin with DSC_.

Alternate Usage
If you have a text file containing a list of S3 objects that you want to delete, you can simply change print $4 to print $1 and then do something like this:

cat list.txt | ./s3delmany.sh

By the way, print $4 simply tells s3delmany.sh that the S3 objects are in the 4th column of the data passed to it. The ./s3cmd ls command outputs a list and the object names are in the 4th column. The awk command expects the columns to be separated by tabs (\t).

If you have any questions or comments, please don’t hesitate to use the comment form below!

My own version of the $1 Image Stabilizer

I’ve seen the $14 Steadycam “The Poor Mans Steadicam” in the past and always thought it would be a cool project to tackle whenever I have some free time (aka never). Even though I own more tools than I’m able to name (when you take care of your own rental properties you need a wide array of tools!), the large number of tools and parts required for the “Poor Mans Steadicam” always turned me off.

Then I saw this cool $1 Image Stabilizer video from MetaCafe:

$1 Image Stabilizer For Any Camera – Lose The Tripod – The best free videos are right here

Sure enough, I found an immediate need for an image stabilizer when I took pictures of the box for my new MacBook Pro power supply. I had some rope in my closet and I remembered the tripod adapter which screws into the bottom of my camera already had a little hole that I could loop the rope through. It took two minutes of my time to tie the rope around the tripod adapter. I then stood on the excess rope and pulled up to steady the camera:

It worked great! Here is the first picture I took without the stabilizer and here is the second picture taken with the stabilizer. You can clearly see the difference!

My co-worker, Raf, spontaneously came up with the idea of using a giant rubber band instead of string or rope. Just as he was laughing it off as a silly idea, I told him I actually have giant rubber bands that I use for fitness. As I was writing this post, I decided to try out his idea:

It also worked well, however the because the rubber band needs to loop around the camera, it was uncomfortable and also difficult to access the camera controls. Besides, not only would I not want to carry around a giant rubber band, I also wouldn’t want the camera to slip out of my hands and go crashing to the ground just as I lifted it to take a picture!

I love the way pictures come out when taken with natural light (no flash). However, when the camera detects low light it slows the shutter speed down so it can absorb enough light to make a clear picture. Even the slightest movement during this slow shutter speed will cause the picture to come out blurry. It’s quite amazing how such a simple image stabilizer fixes that problem!

Attic Studio Renovation

Back in 2003 I spent about $30,000 turning the attic of my 100 year old house into a beautiful studio apartment. I found these pictures while browsing my computer today.

Most of the framing and insulation was done by a contractor. I helped my uncle sheetrock the entire place. My dad and I did the painting, built the two closets, installed custom kitchen counters, and did some of the plumbing. I installed the lights, the hardwood floor, and the shower.

I lived in this studio for a few years until I moved into the partially renovated basement (where I happened to install the exact same shower). This attic studio is now rented for $750 a month.

My First (Real) Electronics Project / Invention

When I was 13 or 14 years old I discovered the need for something to easily change the connections made between VCR’s, tape decks, and speakers. I needed to change inputs and move cables around to different ports. Constantly moving around equipment to access the ports on the back was becoming very cumbersome. I needed an AV Selector. Although one probably existed at the time, I didn’t know of its existence and so I marched on with inventing one.

I had never soldered before, and this was the first time I found a real purpose to solder. My solution was simple: Connect each of the cables coming from the equipment into a box that was easily accessible. For each port on the equipment, the box would have two ports: one that connects from the equipment to the box and then another on the box that connects to another one of these boxes, which is connected to yet more equipment. This would allow me to be able to unplug and change the connections easily by simply manging the connections on the boxes.

To help you understand how this actually worked, I whipped up a quick diagram:

So I bought some parts from the local RadioShack, including a little plastic project box which needed to have openings cut out for the ports (very difficult without the right tools — I think I used a razor and broke a few blades). A few hours later, and my invention was complete. It certainly wasn’t pretty, but it worked exactly as I planned!

Of course there are much more professional AV Switchers out there now, such as this one made by Sony:

ImgListGenerator

Introduction

I’ve been doing a lot of eBay auctions lately and one of the most time consuming parts was creating the HTML for all the images in my auction description. I could reuse a lot the HTML, simply changing the directory and image names, but it was still a lot of repetitive work. This week I had 25 items to list and the repetitive work really got to me, so I stopped and spent 30 minutes putting together a script that would help me.

Usage

To simplify things, I decided not to support the ability to choose the directory with the images for which you want to generate HTML. Instead, you simply upload the index.php file to the directory that contains your images, visit that directory with your web browser, and the HTML is generated. Since your web browser reads HTML, the images will be displayed just as they would in your eBay auction. Simply right click on the page, choose View Source, copy the nicely formatted HTML and paste it into your eBay description.

You can view this script in action by browsing some of my images here.

Details

Here’s a sample output from this script:

<P align="center"><IMG src="http://www.ekarma.net/demo/pics/sample/DSC_0001.jpg"></P>
<P align="center"><IMG src="http://www.ekarma.net/demo/pics/sample/DSC_0003.jpg"></P>
<P align="center"><IMG src="http://www.ekarma.net/demo/pics/sample/DSC_0004.JPG"></P>
<P align="center"><IMG src="http://www.ekarma.net/demo/pics/sample/DSC_0010.JPG"></P>

The code I had to write for this script was rather simple. The real meat of the work is done by a very nice function called preg_find() by Paul Gregg. His code is too much to show here, but I’ll show you the code I wrote for this little script:

// Find all .jpg or .JPG files in the current directory using preg_find()
$files = preg_find('/.jpg|.JPG/', '.', PREG_FIND_SORTBASENAME);

// Store the path to the current directory
// (PHP_SELF includes index.php, so we use substr to remove that)
$link_dir = substr($_SERVER['PHP_SELF'], 0, -9);

// Loop through each of the files and generate the HTML
foreach($files as $file){
	$my_file = substr($file, 2, strlen($file));
	echo "<P align=\"center\"><IMG src=\"http://". $_SERVER['SERVER_NAME'] . $link_dir . $my_file ."\"></P>\n";
}

That’s it! Of course it would be much nicer if you could upload this script to the root directory and either enter or choose a path with images, then click generate. However, this script does exactly what I need, so I don’t plan to make any changes to it.

Download

This script can be downloaded here: index.php.zip (4KB)