I’ve been working day and night for the past few days on an already-late project for work. The fact that the project requires a computing language that was very unfamiliar to me just a few short weeks ago certainly adds to the brain drain. But, it’s a language I’ve been wishing I had a reason to learn for years now so I’m definitely not complaining!
Posts Tagged: Bash
Fixing Boot-Time Dialog Display Issues
Dialog is a really useful utility for creating professional looking dialog boxes and menus within a shell script. I’m working on a boot-time script that allows the user to make system-level changes before the system has fully booted.
When testing my script from the command line, the dialog menu looked fine. However, whenever I set the script to start during boot (update-rc.d myscript.sh defaults, on Debian-based systems) here is what the menu looked like:
UGH! It was barely usable. At first, I thought this would be an endlessly difficult problem to solve given my limited in-depth knowledge of Linux (I’m getting there!), but then I realized the main difference between the script running during boot and the script running after I had logged in was that my environment variables had not been loaded.
From the command line, I ran the env command to display all my current environment variables:
debian_vm:~# env TERM=xterm-color SHELL=/bin/bash SSH_CLIENT=172.16.168.1 61315 22 SSH_TTY=/dev/pts/0 USER=root MAIL=/var/mail/root PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/bin/X11 PWD=/root LANG=en_US.UTF-8 PS1=\h:\w\$ SHLVL=1 HOME=/root LANGUAGE=en_US:en_GB:en LOGNAME=root SSH_CONNECTION=172.16.168.1 61315 172.16.168.132 22 _=/usr/bin/env
The three variables that caught my eye were TERM, SHELL, and LANG. After a little trial and error, I discovered setting the LANG variable fixed the display issues with dialog! I added the following near the top of my script:
export LANG=en_US.UTF-8
Now when my script loads during boot, everything looks correct:
s3delmany.sh – Delete Many S3 Objects With One Command
Update: The Amazon S3 service API now allows for deleting multiple objects with one request (up to 1,000 objects per request). Please see the Amazon S3 Developer Guide for more information.
I’ve been doing some stuff at work using Amazon S3 to store files and during my testing I uploaded a ton of files that didn’t need to be there. Unfortunately, the command line tool I’m using, s3cmd, does not allow me to delete multiple files at once. There is no way to do a wild-card delete. This means I would need to get the full path to each object and delete them one by one:
./s3cmd del s3://s3.ekarma.net/img/1205794432gosD.jpg
Object s3://s3.ekarma.net/img/1205794432gosD.jpg deleted
./s3cmd del s3://s3.ekarma.net/img/1205794432g34fjd.jpg
Object s3://s3.ekarma.net/img/1205794432g34fjd.jpg deleted
Yea, there’s no way I’m doing that for over 200 objects. I mean come on, there are tools to automate this kind of stuff! So I created s3delmany.sh:
#!/bin/sh
# -------------------------
# s3delmany.sh
# Author: Raam Dev
#
# Accepts a list of S3 objects, strips everything
# except the column containing the objects,
# and runs the delete command on each object.
# -------------------------
# Redirect output to the screen
2>&1
# If not using s3cmd, change this to the delete command
DELCMD="./s3cmd del"
# If not using s3cmd, change $4 to match the column number
# that contains the full URL to the file.
# This basically strips the rest of the junk out so
# we end up with a list of S3 objects.
DLIST=`awk 'BEGIN { print "" } { print $4, "\t"} END { print ""}'`
# Now that we have a list of objects,
# we can delete each one by running the delete command.
for i in "$DLIST"; do $DELCMD $i
done
Download
s3delmany.zip
Installation
1. Extract s3delmany.zip (you can put it wherever, but I put it in the same directory as s3cmd).
2. Edit it with a text editor and make sure DELCMD is set correctly. If you’re not using s3cmd, change it to match the delete object command for that tool.
3. Make it executable: chmod 755 s3delmany.sh
Usage
If you’re using s3cmd and you placed s3delmany.sh in the /s3cmd/ directory, you should be able to use the script without modifying it. The script works by taking a list of objects and running the delete command on each one.
To pass s3delmany.sh a list of objects, you can run a command like this:
./s3cmd ls s3://s3.ekarma.net/img/ | ./s3delmany.sh
This will delete all objects under /img/. Make sure you know the output of your s3cmd ls command before you pass it to s3delmany.sh! There is no prompt asking if you’re sure you want to delete the list, so get it right the first time!
Hint: s3cmd doesn’t allow you do use wild-cards, but when you run the ls command, you can specify the beginning of an object name and it will only return objects starting with that. For example, s3cmd ls s3://s3.ekarma.net/img/DSC_, will return only those objects that begin with DSC_.
Alternate Usage
If you have a text file containing a list of S3 objects that you want to delete, you can simply change print $4 to print $1 and then do something like this:
cat list.txt | ./s3delmany.sh
By the way, print $4 simply tells s3delmany.sh that the S3 objects are in the 4th column of the data passed to it. The ./s3cmd ls command outputs a list and the object names are in the 4th column. The awk command expects the columns to be separated by tabs (\t).
If you have any questions or comments, please don’t hesitate to use the comment form below!
Recursively Renaming Multiple Files
I needed to rename a bunch of files for a customer at work the other day — more than 60 files. The files were on a Linux system, so I knew there was an easy way of doing it. A few days ago I used perl to search and replace a piece of text in a several files , so I decided to find a way to do it with perl.
I found the following script on this site:
#!/usr/local/bin/perl
#
# Usage: rename perlexpr [files]
($regexp = shift @ARGV) || die "Usage: rename perlexpr [filenames]\n";
if (!@ARGV) {
@ARGV = <STDIN>;
chomp(@ARGV);
}
foreach $_ (@ARGV) {
$old_name = $_;
eval $regexp;
die $@ if $@;
rename($old_name, $_) unless $old_name eq $_;
}
exit(0);
After saving the script to a file called rename (and chmod 755‘ing it) I was able to run the following command to change the file extension on all .JPG files from uppercase to lowercase .jpg. To search for all files underneath a particular directory, I used the find command and piped it’s output to the rename script:
find /home/customername/content/images/ | rename 's/JPG$/jpg/'
A few seconds later and all the files were renamed! This script is incredibly versatile, as you can pass it any regular expression! A few quirks I found were that you cannot reference the script; you must use it from the directory you stored it in (find / | ~/rename 's/JPG$/jpg/' won’t work). This is because the script uses itself (on line 17). This also means if you save the script as something other than rename, you must also modify line 17 of the script.
Easily Replace all Occurrences of a String in Multiple Files Using Linux
This came in handy when I needed to change the IP address used in several links on a customer’s site. I simply specified the path to all the web files, and ran a command similar to the one below.
find /path/to/directory -name "*.txt" | xargs perl -pi -e 's/stringtoreplace/replacementstring/g'
Note that you can change *.txt to just * to search inside all files. Also keep in mind the replacement command will be completed recursively (files nested inside directories will also be searched).
