Linux Blog

An rsync primer

Filed under: General Linux,Linux Software — TheLinuxBlog.com at 12:55 am on Tuesday, January 13, 2009

An rsync Primer

rsync is a great tool used to copy or “sync” files locally or remotely. Having just lost a fair amount of important personal data its good to make the point that you can lose data if you use rsync, so make sure that you know what you are doing works before putting it into production.
“Whats so great about rsync?”
I hear you ask.

Well, there are many ways of copying data around, regular old cp, scp and rcp but what sets rsync apart is that it’s capable of syncing those files that have changed, or those that are missing. In other words, why do a full copy of all files, risking further corruption when only some have changed?
rsync also handles compression and verifies that the files were written correctly.
The syntax for rsync is fairly straightforward, those of you familiar with the rcp or scp should be easily able to relate to that of rsync.

The syntax for a local file copy is:

rsync [OPTION...] SRC… [DEST]

-avz is the option that I use most commonly.

-a for archive, -v for verbose and -z for compression. As lame as it may sound, I actually remember the syntax for this as “Alien Vs. Zeus.” Sounds stupid, but it actually works.

Since I do not normally sync directories locally, one of my hosts is usually remote. rsync is configured to use SSH by default, but rsh can be used if preferred. I stick with the default SSH for simplicity. The syntax is very similar to scp’s

To Pull: rsync [OPTION...] [USER@]HOST:SRC… [DEST]

To Push: rsync [OPTION...] SRC… [USER@]HOST:DEST

If you like to back up all files matching a pattern the same syntax applies but you specify a pattern. An example is you could backup all .conf files from /etc (provided you have read access) to another host by doing the following:

rsync /etc/*.conf [USER@]HOST:DEST

If you do the above, I am assuming that you would also want to backup other configuration files within the /etc/ directory. Instinct tells you if you have read the man page to just use -r however you may want to just do a full backup as rsync with a pattern by default does not traverse into directories.

Compression is really a nice feature for transferring large amounts of data. To demonstrate the speed increases that the -z option gives my full /etc/ directory which is 48M. It took 0m12.671s with compression and 0m35.657s without. This was over wireless from a dual core 2GHz laptop to a 1.8GHz wired desktop, so your results may vary. Either way compression or no compression rsync is a handy utility that should be able to make your life of copying files around a little easier.

When Photoshop Fails

Filed under: General Linux,Shell Script Sundays — TheLinuxBlog.com at 9:50 pm on Monday, January 15, 2007

I was recently assigned the task of reducing the quality of over 1000 images. My first instinct was to use photoshop’s batch functions to complete the task. After playing around with it for a little while it became apparent that you cannot save files for web with spaces in. Photoshop had failed me. My next idea was to use a bash script to loop over every file and process the image. This was sure to work. Since I have prior experience with image processing I decided to use ImageMagick to complete the task. The command to adjust the quality is:

convert (FILENAME) -quality 50 (FILENAME)

This command is useless on its own so using a for loop I came up with:

#!/bin/bash
for i in $(ls); do
convert $i -quality 50 $i
done

This script would have worked for me if there were no spaces in the filenames. Because there were it took each filename that had spaces and ran the command on each part of the file name. Unfortunately this was not going to work.

After googling for a while it became apparent that I was not the only person to have the problem of spaces in filenames with for loops. The solution I found was to use the find command and a while loop. Below is working script that successfully completed the task in no time:

#!/bin/bash
find * -iname “*” | \
while read I; do
convert “$I” -quality 50 “$I”
done

This script could be easily modified to take advantage of ImageMagick’s many other functions. For example it could be used to batch resize a folder of images to make thumbnails by changing the command to

convert “$I” -resize 200 “$I”

Or it could be used to overlay text onto an image with the following command:

convert “$I” -gravity southeast -annotate +5+10 “thelinuxblog.com” -fill black “$I”

As shown above ImageMagick is really powerful image editing software which can easily be used with the bash shell to process thousands of images with no trouble.