Linux Blog

Working with CSV files in Bash

Filed under: Shell Script Sundays — at 1:25 pm on Sunday, October 19, 2008

CSV files are very common. Using them with Bash to aid in scripting can be very useful. Here are a some methods of using bash to work with CSV files. These are what I sometimes use when dealing with CSV files in bash, they may work for your CSV’s they may not. I accept no responsibility of the accuracy of these, I’m sure there are better ways of accomplishing the same tasks. If anybody finds other ways or knows better please comment and let me know. Don’t run these on production data, its always a good idea to make backups and test them before using in a production environment. Who knows how your precious data could become mangled due to a wrong number of quotes, terminations or bad regex. Jump for the post (Read on …)

Add Voice to your Shell Scripts

Filed under: Shell Script Sundays — at 3:52 pm on Sunday, October 12, 2008

Text to speech has been around for a number of years. While you may not want to add text to speech to all of your shell scripts you may have a particular script that would be nice to get some audio feedback from. There are a couple of text-to-speech engines that run in the shell, but my favorite is flite. It is really simple to install and use and has a decent set of voices. You can add other voices or compile your own with FestVox.

Here is how to use flite from the shell to directly play the text:

[owen@TheLinuxBlog ~]$ flite -t "Thanks for reading The Linochs Blog"

Notice the misspelling of the word Linux. If flite doesn’t correctly output the words, you can sometimes use phonetics to get a better result.

You can also play text files by using the -f option.

[owen@TheLinuxBlog ~]$ flite -t "Thanks for reading The Linochs Blog"
flite: pcm_params.c:2333: sndrv_pcm_hw_params: Assertion `err >= 0' failed.

(Read on …)

shell script argument lists too long

Filed under: Shell Script Sundays — at 11:19 am on Sunday, September 21, 2008

Sometimes you will run into the problem of having too many files in a folder. This results in basic programs like rm and mv to not be able to process your commands. For example, I recently had to extract 52628 jpgs from seven zip files. Once extracted I realized that I had them in the wrong folder. Now, there are other ways I could have accomplished the same goals, such but for various reasons sometimes the easiest way (renaming the folder) was not feasible. Here is the solution: (Read on …)

Reworking Shell Scripts – Part 2

Filed under: Shell Script Sundays — at 10:00 pm on Sunday, September 14, 2008

In the last Shell Script Sunday’s article I wrote, I said that I’d show you some more ways to rework shell scripts to make them easier to use. I’ve got some more tricks up my sleeve that I’d like to share, its  been rather busy this site of the internet at So why I write up some more shell scripting methods, here are some previous posts that can enhance your shell scripts. Be sure to comment on them if you find them useful, or would like more information.

Creating Dialogs with Dialog
If you have a shell script that you use on a regular basis, you may want to consider using dialog to make it more user friendly. Dialog makes it easy to create easy to use dialogs that are intuitive and easy to use. There are so many combinations of dialogs that can be created that the possibilities are ended. Dynamically create dialogs for select lists, input boxes, progress bars and much much more.

Graphical Shell Scripting
This article I wrote introduces graphical shell scripting. Similar to dialog this is an updated “Dialog” and works within X. If you support end users, or your target audience is Ubuntu/Linspire users (j/k) then XDialog may be the better choice. Its got most of the same functionality as Dialog except it depends on X. You can even support both Dialog and XDialog as they pretty much use the same syntax.

Reworking Shell Scripts

Filed under: Shell Script Sundays — at 7:02 am on Sunday, August 24, 2008

To me shell Scripts are all about automation, their primary purpose is to get stuff done. If you have a one liner, snippet or other script you use on a regular basis, have you thought about how you could rework it for it to become more in handy?

Lets take last weeks snippet from this column. It was a simple one liner to reconnect to a host. Now, I knew when I posted this article that it was a helpful snippet of code. Now, how can this script be adapted to be a neat utility that we use on a regular basis? Over the next few week’s we’ll find out.

The first thing that I will note on is that this script or shell snippet is a pain to remember. Does a script save you time if you can’t remember how it works? Is it worth the hassle? Not exactly. So, in order to make this snippet a little better the first thing we are going to do is add something that it needs: parameters. Adding parameters to shell scripts is actually easy, much easier than adding parameters to some other languages that we wont mention. although this script does not use it getopts can be used. I’ve covered how to do this with getopts in other posts. Just do a site search (located at the bottom of the right bar for getopts.)

So, here is the modified script that automatically reconnects to a host by using ping and SSH:

# Sleep Time Default: 15 seconds
# Set a default user up
#usage function
usage () {
echo -e "Usage: $0 host [user] [Sleep Time]"
# display usage if no host is specified
[ -z $1 ] && { usage && exit 1; }
# set the variables
[ $1 ] && { HOST=$1; }
[ $2 ] && { USERNAME=$2; }
[ $3 ] && { STIME=$3; }
# trying:
echo -e "host: $HOST \nuser: $USERNAME \ndelay: $STIME"
while ! ping -W 1 -c 1 $HOST 2>&1 >/dev/null; do true; done && echo "Successful waiting: $STIME secs"; sleep $STIME; ssh $USERNAME@$HOST

Now that you have that done, all you need to do is give the file a name (I called mine ssh_auto) and put it in a folder in your path. Use the filename and parameters defined in the script to connect to the host.

The next shell scripting article I demonstrate how you can further rework shell scripts to better suit your needs.

Automatically reconnecting to a host

Filed under: Shell Script Sundays — at 9:15 pm on Sunday, August 17, 2008

If you follow me on Twitter: then you may know that I regularly update a bunch of Linux PC’s and servers. Now, since I’m sort of lazy and don’t like manually doing anything I don’t have to I thought I’d post the one liner I use to automatically reconnect to a host.

while ! ping -W 1 -c 1 [hostname or IP] 2>&1 >/dev/null; do true; done && sleep 15; ssh [user]@[hostname or IP]

This script uses the ping command to ping the server once (-c 1) with the timeout of 1 second (-W 1) ping a host or IP with a timeout of one second. Once the ping loop is broken (ping returns true) I let it sleep for 15 seconds to enable SSH to come up. Then the inevitable happens. I use SSH to reconnect to the host.

There you have it, a quick way to reconnect to a host without typing the command or pressing the up arrow every time. Enjoy!

Automated scanning with the shell

Filed under: Shell Script Sundays — at 9:25 am on Sunday, July 27, 2008

I recently needed to scan a lot of images on my desktop PC. Unfortunately I am not the owner of an automatic document feed printer, and if I were it wouldn’t have helped this time because the documents I needed to scan were not feed able. XSANE is a great way to scan documents visually in Linux. Its not the easiest to use, but it has plenty of options. Part of the SANE package is scanimage, scanimage can be used from the shell.

The first thing that I did was a few test images with scanimage. I quickly found out that scanimage outputs in pnm format, and at a high resolution if the correct options are used. Once I found out the good options for my scanner (scanimage –resolution 400 > file.pnm) I wrote a quick shell script to scan up to 1000 times or until I don’t give the script any input. To do this, I used a combination of snippets that can be found in this blog column.

Here is a direct link to the script, and the shell script source below

for i in `seq 1 1000`; do
#get input line
read inputline;
if [ $inputline ]; then
#Process Scanned Image in BG
echo Scanning Pg$i;
scanimage --resolution 400 > Pg$i.pnm;
echo Next;

To use it all I do is execute the script, and I get to scan up to 1000 documents providing I type something after it prompts “Next”, and then hit enter. Once I was done scanning, I just hit enter to stop the script execution and then moved on to manipulating the images with the shell.

Hope this shell script scanning script is useful, if it is then drop me a comment, or if you have any suggestions or it was not at all helpful still drop me a comment.

Timing your reboots with Twitter support!

Filed under: Shell Script Sundays — at 12:01 am on Sunday, July 20, 2008

Firstly, I’d like to start off by saying that all of the concepts in this post should have been covered in other posts, so I will not go into great detail on the specifics of this script. If you need to know more information about any of the commands, check the man page section at the bottom of this page, from the man pages will be examples of other posts covering similar topics.

The purpose of this script for me was to time my reboot times. It could be modified to log the time it takes to replace hardware or add memory, but thats another post. Since we are logging reboot times, we are (hopefully) dealing with small numbers and therefore don’t have to deal with formatting time (at least not for now.)

The script should work on multiple systems that have bash. There is nothing too special about it. It uses the reboot command so the user this is launched as will have to have access to that command. You put the script in the users bin directory and chmod it. The user must also have write access to this. Also, they must have write access to their home directory, but this should not be a problem for most. Line 8 of the script needs to be changed to the user you plan on running this as.

After that test that the timereboot command works by typing timereboot:

[owen@linuxblog ~]$ timereboot
Usage: /home/linuxblog/bin/timereboot {time|ttime|back}

Once that is done, thats a pretty good indication that the script is working. Next, I suggest commenting out the reboot command on line #25 if this is a critical mission and you don’t want to reboot multiple times to get it working. If not go ahead and try the time command. Once your system is back up and your logged in you type the “timereboot back” command, it will then tell you the time taken since your system was done.

Once you have verified that the time works, you can go ahead and add it to your bashrc to automatically perform the action once your logged in. All you need to do is add a line like this:

home/linuxblog/bin/timereboot back

Now, if you want you can try again and see the results automatically.

“Thats great, but how do I post it to twitter?”

Well, there is one last thing that you have to do to get your reboot time posted to twitter. Edit line 55 and change to your twitter username and password. Do the same thing as before to reboot, but use the ttime parameter to log to twitter.

This script, does not post to twitter that you are rebooting (although it could) nor does it format the time, but it works and should give you a starting point if you are interested in doing this. It doesn’t really serve a real purpose other than to inform people how quickly or how slow you reboot. Also, please note that this is not a start up time. This times from when you issue the command until you issue the back command, or log in using the .bashrc method.

If you have any questions about this script or any other idea’s let me know and I’ll be happy to help or implement them for fun.

And here is the Twitter reboot script

Adding a service in Fedora

Filed under: Shell Script Sundays — at 2:08 pm on Sunday, July 6, 2008

This week on Shell Script Sundays I’ll show you how to add a service to Fedora. This is very useful if you don’t happen to use yum for every service you want to run, and xinetd doesn’t really work for you.

Firstly there are three main parts to a Fedora service script. Start, Stop and Restart. They are pretty much self explanatory, but you don’t have to worry about the restart action since all it does is stop’s and then starts the service.

Without further ado here is the script:

# Fedora-Service Update notification daemon
# Author:
# chkconfig:    1000 50 50
# description:  This is a test Fedora Service \
#               Second line of the fedora service template.
# processname:  FedoraTemplate
start() {
echo "Starting Fedora-Service"
stop() {
echo "Stopping Fedora-Service"
restart() {
case "$1" in
echo $"Usage: $0 {start|stop|restart}"
exit 1
exit $RETVAL

Now that you have a template for the script, you will want to modify it for your service. You need to keep the header at the top. This is how the Fedora Knows about your service. The three numbers indicate what order the scripts should start up and shut down in. The first seems to be a identification number and the other two are the startup and shutdown order. These can be adjusted depending on when you want the service to start up.Once you are done modifying the script put the script in /etc/init.d/

To make sure it works you can call it with service using the following actions:

service start
service stop
service restart

If all of the actions work, you are ready to add the service to the system. If you use the setup command as root it seems to do this step for you, but if you just want to add the service quickly without bothering to scramble through configuration menu’s you can do the following:

chkconfig --add [script name]

If you want the service to start automatically at boot up you can use ntsysv. For more information read my post on Managing Services on Fedora

tweeting from the command line

Filed under: Shell Script Sundays — at 12:25 pm on Sunday, June 29, 2008

This is a subject that has been covered time and time again but I don’t think that it will hurt one more time. Twitter is a very popular “Microblogging” site where you can constantly change your status to let those who “follow” you know what you are doing. Since I just signed up for twitter for The Linux Blog I figured I’d write this post on how I update my twitter feed. While I’m at it I might as well invite you over to my feed URL:

So here is the script:

echo "Enter Tweet: ";
read inputline; TWEET="$inputline";
curl -u user:password -s -F status="$inputline"

This is a very basic twitter script, it does no error checking and probably doesn’t escape characters properly. None the less it works. The part that gets input from the shell is the following line:

read inputline; TWEET="$inputline";

If you’d like more information on how this works read this article: Shell Script to get user input

Curl is used to send the data to Twitter, to view curl tutorials and how-to’s visit the Curl Man Page which has a wealth of information at the bottom.

Until next time, happy tweeting!

Parse ifconfig data with shell scripts

Filed under: Shell Script Sundays — at 2:25 pm on Sunday, June 8, 2008

This week in’s Shell Script Sundays article I’m going to show you how you can use basic UNIX commands to parse networking data. As always there are a number of different methods of achieving this, and I am in no way saying that this is absolutely the way you must do it, or the best way. Its just an example of how you can use shell scripts to your advantage.

Firstly most know that Linux uses the ifconfig command to get information about networking interfaces. If you issue the ifconfig followed by the interface name you get information just about that interface as follows:

# /sbin/ifconfig eth1
eth1 Link encap:Ethernet HWaddr 00:0E:35:7F:E2:98 inet addr: Bcast: Mask: inet6 addr: fe80::20e:35ff:fe7f:e298/64 Scope:Link UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:1146 errors:0 dropped:39 overruns:0 frame:0 TX packets:1 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:23748601 (22.6 MiB) TX bytes:507899 (495.9 KiB) Interrupt:11 Base address:0x4000 Memory:fceff000-fcefffff

This information is not in the best format to parse (it has also been distorted by my blogging software.) To solve this problem we are going to search for the whitespaces at the beginning of each line and replaces them with commas. By doing this:

# /sbin/ifconfig eth1 | sed 's/          /,/'
eth1 Link encap:Ethernet HWaddr 00:0E:35:7F:E2:98 ,inet addr: Bcast: Mask: ,inet6 addr: fe80::20e:35ff:fe7f:e298/64 Scope:Link ,UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 ,RX packets:1344 errors:0 dropped:39 overruns:0 frame:0 ,TX packets:1 errors:0 dropped:0 overruns:0 carrier:0 ,collisions:0 txqueuelen:1000 ,RX bytes:23809630 (22.7 MiB) TX bytes:507899 (495.9 KiB) ,Interrupt:11 Base address:0x4000 Memory:fceff000-fcefffff

That gives us a nice comma after every line. In order to grab fields from this line the tr command can be used to replace spaces with pipes.

#/sbin/ifconfig eth1 | sed ‘s/ /,/’ | tr [:space:] \|

Now that the fields are all delimited properly, lets use the cut command to grab a line from this. Since I am interested in the RX and TX bytes I’m going to grab data from line 8 by using the cut command as follows:

#/sbin/ifconfig eth1 | sed 's/          /,/' | tr [:space:] \| | cut -d , -f 8

That gave us a nice line of output which is easy to parse even further by using the cut command. You will notice the fields are delimited by a pipe (the | character) and are not always consistent since we replaced all spaces with a pipe. Take a look at the first two fields RX|bytes: This means that to get the RX bytes in bytes we need to cut yet again. Since I’m not to bothered about Bytes and the largest number is delimeted in fields 3 and 4 I will concentrate on those.

#/sbin/ifconfig eth1 | sed 's/          /,/' | tr [:space:] \| | cut -d , -f 8 | cut -d \| -f 3-4

This is a nice RX MiB output yet it has one last problem, the pipe between the characters. Sed can be used to replace this and any other characters if you wish. Just issue a sed find and replace like this:

#/sbin/ifconfig eth1 | sed 's/          /,/' | tr [:space:] \| | cut -d , -f 8 | cut -d \| -f 3-4 | sed 's/|/ /'
(23.0 MiB)

That looks good for now. If you would like more information on how to parse data regarding this post or any other you can always leave me a comment and I’ll try my best to help. Especially if we can post the results on TheLinuxBlog in another Shell Script Sundays Article. Thanks for reading The Linux Blog and come back soon!

« Previous PageNext Page »