Linux Blog

Graphical Shell Scripting

Filed under: Shell Script Sundays — TheLinuxBlog.com at 4:07 pm on Sunday, December 16, 2007

From all of my other previous shell scripting articles you can see that shell scripting is a very good way to get a task done or to automate. This is great but some times a little bit of that three letter acronym “GUI” is a nice touch. Well my friends, if you’ve been reading my articles and following my dialog examples then you are in luck. If not don’t worry, you can view all of the articles in the Shell Script section to the right.

The program to make your GUI’s for your shell scripts is the exact same thing as dialog except its graphical. Its called Xdialog. Once installed you basically use it the same way as dialog. If your thinking about upgrading a script thats written in dialog you might want to think twice because some window managers will display the windows differently.

Check out these Xdialog examples and corresponding screenshots:

Xdialog

Xdialog -yesno “Do you like Xdialog?” 5 50 && echo “Thats nice” || echo “Yea, some times its better to stick to CLI”

Xdialog ExampleXdialog Example

Recursive MD5 Sum Script

Filed under: Shell Script Sundays — TheLinuxBlog.com at 12:08 am on Sunday, December 9, 2007

This week I made this shell script to search one level deep and MD5 all of the files. I did this because I had multiple images and I wanted to see what images were the same so that I could merge them together. Its a pretty simple script & the output is the same as md5suming a file except there is more than one sum generated.

#MD5 Files in the directories
md5Dir () {
echo $directory;
for x in $(ls -1 $directory); do
md5sum $directory’/'$x;
done;
}
#Lists The Directories
for i in $(ls | grep active); do
directory=$i;
md5Dir;
done;

It only does one level deep but thats good enough for now. I am going to make it search recursively depending on the levels given by the user. I would also like to make it display files that are the same at the end.

It gets the job done for small directories, but if I wanted to run it on large multiple directories with lots of files in them I would definitely redirect the output to a file because it can be quite overwhelming. To run it just copy the code into a file and do the following:

sh <filename>

I hope this helps some one who is trying to MD5 multiple files in different directories!

Making your scripts user and sysadmin friendly

Filed under: Shell Script Sundays — TheLinuxBlog.com at 11:08 pm on Sunday, November 18, 2007

When designing a shell script it is important to make them easy to use but also to make it easily automated for deployment. One example of this that comes to mind is the NVIDIA installer. It has command line options to allow for deployment but also gives a nice interface for the end user.

To implement this “dialog” can be used for the user interface and “getopts” can be used for the command line options. The script may look something like:

#help function
help () {
echo “Linux Blog – getopts and dialog example”;
echo “Usage:”;
echo -e “\t -h shows this help”;
echo -e “\t -a [y/n][other] ANSWER (Yes, No or Other)”;
}

#show dialog to get the answers
showDialog () {
dialog –yesno “Do you want to enter y?” 5 50 && \
ANS=”Yes was entered using dialog” ||\
ANS=”No was entered using dialog”
showAnswer;
}

#actually show the answer
showAnswer() {
echo $ANS;
}

#check answer for command line processing
checkAns() {
if [ "${OPT1}" == "y" ]
then
ANS=”Yes sent by getopts”;
elif [ "${OPT1}" == "n" ]
then
ANS=”No was sent getopts”;
else
ANS=”This: $OPT1 was sent by getopts”;
fi
#call showAnswer
showAnswer;
}

#get the options
while getopts “a:h” ARG;
do case “${ARG}” in
a) OPT1=”${OPTARG}”;;
h) HELP=”TRUE”;;
esac;
done

#see if help was entered
if [ "${HELP}" ]
then
#display help and quit
help;
exit;
fi
#if the options are empty
if [ -z "${OPT1}" ]
then
showDialog;
else
checkAns;
fi

Keep this getopts and dialog post in mind next time your shell scripting. It will take a little extra time to implement but the result will be a user and sysadmin friendly script.

Creating Dialogs with Dialog

Filed under: Shell Script Sundays — TheLinuxBlog.com at 5:48 pm on Sunday, October 21, 2007

Have you ever seen those pretty dialogs used in Shell Scripts such as the Slackware installation, the slackpkg program or even the NVIDIA driver installer? Well, my friends to display dialog boxes from shell scripts is very easy with… you guessed it – Dialog.

First of all, there are many different types of dialogs that you can create they are as follows: calendar, checklist, fselect, gauge, infobox, inputbox, menu, msgbox (message), password, radiolist, tailbox, tailboxbg, textbox, timebox, and yesno (yes/no).

This blog post is intended to be a primer on using dialog. More examples will be posted in future blog posts in the Shell Script Sunday’s column.

The simplest form of a dialog in a shell script is probably the msgbox. All this really does is displays text. To display text in a dialog you would do the following:

owen@the-linux-blog:$ dialog –msgbox “Hello from the Linux Blog!” 5 50

The numbers after the text in quotes are the widths and heights of the box. The minimum height that I like to use is 5. The width doesn’t really matter as long as it is big enough. It is good to keep the box sizes standard across a whole script because it gets annoying with constantly resizing boxes.
If the text in a message box is too long it will auto wrap around and give you a type of scroll bar. As follows:

owen@the-linux-blog:$ dialog –msgbox “Hello from The Linux Blog. This text is so long it wraps it to a New Line” 5 50

Dialogs can be canceled. Clicking Ok or pressing enter/return returns “true” and pressing escape or Ctrl+C returns a false.
The simple shell scripting syntax shown in Shell Scripting 101

is used for this:

owen@the-linux-blog:$ dialog –msgbox “Dialog Exit Example” 5 50 && echo “ok” || echo “false”

Another simple dialog example is the Yes/No box. The syntax for this is exactly the same as the msgbox example except instead of using –msgbox, –yesno is used. The difference between a msgbox and a yesno box is that there a two buttons. It is pretty obvious as to what they are labeled, but for those in the back, I’ve included an example and some screen shots anyway.

owen@the-linux-blog:$ dialog –yesno “Are you learning anything from this blog” 5 50 && echo “Yes, thanks Owen.” || echo “No, Write some better Linux Related Posts”

The Linux Blog - Dialog Example - Yes / No

Thats about all I have time for this week. Check back next week!

Creating Script Parameters With getopts

Filed under: Shell Script Sundays — TheLinuxBlog.com at 9:06 am on Sunday, October 14, 2007

Many programs for Linux have parameters that can be given at run time. These are also known as switches, arguments or options. These parameters make it easy to tell a program or script what to do and what options to use. In this Shell Script Sundays Blog post I will show you how to implement these in a script by using getopts.

For this example to work the following must be placed in a script:

while getopts “:e:” ARG;
do case “${ARG}” in
e) echo “${OPTARG}”;;
esac;
done

This code basically gets loops around the arguments. All this script does is take the value after the -e and echo’s it out. In the example below I named the script getopts.

owen@the-linux-blog:$ ./getopts -e “Hi There Linux Blog Viewers”
Hi There Linux Blog Viewers

For each extra parameter that is needed a new constant to the getopts and do loop need to be added. For example if the option -e and -q need to be in the script then the following getopts loop would be created:

while getopts “:e:q” ARG;
do case “${ARG}” in
e) echo “${OPTARG}”;;
q) echo “${OPTARG}”;;
esac;
done

Of course the above script for -q also only echo’s out the value for -q as seen below:

owen@the-linux-blog:$ ./getopts -e “Hi There Linux Blog Viewers” -q “Another Option”
Hi There Linux Blog Viewers
Another Option

This is all very well, but documentation is always nice, even if you think you know how to use your script you may forget in the future. Its also nice if you have a script that other people can easily use. That being said its good to have a way to show users how to run the script.

usage () {
echo -e “Usage: $0 -e \”What To Echo\” [ -q \"Output\" ]”
echo -e “\t-e: specifies what to echo (required)”
echo -e “\t-q: Where to write output to. If not specified the output is written to the console”
}

while getopts “:e:q:” ARG;
do case “${ARG}” in
e) ECHO=”${OPTARG}”;;
q) OUTPUT=”${OPTARG}”;;
esac;
done

[ -z "${ECHO}" ] && { usage && exit 1; }

[ "${OUTPUT}" ] && { echo $ECHO > $OUTPUT; } || { echo $ECHO; }

The code above takes the options and assigns a variable to each of the options $ECHO is what to echo and $OUTPUT is where to write the output to. The script calls the usage() function and exits whenever the required option ($ECHO) is not set. If $ECHO is set it checks to see if $OUTPUT is set, if so it echo’s the contents of $ECHO to the $OUTPUT variable (file or device). If $OUTPUT is not set then it just echo’s the $ECHO variable normally. This is the script running with its various different actions:

owen@the-linux-blog:$ ./getopts
Usage: ./getopts -e “What To Echo” [ -q "Output" ]
-e: specifies what to echo (required)
-q: Where to write output to. If not specified the output is written to the console

owen@the-linux-blog:$ ./getopts -e “The Linux Blog getopts Example”
The Linux Blog getopts Example

owen@the-linux-blog:$ ./getopts -e “The Linux Blog getopts Example. Output To Null” -q /dev/null

owen@the-linux-blog:$ ./getopts -e “The Linux Blog getopts Example” -q Write_To_This_File
owen@the-linux-blog:$ ls
Write_To_This_File getopts
owen@the-linux-blog:$ cat Write_To_This_File
The Linux Blog getopts Example

As there are many different variations and each implementation would be different I can not cover each individual getopts scenario but by assigning variables your option arguments you should be able to get switches working in your own shell scripts.

Linux+ Certification

Filed under: General Linux,Shell Script Sundays,The Linux Blog News — TheLinuxBlog.com at 4:35 pm on Monday, October 1, 2007

My Shell Script Sundays article was not posted yesterday as I was reviewing for the Linux+ Certification by CompTIA. I have been meaning to take the exam for a while not but just never got around to doing it.

I took the exam today and passed. It was comparable to the other CompTIA exams I have taken except the topic was mostly on Linux instead of networking or hardware. The questions were very reasonable some were a little tricky and had me confused about the correct way to complete a task.

I only had the LPIC certification book to review from but had read the XK0-001 book about 8 months ago. The exam description explains that it was designed to measure the competencies of the Linux Professional with six to twelve months experience with Linux. Some of the other CompTIA exams say that you should have X years of experience in a certain field but is not needed, I believe that the experience for this exam is needed, maybe not half a years worth but at least a month of using nothing but Linux full time.

I would recommend to anyone interested in collecting certifications to take this exam. Even if Linux isn’t your main concentration it will still be a valuable asset in your certification portfolio. Now I’ve passed I have to find another certification to study for and beat around the bush to take the exam. I’m wondering if I should take another Linux certification (such as Redhat, LPIC or Novell’s) or go another route like Server+ or Security+. Either way it gives me something to do and keeps my brain active.

My apologies for not getting the Shell Script Sundays article out but I have a good one lined up for next week so check back!

Using BASH to sort a book collection. ISBN Data Mining – Part 2

Filed under: Shell Script Sundays — TheLinuxBlog.com at 8:49 pm on Sunday, September 23, 2007

Last weeks article used a bit of data mining to grab information on ISBN’s from Barnes and Noble and dumped the information into a text file. This week we’ll take those text files and format the data for input into a MySQL database. This tutorial assumes that you are familiar with MySQL.

The data that was written to the text files was in the following format:

Title: DNS and BIND, Third Edition
Title:
Author(s): Paul Albitz, Mike Loukides (Editor)
Publisher: O’Reilly Media, Incorporated
Pub. Date: September 1998
ISBN: 1565925122
Format: Paperback, 480pp
Edition Number: 3
Edition Description: Older Edition
Customer Rating:

This can easily be parsed and formatted for insertion into a MySQL table.

Firstly a database has to be created and then a table structure has to be decided upon. Since this example already has the titles, I’ll just use simular ones for the field names.
Create a database called book_info:

mysqladmin -u root create book_info;

and now create a table within the book_info database that is to contain all of the data:

CREATE TABLE `books` (
`ISBN` INT( 10 ) NOT NULL ,
`title` VARCHAR( 50 ) NOT NULL ,
`author` VARCHAR( 80 ) NOT NULL ,
`publisher` VARCHAR( 50 ) NOT NULL ,
`pub_date` VARCHAR( 50 ) NOT NULL ,
`format` VARCHAR( 30 ) NOT NULL ,
`edition` INT( 2 ) NOT NULL ,
INDEX ( `ISBN` )
) ENGINE = MYISAM ;

This isn’t the best MySQL table structure ever, but it will do for the purposes of this artice and besides it can always be tuned later.

With a directory full of .txt’s files from the last issue of shell script sundays the following can be ran to create a text file called bookQuery.sql.

for i in $(ls); do
echo “INSERT INTO \`book_info\`.\`books\` (\`ISBN\`, \`title\`, \`author\`, \`publisher\`, \`pub_date\`, \`format\`, \`edition\`) VALUES (‘$(cat $i | grep ISBN:)’, ‘$(cat $i | grep Title | sed ‘q’ | sed “s/’/\\\\’/”)’, ‘$(cat $i | grep Author\(s\): | sed “s/’/\\\\’/”)’, ‘$(cat $i | grep Publisher: |sed “s/’/\\\\’/”)’, ‘$(cat $i | grep Date:| sed “s/’/\\\\’/”)’, ‘$(cat $i | grep Format: | sed “s/’/\\\\’/”)’, ‘$(cat $i | grep Edition\ Number: | sed “s/’/\\\\’/”)’);” >> bookQuery.sql; done;

In turn this file can be imported into the table that was created by running the following:

mysql -u root < bookQuery.sql

Whats happening is pretty simple, cat reads the file and grep is used to find the line of text we want to import. After that sed is used. It is used twice in the title field. The first time is to use the first title from the text file. It is also used on every other appropriate field to escape the string so that it does not break the query. This example does not take the titles out of the line, but this could be easily done with cut.

Its easy to import text files into MySQL with shell scripting but the language I feel is best suited for this task is PHP. Some time I’ll go over how to do this with PHP.

Bringing The Internet Up After Failure

Filed under: Shell Script Sundays — TheLinuxBlog.com at 9:58 pm on Sunday, September 9, 2007

This Shell Script Sunday is a short one but don’t let that fool you to the power of the shell. This script I wrote earlier in the week due to power spikes at the office. All of our equipment would stay powered on due to UPS’s but unfortunately something with the ISP was not staying on. Once the brownout occurred our router box would still have an IP and seem to be working but it wouldn’t. We had our suspicions about what piece of equipment it was but had no power to fix it. I would renew the IP from the ISP bring the public interface down by using eth0 down and then eth0 up but this was not successful. To fix it from the router I had to actually reset the network. This worked, but we have some services running at the office that I like to access from home. So to fix the problem I wrote a one liner to reset the network if the connection goes down.

ping -c 1 OurISP.com 2> /dev/null > /dev/null && echo > /dev/null || sudo /etc/rc.d/network restart

The techniques in this script are covered in Shell Scripting 101. All this does is ping OurISP.com one time and output the error & standard output to /dev/null. If the ping was successful it does nothing and if the ping failed then it restarts the network. To get it to repeat at an interval I just set it up as a cron job. This did the trick and I now do not have to worry about brownouts.

For, While and Until Loops in Bash

Filed under: Shell Script Sundays — TheLinuxBlog.com at 3:45 pm on Sunday, August 12, 2007

Normally in a shell script it is desirable to do something repetitive.
I have touched on some loops in other posts but now would like to go over them in a bit more detail. Each of the examples in this post are intended to give an introduction to looping in bash.

For Loops
For loops allow you to repeat a section of code a number of times. Its very similar to other languages syntax but works a little differently. The for loop in bash only allows you to give a fixed list of values to loop over. A good way to remember how a for loop works is “For each of the dishes: clean and dry.”
For Syntax:

for i [in list]
do
statements [use $i]
done

For Example:

for x in 1 2 3
do
echo “Number: $x”
done

echo “Finished!”

This is a very simple script that just counts to 3 and then prints “Finished!”

While and until Loops
In essence while and until are the same in bash. The titles are pretty much self explanatory. A while loop would be explained in real life as “While the sink is still full: wash dishes” and a until loop would be “Until the sink is empty: Wash dishes.”
While and Until Syntax:

until/while [condition] do
statements
done

Example of a While loop:

count=1
while [ $count -lt 10 ]; do
echo $count
let count=$count+1
done
echo “Finished!”

Basically this loop will loop over the code while the count variable is less than 10. If we didn’t put the let statement in the script it would get stuck in the loop causing the user to press CTRL+C to end the script.

Doing the same thing can be done in a until loop except the condition has to be modified to get the same result.
Until example:

count=1
until [ $count -gt 9 ]; do
echo $count
let count=$count+1
done
echo “Finished!”

Now that you’ve figured out how to loop over something its probably a good idea to know how to stop the loop.
All that needs to be done to stop a loop is:

break

Break Example:

for x in 1 2 3 4 5
do
if [ $x = 3 ]; then
echo “Number is 3. Quitting”
break;
fi
echo “Number: $x”
done

This is a very easy to follow example. Its the same as the basic for loop except that if x is 3 it will stop the loop. This example has no real practical purpose. Since its a for loop the number 3 could just have been omitted.

Real World For Loop Example
Looping over all files in /etc and printing all of those that match “grep conf” and putting them in quotes.
The code to do this in a loop is:

for x in $(ls /etc -1|grep conf);
do
echo “$x”
done

The situation for many bash scripts is that there is normally a shorter way to do something. Take the Real World For Loop Example in this tutorial the same results can be achieved with:

x=$(ls /etc |grep conf); echo “$x”\n

This will get the job done but a loop may be better for esthetic purposes or for additional logic.

Decision making in Bash If / Then / Else statements and Cases

Filed under: Shell Script Sundays — TheLinuxBlog.com at 3:44 pm on Sunday, August 5, 2007

There comes a time in every shell script where a decision has to be made.

To make a decision in bash the following if then else syntax can be used:

if [ condition ]
then

statements

[ elif [ condition ]

then

statements ]

[ else

statements ]

fi

Anything in non bold [ brackets ] is optional, and they should not be typed but the bold in them is required. Anything in bold must be typed. Statements are generally other commands or flow control statements.

To give an example of how to use bash if then else statements in the real world take the following scenario:

A system administrator has a strict habit of firing people that have too many .png files. He checks the systems regularly and makes sure that nobody has too many. The following script will display a message depending on the number of .png’s that are in the directory.

#!/bin/bash
gif_count=$(ls | grep .png | wc -l)
echo “Number of PNG’s: $gif_count”
if [ $gif_count -lt 10 ]
then
echo “He will be happy, you have less than 10 files”
elif [ $gif_count -gt 10 ] && [ $gif_count -lt 20 ]
then
echo “Consider deleting some files”
else
echo “you have too many files, DELETE SOME BEFORE HE FINDS OUT!”;
fi

Using Cases.

Cases are similar to if statements in that they match a expression and perform a task if the pattern is matched.

bash case syntax:

case expression in

pattern1 ) statements ;;

pattern2 ) statements ;;

esac

This is fairly simple and some people find this easier than doing if statements for simple logic. Take the following real world example:

The system administrator has recently gone on a bigger power trip than before. Since people got wise about using png’s and started saving images in other file formats he is now monitoring png’s gif’s and jpg’s. To combat the problem, you can use a case to count how many files you have of each type. (This is intended as an example, there are many ways to accomplish this task, this is just to demonstrate how cases work)

#!/bin/bash
#set all variables to 0
png=0
gif=0
jpg=0

# start loop
for wc in $(ls); do

case “$wc” in
*png ) let png=$png+1 ;;
*gif ) let gif=$gif+1 ;;
*jpg ) let jpg=$jpg+1 ;;
esac

# end loop
done

echo “Png’s $png”;
echo “gif’s $gif”;
echo “jpg’s $jpg”;

There you have it, two ways to make basic decisions in bash. Just figure out what you want to do then use an if then else, or a case statement to do the logic. I myself prefer if statements over cases as they make more sense to me and I find it easier to perform logic within ifs.

Programs used in this post
ls, echo, grep, wc

Recent changes to The Linux Blog.

Filed under: The Linux Blog News — TheLinuxBlog.com at 10:39 pm on Sunday, July 29, 2007

This article is mostly just updates on the site.

Server Changes
I have recently moved the site to a new server. This was a fairly simple task which took longer than it should because of DNS issues.

Now that it is on the new Linux based web host all should be dandy. The fact that I have more flexibility over the old host is an added bonus.

Bugs

Some problems that surfaced after I moved the site have just been resolved in a matter of minutes ago.

The biggest problem the site had was that the detailed/archive WordPress URL’s were not getting processed correctly. I was aware of the problem last night but was too tired to fix it. The fix was simple. I had to place the .htaccess file in the directory. The problem was that when I used lftp to mirror the old code, it didn’t download the hidden .htaccess file. I should have checked this before I updated the DNS but I guess something will always go wrong and I’m glad it was a simple fix. When I figure out how to mirror a directory in lftp including hidden files I will be sure to let everyone know. After scanning the help for the mirror command it didn’t jump out at me, but maybe thats just because I’m tired.
The URL’s that were affected by the bug were:

http://www.thelinuxblog.com/2007/07/29/shell-script-sundays/

http://www.thelinuxblog.com/2007/07/28/phones-meet-linux/

New Category Created
Not only did I fix this little problem, I also created the Shell Script Sundays section and moved the related posts to that category. The When Photoshop Fails article was posted on a Monday but I did most of the writing on Sunday, so I believe that since its mostly a shell scripting article it is worthy of this category.

After writing a paragraph in the description section of WordPress, I realized that it doesn’t actually show up anywhere on the site. I’m unsure if it shows up in the RSS feeds or not but I’m going to post it here anyway for the web browsers.

Shell Script Sundays Description
This section of the site is dedicated to Linux shell scripts. Twice a month I will post a nifty shell script that will perform a certain task. Most of the scripts will be written in Bash or the Korn shell and occasionally a CLI PHP or Perl script may surface. Some scripts will be more advanced than others and some will require additional software to be installed. This section will show how powerful scripting can be and I hope it educates people on how to shell script with Linux.

More to come!

Expect a Linux related post within the next few days. I have hundreds of topics to choose from, but I am always willing to take peoples suggestions on what to blog about. So if there is a particular topic that interests you, just contact us and we’ll do our best to cater to your needs.

Ciao,

- Owen.

« Previous PageNext Page »