Linux Blog

timestamps in the shell

Filed under: Shell Script Sundays — TheLinuxBlog.com at 12:02 pm on Sunday, March 2, 2008

Time and Date functions are very important when writing shell scripts. I mostly use them for logging reasons, for example to know when something was run last. As much as I dislike time stamps they are still used (at least for now) and therefore I am giving and example.

I am unsure of a way to get a time stamp in Bash. If you have PHP installed you can do the following to get a UNIX time stamp (suitable for inserting into a DB):

php -r ‘echo time().”\n”;’

php -r executes PHP code inside quotes. the time() function just creates a time stamp for the current time. If you need to format a string based on a time stamp you can use the date() function. Here is an example of turning a timestamp into a readable string:

bash-3.1$ php -r ‘echo date(“l dS \of F Y h:i:s A”,”1204476759″).”\n”;’
Sunday 02nd of March 2008 11:52:39 AM

Take a look at the date() function page on php.net if you wish to use this method of using time stamps in the shell.

If any one has any other methods of using time stamps in the shell or needs any help as usual leave a comment :)

Shell Script to get user input

Filed under: Shell Script Sundays — TheLinuxBlog.com at 2:06 am on Sunday, January 27, 2008

Creating a shell script to get input is rather easy. Shell scripts prompting input are generally more user friendly too. In this article I’ll show you how to read input from the bash shell. Take the example below:

#!/bin/bash

echo "Shell Script To Get User Input";

while read inputline
do
what="$inputline"
echo $what;

if [ -z "${what}" ];
then
exit
fi

done

All it does is echo’s the introduction “Shell Script To Get User Input” and then goes right to the bash read input loop. The next line makes a variable so that we can echo it out and also check if its empty with the if [ -z part. If the script is empty we exit, if not we loop around another time.

This is a very basic example but it can easily be modified so that you can use bash to grab user input. If you have any trouble with this script drop me a comment and I’ll be happy to help you out.

What to do when you run out of disk space

Filed under: General Linux,Linux Software,Quick Linux Tutorials — TheLinuxBlog.com at 11:27 am on Tuesday, January 8, 2008

Some times you run out of disk space. It just happens. So, what do you do when it does happen?

Well, it just happened to me and I’ll write about what I did. I’ll first start off with how I discovered that I was out of disk space in the first place. It was about 10:30 last night when for some reason that I can’t remember now I decided I’d start up my good old XP Virtual Machine (Probably to use some quirky Windows program.) Anyhow the VMWare console reported that I did not have enough disk space. This came as a bit of a shock to me as I have a 100GB hard drive. I had been downloading ISO’s of Linux Distributions but not that much. So, here is what to do when you run out of disk space:

Step 1) Don’t panic
Step 2) Take a look at your processes and shutdown anything that is not needed. init to single user mode if it makes you feel better.
Step 3) Use the disk free utility to figure out how much space you have:

df -h /

Step 4) Make a couple of megabytes of free space so that you can install a program.
Step 5) Download and install xdiskusage from source or from your favorite package manager.
Step 6) Run xdiskusage from the terminal as root
Step 7) Select a disk / partition
Step 8) Wait
Step 9) View the results
Step 10) Rinse wash repeat. (Browse Partitions / Delete / Move files to another disk & do it again)

Here are some screen shots of my xdiskusage:

xdiskusage example screenshot
xdiskusage example screenshot xdiskusage example screenshot xdiskusage example screenshot
Click For xdiskusage screenshots

As you can see from the root screen shot that my root partition that I have 60GB used between my /var and /home directories. On closer inspection, the var has 40GB, 20GB in virtual machines and 20GB in the logs directory. 20GB’s of logs is quite a lot, this is where my problem is. After finding the problem I was able to backup my log files and remove them. I know that this can be done with shell scripts xdiskusage has helped me in the past so I thought I’d pass on the information. If you have a favorite utility or script what you use when you run out of disk space let me know!

Project URL: http://xdiskusage.sourceforge.net/

Graphical Shell Scripting

Filed under: Shell Script Sundays — TheLinuxBlog.com at 4:07 pm on Sunday, December 16, 2007

From all of my other previous shell scripting articles you can see that shell scripting is a very good way to get a task done or to automate. This is great but some times a little bit of that three letter acronym “GUI” is a nice touch. Well my friends, if you’ve been reading my articles and following my dialog examples then you are in luck. If not don’t worry, you can view all of the articles in the Shell Script section to the right.

The program to make your GUI’s for your shell scripts is the exact same thing as dialog except its graphical. Its called Xdialog. Once installed you basically use it the same way as dialog. If your thinking about upgrading a script thats written in dialog you might want to think twice because some window managers will display the windows differently.

Check out these Xdialog examples and corresponding screenshots:

Xdialog

Xdialog -yesno “Do you like Xdialog?” 5 50 && echo “Thats nice” || echo “Yea, some times its better to stick to CLI”

Xdialog ExampleXdialog Example

Fetching Online Data From Command Line

Filed under: Shell Script Sundays — TheLinuxBlog.com at 6:12 pm on Sunday, December 2, 2007

Shell Scripts can come in handy for processing or re-formatting data that is available from the web. There are lots of tools available to automate the fetching of pages instead of downloading each page individually.

The first two programs I’m demonstrating for fetching are links and lynx. They are both shell browsers, meaning that they need no graphical user interface to operate.

Curl is a program that is used to transfer data to or from a server. It supports many protocols, but for the purpose of this article I will only be showing the http protocol.

The last method (shown in other blog posts) is wget. wget also fetches files from many protocols. The difference between curl and wget is that curl by default dumps the data to stdout where wget by default writes the file to the remote filename.

Essentially the following do the exact same thing:

owen@linux-blog-:~$ lynx http://www.thelinuxblog.com -source > lynx-source.html
owen@linux-blog-:~$ links http://www.thelinuxblog.com -source > links-source.html
owen@linux-blog-:~$ curl http://www.thelinuxblog.com > curl.html

Apart from the shell browser interface links and lynx also have some differences that may not be visible to the end user.
Both lynx and links re-format the code received into a format that they understand better. The method of doing this is -dump. They both format it differently so which ever one is easier for you to parse I would recommend using. Take the following:

owen@linux-blog-:~$ lynx -dump http://www.thelinuxblog.com > lynx-dump.html
owen@linux-blog-:~$ links -dump http://www.thelinuxblog.com > links-dump.html
owen@linux-blog-:~$ md5sum links-dump.html
8685d0beeb68c3b25fba20ca4209645e links-dump.html
owen@linux-blog-:~$ md5sum lynx-dump.html
beb4f9042a236c6b773a1cd8027fe252 lynx-dump.html

The md5 indicates that the dumped HTML is different.

wget does the same thing (as curl, links -source and lynx -source) but will create the local file with the the remote filename like so:

owen@linux-blog-:~$ wget http://www.thelinuxblog.com
–17:51:21– http://www.thelinuxblog.com/
=> `index.html’
Resolving www.thelinuxblog.com… 72.9.151.51
Connecting to www.thelinuxblog.com|72.9.151.51|:80… connected.
HTTP request sent, awaiting response… 200 OK
Length: unspecified [text/html][ <=> ] 41,045 162.48K/s

17:51:22 (162.33 KB/s) – `index.html’ saved [41045]

owen@linux-blog-:~$ ls
index.html

Here is the result md5sum on all of the files in the directory:

owen@linux-blog-:~$ for i in $(ls); do md5sum $i; done;
a791a9baff48dfda6eb85e0e6200f80f curl.html
a791a9baff48dfda6eb85e0e6200f80f index.html
8685d0beeb68c3b25fba20ca4209645e links-dump.html
a791a9baff48dfda6eb85e0e6200f80f links-source.html
beb4f9042a236c6b773a1cd8027fe252 lynx-dump.html
a791a9baff48dfda6eb85e0e6200f80f lynx-source.html

Note: index.php is wget’s output.
Where ever the sum matches, the output is the same.

What do I like to use?
Although all of the methods (excluding dump) produce the same results I personally like to use curl because I am familiar with the syntax. It handles variables, cookies, encryption and compression extremely well. The user agent is easy to change. The last winning point for me is that it has a PHP extension which is nice to avoid using system calls to the other methods.

Putting it all together

Filed under: Shell Script Sundays — TheLinuxBlog.com at 12:40 pm on Sunday, October 28, 2007

Since the start of this column I’ve shown many techniques that can be used in Shell Scripts. Today I will show an example of what can be done when it is all put together.

To start off, I recently made a photo tagging script for a friend of mine. The purpose of this script is to basically gather information about the pictures such as the month, year and who are in the photo. It also allows people to upload new pictures. The information is stored in a MySQL database and photos are kept in one folder as a unique hash of the file.

The script that I am going to show this week basically hashes the filenames and creates the SQL code that is needed to import the records into the database.

The full script source code can be found here.

The first thing the script does is use getopts to put the options into variables. For more information on how to do this can be found in my blog post on Creating Script Parameters With getopts.

### Get passed options.
while getopts “:s:d:f:” ARG; do
case “${ARG}” in
s) dirSource=”${OPTARG}”;;
d) dirName=”${OPTARG}”;;
f) sqlDump=”${OPTARG}”;;
esac
done

Once the options have been put into variables the following is ran:

The only thing that is needed for this script to run is a path. All that the above does is make sure that a path was specified. If not it calls the function usage and exits the script.
Here is the usage function:

usage () {
echo -e “\nUsage: $0 -s [ -d -f ]\n”
echo -e “\t-s: specifies the source directory. (required)”
echo -e “\t-d: specify the destination.”
echo -e “\t-f: create an sql dump to import into MySQL.\n”
}

The main loop is very simular to my When Photoshop Fails article as it uses find. Except it finds all jpg files regardless of case. It does this by using [j,J][p,P][g,G] as the file mask.

find “${dirSource}” -name *.[j,J][p,P][g,G] | while read sourceFile; do

The following checks to make sure that dirName is not empty. The dirname command takes a file and will extract the directory name from it. So, if a directory to put the moved files in wasn’t given at run time, the directory that the new picture gets put in is the same as the old one.

### We don’t want to overwrite this var by mistake.
if [ -z "${dirName}" ]; then
dirName=`dirname “${sourceFile}”`
fi

This is the last part of the loop. It uses md5sum to create a hash of the file is piped to awk which is used to print the actual hash. It then moves the file to the directory that was created from above. Create SQL and cleanVar are both functions that are called to do another task.

fileHash=`md5sum -b “${sourceFile}” | awk ‘{print $1}’`
mv “${sourceFile}” “${dirName}/${fileHash}.jpg”
createSQL; cleanVar

The createSQL function is used in this script to take the hash and insert it into the table that the PHP web app uses. At the moment the script just creates the SQL. The importing has to be done manually.

createSQL () {
if [ -n "${sqlDump}" ]; then
echo “INSERT INTO \`pictures\` (\`img_name\`) VALUES(‘${fileHash}’);” >> “${sqlDump}”
fi
}

The cleaning function unsets fileHash which is the hash of the file that was just copied. It also does a check to see if the $dirName variable needs to be unset. It does this by checking to see if the dirName is the same as the directory name of the source file and unsets $dirName if it is.

### Clean up variables before next loop (precautionary).
cleanVar () {
unset fileHash

### We don’t want to clean this var unless we have to.
if [ "${dirName}" == "$(dirname ${sourceFile})" ]; then
unset dirName
fi
}

Its a pretty simple script and uses a lot of what I’ve gone over in previous posts. Heres the script in action (the directory that the pictures are to be copied to must exist.)

owen@the-linux-blog:$ ls
JPG-hasmove.sh NewPics Pictures
owen@the-linux-blog:$ ls Pictures/
1.jpg 2.jpg 3.jpg 4.jpg 5.jpg
owen@the-linux-blog:$ ls NewPics/
owen@the-linux-blog:$ ./JPG-hasmove.sh -s Pictures/ -d NewPics -f SQLFile.sql
owen@the-linux-blog:$ ls
JPG-hasmove.sh NewPics Pictures SQLFile.sql
owen@the-linux-blog:$ ls Pictures/
1.jpg 2.jpg 3.jpg 4.jpg 5.jpg
owen@the-linux-blog:$ ls NewPics/
294e8b2c5a9c4d61de67a166cd8e8e29.jpg 9b3270fc78fdf3920d0e44197da52944.jpg
6ef86ca6dcfb2467b46ed9d929fd4070.jpg f353aa857522586d08ad7956090c8566.jpg
8aa35639bc0d3e7abc23c0b70acce08d.jpg
owen@the-linux-blog:$ cat SQLFile.sql
INSERT INTO `pictures` (`img_name`) VALUES(’8aa35639bc0d3e7abc23c0b70acce08d’);
INSERT INTO `pictures` (`img_name`) VALUES(‘f353aa857522586d08ad7956090c8566′);
INSERT INTO `pictures` (`img_name`) VALUES(’294e8b2c5a9c4d61de67a166cd8e8e29′);
INSERT INTO `pictures` (`img_name`) VALUES(’6ef86ca6dcfb2467b46ed9d929fd4070′);
INSERT INTO `pictures` (`img_name`) VALUES(’9b3270fc78fdf3920d0e44197da52944′);

If you have a use for the script or this has helped you some please let me know, I always like to hear from people that find my posts useful. Until next time: Happy Shell Scripting!

Creating Dialogs with Dialog

Filed under: Shell Script Sundays — TheLinuxBlog.com at 5:48 pm on Sunday, October 21, 2007

Have you ever seen those pretty dialogs used in Shell Scripts such as the Slackware installation, the slackpkg program or even the NVIDIA driver installer? Well, my friends to display dialog boxes from shell scripts is very easy with… you guessed it – Dialog.

First of all, there are many different types of dialogs that you can create they are as follows: calendar, checklist, fselect, gauge, infobox, inputbox, menu, msgbox (message), password, radiolist, tailbox, tailboxbg, textbox, timebox, and yesno (yes/no).

This blog post is intended to be a primer on using dialog. More examples will be posted in future blog posts in the Shell Script Sunday’s column.

The simplest form of a dialog in a shell script is probably the msgbox. All this really does is displays text. To display text in a dialog you would do the following:

owen@the-linux-blog:$ dialog –msgbox “Hello from the Linux Blog!” 5 50

The numbers after the text in quotes are the widths and heights of the box. The minimum height that I like to use is 5. The width doesn’t really matter as long as it is big enough. It is good to keep the box sizes standard across a whole script because it gets annoying with constantly resizing boxes.
If the text in a message box is too long it will auto wrap around and give you a type of scroll bar. As follows:

owen@the-linux-blog:$ dialog –msgbox “Hello from The Linux Blog. This text is so long it wraps it to a New Line” 5 50

Dialogs can be canceled. Clicking Ok or pressing enter/return returns “true” and pressing escape or Ctrl+C returns a false.
The simple shell scripting syntax shown in Shell Scripting 101

is used for this:

owen@the-linux-blog:$ dialog –msgbox “Dialog Exit Example” 5 50 && echo “ok” || echo “false”

Another simple dialog example is the Yes/No box. The syntax for this is exactly the same as the msgbox example except instead of using –msgbox, –yesno is used. The difference between a msgbox and a yesno box is that there a two buttons. It is pretty obvious as to what they are labeled, but for those in the back, I’ve included an example and some screen shots anyway.

owen@the-linux-blog:$ dialog –yesno “Are you learning anything from this blog” 5 50 && echo “Yes, thanks Owen.” || echo “No, Write some better Linux Related Posts”

The Linux Blog - Dialog Example - Yes / No

Thats about all I have time for this week. Check back next week!

Shell Script Sundays

Filed under: The Linux Blog News — TheLinuxBlog.com at 12:06 am on Sunday, July 29, 2007

Well, I have been planning content for this blog for a while and I now have a rather healthy long list of Linux related subjects that I can get crunching on.

One of the new sections I am going to add is going to be on shell scripts. It will be called Shell Script Sundays. Expect useful and handy shell scripts that anyone can use to do specific tasks.

This is going to be a good way to show off how powerful the shell is and what can be achieved with one. I hope to educate and help people along the way. So tune in next week for the first ever edition of Shell Script Sundays and feel the power of the almighty shells.

« Previous Page