Linux Blog

Using cut – Shellscript string manipulation

Filed under: Shell Script Sundays — TheLinuxBlog.com at 1:21 am on Sunday, August 26, 2007

This post is designed to be a refresher, reference or quick intro into how to manipulate strings with the cut command in bash.

Some times its useful to take the output of a command and reformat it. I sometimes do this for aesthetic purposes or tor format for use as input into another command.
Cut has options to cut by bytes (-b), characters (-c) or fields (-f). I normally cut by character or field but byte can come in handy some times.
The options to cut by are below.

N          N’th byte, character or field, counted from 1
N-         from N’th byte, character or field, to end of line
N-M      from N’th to M’th (included) byte, character or field
-M         from first to M’th (included) byte, character or field

The options pretty much explain themselves but I have included some simple examples below:
Cutting by characters (command on top, output below)

echo "123456789" | cut -c -5
12345

echo "123456789" | cut -c 5-
56789

echo "123456789" | cut -c 3-7
34567

echo "123456789" | cut -c 5
5

Sometimes output from a command is delimited so a cut by characters will not work. Take the example below:

echo -e "1\t2\t3\t4\t5" |cut -c 5-7
3 4

To echo a tab you have to use the -e switch to enable echo to process back slashed characters. If the desired output is 3\t4 then this would work great if the strings were always 1 character but if anywhere before field 3 a character was added the output would be completely changed as followed:

echo -e "1a\t2b\t3c\t4d\t5e" | cut -c 5-7
b 3

This is resolved by cutting by fields.
Cutting by fields

The syntax to cut by fields is the same as characters or bytes. The two examples below display different output but are both displaying the same fields (Fields 3 Through to the end of line.)

echo -e "1\t2\t3\t4\t5" | cut -f 3-
3 4 5

echo -e "1a\t2a\t3a\t4a\t5a" | cut -f 3-
3a 4a 5a

The default delimiter is a tab, if the output is delimited another way a custom delimiter can be specified with the -d option. It can be just about any printable character, just make sure that the character is escaped (back slashed) if needed. In the example below I cut the string up using the pipe as the delimiter.

echo "1|2|3|4|5" | cut -f 3- -d \|
3|4|5

One great feature of cut is that the delimiter that was used for input can be changed by the output of cut. In the example below I change the format of the string from a dash delimited output and change it to a comma.

echo -e "1a-2a-3a-4a-5a" | cut -f 3- -d – --output-delimiter=,
3a,4a,5a

Formatting with Cut Example

Sometimes certain Linux applications such as uptime do not have options to format the output. Cut can be used to pull out the information that is desired.
Normal up-time Command:

owen@the-linux-blog:~$ uptime
19:18:40 up 1 day, 22:15, 4 users, load average: 0.45, 0.10, 0.03

Time with up-time displayed:

owen@the-linux-blog:~$ uptime |cut -d , -f 1,2 | cut -c 2-
19:19:36 up 1 day, 22:22

For the above example I pipe the output of uptime to cut and tell it I want to split it with a comma , delimiter. I then choose fields 1 and 2. The output from that cut is piped into another cut that removes the spaces in front of the output.
Load averages extracted from uptime:

owen@the-linux-blog:~$ uptime |cut -d , -f 4- | cut -c 3-
load average: 0.42, 0.10, 0.03

This is about the same as the previous example except the fields changed. Instead of fields 1 and 2 I told it to display fields 4 through the end. The output from that is piped to another cut which removes the three spaces that were after the comma in "4 users, " by starting at the 3rd character.
The great thing about cutting by fields is that no matter if the field length changes the data stays the same. Take the example below. I now have 17 users logged in which would have broke the output if I had used -c (since there is an extra character due to a double digit number of users being logged in.)

owen@the-linux-blog:~$ uptime
19:25:11 up 1 day, 22:28, 17 users, load average: 0.00, 0.06, 0.04

owen@the-linux-blog:~$ uptime |cut -d , -f 4- | cut -c 3-
load average: 0.00, 0.06, 0.04

That just about covers everything for the cut command. Now you know about it you can use cut to chop up all types of strings. It is one of the many great tools available for string manipulation in bash. If you can remember what cut does it will make your shell scripting easier, you don’t need to memorize the syntax because all of the information on how to use cut is available here, in the man pages and all over the web.

Rotating Videos in Linux

Filed under: General Linux,Linux Software — TheLinuxBlog.com at 11:17 pm on Friday, August 24, 2007

Like many people I receive video media from family members and friends on line. Sometimes for I receive videos that are shot in portrait mode instead of landscape. I used to find this quite annoying until I figured out how to rotate a video under Linux. All that is needed to rotate the video is mencoder. It comes bundled with the king of media players MPlayer.

Command to rotate a video is:

mencoder -ovc lavc -vop rotate=1 -oac copy input.mpg -o output.mpg

The rotate=1 can be replaced with whatever option best suits your needs. Rotating video options are below:

0    Rotate by 90 degrees clockwise and flip (default).
1    Rotate by 90 degrees clockwise.
2    Rotate by 90 degrees counterclockwise.
3    Rotate by 90 degrees counterclockwise and flip.

I have broken down the command and options below for those that are interested.

-ovc Output Video Codec. This is what codec mencoder should use when creating the video. The command above uses the libavcodec. This is known for quality. “mencoder -ovc help” will display all of the video codecs available

-vop still works but has been replaced with -vf. It is used to setup a chain of video filters in our case it is used to rotate each frame. See the above table to find out what rotate mode you need. Up-side-down videos can be rotated by doing rotate=1 twice.

-oac Output Audio Codec. If you would like to specify an audio codec to use this is where it should be done. “mencoder -oac help” will show all of the available audio codecs. Choosing an audio time adds encoding time but it can greatly reduce or increase file size. I use the copy codec to copy the exact sound from the original first. Then if I wish to reduce file size I can reduce the quality or change the codec afterwards.

input.mpg is the input file and will have to be changed for the file that you wish to rotate. The -o option is used to specify the output file. This must not be left out or mencoder will give an error and your file will not be written.

Now that your done reading all about how to rotate a video with Linux you can give it a try. By having a command line application rotating a whole directory of images can be done in minimal time without human interaction. Try rotating a hundred movies without user interaction in Windows Movie Maker!

Select Statements in Bash

Filed under: Shell Script Sundays — TheLinuxBlog.com at 10:40 pm on Sunday, August 19, 2007

Shell scripts often need a menu so that the user can interact with the script and choose options. An easy way to so this is with the “select” statement.

Syntax:

select selection_var [in list]; do
statements (can use selection_var)
done

Example:

echo "########################################";
echo "#  SELECTS EXAMPLE - TheLinuxBlog.Com  #";
echo "########################################";
echo "#        Please Choose an Option       #";
echo "########################################";
 
select selection_var in Number_1 This_is_Choice_2 Exit; do
case $selection_var in
Number_1 )
echo "You picked $selection_var or #$REPLY"
;;
This_is_Choice_2 )
echo "You picked $selection_var or #$REPLY"
;;
Exit )
echo Goodbye!
break
;;
* )
echo "Invalid Selection"
;;
esac
done

This bash select example has three choices. The first two are just examples, the third exits the script. There is a fourth case but it is not a choice, it tells the user that they have entered an invalid selection. I recommend using a case inside of a select because it will make life easier when adding onto a script. The $REPLY variable is returned from the select statement as a means of knowing what number was pressed. The $REPLY variable can be used in the case statement but I avoid doing so as all of the case blocks will have to be rearranged every time a new option is added.

The select statement in bash is very easy to implement and it can add a whole new range of functionality into your scripts. So try them out and look out for them in future shell scripts from The Linux Blog.

Avoid The Apple Keyboard: Sexy Brushed Aluminum Alternatives

Filed under: General Linux — TheLinuxBlog.com at 10:51 pm on Friday, August 17, 2007

I feel a bit goofy tonight so I figure I’ll make an attempt at a post that might make a few people giggle and catch a few others on fire. This post is going to be about Apples new keyboard, or as I like to call it the iKeyboard.

Don’t get me wrong I do like the look of Apples new keyboard but brushed aluminum keyboards are nothing new. To be quite honest for a company being largely based on style I’m quite surprised that they hadn’t released one sooner. The previous white plastic one was a fantastic keyboard to use and there was also the bluetooth version. The new one is a complete redesign. The plastic has gone and it has a new modified layout. They removed the Apple icon from the Open Apple button and its now been replaced with the word command, they also moved the function key to where the insert button is suppose to be and changed the num pad layout a bit. Now with all of the layout changes and being used to a standard keyboard I don’t think I would be able to use one properly let alone justify buying one.
Having a fancy keyboard is nice. An elegant design feel and a quiet keyboard is what in my books makes a nice keyboard. Most standard keyboards can’t touch the look of the new Apple keyboard so I set on a quest to find the ultimate Linux keyboard.

The Linux Friendly Keyboard Requirements I came up with:

  • Must be compatible with Linux out of the box.
  • Must be brushed aluminum to match the look of the iKeyboard and clash with my other ugly hardware.
  • Must be thin and light but also strong so that I can’t break it when I smash it because my code isn’t working or WordPress fails to format my blog post correctly.
  • Wireless would be a plus but is not needed.

After some searching I found three keyboards that match my criteria. I have my findings listed below:

Speed-link Keyboard
This keyboard from [speed-link] is a pretty slick design. Its pretty like Apples new keyboard, it has media shortcuts that I will never use and they also re-arranged the keys so I’ll have to re-learn a keyboard layout. Thats perfect! Just kidding, they have an U.S layout available and there is a review of it here: [altgamer.org]
Its about the same price as the Apple keyboard and its not completely aluminum the base is actually made of plastic but I don’t know too many people that would actually inspect your keyboard (unless their Apple fan boys.)

The Hiper Clavier Aluminum Keyboard
This keyboard can be purchased from [ipcrepublic.com] for a mere $37. Thats pennies on the Apple keyboard. Its been out for at least two years so it has been tested longer than the Apple keyboard. I don’t know about you, but I like to have my products tested before I buy them, why not buy a product thats been on the market for a while? Like the other keyboards this one also has media shortcuts that nobody ever really uses.
A review can be read here: [phoronix.com] Like the Speed-link keyboard the base is also made of plastic.

Enermax Aurora
Enermax makes keyboards?
Yes they do and this is probably the best alternative I have found. It can be brought from [newegg] for about the same price as the Apple. It has the USB ports on it like the Apple but also has audio jacks which the Apple doesn’t have. Granted its not flat but it has an elegant design to it and also a standard layout. It comes in two colors aluminum and black aluminum. Considering its made of one solid piece of aluminum its quite light. It also doesn’t sit flat which is a plus as I like a raised keyboard.
A full review can be read here: [digitaldingus.com]

Since I’m picky, I would actually have to go to the store and test a keyboard before I purchased one. I have tried many keyboards and to be honest I’m perfectly happy with my $5 keyboard that I purchased on sale. Its the keyboard that makes a difference, its how its used. Its clear that any skilled Linux user is going to have the best keyboard for their needs since they can modify the layout as needed. This also allows the consumer to use the keyboard the way that they want it to work not how a hardware manufacturer has decided it should work.

Perhaps this is the best keyboard for Linux users: [thinkgeek.com] I would love to be able to type in DVORAK and have one of these keyboards! With this keyboard nobody would ever know your top secret shortcuts. It would also give you the flexibility of customizing it as you needed without any markers on the keys. This would stop people saying

“Well it has the windows logo on it, why did it lock your screen?”.

Ubuntu & Gentoo Servers compromised

Filed under: General Linux — TheLinuxBlog.com at 11:00 pm on Wednesday, August 15, 2007

The case of the Ubuntu servers being breached [wiki.ubuntu.com]
Missing security updates and system administrators not running updates on servers is a problem. I don’t know why they didn’t do any updates past Breezy. They suggest that it was because of problems with network cards and later kernels but I don’t get it. Since when do software updates for an operating system have anything to do with what kernel is running? If there is a problem with hardware support for the network card you have two choices. The first is to fix the driver yourself or pay some one to do it. The second is to replace the network card to a better supported device. Both situations could be costly but it would get the problem fixed and five of the servers wouldn’t have been taken down at the same time.
If the kernels were configured correctly, the boxes probably wouldn’t of even had to have been rebooted.
Running FTP instead of a more secure version is not so bad unless they were running accounts with higher privileges than guest or using system accounts. In which case thats just stupid.

The Gentoo Situation [bugs.gentoo.org]
Apparently there is a problem in the packages.gentoo.org script. The bugzilla article goes into deeper explanation but basically there is some pretty unsafe code which could have allowed any one to run any command. I understand that the code is old but it probably should have been audited at some point. The problem would have stuck out like a soar thumb if looked at by a python coder and they probably would have fixed it, or at least suggested a fix. The problem was found on Tuesday the 7th. All of the infra- (I assume they mean infrastructure?) guys were at a conference last week so they couldn’t work on it. It still seems that if they were at the conference until midnight on the 12th they would still have been able to put up a coming back soon placeholder on the packages site by now. Hey, if they put some pay per click ads up there maybe they will get some additional funds during the down time. I would like to see what products would be pushed thru the advertising on that one. I believe that they could have reduced the downtime by releasing the code for the packages.gentoo.org site as open source or by asking for help from developers to review and upgrade the code as needed.

Its not strange for web servers get hacked. They get hacked all the time but who’s fault is it in the open source community? I really think that there is a problem in the community when it comes to situations like this but the blame can’t be placed on any one person. I would offer any assistance I could into getting these situations resolved but its not as easy as that. There has to be a certain level of trust for those working within a project. If they gave out keys to their servers to anyone the servers probably would have been compromised a long time ago. I hope that the affected sites can pull them selfs together and get back up and running as normal. It seems that Ubuntu did not have complete down time, but the Gentoo site is still down and there is no indication of when it will be back up.

For, While and Until Loops in Bash

Filed under: Shell Script Sundays — TheLinuxBlog.com at 3:45 pm on Sunday, August 12, 2007

Normally in a shell script it is desirable to do something repetitive.
I have touched on some loops in other posts but now would like to go over them in a bit more detail. Each of the examples in this post are intended to give an introduction to looping in bash.

For Loops
For loops allow you to repeat a section of code a number of times. Its very similar to other languages syntax but works a little differently. The for loop in bash only allows you to give a fixed list of values to loop over. A good way to remember how a for loop works is “For each of the dishes: clean and dry.”
For Syntax:

1
2
3
4
for i [in list]
do
statements [use $i]
done

For Example:

1
2
3
4
5
6
for x in 1 2 3
do
echo "Number: $x"
done
 
echo "Finished!"

This is a very simple script that just counts to 3 and then prints “Finished!”

While and until Loops
In essence while and until are the same in bash. The titles are pretty much self explanatory. A while loop would be explained in real life as “While the sink is still full: wash dishes” and a until loop would be “Until the sink is empty: Wash dishes.”
While and Until Syntax:

1
2
3
until/while [condition] do
statements
done

Example of a While loop:

1
2
3
4
5
6
count=1
while [ $count -lt 10  ]; do
echo $count
let count=$count+1
done
echo "Finished!"

Basically this loop will loop over the code while the count variable is less than 10. If we didn’t put the let statement in the script it would get stuck in the loop causing the user to press CTRL+C to end the script.

Doing the same thing can be done in a until loop except the condition has to be modified to get the same result.
Until example:

1
2
3
4
5
6
count=1
until [ $count -gt 9 ]; do
echo $count
let count=$count+1
done
echo "Finished!"

Now that you’ve figured out how to loop over something its probably a good idea to know how to stop the loop.
All that needs to be done to stop a loop is:

break

Break Example:

1
2
3
4
5
6
7
8
for x in 1 2 3 4 5
do
if [ $x = 3 ]; then
echo "Number is 3. Quitting"
break;
fi
echo "Number: $x"
done

This is a very easy to follow example. Its the same as the basic for loop except that if x is 3 it will stop the loop. This example has no real practical purpose. Since its a for loop the number 3 could just have been omitted.

Real World For Loop Example
Looping over all files in /etc and printing all of those that match “grep conf” and putting them in quotes.
The code to do this in a loop is:

1
2
3
4
for x in $(ls /etc -1|grep conf);
do
echo "$x"
done

The situation for many bash scripts is that there is normally a shorter way to do something. Take the Real World For Loop Example in this tutorial the same results can be achieved with:

x=$(ls /etc |grep conf); echo "$x"\n

This will get the job done but a loop may be better for esthetic purposes or for additional logic.

Low cost PC’s in China. Is Linux about?

Filed under: General Linux — TheLinuxBlog.com at 11:56 pm on Monday, August 6, 2007

I was listing to NPR late yesterday evening on my way home from work. There was an interesting tech talk going on about computer manufacturers in China. Notably Dell and Lenovo (IBM). They are both offering PC’s for around $199. The Dell ships with Windows, but I am unclear on what the Lenovo ships with. Since the dirt cheap Dell ships with a monitor, keyboard & mouse but its competitor the Lenovo desktop only ships with a keyboard & mouse. A television is used for the monitor. It makes me wonder what OS the Lenovo is running.

Apparently the Lenovo shipped in April. There are rumors that Microsoft, one of Lenovos partners has been cooperating with the company to deploy Linux machines. Microsoft pushing Linux in China? Thats a interesting concept, but in my opinion there is nothing wrong with it. It does make me wonder if even Microsoft has come to the conclusion that the cost of Windows is too high.

There is a huge market for low end PC’s in China the NPR radio show reported. With 800 million potential customers who wouldn’t attempt to make a profit if given the chance? What better way to maximize profit then to use an operating system with a such a low cost to maintain? Given the potential customers are mostly rural farmers that earn only $600 a year its possible that they could still afford a computer if they saved up for a while. The show also went into say that the computer could be a powerful tool for the farmers. I agree with this in that they would be better able to predict when to buy and sell produce. They could check the weather to make sure that grounds are soft for planting crops the next day.

The low end computers would be a good asset to own. I am proud of the Linux operating system if the rumors are true. I wish the best to Lenovo and Dell and hope they are successful in their business in China.

Decision making in Bash If / Then / Else statements and Cases

Filed under: Shell Script Sundays — TheLinuxBlog.com at 3:44 pm on Sunday, August 5, 2007

There comes a time in every shell script where a decision has to be made.

To make a decision in bash the following if then else syntax can be used:

if [ condition ]
then

statements

[ elif [ condition ]

then

statements ]

[ else

statements ]

fi

Anything in non bold [ brackets ] is optional, and they should not be typed but the bold in them is required. Anything in bold must be typed. Statements are generally other commands or flow control statements.

To give an example of how to use bash if then else statements in the real world take the following scenario:

A system administrator has a strict habit of firing people that have too many .png files. He checks the systems regularly and makes sure that nobody has too many. The following script will display a message depending on the number of .png’s that are in the directory.

#!/bin/bash
gif_count=$(ls | grep .png | wc -l)
echo “Number of PNG’s: $gif_count”
if [ $gif_count -lt 10 ]
then
echo “He will be happy, you have less than 10 files”
elif [ $gif_count -gt 10 ] && [ $gif_count -lt 20 ]
then
echo “Consider deleting some files”
else
echo “you have too many files, DELETE SOME BEFORE HE FINDS OUT!”;
fi

Using Cases.

Cases are similar to if statements in that they match a expression and perform a task if the pattern is matched.

bash case syntax:

case expression in

pattern1 ) statements ;;

pattern2 ) statements ;;

esac

This is fairly simple and some people find this easier than doing if statements for simple logic. Take the following real world example:

The system administrator has recently gone on a bigger power trip than before. Since people got wise about using png’s and started saving images in other file formats he is now monitoring png’s gif’s and jpg’s. To combat the problem, you can use a case to count how many files you have of each type. (This is intended as an example, there are many ways to accomplish this task, this is just to demonstrate how cases work)

#!/bin/bash
#set all variables to 0
png=0
gif=0
jpg=0

# start loop
for wc in $(ls); do

case “$wc” in
*png ) let png=$png+1 ;;
*gif ) let gif=$gif+1 ;;
*jpg ) let jpg=$jpg+1 ;;
esac

# end loop
done

echo “Png’s $png”;
echo “gif’s $gif”;
echo “jpg’s $jpg”;

There you have it, two ways to make basic decisions in bash. Just figure out what you want to do then use an if then else, or a case statement to do the logic. I myself prefer if statements over cases as they make more sense to me and I find it easier to perform logic within ifs.

Programs used in this post
ls, echo, grep, wc

Shell Scripting 101

Filed under: Shell Script Sundays — TheLinuxBlog.com at 2:27 am on Sunday, August 5, 2007

In my last shell scripting article I made a script to manipulate images with imagemagick. The script was only five lines long and has saved me endless hours of manual work. The script can be adapted to do anything repetitive on any types of files.

Since this is my first official shell scripting post I am going to introduce some common basic shell scripting. This is intended for people that want to learn the basics of shell scripting, so those that have experience already may want to skip this post.

Basic applications used in this tutorial
In order to teach a little about shell scripting, I need to use a couple of programs other than bash its self. The ones that I will use in this post are:
echo
This program displays a line of text
cat
Cat likes to read files and print what it finds out.
grep
Search through files with an expression to find a line that matches the pattern. Sounds complicated but really is not.
less
A paging utility is used to display data. Its kind of like opening a file in notepad Windows except with most utilities more can be done.
less is probably my favorite pager. There are many of these including tail and more. Less is more is the tag line of this pager because it does more than less but is the same thing.

Basic standard output redirecting
Often abbreviated as stdout or stderr for standard out and standard error.
This is normally used to take the output of one application and direct it somewhere. You can overwrite or append to the stdout. The symbol to do this is > to overwrite and >> to append.
stdin example:

echo "This is the new text" > File_To_Be_Written
echo "This is appended" >> File_To_Be_Written

“This is the new text” (without quotes) is redirected to File_To_Be_Written and overwrites the file.

“This is appended” is appended to the end of the file that was just created. If the file did not exist before hand, the file would have been created.

Redirecting Errors
Redirecting errors is achieved the same way as redirecting stdout except it has a 2 in front of the greater than symbol.
stderror example:

echo "This is the new text" 2> File_To_Be_Written
echo "This is appended" 2>> File_To_Be_Written

To append to stderror just think “To redirect to stderror is to redirect with two” Meaning, to redirect stderror is just to redirect with two in front. Cheesy, but its easy to remember.

Pipes
Piping is a common technique I normally use in all of my scripts. The way this works is to send the stdout of one program to the stdin of another.
Example:

cat Big_File | grep search | less

This basically reads the Big_File and then grep’s the file for lines that have search in them. It then pipes that output into the paging utility less.

Lists
A list is just a sequence of pipes that are separated by control operators. Basically
what that means is you have operators that do specific things depending on return codes

&
This operator puts the command into the background, gives a return status of 1 and moves on to the next task.
Example:

owen@amd:~$ echo "tails" > heads
owen@amd:~$ cat heads & echo "done"
tails
[1] 15385
done
[1]   Done                    cat heads

;
The ; operator waits until the process is finished before moving onto the next task.
Example:

owen@amd:~$ cat heads; echo "done"
tails
done

&&
command1 && command2
command2 will only be executed if command1 returns an exit status of zero
Example:

owen@amd:~$ cat heads && echo "done"
tails
done

Note this works because cat has returned the correct exit status

owen@amd:~$ cat eads && echo "done"
cat: eads: No such file or directory

Notice that this does not work because cat did not return the correct exit status.

||
command1 || command2
command2 will be executed if command1 returns a non zero return code.
Example:

owen@amd:~$ cat heads || echo "Did not work"
tails
Did not work is not printed because cat exited properly
owen@amd:~$ cat eads || echo "Did not work"
cat: eads: No such file or directory
Did not work

Did not work was printed because the eads file does not exist.

Throwing it all together:

echo "This is whats in the output file" > output;
echo "Now that has done, lets add something to the end" >> output;
cat output > normal_output && echo "This is not the error log. Press Q to exit this pager and view the error log" >> normal_output;
cat put >> normal_output 2> error_output;
cat normal_output | less ; cat error_output  | less

This short example basically echos some text to the output file. It then adds some more to the end of the output file. Next it reads from the output file into the normal_output file. If that executes successfully it goes on to echo some instructions to the normal_output file. The script moves on to read the put file (which does not exist) and directs the error of that command to a error_output file. After that, it reads the normal output and sends it to the less pager. The first prompt has instructions for ending the pager, and once the pager has been stopped it reads the error log.

Variables
Variables are used to temporarily store data in memory. To read a variable you can just just the echo command. An example of setting and reading a bash variable is below:

owen@amd:~$ variable='This is my variable'
owen@amd:~$ echo $variable
This is my variable

Flow Control
There are ways of controlling how a script is going to work. This is the logic part of the script, the intelligence if you will. There are many different ways to control flow with bash. The following topics should get you started:
Bash If / Then / Else statements & Cases
Bash Loops For, While and Until
Creating menus in Bash with select

Where to from here?
If you followed this example and understand everything so far then you have got your feet wet in shell scripting. You should now know a little on how to read/write files, redirect output and use conditional operators.
If you would like more help there are plenty of resources on-line and probably on the computer you are using to try the examples. Check the man pages for more help on the utility and “man bash” for bash help. If your looking for more tutorials go to Google and type “bash shell scripting tutorial”.
If you have a specific question that needs answering leave a comment or contact me.

KDE Wall Paper Contest

Filed under: General Linux — TheLinuxBlog.com at 8:56 am on Thursday, August 2, 2007

A wall paper contest was posted this morning on Digg and many other blogs. The contest is simple, upload wall papers that you created and you have the possibility of getting them included in the next release of KDE.

After viewing the blog and homepage of the link submitted in the article, I was a bit skeptical as to whether this was just a scheme for a blogger to get some publicity, so I did some research. Turns out that Riccardo Laconelli is not listed in the main credits for the KDE project, but is listed in the KDE lists [lists.kde.org]. It also seems as if he is the developer of the Oxygen theme and has committed various icons to the KDE project.

Should you participate in this contest?

I believe if you have any artistic skills that you should design a wall paper and submit it to the competition. For your efforts you may have your wall paper included one of the most popular window managers and it will make you feel good for contributing. If you enter the competition and your entries are not specific to KDE I urge you to also post your wallpapers on your own site/blog/photo gallery and other Linux wall paper sites. Some people like to be able to use wall papers on other window managers. If I had any artistic skills I would most defiantly be entering this competition myself.

KDE Wallpaper Contest is [http://pd.ruphy.org/]

Hard drive set up and tuning.

Filed under: General Linux — TheLinuxBlog.com at 11:49 am on Wednesday, August 1, 2007

One question people often ask me is
“What is the best way to partition a hard drive?”
With graphical installations becoming more popular often hard drive partitioning is overlooked.

If you already have a working installation you can check how your system is currently partitioned by running the command:

bash-3.1# fdisk -l /dev/hdX (depending on your drive)

If you are happy with the performance of your system then reinstalling is not really needed. If you would like to possibly gain a performance increase then you may want to consider reinstalling with a better partitioning method.

Repartitioning your drive
The first thing that I recommend doing when deciding how to partition the hard drive is to figure out what the system is for. Is the system going to be a Linux workstation, desktop, server or application. Knowing this information is not necessary as sometimes computer often have multiple roles but it is good start into setting up your hard drive properly.
There are many ways that a hard drive can be partitioned. Depending on the role of the computer I have different partitioning methods.

bash-3.1# fdisk -l /dev/hda
Disk /dev/hda: 60.0 GB, 60011642880 bytes
255 heads, 63 sectors/track, 7296 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
 
Device Boot Start End Blocks Id System
/dev/hda1 1 124 995998+ 82 Linux swap
/dev/hda2 7236 7296 489982+ 5 Extended
/dev/hda3 125 7235 57119107+ 83 Linux
/dev/hda5 * 7236 7296 489951 83 Linux

Firstly, my hda1 is swap. I always make my swap disk the first primary partition on the hard drive. This is because its the fasted to access. If your currently running an install, check to make sure your swap is hda1.

My hda2 is not used, this is just a preference that I have, to my knowledge there is nothing wrong with using hda2 as root. I just like to use /dev/hda3 as my root. My root file system for this particular machine is about 95% of the total drive. The other 5% is used for my swap partition and my /boot/ partition. I used /dev/hda5 for my boot partition as this is not so important as it only gets accessed at boot time and when copying kernels to the partition. You don’t have to have a separate boot partition, but I like to just in case I want to export it as an NFS share.

Once your happy with your disk partitioning scheme, the next thing that needs to be done in the partitioning manager is to choose a file system. The file system type for Linux Swap partition is 82 and 83 for the Linux type.

Choosing a filesystem and formatting it.
Most installations will take care of the formatting and adding swap, you just have to choose what type of file system you want to use. I would advise to use the ext3 or Reiser file systems for your root and boot partition. Ext2 is depreciated and has been replaced with ext3. If ext2 is still in your distributions setup it is for legacy reasons and should not be used. I like to use ext3 because its more mature, but I have no problem with using Reiser. Choose one and install your distro as usual.

Basic Tuning
Once your install has finished and is booted it is good to check to make sure your disk is being used correctly.
I like to double check this because I want to make sure that I am getting the most out of my system. Some times certain distributions don’t set up the drive correctly for stability issues (I’m not pointing any fingers here).

To do this I use the hdparm tool. Quote [man pages]:

DESCRIPTION
hdparm provides a command line interface to various hard disk ioctls
supported by the Linux SATA/PATA/SAS "libata" subsystem and the older
IDE driver subsystem. Some options may work correctly only with the
latest kernels.

The first thing to do is run hdparm /dev/hdX (depending on your drive)
I have changed some options for the purpose of this tutorial but here is my current setup:

bash-3.1# hdparm /dev/hda
/dev/hda:
multcount = 0 (off)
IO_support = 0 (default 16-bit)
unmaskirq = 0 (off)
using_dma = 0 (off)
keepsettings = 0 (off)
readonly = 0 (off)
readahead = 256 (on)
geometry = 16383/255/63, sectors = 117210240, start = 0

Turning DMA on.
Notice, that IO_support and using_dma are set to off.
For performance reasons this should change.
As I know my controller supports DMA I should turn this on. If your unsure if your controller supports DMA you should refer to your motherboard specifications or try it at your own risk.

bash-3.1# hdparm -d1 /dev/hda
/dev/hda:
setting using_dma to 1 (on)
using_dma = 1 (on)

If you just try it without knowing if DMA is supported, it will either error or may become unstable. Whenever installing a new system, if DMA or 32-bit transfers are turned off, I like to keep the system running for a while and copy large files around to make sure there are no stability issues.

Turning 32-bit transfers on
Now that DMA has been turned on successfully and the system is still stable, I will turn 32-bit transfers on. The same applise for 32-bit transfers as to DMA. The command is the same as before but with the -c1 switch.

bash-3.1# hdparm -c1 /dev/hda
/dev/hda:
setting 32-bit IO_support flag to 1
IO_support = 1 (32-bit)

Now the basic hard drive tuning has been set up I will run stability tests for a while and make sure the settings work as desired. If the system is not stable, you can try using 32-bit transfers or DMA independently and try to find out which one is guilty or if they both work separately. Which one to use should probably be measured with a bench mark test. This can be done with a benchmarking script, or just by copying a file and measuring the time it takes with both.

Final Notes
Check out your hard drive settings and partitioning scheme and make sure you are happy with the way its set up. If you have the time and want to tweak your system these basic steps should help you with your quest of having lean mean Linux machine.