Linux Blog

I’ll be honest, I’m no expert on optimizing shell scripts. I’m hoping that readers will chime in with their tips / experiences. With that being said I do have a few tricks up my sleeve from hands on experience with code optimization using other languages.

Use time to get a baseline

Any performance testing normally starts with a baseline. It can be hard to tell which direction you need to go, when you don’t know where you started. Using the Linux time command, you can get an baseline which you can use to track any performance increases.

Consider changing types of loops

I’m not sure how much this makes a difference in bash, but it can make a huge difference in other languages. It is also important to make sure there is no repetition of operations within the clause, because this too will get repeated on every iteration.

Remove unneeded output

Backgrounding

If you have processes that may take a while, you can always use backgrounding to put them in the background while you perform other operations. This may or may not work depending on the situation.

Change Shell

Remove needed Comments and lines

Most compiled languages remove comments so that they don’t appear in the binary, since bash is an interpreted language this is not the case. If there is a huge number of comments in a script it can cause some sluggishness as it interprets each line.

Use sed to remove them, keep another copy or branch for development and distribute the de-commented version.

Database Vs. Files

Using a database versus. files can give you a performance boost. Think about it, writing files takes up processing time, it then takes time to read those files. Processing and performing lookups on data using text files is slow, using a database such as MySQL or SQLite will work and give you results.

Inserting into SQLite can take longer than writing to a file (at least the way I was doing it) which may or may not be a problem depending on what you’re trying to do. The resulting file is also larger than a plain text file, which I assume is due to indexes.

Here is an example of reading from a text file versus reading from sqlite:

sqlite sqlite.db “select * from test”;

While this may not seem significant, try selecting lines 1,10,100,150,200 and 9000-10000 delimited by pipes.

Commands Used:

time awk ‘NR==1;NR==10;NR==100;NR==150;NR==200;NR==9000,NR==10000;’ bash.txt | sed ‘s/ /\|/’
time sqlite sqlite.db “SELECT * from test WHERE one IN (1,10,100,150,200) OR one BETWEEN 9000 AND 10000”

Not much of a difference here, but if you’re working with real world data and need to select certain rows it can make a huge difference take the example of looking for the string Hello within 10000 rows:

time grep Hello bash.txt
time sqlite sqlite.db “SELECT * from test where two LIKE ‘%Hello%'”

Use the right tool for the job

You may try to hammer a nail into the wall, but a screwdriver or drill will work much better. Using the correct application is key, knowing what tool is best to use and for what purpose is the tricky part.

Take a look at cut vs sed vs awk:

Perhaps this one isn’t fair. The sed expression doesn’t do it properly. If some one with uber sed-fu can do it a better way, let me know in the comments and I’ll bench mark it. This is the closest to awk and cut that I came up with so this is whats represented for now.

Use Better Syntax

The same goes for a lot of other programs including grep:

Process in Parallel

If performance is a huge concern, you could consider performing operations in parallel. Projects like distcc achieve awesomely fast compile times when distributing load over a number of hosts. You’d think that using this kind of technique with shell scripts would result in a considerable performance boost. From my testing it seems to produce varying results, you can use the techniques outlined here: http://pebblesinthesand.wordpress.com/2008/05/22/a-srcipt-for-running-processes-in-parallel-in-bash/ to see if your scripts can benefit from parallel processing.

Consider Changing Languages

This may be sacrilegious to die hard scripters but I’ve written before about when not to script it, this was more along the lines of why make something so complex when a shell script can do it? When performance matters is the answer.

Simple test to echo hello 10,000 times