Linux Blog

Last 50 Characters of Each line

Filed under: Shell Script Sundays — TheLinuxBlog.com at 4:18 pm on Sunday, May 24, 2009

I got a question from a user called Bastiaan. He had found my site while searching for ‘cut from end of line Linux’ and landed on the Using cut – shellscript string manipulation article. I haven’t received a lot of feedback on it, but am happy with the feedback I have and the amount of visits it gets. As I’ve said before if no one else reads The Linux Blog I still use it as a reference, so I am glad people are finding it useful. Anyways, Bastiaan’s problem was he works in a University and has a file with A LOT of DNA records in it. He needed to grab the last 50 characters of each line, regardless of the line length. After some correspondence we came up with a solution.

I have experience in doing this sort of thing in other languages such as PHP but not bash. Here is what I came up with for bash:

cat find.txt | while read i; do echo $i | \
cut -b $((`echo $i | wc -c` - 50))-; done;

While this was really quick to write it is not the most efficient way in the world. It has to read each line, echo it out, calculate the length of the line, subtract 50 from it. Again, does the job but not very gracefully.

Bastiaan then had told me he reversed the whole file and then was processing that with cut. I have heard of tac, to reverse entire files, but not had never heard of rev. Using rev I assumed that he was running something like the following:

rev file.txt > rev_file.txt
cat rev_file.txt | cut -c -50 | rev

That will get you the last 50 characters from each line (well, really the first 50 of a reversed file) That works pretty good so the final solution was to try to stream line it a little bit so that it could be done in one step.

rev file.txt | cut -c -50 | rev > out.txt

So there you have it, if you’re looking to use cut to “cut” characters from the end of the line, the above will cut 50 characters off of the end. Obviously you can remove the last “> out.txt” to get the output on the screen.

Hope this helps some one, and thanks to Bastiaan for the question!

Shell Script Flow Control – my most refered to articles

Filed under: Shell Script Sundays — TheLinuxBlog.com at 2:38 pm on Sunday, November 9, 2008

This week I do not have time to write a full fledged article on shell scripting, so I am going to some flow control and logic articles that I wrote that I read the most.

One script that I continuously refer to is one of the first shell scripting articles I wrote titled “When Photoshop Fails.” The reason I refer to this article is because it describes a couple of looping techniques and a loop that works with spaces in filenames. For those interested but do not want to read the whole thing here is the loop in short:

find * -iname "*" | while read i; do echo "$i"; done

The next article I’m linking to is the one I wrote on loops: http://www.thelinuxblog.com/for-while-and-until-loops-in-bash/ it outlines some of the basic looping techniques. What I fail to mention in this article is that seq can be used to generate sequences of numbers for the [in list] section. The following is how you would create a loop to loop from 1 to 10 echoing out each number:

for i in `seq 1 10`; do echo $i; done;

Select Statements in Bash is exactly what the title implies. How to implement select statements in bash scripting. Also known as switch’s and case’s in other languages many will be familiar with this sort of logic. Even though I do not refer to it as often as some of the other articles I think it is worth of a mention in this list.

Decision making using if statements is something that every shell scripter is bound to come across. While not exactly the most in depth article on bash if then else statements it is a good start for any one wanting to learn more.

I love dialog’s and it just so happens that dialog and xdialog do a great job of making dialogs for your shell scripts. Creating Dialogs with Dialog has some basic uses of dialog. While it is not exactly logic or flow control, it can be used to prompt the user for input or just to pretty things up a little. It just so happens that it has an X front end called xdialog which does a good job too.

This is not all of the articles I refer to, but its the ones I wrote. If any one else has a list of articles they refer to that are bookmarked that I wrote or not I’d be interested to see them. Just paste them in a comment.

Reworking Shell Scripts

Filed under: Shell Script Sundays — TheLinuxBlog.com at 7:02 am on Sunday, August 24, 2008

To me shell Scripts are all about automation, their primary purpose is to get stuff done. If you have a one liner, snippet or other script you use on a regular basis, have you thought about how you could rework it for it to become more in handy?

Lets take last weeks snippet from this column. It was a simple one liner to reconnect to a host. Now, I knew when I posted this article that it was a helpful snippet of code. Now, how can this script be adapted to be a neat utility that we use on a regular basis? Over the next few week’s we’ll find out.

The first thing that I will note on is that this script or shell snippet is a pain to remember. Does a script save you time if you can’t remember how it works? Is it worth the hassle? Not exactly. So, in order to make this snippet a little better the first thing we are going to do is add something that it needs: parameters. Adding parameters to shell scripts is actually easy, much easier than adding parameters to some other languages that we wont mention. although this script does not use it getopts can be used. I’ve covered how to do this with getopts in other posts. Just do a site search (located at the bottom of the right bar for getopts.)

So, here is the modified script that automatically reconnects to a host by using ping and SSH:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
#!/bin/bash
 
# Sleep Time Default: 15 seconds
STIME=15
 
# Set a default user up
USERNAME=LinuxBlog
 
#usage function
usage () {
echo -e "Usage: $0 host [user] [Sleep Time]"
}
 
# display usage if no host is specified
[ -z $1 ] && { usage && exit 1; }
 
# set the variables
[ $1 ] && { HOST=$1; }
[ $2 ] && { USERNAME=$2; }
[ $3 ] && { STIME=$3; }
 
# trying:
echo -e "host: $HOST \nuser: $USERNAME \ndelay: $STIME"
 
while ! ping -W 1 -c 1 $HOST 2>&1 >/dev/null; do true; done && echo "Successful waiting: $STIME secs"; sleep $STIME; ssh $USERNAME@$HOST

Now that you have that done, all you need to do is give the file a name (I called mine ssh_auto) and put it in a folder in your path. Use the filename and parameters defined in the script to connect to the host.

The next shell scripting article I demonstrate how you can further rework shell scripts to better suit your needs.

For, While and Until Loops in Bash

Filed under: Shell Script Sundays — TheLinuxBlog.com at 3:45 pm on Sunday, August 12, 2007

Normally in a shell script it is desirable to do something repetitive.
I have touched on some loops in other posts but now would like to go over them in a bit more detail. Each of the examples in this post are intended to give an introduction to looping in bash.

For Loops
For loops allow you to repeat a section of code a number of times. Its very similar to other languages syntax but works a little differently. The for loop in bash only allows you to give a fixed list of values to loop over. A good way to remember how a for loop works is “For each of the dishes: clean and dry.”
For Syntax:

1
2
3
4
for i [in list]
do
statements [use $i]
done

For Example:

1
2
3
4
5
6
for x in 1 2 3
do
echo "Number: $x"
done
 
echo "Finished!"

This is a very simple script that just counts to 3 and then prints “Finished!”

While and until Loops
In essence while and until are the same in bash. The titles are pretty much self explanatory. A while loop would be explained in real life as “While the sink is still full: wash dishes” and a until loop would be “Until the sink is empty: Wash dishes.”
While and Until Syntax:

1
2
3
until/while [condition] do
statements
done

Example of a While loop:

1
2
3
4
5
6
count=1
while [ $count -lt 10  ]; do
echo $count
let count=$count+1
done
echo "Finished!"

Basically this loop will loop over the code while the count variable is less than 10. If we didn’t put the let statement in the script it would get stuck in the loop causing the user to press CTRL+C to end the script.

Doing the same thing can be done in a until loop except the condition has to be modified to get the same result.
Until example:

1
2
3
4
5
6
count=1
until [ $count -gt 9 ]; do
echo $count
let count=$count+1
done
echo "Finished!"

Now that you’ve figured out how to loop over something its probably a good idea to know how to stop the loop.
All that needs to be done to stop a loop is:

break

Break Example:

1
2
3
4
5
6
7
8
for x in 1 2 3 4 5
do
if [ $x = 3 ]; then
echo "Number is 3. Quitting"
break;
fi
echo "Number: $x"
done

This is a very easy to follow example. Its the same as the basic for loop except that if x is 3 it will stop the loop. This example has no real practical purpose. Since its a for loop the number 3 could just have been omitted.

Real World For Loop Example
Looping over all files in /etc and printing all of those that match “grep conf” and putting them in quotes.
The code to do this in a loop is:

1
2
3
4
for x in $(ls /etc -1|grep conf);
do
echo "$x"
done

The situation for many bash scripts is that there is normally a shorter way to do something. Take the Real World For Loop Example in this tutorial the same results can be achieved with:

x=$(ls /etc |grep conf); echo "$x"\n

This will get the job done but a loop may be better for esthetic purposes or for additional logic.