Linux Blog

Last 50 Characters of Each line

Filed under: Shell Script Sundays — TheLinuxBlog.com at 4:18 pm on Sunday, May 24, 2009

I got a question from a user called Bastiaan. He had found my site while searching for ‘cut from end of line Linux’ and landed on the Using cut – shellscript string manipulation article. I haven’t received a lot of feedback on it, but am happy with the feedback I have and the amount of visits it gets. As I’ve said before if no one else reads The Linux Blog I still use it as a reference, so I am glad people are finding it useful. Anyways, Bastiaan’s problem was he works in a University and has a file with A LOT of DNA records in it. He needed to grab the last 50 characters of each line, regardless of the line length. After some correspondence we came up with a solution.

I have experience in doing this sort of thing in other languages such as PHP but not bash. Here is what I came up with for bash:

cat find.txt | while read i; do echo $i | \
cut -b $((`echo $i | wc -c` - 50))-; done;

While this was really quick to write it is not the most efficient way in the world. It has to read each line, echo it out, calculate the length of the line, subtract 50 from it. Again, does the job but not very gracefully.

Bastiaan then had told me he reversed the whole file and then was processing that with cut. I have heard of tac, to reverse entire files, but not had never heard of rev. Using rev I assumed that he was running something like the following:

rev file.txt > rev_file.txt
cat rev_file.txt | cut -c -50 | rev

That will get you the last 50 characters from each line (well, really the first 50 of a reversed file) That works pretty good so the final solution was to try to stream line it a little bit so that it could be done in one step.

rev file.txt | cut -c -50 | rev > out.txt

So there you have it, if you’re looking to use cut to “cut” characters from the end of the line, the above will cut 50 characters off of the end. Obviously you can remove the last “> out.txt” to get the output on the screen.

Hope this helps some one, and thanks to Bastiaan for the question!

Bulk Editing Text Files

Filed under: Shell Script Sundays — TheLinuxBlog.com at 1:00 am on Sunday, May 10, 2009

A Co-Worker wanted to edit a number of files in a directory that contained a lot of files. Each file that needed to be edited contained a function that needed to be replaced. Since it was production data we did not want to do a backup and run a sed find and replace for all files and risk screwing something up we decided to use vi to edit a list of files. Here is what I came up with to do that:

vi `grep function\_name * -n |cut -d : -f 1 | uniq`

If it were me, I would not have wanted to type sed find and replaces and would have done something like this because I’m lazy and I like to live on the edge:

 grep function\_name * -n | cut -d : -f 1 | uniq | while read i; do cp $i $i-bak; sed 's/function_name/new_function_name/g' $i-bak > $i; done;

Rather than editing them with vi it makes a -bak file, and uses sed to replace function_name with new_function_name. It does this from the bak file into the original. Some may think it’s kind of scary not making a backup, but I figure the -bak file should be enough depending on the operation. Make a backup if you value your data though.

Screen Script, Part Deux

Filed under: Shell Script Sundays — TheLinuxBlog.com at 6:09 pm on Sunday, May 3, 2009

So, the last script for handling the screen didn’t quite work out as I had liked so I have stopped using it. Basically if you try to SCP something, it would flip out because there was no terminal. Also, if you were trying to do anything in X after logging in with SSH you would have to detach your screen session first, which could get annoying. I’ve came up with this snippet, its rather ugly but is intended to be run after you log in if you want to start a screen session. I’d like a solution that logs me in with this sort of dialog selection but also allows X and scp transfers to work if a selection is not made. Perhaps it could be done on a timer. Anyhow, here is the script:

screen -r $(dialog --stdout --menu Test 20 50 10 `echo $(screen -ls | grep \( | sed -e 's/\t/,/g' | cut -d , -f 2- | awk -F ',' '{print $1,$2}' | awk -F '.' '{print $1,$3}' | cut -d-f 1,3)`)

It errors if no selection was made but that can be changed pretty easily by adding some if logic. Perhaps I’ll tweak it and repost, or perhaps I’ll find a better solution. Who knows.

Reattach Screen Script

Filed under: Shell Script Sundays — TheLinuxBlog.com at 2:02 pm on Sunday, April 12, 2009

A friend of mine who happens to be an avid screen user sent me this snippet below:

### Reattach to a screen if one exists ###
if [[ $TERM != 'screen' ]] ; then
if [[ `screen -list | grep -v "No" | awk '$2 { print }' | wc -l` == 0 ]] ; then
screen
else
screen -dr
fi
fi

What this handy snippet does is looks for a screen session, if it finds one it detaches the running screen, and reattaches it(-dr) if it isn’t lucky enough to find one, then it just starts a session up for you. Its rather handy to put in your .bashrc file to auto launch a screen session. The only thing I have modified for my use is replacing -dr for -x to enable me to reattach the screen without detaching the session I may have had open on another terminal. It works pretty well, although when you open a new “screen” CTRL-a + c, the tab doesn’t show up on the other sessions until you change to it, or cycle through them. It isn’t a big deal and could even be a local configuration issue. Anyway, enjoy this snippet and as always let me know if you found it useful.

Basic Scripting Snippets Repository

Filed under: Shell Script Sundays — TheLinuxBlog.com at 11:04 pm on Sunday, April 5, 2009

Everyone has their own favorite snippets of code that they use. With command line fu you can share your snippets and look at snippets that other people have posted. I’ve known about it for quite some time and frequently check it out, it’s really shaping up nicely.


Command-Line-Fu is the place to record those command-line gems that you return to again and again.

Delete that bloated snippets file you’ve been using and share your personal repository with the world. That way others can gain from your CLI wisdom and you from theirs too. All commands can be commented on and discussed – digg-esque voting is also encouraged so the best float to the top.

I don’t think that you should delete your snippets file, just in-case you don’t have the intrawebs but I think you could get something out of browsing through the highest rated if nothing else.

APC Access Temperature Query and Conversion. (2 of 2)

Filed under: Shell Script Sundays — TheLinuxBlog.com at 6:55 pm on Sunday, March 29, 2009

This second part of the script APC Access Temperature Query Script and its been a long time coming. Basically this script is the part that runs as a cron and will e-mail me if the temperature goes over a certain threshold. Once it returns to normal it e-mails me again. It has the option to send a text message to me via my SMS gateway, but it is commented out.

#!/bin/bash
 
temp=$(/home/linux/bin/temp f)
threshold=76
 
if [ "`echo \"$temp > $threshold\" | bc`" == 1 ]; then
echo $(date +%s) $temp >> /home/linux/thermal-over.log
echo "High Temp";
 
if [ "$(cat temp.txt)" == "norm" ]; then
echo "Sending E-Mail, High Temp";
echo "Current Temperature Is: $(/home/linux/bin/temp f)" | mail -s "Thermal Overload" owen@linuxblog                #echo "Current Temp Is: $(/home/linux/bin/temp f)" | mail -s "Thermal Overload" mynumber@cingularme.com
echo "high" > temp.txt
fi
elif [ "`echo \"$temp < $threshold\" | bc`" == 1 ]; then
echo "Low Temp";
 
if [ "$(cat temp.txt)" == "high" ]; then
echo "Temp Resumed, Sending E-Mail";
echo $(date +%s) Resumed at: $temp | mail -s "Thermal Normal" owen@linuxblog
echo "norm" > temp.txt
fi
 
fi
 
echo $(date +%s) $temp >> /home/linux/thermal.log

When I first wrote the script, I did not do any temperature checking. I found out that I needed to when I came back one morning with a bunch of emails that I needed to delete. Its pretty simple to figure out, temp.txt holds a value that is either norm or high. It gets switched when the temperature changes, this will in turn stop it from e-mailing me repeatedly. Once the temperature drops it flips it back. It will still e-mail if your temperature fluctuates between 75 and 77 which can be annoying, but you can adjust the threshold with the variable and set it to what you need. Thankfully our chiller has been fixed and I no longer have to worry about the temperature, but it still runs on a cron just in case.

APC Access Temperature Query and Conversion. (1 of 2)

Filed under: Shell Script Sundays — TheLinuxBlog.com at 10:54 pm on Sunday, February 1, 2009

This week I present to you; dearest reader part of a script I wrote to monitor the temperature on APC brand UPS’s. The script requires the apcupsd package to be installed and properly configured.

Here is the script:

#!/bin/bash
 
f () {
echo $(echo "($(/usr/sbin/apcaccess | grep ITEMP | cut -d : -f 2 | cut -d \  -f 2)*1.8)+32" | bc);
}
 
c () {
echo "$(/usr/sbin/apcaccess | grep ITEMP | cut -d : -f 2 | cut -d \  -f 2)"
}
 
case "$1" in
c)
c
;;
f)
f
;;
b)
echo "$(/usr/sbin/apcaccess | grep ITEMP | cut -d : -f 2 | cut -d \  -f 2)" C
echo $(echo "($(/usr/sbin/apcaccess | grep ITEMP | cut -d : -f 2 | cut -d \  -f 2)*1.8)+32" | bc) F;
;;
*)
echo $"Usage $0 {f|c|b}"
;;
esac

The script uses basic bash, grep, cut and bc. It requires only one input, and that is f, c or b. f for Fahrenheit, c for Celsius and b for both. The second part of the script is a cron, with some more basic bash. It write a log and e-mails me if the temperature goes over a certain threshold, and it e-mails me again once the temperature has been resumed. I will post the cron script next week as it is a major portion. The tricky part of the cron was making it e-mail me only once.

Until next time, Happy Scripting!

APCUPSD

thinkpad_handler ACPI Script for Lenovo T61 hotkeys

Filed under: Shell Script Sundays — TheLinuxBlog.com at 2:54 pm on Sunday, January 18, 2009

Lenovo T61 Thinkpad_Handler ACPI hotkey

This weeks shell scripting article showcases the modifications I have made to the thinkpad_handler script that normally resides in /usr/lib/acpid on an OpenSuse distribution. For any one not using a Lenovo T61 this may or may not be useful, however anyone interested in shell scripting can take a look at the file and see the simplicity of shell scripts that are used in production environments.

Here is my modified thinkpad_handler script

One of the reasons I chose to post this was because an OpenSuse upgrade broke my brightness keys support, so I figured it would be another good place to back it up while providing a reference. My modifications are very minimal at the moment, I still have a lot of work to do.

Here are my minor modifications:

Line 102 to add support for suspendon Fn+F4

Line 127 to eject /dev/cdrom on Fn+F9

I used to have a special script for Fn+F8 which enabled me to toggle support for the touchpad and track point mouse. Until I find that script I’ll endure excruciating mental anguish every time I accidentally hit the touchpad. I’ll have to make an update to this script once I change all of the other hotkeys for OpenSuse on the Lenovo T61 or other similar Lenovo notebook computer. I should also probably upgrade to OpenSuse 11.1 and see if that makes a difference in the keyboard shortcuts.

Until next week, happy scripting!

Crontabs 101

Filed under: Shell Script Sundays — TheLinuxBlog.com at 11:42 pm on Sunday, January 4, 2009

Although not necessarily classified as shell scripting its self cron’s are very useful to any Linux administrators arsenal. The ability to run tasks at a specific interval is a great way to schedule things to run later or when the system load is lower. Many applications use crontab to schedule tasks so its hard to say what yours will look like.

crontab -l

will list all of the cron jobs scheduled for the currently logged in user mine has an entry for kpodder

# (Cron version V5.0 -- $Id: crontab.c,v 1.12 2004/01/23 18:56:42 vixie Exp $)
#KPodder entries
0 0 * * * kpodder.sh -c "/home/owen/.kde/share/apps/kpodder/" -s "global.casts" -o "/home/owen/podcasts" -d 0
#KPodder End

The first five fields are to tell the task when to run. They are in the following order: minute, hour, day, month, day of the week. Asterisks are used to say any valid value and a forward slash can be used to make intervals such as five minutes, hours, days or months (*/5). A comma can be used for or values, so if used as 2,4,6 the cron would run at 2, 4 or 6 o’clock. Dashes are used for time spans, If you have an 8-5 work day you can use 8-17 if in the hours field.

Next, the sixth field is the actual program to run. It will look in the $PATH for the user, but for safety’s sake, I try to use the full path if possible. In my example of the kpodder script there are many arguments. I only really use simple crons and the number of arguments here seems rather excessive.

To edit the crontab do crontab -e. This will edit the current users crontab. If you are root and wish to edit a naughty cron from another user you use -u and specify the user.

Thats pretty much all there is to it. I’d love to here tips and how much people love/hate cron and why.

Happy Shell Scripting New Year,

- Owen.

To Script, or not To Script

Filed under: Shell Script Sundays — TheLinuxBlog.com at 3:23 pm on Sunday, December 21, 2008

I hate doing manual repetitive labor. There is something about it that just bores me to death. Perhaps this is why I like shell scripting so much. To me it is one of the most useful things about the terminal. All the time I’m working I’m thinking about automation.

“Can this be automated?”
“Can I improve this process?”

Most of the time tasks can be automated in some fashion, and a simple shell script can be written. Its very useful. One of the problems about being able to shell script is the self discipline to say, no I’ll do this by hand. The reason being is that a lot of the time the learning curve and potential problems of automating a process are costly upfront, therefore taking you more time in the long run. So, next time you think about using shell scripting for automation ask yourself if it is really worth it. Depending on your workload, perhaps on the third of fourth time round it will be worth considering it. If you’re not busy and are up for a challenge, by all means knock yourself out!

Check if SELinux is Enabled

Filed under: Shell Script Sundays — TheLinuxBlog.com at 10:41 am on Monday, December 8, 2008

This weeks (now late) Shell Script Sundays (posted on Monday) article is a short one on a check to see if SELinux is enabled. While SELinux has some great security enhancements it can present a number of problems in applications and shell scripts alike. There is a simple utility that comes with many Linux distributions called “selinuxenabled”

selinuxenabled exits with a status of 1 if it is not enabled and 0 if it is. Zero normally means false but in this case since it is an exit status it is an exception. So, if you need to do a quick check, you may just run selinuxenabled. You will quickly find that it returns nothing. To figure out the exit status for your quick check, put an ampersand (&) at the end, and it will tell you the exit status. eg:

[root@thelinuxblog.com ~]# selinuxenabled &
[1] 28417
[1]+  Exit 1                  selinuxenabled

As we can see from the example above SELinux is disabled.

To use selinuxenabled in your scripts you would use it like any other command. Refer to Shell Scripting 101 for some more information. selinuxenabled can also be used in your scripts to make sure that selinux is enabled, which can be useful if you are trying to do security audits across multiple machines.

« Previous PageNext Page »