Linux Blog

Remove lines that are in another file

Filed under: Shell Script Sundays — TheLinuxBlog.com at 6:52 pm on Sunday, February 7, 2010

Remove lines from a file that exist in another fileI had an issue this week where I needed to remove lines from one file if they existed in another file. Looking back it was frustrating as such a task should be simple.

I tried all sorts of things. Differencing the two files and using grep to grab the lines I wanted. Whatever I tried just did not produce the expected results. Thanks to a buddy I found the solution which ended up being to sort the two files before using diff.

Example:
Assuming two files exist, File_1 and File_2. File_1 containing lines with a, b, c and. File_2 containing b and d. If we want to remove b and d from File_1 because they exist in File_2 you could use something like the this:

1
2
3
4
5
6
7
8
9
10
11
12
owen@linuxblog:~$ cat File_1.txt
a
b
c
d
owen@linuxblog:~$ cat File_2.txt
b
d
 
owen@linuxblog:~$ diff File_1.txt File_2.txt | grep \< | cut -d \  -f 2
a
c

That’s all fine and dandy until File_2.txt contains the same lines in a different order. Running the same command produces different results. See Below:

1
2
3
4
5
6
7
8
owen@linuxblog:~$ cat File_2.txt
d
b
 
owen@linuxblog:~$ diff File_1.txt File_2.txt | grep \< | cut -d-f 2
a
b
c

The solution as noted above is to use sort before hand and then difference them:

1
2
3
4
owen@linuxblog:~$ sort File_1.txt >> File_1-sorted; sort File_2.txt >> File_2-sorted;
owen@linuxblog:~$ diff File_1-sorted File_2-sorted | grep \< | cut -d \  -f 2
a
c

Obviously the example has been simplified, when dealing with thousands of lines the sort could take a while. With that said I’m sure there are more efficient ways to achieve the same results. I wouldn’t doubt there being a command better suited to do this. Have at it in the comments.

Searching for multiple strings with grep

Filed under: Shell Script Sundays — TheLinuxBlog.com at 9:36 pm on Sunday, August 23, 2009

Sometimes when using grep it’s nice to be able to search for more than one string in a file. It’s actually pretty easy to do, if you can remember the simple syntax. Basically, you pipe the terms together in double quotes and escape the pipe.

grep "gnome\|kde" install.log

The above example will search for gnome or KDE. It works for more more than one term, and also works with the invert match -v option, to exclude lines that include gnome or kde.

Todays article extra short, and late, but better late and short than none at all I guess.

CPU Flags and Meanings

Filed under: General Linux — TheLinuxBlog.com at 11:38 pm on Tuesday, August 11, 2009

Having a brain that is somewhat comparable to a sift for remembering acronym’s, I’m always asking a friend which (I, probably incorrectly call) CPU extension is for virtualization. So I figured I’d blog about this topic. This page CPU feature flags and their meanings clearly explains each of the flags that may be on your machine, which you can check what your CPU supports by typing:

cat /proc/cpuinfo | grep flags

Secure Virtual Machine. (AMD’s virtualization extensions to the 64-bit x86 architecture, equivalent to Intel’s VMX, both are also known as HVM in the Xen hypervisor.)

So to answer my own question SVM and VMX are what to look for when considering a CPU for virtualization. My laptop supports VMX, but my desktop doesn’t. Considering I’d rather do my virtualization on my desktop, I’ll have to upgrade my CPU. Now all I need to do is find out where to find this information for AMD processors and hope I can get a CPU for my outdated socket type. I don’t even think that it is AM2 :(

Bulk Editing Text Files

Filed under: Shell Script Sundays — TheLinuxBlog.com at 1:00 am on Sunday, May 10, 2009

A Co-Worker wanted to edit a number of files in a directory that contained a lot of files. Each file that needed to be edited contained a function that needed to be replaced. Since it was production data we did not want to do a backup and run a sed find and replace for all files and risk screwing something up we decided to use vi to edit a list of files. Here is what I came up with to do that:

vi `grep function\_name * -n |cut -d : -f 1 | uniq`

If it were me, I would not have wanted to type sed find and replaces and would have done something like this because I’m lazy and I like to live on the edge:

 grep function\_name * -n | cut -d : -f 1 | uniq | while read i; do cp $i $i-bak; sed 's/function_name/new_function_name/g' $i-bak > $i; done;

Rather than editing them with vi it makes a -bak file, and uses sed to replace function_name with new_function_name. It does this from the bak file into the original. Some may think it’s kind of scary not making a backup, but I figure the -bak file should be enough depending on the operation. Make a backup if you value your data though.

Reattach Screen Script

Filed under: Shell Script Sundays — TheLinuxBlog.com at 2:02 pm on Sunday, April 12, 2009

A friend of mine who happens to be an avid screen user sent me this snippet below:

### Reattach to a screen if one exists ###
if [[ $TERM != 'screen' ]] ; then
if [[ `screen -list | grep -v "No" | awk '$2 { print }' | wc -l` == 0 ]] ; then
screen
else
screen -dr
fi
fi

What this handy snippet does is looks for a screen session, if it finds one it detaches the running screen, and reattaches it(-dr) if it isn’t lucky enough to find one, then it just starts a session up for you. Its rather handy to put in your .bashrc file to auto launch a screen session. The only thing I have modified for my use is replacing -dr for -x to enable me to reattach the screen without detaching the session I may have had open on another terminal. It works pretty well, although when you open a new “screen” CTRL-a + c, the tab doesn’t show up on the other sessions until you change to it, or cycle through them. It isn’t a big deal and could even be a local configuration issue. Anyway, enjoy this snippet and as always let me know if you found it useful.

APC Access Temperature Query and Conversion. (1 of 2)

Filed under: Shell Script Sundays — TheLinuxBlog.com at 10:54 pm on Sunday, February 1, 2009

This week I present to you; dearest reader part of a script I wrote to monitor the temperature on APC brand UPS’s. The script requires the apcupsd package to be installed and properly configured.

Here is the script:

#!/bin/bash
 
f () {
echo $(echo "($(/usr/sbin/apcaccess | grep ITEMP | cut -d : -f 2 | cut -d \  -f 2)*1.8)+32" | bc);
}
 
c () {
echo "$(/usr/sbin/apcaccess | grep ITEMP | cut -d : -f 2 | cut -d \  -f 2)"
}
 
case "$1" in
c)
c
;;
f)
f
;;
b)
echo "$(/usr/sbin/apcaccess | grep ITEMP | cut -d : -f 2 | cut -d \  -f 2)" C
echo $(echo "($(/usr/sbin/apcaccess | grep ITEMP | cut -d : -f 2 | cut -d \  -f 2)*1.8)+32" | bc) F;
;;
*)
echo $"Usage $0 {f|c|b}"
;;
esac

The script uses basic bash, grep, cut and bc. It requires only one input, and that is f, c or b. f for Fahrenheit, c for Celsius and b for both. The second part of the script is a cron, with some more basic bash. It write a log and e-mails me if the temperature goes over a certain threshold, and it e-mails me again once the temperature has been resumed. I will post the cron script next week as it is a major portion. The tricky part of the cron was making it e-mail me only once.

Until next time, Happy Scripting!

APCUPSD

Logging Sessions to Twitter

Filed under: Linux Software — TheLinuxBlog.com at 12:01 am on Wednesday, August 20, 2008

If you follow me on Twitter you would know I asked everyone if there was anything that they wanted me to write about. @Ben_Marvin responded and asked about logging the commands you type to Twitter.At first, I thought that this could be done with history, which it probably can be, but does everything you type really have to be echo’ed to Twitter? I don’t think that you’d have many friends, Twitter would probably hate you and you’d most likely hit your maximum requests per hour pretty quickly.The Script command is another option, and this works quite well for this purpose. Read the script man page to find out more about this program. It basically (when ran) takes the I/O from your terminal and logs it to a file. Its a very handy utility.So, how do we get this data into Twitter? First of all, Twitter doesn’t allow very long posts so echoing out the data that the commands you type is not really practical. The best way to do it is to use script to log the session, exit the session and grep for the “]0\;” string for stuff you typed & not the responses.Here is the code:

script; grep ]0\; typescript 

You can then copy and paste it to your favorite Twitter application or pipe it to a scrubbing script to remove the formatting and do the correct HTML stuff, then post it via the http interface. Either way, it can be done even though I don’t think it really should.

Apache “Directory index forbidden by Options directive:”

Filed under: Quick Linux Tutorials — TheLinuxBlog.com at 9:29 pm on Friday, June 27, 2008

This is a common problem with the Apache web server. It is considered a security risk to show the directory contents by default. Especially those in the root directory. The way to fix this is pretty simple, you first need to make sure that there are no -Index directives in any of the configuration files. In your httpd configuration directory grep with a line number for Options like so:

grep -n Options *

If you don’t see any -Indexes then its possible there are included files. Check these directories if you know them, if you don’t grep your configuration file for the Includes:

grep -n Include *

If you’re still getting the test page or a permission denied error after removing the directive that disables Indexes then it may be a permissions issue. Apache needs +x access for all users to enable directory listings. Change the permissions and it should work.

On Fedora there is a file called welcome.conf in the conf.d directory. This can be removed and if you have Options All set, then you should be good to go. On other distributions like Slackware it is easier to accomplish. I wouldn’t recommend allowing any sort of directory listing in a production environment but in my development case, where I only allow access to my test server on a per host basis this is not a big problem.

Also, remember to restart the server after making changes.

Hope this helps, if it doesn’t be sure to drop a comment!

Suspend Scripts for the Toshiba Tecra M2

Filed under: Quick Linux Tutorials,Shell Script Sundays — TheLinuxBlog.com at 12:15 am on Sunday, March 30, 2008

As you may know if you are a regular reader I own a Toshiba Tecra M2. One of the things that annoyed me was I had to turn the brightness up every time my computer came out of standby mode. A fix for this is to adjust the brightness every time the computer comes out of standby mode.

The script is intended to be run under cron. I have mine set up to suspend after 5 minutes of the lid being closed.

if [ $(cat /proc/acpi/button/lid/LID/state | sed 's/state:      //') == "closed" ]; then
VAR=$(cat /proc/acpi/toshiba/lcd | sed 's/brightness:              //' grep -v levels);
sudo su -c "echo mem > /sys/power/state";
if [ $VAR -eq 1 ]; then
ACTION=ADD;
elif [ $VAR -eq 7 ]; then
ACTION=SUB;
else
ACTION=ADD;
fi;
if [ $ACTION == "ADD" ]; then
VAR=$(($VAR + 1));
else
VAR=$(($VAR - 1));
fi;
sudo su -c "echo brightness:$(echo $VAR) > /proc/acpi/toshiba/lcd";
fi;

I run this with the following cron entry:

*/5 * * * * sh hibernate.sh

The script first checks the current brightness. If the brightness is currently 1 or 7 it adjusts the mathematic operation so that when the laptop is opened the brightness is adjusted. Basically if the brightness is one, it adds one. If the brightness is 7 or any other value it subtracts one. This is currently working out quite well for me. I don’t know how useful this is to any body else, unless you happen to have a Toshiba that is doing the same thing but it should give you a good overall idea of how to perform basic mathematic operations in bash.

Snippet: Keeping SSH Running

Filed under: Quick Linux Tutorials,Shell Script Sundays — TheLinuxBlog.com at 12:53 am on Sunday, November 4, 2007

I wrote a post not so long back called Bringing The Internet Up After Failure that explained how I was restarting my network services after the internet went down.

Shortly after this I was remotely working when I thought it would be a good idea to restart my SSHD to enable X11 forwarding. After running the script that normally restarts the service I tried to reconnect. Unfortunately the service never restarted after being stopped. This is not a good situation for any one to be in so I added something like this to my cron along at the end of my network services script:

ps ax | grep \usr\/sbin\/sshd | grep Ss && echo "SSHD Running" || echo "Starting SSH"; sudo /etc/rc.d/rc.sshd start

The only difference between the version I am running and the version above is that I don’t echo anything out. All the command does is use the ps command and grep for usr/sbin/sshd then grep for the STAT field of sS. I do this because sshd shows up in the process list as shown below:

owen@the-linux-blog$ ps ax | grep sshd
3463 pts/0    R+     0:00 grep sshd

The rest is self explanitory, it echo’s “SSHD Running” or sudo starts the SSHD.

If SSHD ever terminates on me or those of us who run this snippet we will be able to rightfully regain access to our systems, Hurray!

Comment spam vent and an idea to combat it

Filed under: General Linux,The Linux Blog News — TheLinuxBlog.com at 10:48 am on Thursday, September 27, 2007

Ok I just have to vent a little. Spam Spam Spam. Spam bots and blog spammers in general are stupid. Why would you spam a site that doesn’t even have comments on blog posts? I don’t get it. I would love to get my hands on the code of these spam bots to see how they are working. It would be so simple for them to actually load the page and grep for the comment link. That way they could stop wasting mine and their time by spamming this site. All comments are marked as spam at the moment since that’s all I’m getting. If some one wants to send me a comment they could go trough the trouble of creating a <FORM> to send one but I think they would just send me mail at owen -at- <thelinuxblog.com>. This generally means that I can delete all of the comments I get as I doubt any one is really going to create a form just to send me a comment.

IP addresses are being recorded and I’ll continue to monitor the situation. Eventually I might do a security scan on repeat offenders and let the ISP of the host know what’s what. It would be nice to catch the spammer but I will settle for the knowing that the owners of the Zombie PC have been warned about the consequences and might send their computer in for repair.

After writing this post I have an idea for a Linux Blog project. The Linux Blog can be the first to submit data. Basically I propose a plugin for WordPress that would monitor comments and send the IP’s of those that were spam (either flagged manually or automatically) to a host. This host would log all of the IP’s submitted and when appropriate take action against them.
This reminds me of a project that I saw a presentation on at CarolinaCon 05. Except this project was targeted at bot nets and aimed to be malicious back or ban the host / subnet completely. This would not be necessary for this project; just a simple mail merge application written in Linux would work.
I suppose the real first step is to see if this application already exists. I’m sure I’m not the first person to think of this.
I guess I need to come up with a quirky name for the project and a plan. It goes without saying that the platform will be Linux and will use open source applications to achieve the final goal. Once research has been done I can either try to add it to my never ending list of projects to complete or find a team of developers willing to work on it.