Linux Blog

Last 50 Characters of Each line

Filed under: Shell Script Sundays — TheLinuxBlog.com at 4:18 pm on Sunday, May 24, 2009

I got a question from a user called Bastiaan. He had found my site while searching for ‘cut from end of line Linux’ and landed on the Using cut – shellscript string manipulation article. I haven’t received a lot of feedback on it, but am happy with the feedback I have and the amount of visits it gets. As I’ve said before if no one else reads The Linux Blog I still use it as a reference, so I am glad people are finding it useful. Anyways, Bastiaan’s problem was he works in a University and has a file with A LOT of DNA records in it. He needed to grab the last 50 characters of each line, regardless of the line length. After some correspondence we came up with a solution.

I have experience in doing this sort of thing in other languages such as PHP but not bash. Here is what I came up with for bash:

cat find.txt | while read i; do echo $i | \
cut -b $((`echo $i | wc -c` - 50))-; done;

While this was really quick to write it is not the most efficient way in the world. It has to read each line, echo it out, calculate the length of the line, subtract 50 from it. Again, does the job but not very gracefuly.

Bastiaan then had told me he reversed the whole file and then was processing that with cut. I have heard of tac, to reverse entire files, but not had never heard of rev. Using rev I assumed that he was running something like the following:

rev file.txt > rev_file.txt
cat rev_file.txt | cut -c -50 | rev

That will get you the last 50 characters from each line (well, really the first 50 of a reversed file) That works pretty good so the final solution was to try to stream line it a little bit so that it could be done in one step.

rev file.txt | cut -c -50 | rev > out.txt

So there you have it, if you’re looking to use cut to “cut” characters from the end of the line, the above will cut 50 characters off of the end. Obviously you can remove the last “> out.txt” to get the output on the screen.

Hope this helps some one, and thanks to Bastiaan for the question!

Bulk Editing Text Files

Filed under: Shell Script Sundays — TheLinuxBlog.com at 1:00 am on Sunday, May 10, 2009

A Co-Worker wanted to edit a number of files in a directory that contained a lot of files. Each file that needed to be edited contained a function that needed to be replaced. Since it was production data we did not want to do a backup and run a sed find and replace for all files and risk screwing something up we decided to use vi to edit a list of files. Here is what I came up with to do that:

vi `grep function\_name * -n |cut -d : -f 1 | uniq`

If it were me, I would not have wanted to type sed find and replaces and would have done something like this because I’m lazy and I like to live on the edge:

grep function\_name * -n | cut -d : -f 1 | uniq | while read i; do cp $i $i-bak; sed ‘s/function_name/new_function_name/g’ $i-bak > $i; done;

Rather than editing them with vi it makes a -bak file, and uses sed to replace function_name with new_function_name. It does this from the bak file into the original. Some may think it’s kind of scary not making a backup, but I figure the -bak file should be enough depending on the operation. Make a backup if you value your data though.

Fedora 11 Beta Release

Filed under: General Linux — TheLinuxBlog.com at 11:14 am on Tuesday, March 31, 2009

The Fedora Project Announced today that they were releasing the Beta version of Fedora 11. Stupidly I went and downloaded it. After installing from the live CD which I had never done before (It went really smoothly) I had realized I’d installed the Alpha release. Nice!

So now I’m trying to figure out how to update from the Alpha to the Beta and reading over the Fedora 11 Beta Release Notes. A lot has changed and I really wanted to give ext4 a shot but I guess I’ll just wait till the stable comes out.

Whats old is New!

Filed under: General Linux,Linux Hardware — TheLinuxBlog.com at 1:41 am on Tuesday, March 10, 2009

There was a display in a local library showcasing “What’s old is new again” Basically showing the similarities between new and older newspapers and the cartoons in them. It may have something to do with the NY Times tour, but maybe not. All I know is it was there the other day, and now I go to write about it – GONE. Many of the same concepts such as recession “funnies” are popping up again. What so these journalists and cartoonist’s have it easy eh? All they have to do to get their job done is find some old drawing and change it a little. Well, I’m jumping on the bandwagon folks. Here are two “What’s old is New!” Articles:
Living without Windows
Shell Scripting 101
Since they’ve been written we’ve learned a lot, new concepts are out there but while they are somewhat older, the concepts still apply. As everything old they enjoy receiving comments but don’t take criticism well. They know their faults such as bad spelling and grammar but their ageing and refuse to acknowledge their flaws unless you directly point them out. Give them a read and talk to them before they die and go to internet Heaven (or Hell).

KDE End User Wiki

Filed under: Linux Software — TheLinuxBlog.com at 10:27 am on Friday, September 19, 2008

Today the KDE User base wiki (http://userbase.kde.org/) went live. If you’ve ever searched for information regarding an issue with KDE, its possible you would have come across the techbase (http://techbase.kde.org) The KDE Techbase is aimed more at developers and has some really good content regarding development. The KDE userbase Wiki also has some great information and you can learn a lot about KDE just by reading it. If you are a developer or an end user the userbase can be of great help, I just learned something new about plasma application launchers, check it out!

rsync to smb share

Filed under: General Linux — TheLinuxBlog.com at 4:30 pm on Thursday, September 11, 2008

If you happen to have a SMB share with a lot of disk space laying around, then you may have considered backing up to it. There is more than one methods that you could back up to a SMB share but this article will show how to rsync to a smb share. This blog post assumes that you have successfully set up your SMB share and have installed RSync.
(Read on …)

htaccess allow from

Filed under: General Linux — TheLinuxBlog.com at 10:32 am on Tuesday, September 9, 2008

htaccess allow from gives you the ability to allow (or deny) specific IP’s or domain names from a directory on your server. To do this the syntax is quite simple. Using VIM or nano open up the .htaccess file in the directory that you want to restrict access to. You need to add the following:

Order Deny,Allow
Deny from all
Allow from 127.0.0.1

This allows access from your local host and the IP address you specify. Using .htaccess you can also allow by host name. This is useful if you wish to allow or deny a friend access to a directory. (note: it will also work if you have them in your hosts file)

Order Deny,Allow
Deny from all
Allow from LinuxBlog
Allow from .thelinuxblog.com

Using htaccess to allow from your LAN is also pretty easy. You use your CIDR address (ip/subnet) to do this try something like this (changing to match your LAN):

Order Deny,Allow
Deny from all
Allow from 192.168.1.1/24

I run into htaccess allow problems a lot, and hope that this will clear the air up for me. htaccess can be very handy if you do not want to keep turning your firewall on and off, but do not want your directories wide open. Just remember, if you want to stop everyone except those you choose to access your apache web directories, use htaccess allow from!

Managing Services on Fedora

Filed under: Linux Software — TheLinuxBlog.com at 12:01 am on Wednesday, July 2, 2008

Fedora is a great desktop system, it has a pretty good services manager called service. It can be used to turn on, off and restart most services that run on the system. Should you want to manage running services all you have to do is type:

service <service name> <stop | start | restart>

If you don’t know the service name it can normally be found by issuing the –status-all switch. Doing so will output a long list of services so you may want to grep it or use a pager such as less:

[root@linux-blog ~]# service –status-all | less

[root@linux-blog ~]# service –status-all | grep snmp
capi not installed – No such file or directory (2)
JAVA_EXECUTABLE or HSQLDB_JAR_PATH in ‘/etc/sysconfig/hsqldb’ is set to a non-file.
snmpd is stopped
snmptrapd is stopped

While service is a great utility to manage services it does not stop them from running at startup. To manage services that start up when you boot you can use a handy little dialog script called “ntsysv”:

[root@linux-blog ~]# ntsysv

This will give you a nice dialog which is intuitive and similar to those of text based installers. Select the services you want to start up and then tab over to OK. I turn a lot of services off that I don’t use, if you’re unsure of what you need, try stopping it with the services command first and see if anything breaks. If after test it works well just turn it off.

Since I don’t use SELinux on my development machine, I always turn off setroubleshootd. I used ntsysv to stop it from starting at bootup and if I need it I can use the service command to start it.

Here is a screenshot of ntsysv in action:

managing services with ntsysv on Fedora