For those a little scared of the terminal using the Linux find command may seem a little daunting. To be honest though the find command really isn’t that hard to get the hang of. By effectively learning and using the Linux find command you’ll open up a whole new can of searching capabilities. You’ll increase your capabilities, boost productivity, and be more likely to find what your looking for. Alright, enough of the pep talk already and lets get to the core that is the powerful Linux find command.
Sometimes when using grep it’s nice to be able to search for more than one string in a file. It’s actually pretty easy to do, if you can remember the simple syntax. Basically, you pipe the terms together in double quotes and escape the pipe.
grep “gnome\|kde” install.log
The above example will search for gnome or KDE. It works for more more than one term, and also works with the invert match -v option, to exclude lines that include gnome or kde.
Todays article extra short, and late, but better late and short than none at all I guess.
rsync is a great tool used to copy or “sync” files locally or remotely. Having just lost a fair amount of important personal data its good to make the point that you can lose data if you use rsync, so make sure that you know what you are doing works before putting it into production.
“Whats so great about rsync?”
I hear you ask.
Well, there are many ways of copying data around, regular old cp, scp and rcp but what sets rsync apart is that it’s capable of syncing those files that have changed, or those that are missing. In other words, why do a full copy of all files, risking further corruption when only some have changed?
rsync also handles compression and verifies that the files were written correctly.
The syntax for rsync is fairly straightforward, those of you familiar with the rcp or scp should be easily able to relate to that of rsync.
The syntax for a local file copy is:
rsync [OPTION...] SRC… [DEST]
-avz is the option that I use most commonly.
-a for archive, -v for verbose and -z for compression. As lame as it may sound, I actually remember the syntax for this as “Alien Vs. Zeus.” Sounds stupid, but it actually works.
Since I do not normally sync directories locally, one of my hosts is usually remote. rsync is configured to use SSH by default, but rsh can be used if preferred. I stick with the default SSH for simplicity. The syntax is very similar to scp’s
To Pull: rsync [OPTION...] [USER@]HOST:SRC… [DEST]
To Push: rsync [OPTION...] SRC… [USER@]HOST:DEST
If you like to back up all files matching a pattern the same syntax applies but you specify a pattern. An example is you could backup all .conf files from /etc (provided you have read access) to another host by doing the following:
rsync /etc/*.conf [USER@]HOST:DEST
If you do the above, I am assuming that you would also want to backup other configuration files within the /etc/ directory. Instinct tells you if you have read the man page to just use -r however you may want to just do a full backup as rsync with a pattern by default does not traverse into directories.
Compression is really a nice feature for transferring large amounts of data. To demonstrate the speed increases that the -z option gives my full /etc/ directory which is 48M. It took 0m12.671s with compression and 0m35.657s without. This was over wireless from a dual core 2GHz laptop to a 1.8GHz wired desktop, so your results may vary. Either way compression or no compression rsync is a handy utility that should be able to make your life of copying files around a little easier.
There is no doubt about it that SSH is a handy administration and remote access tool. Have you ever wanted to add port forwarding or cancel port forwarding from within an active SSH connection? Well, a feature that many do not know about is the ssh escape feature. With this pseudo terminal you can do some nifty things, such as forward ports over an active SSH connection.
To do this, follow these instructions:
(Read on …)
In the last Shell Script Sunday’s article I wrote, I said that I’d show you some more ways to rework shell scripts to make them easier to use. I’ve got some more tricks up my sleeve that I’d like to share, its been rather busy this site of the internet at TheLinuxBlog.com. So why I write up some more shell scripting methods, here are some previous posts that can enhance your shell scripts. Be sure to comment on them if you find them useful, or would like more information.
Creating Dialogs with Dialog
If you have a shell script that you use on a regular basis, you may want to consider using dialog to make it more user friendly. Dialog makes it easy to create easy to use dialogs that are intuitive and easy to use. There are so many combinations of dialogs that can be created that the possibilities are ended. Dynamically create dialogs for select lists, input boxes, progress bars and much much more.
Graphical Shell Scripting
This article I wrote introduces graphical shell scripting. Similar to dialog this is an updated “Dialog” and works within X. If you support end users, or your target audience is Ubuntu/Linspire users (j/k) then XDialog may be the better choice. Its got most of the same functionality as Dialog except it depends on X. You can even support both Dialog and XDialog as they pretty much use the same syntax.
htaccess allow from gives you the ability to allow (or deny) specific IP’s or domain names from a directory on your server. To do this the syntax is quite simple. Using VIM or nano open up the .htaccess file in the directory that you want to restrict access to. You need to add the following:
Deny from all
Allow from 127.0.0.1
This allows access from your local host and the IP address you specify. Using .htaccess you can also allow by host name. This is useful if you wish to allow or deny a friend access to a directory. (note: it will also work if you have them in your hosts file)
Deny from all
Allow from LinuxBlog
Allow from .thelinuxblog.com
Using htaccess to allow from your LAN is also pretty easy. You use your CIDR address (ip/subnet) to do this try something like this (changing to match your LAN):
Deny from all
Allow from 192.168.1.1/24
I run into htaccess allow problems a lot, and hope that this will clear the air up for me. htaccess can be very handy if you do not want to keep turning your firewall on and off, but do not want your directories wide open. Just remember, if you want to stop everyone except those you choose to access your apache web directories, use htaccess allow from!
I made this little script to check how many packages were available on the web from the Cygwin Package Repository located at http://www.cygwin.com/packages
Its a one liner but it does its job well.
CYGLIST=$(curl http://www.cygwin.com/packages/ | grep \<tr | grep ball | wc -l); echo $CYGLIST;
All the above is doing is creating a variable called CYGLIST that is the result of grabbing the cygwin.com/packages/ page, grepping all of the TR’s that also have the word “ball” in it (for the image) and then using the wc -l (L) command to count how many results are found. Then the list is echoed out.
wc is a very useful command for printing newline, word and byte counts. This is a good example of how to use wc to count lines in a shell script. wc can also be used to print all of these values in one line of a file. The syntax is below:
bash-3.1# wc file.txt
9 20 184 file.txt
The above shows the number of lines in the file.txt, it shows how many words are in the file and also how many bytes. In my first example wc uses the -l switch to display the number of lines. This script can also be used with a little bit of bash math to calculate how many items are in an HTML list. I’m working on a script that automatically does this, when its finished I will be sure to post it here on The Linux Blog.
I was reading a post on Tech.Blorge and thought I’d write a little about it.
I think that this keyboard is a little over priced at the moment, but the price is sure to drop. If I got my hands on one of these here are the things that I would do with it:
- Impress my friends
- Make it have some kind of fireworks effect where every key you hit makes some sort of explosion on the keys around it. Who doesn’t love fireworks?
- Matrix Keyboard. This would be pretty sweet. Just have it constantly waste power by scrolling the Matrix code down it all the time. No one will ever use your computer because they won’t be able to figure out where that blasted key they are looking for is.
- Write a movie player so that I could watch a movie on my keyboard instead of working.
- Make a game that tests hand eye-coordination. Kind of like BrainAge except you use your hands and don’t have a 1″ screen.
- Gaming. I’d love for my keys to change when I play any game. Lets take Quake or Halo for example. It should always flash the mother of all weapons the either the B.F.G or the Rocket Launcher.
I love the shift feature and the special characters feature is pretty cool too but there are better practical uses of this keyboard:
- System Stats. Lets say WiFi signal strength, CPU load and memory usage Why display them in Torsmo, gkrellm or anything else you can just move your fingers and see the percentage of memory being used.
- Syntax highlighting while programming. I think that this would be a nice feature for new developers and for programmers not familiar with a particular programming language. An example would be the IF syntax. Once you type if, the space bar highlights. Next the left parenthesis then the quote depending on which programming language you use.
If any one gets one of these working I’d love to see it in action. I also thing it should be mandatory that when your OS crashes (no matter what OS your running) it displays the B.S.O.D.
Have you ever seen those pretty dialogs used in Shell Scripts such as the Slackware installation, the slackpkg program or even the NVIDIA driver installer? Well, my friends to display dialog boxes from shell scripts is very easy with… you guessed it – Dialog.
First of all, there are many different types of dialogs that you can create they are as follows: calendar, checklist, fselect, gauge, infobox, inputbox, menu, msgbox (message), password, radiolist, tailbox, tailboxbg, textbox, timebox, and yesno (yes/no).
This blog post is intended to be a primer on using dialog. More examples will be posted in future blog posts in the Shell Script Sunday’s column.
The simplest form of a dialog in a shell script is probably the msgbox. All this really does is displays text. To display text in a dialog you would do the following:
owen@the-linux-blog:$ dialog –msgbox “Hello from the Linux Blog!” 5 50
The numbers after the text in quotes are the widths and heights of the box. The minimum height that I like to use is 5. The width doesn’t really matter as long as it is big enough. It is good to keep the box sizes standard across a whole script because it gets annoying with constantly resizing boxes.
If the text in a message box is too long it will auto wrap around and give you a type of scroll bar. As follows:
owen@the-linux-blog:$ dialog –msgbox “Hello from The Linux Blog. This text is so long it wraps it to a New Line” 5 50
Dialogs can be canceled. Clicking Ok or pressing enter/return returns “true” and pressing escape or Ctrl+C returns a false.
The simple shell scripting syntax shown in Shell Scripting 101
is used for this:
owen@the-linux-blog:$ dialog –msgbox “Dialog Exit Example” 5 50 && echo “ok” || echo “false”
Another simple dialog example is the Yes/No box. The syntax for this is exactly the same as the msgbox example except instead of using –msgbox, –yesno is used. The difference between a msgbox and a yesno box is that there a two buttons. It is pretty obvious as to what they are labeled, but for those in the back, I’ve included an example and some screen shots anyway.
owen@the-linux-blog:$ dialog –yesno “Are you learning anything from this blog” 5 50 && echo “Yes, thanks Owen.” || echo “No, Write some better Linux Related Posts”
Thats about all I have time for this week. Check back next week!
This post is designed to be a refresher, reference or quick intro into how to manipulate strings with the cut command in bash.
Some times its useful to take the output of a command and reformat it. I sometimes do this for aesthetic purposes or tor format for use as input into another command.
Cut has options to cut by bytes (-b), characters (-c) or fields (-f). I normally cut by character or field but byte can come in handy some times.
The options to cut by are below.
N N’th byte, character or field, counted from 1
N- from N’th byte, character or field, to end of line
N-M from N’th to M’th (included) byte, character or field
-M from first to M’th (included) byte, character or field
The options pretty much explain themselves but I have included some simple examples below:
Cutting by characters (command on top, output below)
echo "123456789" | cut -c -5
echo "123456789" | cut -c 5-
echo "123456789" | cut -c 3-7
echo "123456789" | cut -c 5
Sometimes output from a command is delimited so a cut by characters will not work. Take the example below:
echo -e "1\t2\t3\t4\t5" |cut -c 5-7
To echo a tab you have to use the -e switch to enable echo to process back slashed characters. If the desired output is 3\t4 then this would work great if the strings were always 1 character but if anywhere before field 3 a character was added the output would be completely changed as followed:
echo -e "1a\t2b\t3c\t4d\t5e" | cut -c 5-7
This is resolved by cutting by fields.
Cutting by fields
The syntax to cut by fields is the same as characters or bytes. The two examples below display different output but are both displaying the same fields (Fields 3 Through to the end of line.)
echo -e "1\t2\t3\t4\t5" | cut -f 3-
3 4 5
echo -e "1a\t2a\t3a\t4a\t5a" | cut -f 3-
3a 4a 5a
The default delimiter is a tab, if the output is delimited another way a custom delimiter can be specified with the -d option. It can be just about any printable character, just make sure that the character is escaped (back slashed) if needed. In the example below I cut the string up using the pipe as the delimiter.
echo "1|2|3|4|5" | cut -f 3- -d \|
One great feature of cut is that the delimiter that was used for input can be changed by the output of cut. In the example below I change the format of the string from a dash delimited output and change it to a comma.
echo -e "1a-2a-3a-4a-5a" | cut -f 3- -d – --output-delimiter=,
Formatting with Cut Example
Sometimes certain Linux applications such as uptime do not have options to format the output. Cut can be used to pull out the information that is desired.
Normal up-time Command:
19:18:40 up 1 day, 22:15, 4 users, load average: 0.45, 0.10, 0.03
Time with up-time displayed:
owen@the-linux-blog:~$ uptime |cut -d , -f 1,2 | cut -c 2-
19:19:36 up 1 day, 22:22
For the above example I pipe the output of uptime to cut and tell it I want to split it with a comma , delimiter. I then choose fields 1 and 2. The output from that cut is piped into another cut that removes the spaces in front of the output.
Load averages extracted from uptime:
owen@the-linux-blog:~$ uptime |cut -d , -f 4- | cut -c 3-
load average: 0.42, 0.10, 0.03
This is about the same as the previous example except the fields changed. Instead of fields 1 and 2 I told it to display fields 4 through the end. The output from that is piped to another cut which removes the three spaces that were after the comma in "4 users, " by starting at the 3rd character.
The great thing about cutting by fields is that no matter if the field length changes the data stays the same. Take the example below. I now have 17 users logged in which would have broke the output if I had used -c (since there is an extra character due to a double digit number of users being logged in.)
19:25:11 up 1 day, 22:28, 17 users, load average: 0.00, 0.06, 0.04
owen@the-linux-blog:~$ uptime |cut -d , -f 4- | cut -c 3-
load average: 0.00, 0.06, 0.04
That just about covers everything for the cut command. Now you know about it you can use cut to chop up all types of strings. It is one of the many great tools available for string manipulation in bash. If you can remember what cut does it will make your shell scripting easier, you don’t need to memorize the syntax because all of the information on how to use cut is available here, in the man pages and all over the web.
Normally in a shell script it is desirable to do something repetitive.
I have touched on some loops in other posts but now would like to go over them in a bit more detail. Each of the examples in this post are intended to give an introduction to looping in bash.
For loops allow you to repeat a section of code a number of times. Its very similar to other languages syntax but works a little differently. The for loop in bash only allows you to give a fixed list of values to loop over. A good way to remember how a for loop works is “For each of the dishes: clean and dry.”
for i [in list]
statements [use $i]
for x in 1 2 3
echo “Number: $x”
This is a very simple script that just counts to 3 and then prints “Finished!”
While and until Loops
In essence while and until are the same in bash. The titles are pretty much self explanatory. A while loop would be explained in real life as “While the sink is still full: wash dishes” and a until loop would be “Until the sink is empty: Wash dishes.”
While and Until Syntax:
until/while [condition] do
Example of a While loop:
while [ $count -lt 10 ]; do
Basically this loop will loop over the code while the count variable is less than 10. If we didn’t put the let statement in the script it would get stuck in the loop causing the user to press CTRL+C to end the script.
Doing the same thing can be done in a until loop except the condition has to be modified to get the same result.
until [ $count -gt 9 ]; do
Now that you’ve figured out how to loop over something its probably a good idea to know how to stop the loop.
All that needs to be done to stop a loop is:
for x in 1 2 3 4 5
if [ $x = 3 ]; then
echo “Number is 3. Quitting”
echo “Number: $x”
This is a very easy to follow example. Its the same as the basic for loop except that if x is 3 it will stop the loop. This example has no real practical purpose. Since its a for loop the number 3 could just have been omitted.
Real World For Loop Example
Looping over all files in /etc and printing all of those that match “grep conf” and putting them in quotes.
The code to do this in a loop is:
for x in $(ls /etc -1|grep conf);
The situation for many bash scripts is that there is normally a shorter way to do something. Take the Real World For Loop Example in this tutorial the same results can be achieved with:
x=$(ls /etc |grep conf); echo “$x”\n
This will get the job done but a loop may be better for esthetic purposes or for additional logic.