Linux Blog

An rsync primer

Filed under: General Linux,Linux Software — TheLinuxBlog.com at 12:55 am on Tuesday, January 13, 2009

An rsync Primer

rsync is a great tool used to copy or “sync” files locally or remotely. Having just lost a fair amount of important personal data its good to make the point that you can lose data if you use rsync, so make sure that you know what you are doing works before putting it into production.
“Whats so great about rsync?”
I hear you ask.

Well, there are many ways of copying data around, regular old cp, scp and rcp but what sets rsync apart is that it’s capable of syncing those files that have changed, or those that are missing. In other words, why do a full copy of all files, risking further corruption when only some have changed?
rsync also handles compression and verifies that the files were written correctly.
The syntax for rsync is fairly straightforward, those of you familiar with the rcp or scp should be easily able to relate to that of rsync.

The syntax for a local file copy is:

rsync [OPTION...] SRC… [DEST]

-avz is the option that I use most commonly.

-a for archive, -v for verbose and -z for compression. As lame as it may sound, I actually remember the syntax for this as “Alien Vs. Zeus.” Sounds stupid, but it actually works.

Since I do not normally sync directories locally, one of my hosts is usually remote. rsync is configured to use SSH by default, but rsh can be used if preferred. I stick with the default SSH for simplicity. The syntax is very similar to scp’s

To Pull: rsync [OPTION...] [USER@]HOST:SRC… [DEST]

To Push: rsync [OPTION...] SRC… [USER@]HOST:DEST

If you like to back up all files matching a pattern the same syntax applies but you specify a pattern. An example is you could backup all .conf files from /etc (provided you have read access) to another host by doing the following:

rsync /etc/*.conf [USER@]HOST:DEST

If you do the above, I am assuming that you would also want to backup other configuration files within the /etc/ directory. Instinct tells you if you have read the man page to just use -r however you may want to just do a full backup as rsync with a pattern by default does not traverse into directories.

Compression is really a nice feature for transferring large amounts of data. To demonstrate the speed increases that the -z option gives my full /etc/ directory which is 48M. It took 0m12.671s with compression and 0m35.657s without. This was over wireless from a dual core 2GHz laptop to a 1.8GHz wired desktop, so your results may vary. Either way compression or no compression rsync is a handy utility that should be able to make your life of copying files around a little easier.

Getting a Hand With Bash

Filed under: General Linux — TheLinuxBlog.com at 8:27 am on Wednesday, October 1, 2008

If you use the bash shell, or as many know it the terminal on a regular basis then there is a package that you can install that will make your life a whole lot easier. Firstly I’ll tell you a little bit more about it.
When I first started using Linux a friend told me to install this package. I never really understood what it exactly did, but using the terminal was a whole lot easier then using the command prompt in Windows. I couldn’t explain it but things just worked. Features that were missing from Dos were there. There was awesome auto completion that knew what I wanted to type. Over the years I became more accustomed to using Linux and the Bash shell. I upgraded hardware, changed distributions and never really noticed what I was missing out on. The other day I was thinking, wouldn’t it be nice if I could press tab and this would auto complete. Then it hit me, the package my friend had told me to install when I first was starting out. The package happened to be exactly for this purpose. Since I was a novice and he was helping me out I didn’t really have the time to ask about the details of every single package, so here I am now letting you in on a little secret. (Read on …)

Using Subversion with SSH & Custom Ports

Filed under: Linux Software,Quick Linux Tutorials — TheLinuxBlog.com at 9:09 am on Monday, September 15, 2008

Lets say you use subversion on your home PC to keep track of projects and you want to checkout or export your project from a remote location. Here’s the catch, sshd is running on a custom port or forwarded from another. For some reason the command line SVN client does not support a port parameter when using the defacto svn+ssh://

svn co svn+ssh://thelinuxblog.com/owen/svn/project1/trunk project1
ssh: connect to host thelinuxblog.com port 22: Connection refused

Well we know why the error above happens its because I happen to run SSH on port 1337. The following work around requires root privileges, and may mess with your system a bit but if you really need to check something out, then it will work.

As root, login and stop SSH if you run it. With SSH Forward port 22 with a local SSH forwarding connection to the remote host.

[owen@thelinuxblog.com]$ sudo su -
[root@thelinuxblog.com]$ /sbin/service sshd stop
[root@thelinuxblog.com]$ ssh -p 1337 owen@thelinuxblog.com-L 22:<internal ip>:1337

Once this is done, your localhost:22 now forwards to your remote host. With another session (on your local machine) you can verify the connection by using ssh localhost. You will probably get warnings about the hosts identity being changed, or not verifying it, but you can ignore then. Once you’ve tested it, just use SVN as normal. When finished, remember to logout of the SSH session, and start SSH back up again if you run it.

htaccess allow from

Filed under: General Linux — TheLinuxBlog.com at 10:32 am on Tuesday, September 9, 2008

htaccess allow from gives you the ability to allow (or deny) specific IP’s or domain names from a directory on your server. To do this the syntax is quite simple. Using VIM or nano open up the .htaccess file in the directory that you want to restrict access to. You need to add the following:

Order Deny,Allow
Deny from all
Allow from 127.0.0.1

This allows access from your local host and the IP address you specify. Using .htaccess you can also allow by host name. This is useful if you wish to allow or deny a friend access to a directory. (note: it will also work if you have them in your hosts file)

Order Deny,Allow
Deny from all
Allow from LinuxBlog
Allow from .thelinuxblog.com

Using htaccess to allow from your LAN is also pretty easy. You use your CIDR address (ip/subnet) to do this try something like this (changing to match your LAN):

Order Deny,Allow
Deny from all
Allow from 192.168.1.1/24

I run into htaccess allow problems a lot, and hope that this will clear the air up for me. htaccess can be very handy if you do not want to keep turning your firewall on and off, but do not want your directories wide open. Just remember, if you want to stop everyone except those you choose to access your apache web directories, use htaccess allow from!

Finding a PC on your network

Filed under: General Linux,Linux Software — TheLinuxBlog.com at 4:31 am on Tuesday, September 4, 2007

When I’m at a remote location I sometimes need to gain access to a computer that is not accessible from the internet but is on the same network as another machine that is. I have remote SSH access into the box that is on the same network but I often don’t know the IP address for the computer that I am trying to gain access to since they are assigned via DHCP.
I have a simple solution that will locate the computer I wish to use once logged into the gateway in no time. The program needed is from our friends at insecure.org and is called nmap. It is a pretty standard tool so it should be included with your distribution. However if you do not own the Linux machine then it may not be installed or you may not have the ability to install it or have sufficient privileges to run it.
Basically what nmap does is scan the network.
The command I use to scan a whole subnet for my host is:

nmap -sP 192.168.x.x/24

The type of scan I use is a ping scan, I only determine if the host is online. If I know the DHCP pool starts from 1.1 and ends at 1.100 then I would use:

nmap -sP 192.168.1.1-100

This will yield faster scanning results since it only has to ping 100 hosts not the whole subnet.
Sometimes if the environment is a busy one (one with lots of hosts) a lot of online IP’s will be returned and its hard to identify which one your trying to connect to. I remedy this by just scanning the host range I need that only have port 22 open.

nmap 192.168.3.1-100 -sT -p22

The -sT option doesn’t require the user to be root but if you have root the -sS option is better as it gives detailed information such as the mac address which can come in handy if you happen to know what brand of network card is in the computer you are trying to log into.

This will work to find a computer on a network with Linux but it still requires you to know a little information about the PC your trying to find. If you need a better way of finding your PC’s I would recommend using static IP’s and DNS. Give it a shot if your on location somewhere and need access to your computer.