Automatically Check RSYNC and Restart if Stopped

I occa­sion­al­ly use RSYNC to syn­chro­nize large direc­to­ries of files between servers. This is espe­cial­ly use­ful if you’re mov­ing a client from one server to anoth­er and they have alot of sta­t­ic files that are always chang­ing. You can copy the files and sync them up, all with RSYNC and if your con­nec­tion gets cut off, it will start where it left off. It will also grab changes to files that have already been RSYNCd.

I ran into an issue with RSYNC recent­ly, where­in the RSYNC process was run­ning in the back­ground; but was ter­mi­nat­ing due to errors sim­i­lar to the fol­low­ing. The­se con­nec­tions were prob­a­bly relat­ed to the slow and unsta­ble con­nec­tion to the remote server.

rsync: writefd_unbuffered failed to write 998 bytes to socket [sender]: Broken pipe (32)
rsync: connection unexpectedly closed (888092 bytes received so far) [sender]
rsync error: error in rsync protocol data stream (code 12) at io.c(600) [sender=3.0.6]

Given that I was trans­fer­ring files through a rel­a­tive­ly bad inter­net con­nec­tion and received this error a half dozen times over a cou­ple of days, I decid­ed the best way to han­dle it, would be to write a cron script. This cron script should check for the RSYNC process and start it if it isn’t run­ning.

rsync_check.sh

Cus­tomize this script for your own pur­pose, to check for your RSYNC process and start it if it isn’t run­ning.

#!/bin/bash
echo "checking for active rsync process"
COUNT=`ps ax | grep rsync | grep -v grep | grep -v rsync_check.sh | wc -l` # see how many are running
echo "there are $COUNT rsync related processes running";
if [ $COUNT -eq 0 ] 
then
	echo "no rsync processes running, restarting process"
	killall rsync  # prevent RSYNCs from piling up, if by some unforeseen reason there are already processes running
	rsync -avz -e "ssh" user@host.com:/mnt/syncdirectory/ /home/ccase/syncdirectory/ 
fi

Crontab Entry

Save the script in the appro­pri­ate cron direc­to­ry, or add it to the cron.d direc­to­ry and put a crontab entry in, to run it at the desired inter­val. This will have it run every 10 min­utes.

*/10 * * * * ccase /etc/cron.d/rsync_check.sh

No More Worries

Now you can move onto oth­er things, with the knowl­edge that your RSYNC will not just fail and leave the work undone. It prob­a­bly wouldn’t hurt to check on it at first and from time to time; but there’s alot less to wor­ry about!

Appending to a Remote File via SSH

Most LINUX users know how to copy and over­write a file from one server to anoth­er; but it can also be use­ful to direct­ly append to a file, with­out hav­ing to login to the remote server and make the changes man­u­al­ly. This does not appear to be pos­si­ble with the com­mon­ly used SCP util­i­ty; how­ev­er, there is a way to do this with SSH. Its actu­al­ly quite sim­ple. Con­tin­ue read­ing “Append­ing to a Remote File via SSH”

Tunneling Through a Remote Firewall Using SSH Commands

If you’re deal­ing with sys­tems behind a fire­wall it’s almost inevitable that you will need to tun­nel into those sys­tems from time to time.  For­tu­nate­ly, there are some quick & easy com­mands to accom­plish this.  In this exam­ple, we are going to use a Mac OSX or linux-based sys­tem, to gain access to a web server’s port 80 on a fire-walled server.

Let’s say the domain of the remote server is dfrn.net, the fire-walled server has an IP address of 192.168.1.100 and the fire­walled server has a web server at port 80.  We need to choose an unused port on our own sys­tem, in this case we’ll use 2020.

So our side of the tun­nel is going to be http://localhost:2020/ and the oth­er side of the tun­nel will be http://192.168.1.100:80/.

ssh root@dfrn.net -L 2020:192.168.1.100:80
 
root@dfrn.net's password:

So, now port 80 on the fire-walled server will be acces­si­ble by sim­ply point­ing your web browser to http://localhost:2020/.  To ter­mi­nate the tun­nel, sim­ply exit the shell.

SSH: DSA Key Issue: Unknown code krb5 195

I was attempt­ing to set up an SSH key between two servers, so I could auto­mat­i­cal­ly back up a file from one to the oth­er. This is usu­al­ly a fair­ly straight­for­ward and rou­tine pro­ce­dure; but in this case I had some issues. 

When I got every­thing set-up, with the pub­lic key in the accepted_keys file of the remote server, I saw this error when run­ning in debug mode, then I was asked for a pass­word, which should not hap­pen.

[me@host ~]$ ssh -v root@remotehost
...
debug1: Unspecified GSS failure.  Minor code may provide more information
Unknown code krb5 195

Con­tin­ue read­ing “SSH: DSA Key Issue: Unknown code krb5 195”

Restrict a Linux User’s Access: Only Allowing SCP/SFTP, no SSH

The stan­dard tech­niques for restrict­ing a Lin­ux user account, does not allow for file trans­fers to/from the user’s home direc­to­ry. In my expe­ri­ence it is use­ful to have cer­tain account types which are only allowed to upload/download files from their home direc­to­ry; but not login and run shell com­mands.

This is easy to do with a shell called rssh (Restrict­ed Secure Shell); but you must first install it, because it does not typ­i­cal­ly come pack­aged with most dis­tri­b­u­tions of Lin­ux.

Installing RSSH

Locate the most appro­pri­ate pack­age for your dis­tri­b­u­tion of Lin­ux at the down­load site. Once you have locat­ed the RPM you will need do the fol­low­ing steps, sub­sti­tut­ing your cho­sen pack­age for the RPM.

[root@Internal ~]# <strong>wget http://packages.sw.be/rssh/rssh-2.3.2-1.1.el3.rf.x86_64.rpm</strong>
--2010-10-11 20:36:21--  http://packages.sw.be/rssh/rssh-2.3.2-1.1.el3.rf.x86_64.rpm
Resolving packages.sw.be... 85.13.226.40
Connecting to packages.sw.be|85.13.226.40|:80... connected.
HTTP request sent, awaiting response... 302 Found
Location: http://rpmforge.sw.be/redhat/el3/en/x86_64/rpmforge/RPMS/rssh-2.3.2-1.1.el3.rf.x86_64.rpm [following]
--2010-10-11 20:36:21--  http://rpmforge.sw.be/redhat/el3/en/x86_64/rpmforge/RPMS/rssh-2.3.2-1.1.el3.rf.x86_64.rpm
Resolving rpmforge.sw.be... 85.13.226.40
Reusing existing connection to packages.sw.be:80.
HTTP request sent, awaiting response... 200 OK
Length: 45053 (44K) [application/x-rpm]
Saving to: “rssh-2.3.2-1.1.el3.rf.x86_64.rpm”
100%[====================================================================================================================================================>] 45,053      94.6K/s   in 0.5s
 
2010-10-11 20:36:22 (94.6 KB/s) - “rssh-2.3.2-1.1.el3.rf.x86_64.rpm” saved [45053/45053]
 
[root@Internal ~]# rpm -ivh rssh-2.3.2-1.1.el3.rf.x86_64.rpm
warning: rssh-2.3.2-1.1.el3.rf.x86_64.rpm: Header V3 DSA signature: NOKEY, key ID 6b8d79e6
Preparing...                ########################################### [100%]
1:rssh                   ########################################### [100%]

Updating Access Permissions

Now you should be able to set a user’s login shell to RSSH. Here is what the orig­i­nal line will usu­al­ly look like.

joe:x:501:501::/home/joe:/bin/bash

This is what the updat­ed line will look like.

joe:x:501:501::/home/joe:/usr/bin/rssh

What Happens if the User Attempts to SSH in After Access is Restricted

Now if joe attempts to login via SSH, the fol­low­ing will occur:

[root@Internal ~]# ssh joe@localhost
The authenticity of host 'localhost (127.0.0.1)' can't be established.
RSA key fingerprint is b5:39:02:23:01:a5:ff:b9:c1:aa:01:a9:69:21:a4:e0.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'localhost' (RSA) to the list of known hosts.
joe@localhost's password: 
 
This account is restricted by rssh.
This user is locked out.
 
If you believe this is in error, please contact your system administrator.
 
Connection to localhost closed.