I occasionally use RSYNC to synchronize large directories of files between servers. This is especially useful if you’re moving a client from one server to another and they have alot of static files that are always changing. You can copy the files and sync them up, all with RSYNC and if your connection gets cut off, it will start where it left off. It will also grab changes to files that have already been RSYNCd.
I ran into an issue with RSYNC recently, wherein the RSYNC process was running in the background; but was terminating due to errors similar to the following. These connections were probably related to the slow and unstable connection to the remote server.
rsync: writefd_unbuffered failed to write 998 bytes to socket [sender]: Broken pipe (32)
rsync: connection unexpectedly closed (888092 bytes received so far) [sender]
rsync error: error in rsync protocol data stream (code 12) at io.c(600) [sender=3.0.6]
Given that I was transferring files through a relatively bad internet connection and received this error a half dozen times over a couple of days, I decided the best way to handle it, would be to write a cron script. This cron script should check for the RSYNC process and start it if it isn’t running.
Customize this script for your own purpose, to check for your RSYNC process and start it if it isn’t running.
echo "checking for active rsync process"
COUNT=`ps ax | grep rsync | grep -v grep | grep -v rsync_check.sh | wc -l` # see how many are running
echo "there are $COUNT rsync related processes running";
if [ $COUNT -eq 0 ]
echo "no rsync processes running, restarting process"
killall rsync # prevent RSYNCs from piling up, if by some unforeseen reason there are already processes running
rsync -avz -e "ssh" email@example.com:/mnt/syncdirectory/ /home/ccase/syncdirectory/
Save the script in the appropriate cron directory, or add it to the cron.d directory and put a crontab entry in, to run it at the desired interval. This will have it run every 10 minutes.
*/10 * * * * ccase /etc/cron.d/rsync_check.sh
No More Worries
Now you can move onto other things, with the knowledge that your RSYNC will not just fail and leave the work undone. It probably wouldn’t hurt to check on it at first and from time to time; but there’s alot less to worry about!
Most LINUX users know how to copy and overwrite a file from one server to another; but it can also be useful to directly append to a file, without having to login to the remote server and make the changes manually. This does not appear to be possible with the commonly used SCP utility; however, there is a way to do this with SSH. Its actually quite simple. Continue reading “Appending to a Remote File via SSH”
If you’re dealing with systems behind a firewall it’s almost inevitable that you will need to tunnel into those systems from time to time. Fortunately, there are some quick & easy commands to accomplish this. In this example, we are going to use a Mac OSX or linux-based system, to gain access to a web server’s port 80 on a fire-walled server.
Let’s say the domain of the remote server is dfrn.net, the fire-walled server has an IP address of 192.168.1.100 and the firewalled server has a web server at port 80. We need to choose an unused port on our own system, in this case we’ll use 2020.
So our side of the tunnel is going to be http://localhost:2020/ and the other side of the tunnel will be http://192.168.1.100:80/.
ssh firstname.lastname@example.org -L 2020:192.168.1.100:80
So, now port 80 on the fire-walled server will be accessible by simply pointing your web browser to http://localhost:2020/. To terminate the tunnel, simply exit the shell.
I was attempting to set up an SSH key between two servers, so I could automatically back up a file from one to the other. This is usually a fairly straightforward and routine procedure; but in this case I had some issues.
When I got everything set-up, with the public key in the accepted_keys file of the remote server, I saw this error when running in debug mode, then I was asked for a password, which should not happen.
[me@host ~]$ ssh -v root@remotehost
debug1: Unspecified GSS failure. Minor code may provide more information
Unknown code krb5 195
Continue reading “SSH: DSA Key Issue: Unknown code krb5 195”
The standard techniques for restricting a Linux user account, does not allow for file transfers to/from the user’s home directory. In my experience it is useful to have certain account types which are only allowed to upload/download files from their home directory; but not login and run shell commands.
This is easy to do with a shell called rssh (Restricted Secure Shell); but you must first install it, because it does not typically come packaged with most distributions of Linux.
Locate the most appropriate package for your distribution of Linux at the download site. Once you have located the RPM you will need do the following steps, substituting your chosen package for the RPM.
[root@Internal ~]# <strong>wget http://packages.sw.be/rssh/rssh-2.3.2-1.1.el3.rf.x86_64.rpm</strong>
--2010-10-11 20:36:21-- http://packages.sw.be/rssh/rssh-2.3.2-1.1.el3.rf.x86_64.rpm
Resolving packages.sw.be... 184.108.40.206
Connecting to packages.sw.be|220.127.116.11|:80... connected.
HTTP request sent, awaiting response... 302 Found
Location: http://rpmforge.sw.be/redhat/el3/en/x86_64/rpmforge/RPMS/rssh-2.3.2-1.1.el3.rf.x86_64.rpm [following]
--2010-10-11 20:36:21-- http://rpmforge.sw.be/redhat/el3/en/x86_64/rpmforge/RPMS/rssh-2.3.2-1.1.el3.rf.x86_64.rpm
Resolving rpmforge.sw.be... 18.104.22.168
Reusing existing connection to packages.sw.be:80.
HTTP request sent, awaiting response... 200 OK
Length: 45053 (44K) [application/x-rpm]
Saving to: “rssh-2.3.2-1.1.el3.rf.x86_64.rpm”
100%[====================================================================================================================================================>] 45,053 94.6K/s in 0.5s
2010-10-11 20:36:22 (94.6 KB/s) - “rssh-2.3.2-1.1.el3.rf.x86_64.rpm” saved [45053/45053]
[root@Internal ~]# rpm -ivh rssh-2.3.2-1.1.el3.rf.x86_64.rpm
warning: rssh-2.3.2-1.1.el3.rf.x86_64.rpm: Header V3 DSA signature: NOKEY, key ID 6b8d79e6
Preparing... ########################################### [100%]
1:rssh ########################################### [100%]
Updating Access Permissions
Now you should be able to set a user’s login shell to RSSH. Here is what the original line will usually look like.
This is what the updated line will look like.
What Happens if the User Attempts to SSH in After Access is Restricted
Now if joe attempts to login via SSH, the following will occur:
[root@Internal ~]# ssh joe@localhost
The authenticity of host 'localhost (127.0.0.1)' can't be established.
RSA key fingerprint is b5:39:02:23:01:a5:ff:b9:c1:aa:01:a9:69:21:a4:e0.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'localhost' (RSA) to the list of known hosts.
This account is restricted by rssh.
This user is locked out.
If you believe this is in error, please contact your system administrator.
Connection to localhost closed.