Automatically Check RSYNC and Restart if Stopped

I occa­sion­al­ly use RSYNC to syn­chro­nize large direc­to­ries of files between servers. This is espe­cial­ly use­ful if you’re mov­ing a client from one server to anoth­er and they have alot of sta­t­ic files that are always chang­ing. You can copy the files and sync them up, all with RSYNC and if your con­nec­tion gets cut off, it will start where it left off. It will also grab changes to files that have already been RSYNCd.

I ran into an issue with RSYNC recent­ly, where­in the RSYNC process was run­ning in the back­ground; but was ter­mi­nat­ing due to errors sim­i­lar to the fol­low­ing. The­se con­nec­tions were prob­a­bly relat­ed to the slow and unsta­ble con­nec­tion to the remote server.

rsync: writefd_unbuffered failed to write 998 bytes to socket [sender]: Broken pipe (32)
rsync: connection unexpectedly closed (888092 bytes received so far) [sender]
rsync error: error in rsync protocol data stream (code 12) at io.c(600) [sender=3.0.6]

Given that I was trans­fer­ring files through a rel­a­tive­ly bad inter­net con­nec­tion and received this error a half dozen times over a cou­ple of days, I decid­ed the best way to han­dle it, would be to write a cron script. This cron script should check for the RSYNC process and start it if it isn’t run­ning.

rsync_check.sh

Cus­tomize this script for your own pur­pose, to check for your RSYNC process and start it if it isn’t run­ning.

#!/bin/bash
echo "checking for active rsync process"
COUNT=`ps ax | grep rsync | grep -v grep | grep -v rsync_check.sh | wc -l` # see how many are running
echo "there are $COUNT rsync related processes running";
if [ $COUNT -eq 0 ] 
then
	echo "no rsync processes running, restarting process"
	killall rsync  # prevent RSYNCs from piling up, if by some unforeseen reason there are already processes running
	rsync -avz -e "ssh" user@host.com:/mnt/syncdirectory/ /home/ccase/syncdirectory/ 
fi

Crontab Entry

Save the script in the appro­pri­ate cron direc­to­ry, or add it to the cron.d direc­to­ry and put a crontab entry in, to run it at the desired inter­val. This will have it run every 10 min­utes.

*/10 * * * * ccase /etc/cron.d/rsync_check.sh

No More Worries

Now you can move onto oth­er things, with the knowl­edge that your RSYNC will not just fail and leave the work undone. It prob­a­bly wouldn’t hurt to check on it at first and from time to time; but there’s alot less to wor­ry about!

3 thoughts on “Automatically Check RSYNC and Restart if Stopped”

  1. Hel­lo,

    Thank you for your inspi­ra­tion!
    The prob­lem i had with your solu­tion was that your script kills all rsync process­es. I just want to restart the one that is fail­ing. This is how i tried to solve the prob­lem. I am using this on my NAS so some path and com­mand are a bit dif­fer­ent

    #!/opt/bin/bash

    while true; do
    /opt/bin/rsync –rtv –timeout=30 –delete –pro­gress –modify-window=1 –e ‘ssh –p 22’
    rsync­sta­tus=$?
    if ((rsync­sta­tus == 0));
    then
    echo “direcories zijn in sync, exit script”
    break
    fi

    PID=‘ps | grep ’ [l]ocalhost>’ | awk ‘{print $1}’‘
    wait $PID
    echo “rsyncjob died, restart­ing (returned $rsync­sta­tus)”
    done

    Hope this helps

  2. This is great! How would you mon­i­tor pro­gress?

    Nor­mal­ly I run rsync in screen, with a –pro­gress flag… How would I be able to see pro­gress with this method? I mean, doubt it’s pos­si­bl, but still would be nice to be able to see what’s been hap­pen­ing with it (speed wise per file).

Leave a Reply

Your email address will not be published. Required fields are marked *