I occasionally use RSYNC to synchronize large directories of files between servers. This is especially useful if you’re moving a client from one server to another and they have alot of static files that are always changing. You can copy the files and sync them up, all with RSYNC and if your connection gets cut off, it will start where it left off. It will also grab changes to files that have already been RSYNCd.
I ran into an issue with RSYNC recently, wherein the RSYNC process was running in the background; but was terminating due to errors similar to the following. These connections were probably related to the slow and unstable connection to the remote server.
rsync: writefd_unbuffered failed to write 998 bytes to socket [sender]: Broken pipe (32)
rsync: connection unexpectedly closed (888092 bytes received so far) [sender]
rsync error: error in rsync protocol data stream (code 12) at io.c(600) [sender=3.0.6]
Given that I was transferring files through a relatively bad internet connection and received this error a half dozen times over a couple of days, I decided the best way to handle it, would be to write a cron script. This cron script should check for the RSYNC process and start it if it isn’t running.
Customize this script for your own purpose, to check for your RSYNC process and start it if it isn’t running.
echo "checking for active rsync process"
COUNT=`ps ax | grep rsync | grep -v grep | grep -v rsync_check.sh | wc -l` # see how many are running
echo "there are $COUNT rsync related processes running";
if [ $COUNT -eq 0 ]
echo "no rsync processes running, restarting process"
killall rsync # prevent RSYNCs from piling up, if by some unforeseen reason there are already processes running
rsync -avz -e "ssh" email@example.com:/mnt/syncdirectory/ /home/ccase/syncdirectory/
Save the script in the appropriate cron directory, or add it to the cron.d directory and put a crontab entry in, to run it at the desired interval. This will have it run every 10 minutes.
*/10 * * * * ccase /etc/cron.d/rsync_check.sh
No More Worries
Now you can move onto other things, with the knowledge that your RSYNC will not just fail and leave the work undone. It probably wouldn’t hurt to check on it at first and from time to time; but there’s alot less to worry about!
Linux makes it relatively easy to mount shared drives either manually, at the command line, or automatically, by configuring an entry in /etc/fstab. Here is the basic syntax of our mount command.
[ccase@midas ~]$ sudo mount -t cifs -o username=<share username>,password=<share password>,<additional options> //<name or ip of server>/<share name> <folder to mount to>
Here is an example of mounting our CIFS share to a folder named myshare. We are using the option ro to mount the share read only.
[ccase@midas ~]$ sudo mount -t cifs -o username=admin,password=secret,ro //192.168.1.200/myshare myshare
If we want to make this automatic, it can easily be configured in /etc/fstab/ to mount after the network comes up. Here is the basic syntax you would use in /etc/fstab/
//<name or ip of server>/<share name> <folder to mount to> cifs username=<share username>,password=<share password>,_netdev,<additional options> 0 0
Here is an example of mounting our CIFS share automatically to /mnt/myshare/. We are using the option _netdev, to tell it to attempt the mount only after the network has come up and ro, to mount the share read only.
//192.168.1.200/myshare /mnt/myshare cifs username=admin,password=secret,_netdev,ro 0 0
One of my recent challenges is to go through an archive on a NAS and find all of the .xlsx files, then copy them; preserving as much of the file metadata (date created, folder tree, etc) as possible, to a specified folder. After this copy, they will be gone through with another script, to rename the files, using the metadata, where they will then be processed by an application, which utilizes the name of the file in its process.
The part I want to share here, is finding the files and copying them to a folder, with metadata preserved. This is where the power of the find utility comes in handy.
Since this is a huge archive, I want to first produce a list of the files, that way I will be able to break this up into two steps. This will produce a list and write it into a text file. I am first going to run a find command on the volume I have mounted called data in my Volumes folder.
find /Volumes/data/archive/2012 -name '*.xlsx' > ~/archive/2012_files.txt
Now that the list is saved into a text file, I want to copy the files in the list, preserving the file metadata and path information, to my archive folder. The cpio utility accepts the paths of the files to copy from stdin, then copies them to my archive folder.
cat ~/archive/2012_files.txt | cpio -pvdm ~/archive
I ran into an issue recently, where an existing log4j.xml configuration file was built into a jar file I was referencing and I was unable to get Java to recognize another file that I wanted it to use instead. Fortunately, the solution to this problem is fairly straightforward and simple.
I was running a standalone application in linux, via a bash shell script; but this technique can be used in other ways too. You simply add a parameter to the JVM call like the example below.
So the syntax is basically:
java -Dlog4j.configuration="file:<full path to file>" -cp <classpath settings> <package name where my main function is located>
Lets say I have a file named log4j.xml in /opt/tools/myapp/ which I want to use when my application runs, instead of any existing log4j.xml files. This can be done by passing a JVM flag -Dlog4j.configuration to Java.
Here is an example:
java -Dlog4j.configuration="file:/opt/tools/myapp/log4j.xml" -cp $CLASSPATH my.standalone.mainClass;
With that change, as long as your log4j file is set up properly, your problems should be behind you.
I have recently started using a Toshiba P870 laptop and decided to install Linux Mint 13 Maya (Cinnamon Edition) on it, due to its ease of use and overall security soundness.
Being as the Toshiba P870 is a relatively new laptop, with some components’ drivers not having been included in the installation files of Mint, it has been a little tricky. I’m sharing this, for those who want to install mint on the P870 or similar laptops. This should save you a couple hours of searching. It will get you the drivers you need and get you up and running.
We’re going to discuss howto:
- Download, Burn and Run the Linux Mint Installer
- Install the missing network drivers, both WIFI and Ethernet, so you can connect to the internet
- Fix the internal sound problem that causes the internal speakers not to produce any sound
- Update: Installing the SD Card driver — since this article was originally written, I also figured out that the SD card driver needs to be installed as well.
If you’re dealing with systems behind a firewall it’s almost inevitable that you will need to tunnel into those systems from time to time. Fortunately, there are some quick & easy commands to accomplish this. In this example, we are going to use a Mac OSX or linux-based system, to gain access to a web server’s port 80 on a fire-walled server.
Let’s say the domain of the remote server is dfrn.net, the fire-walled server has an IP address of 192.168.1.100 and the firewalled server has a web server at port 80. We need to choose an unused port on our own system, in this case we’ll use 2020.
So our side of the tunnel is going to be http://localhost:2020/ and the other side of the tunnel will be http://192.168.1.100:80/.
ssh firstname.lastname@example.org -L 2020:192.168.1.100:80
So, now port 80 on the fire-walled server will be accessible by simply pointing your web browser to http://localhost:2020/. To terminate the tunnel, simply exit the shell.
When you’re trying to move a large block of files, its often useful to do so in one command and to be able to close your terminal window (or allow it to time out). If you run a command under normal circumstances, losing the connection can cause your command to terminate prematurely, this is where nohup (No HangUP — a utility which allows a process to continue even after a connection is lost) comes in.
Let’s say we have a large directory to backup, which we want to first tar, then gzip; keeping the command non-dependent on the continuity of the terminal session. Continue reading “Tar/GZip Files in One Operation, Unattached to the Terminal Session”
Today, it came to my attention that a LINUX computer, priced between $25–35, is now available. This computer is called the Raspberry Pi.
It sure looks like computing is going to take on a whole new dimension in the coming years. No longer are there going to be significant financial barriers to acquisition, meaning they will be everywhere and clusters of extremely cheap computers will add yet another dimension to cloud computing.
I think this will mean that computer technical skills are going to eventually be synonymous to literacy. Surely computers and computing will continue to evolve at a feverish pace, eliminating much of the unnecessary human toil.
Continue reading “The Paradigm Shift to Accompany The Advent of Cheap Computing”