bash - Chris Case

Getting the Last Modification Timestamp of a File with Stat

If we want to get just the date modified, for a file, in a format of our choosing. This can be done with a utility called stat.

The syntax is as follows:

stat -f <format> -t "<timestamp format>" <path to file>

In this example, we are printing just the date created in the format YYYYMMDD_HHMMSS.

stat -f "%Sm" -t "%Y%m%d_%H%M%S" filename.txt

We are using the –f “%Sm flag to specify that we want to print out only the date modified. The –t “%Y%m%d_%H%M%S” sets the date format.

In my example, the output was:

20121130_180221

This translates to November 30, 2012 at 18:02:21.

Using the Linux Command Line to Find and Copy A Large Number of Files from a Large Archive, Preserving Metadata

One of my recent challenges is to go through an archive on a NAS and find all of the .xlsx files, then copy them; preserving as much of the file metadata (date created, folder tree, etc) as possible, to a specified folder. After this copy, they will be gone through with another script, to rename the files, using the metadata, where they will then be processed by an application, which utilizes the name of the file in its process.

The part I want to share here, is finding the files and copying them to a folder, with metadata preserved. This is where the power of the find utility comes in handy.

Since this is a huge archive, I want to first produce a list of the files, that way I will be able to break this up into two steps. This will produce a list and write it into a text file. I am first going to run a find command on the volume I have mounted called data in my Volumes folder.

find /Volumes/data/archive/2012 -name '*.xlsx' > ~/archive/2012_files.txt

Now that the list is saved into a text file, I want to copy the files in the list, preserving the file metadata and path information, to my archive folder. The cpio utility accepts the paths of the files to copy from stdin, then copies them to my archive folder.

cat ~/archive/2012_files.txt | cpio -pvdm ~/archive

Appending to a Remote File via SSH

Most LINUX users know how to copy and overwrite a file from one server to another; but it can also be useful to directly append to a file, without having to login to the remote server and make the changes manually. This does not appear to be possible with the commonly used SCP utility; however, there is a way to do this with SSH. Its actually quite simple. Continue reading “Appending to a Remote File via SSH”

Fixing Performance Problems on Your JBoss Web Apps By Diagnosing Blocked Thread Issues

I was once perplexed by a bizarre performance issue, I encountered at seemingly random intervals, in an application I help to maintain. The application kept freezing up, without any log messages to use for diagnosis. This was very frustrating, because it meant the application server typically had to be restarted manually to restore service.

After a bit of research, I learned of thread blocking, as a potential performance issue. Being as I was fairly certain that the database was functioning within acceptable parameters and the server had ample CPU and memory to handle the load. I sought to determine if thread blocking was an issue.

I started by simply running a twiddle command to dump the threads, whenever this performance problem was reported. This showed that the BLOCKED threads were indeed the cause. Continue reading “Fixing Performance Problems on Your JBoss Web Apps By Diagnosing Blocked Thread Issues”

When a List of Files is Too Long for a Typical “rm” Command

I was on a client’s reporting server and noticed that an “ls” of their report logs took about 10 minutes. The directory had a log for every report run since June 2010, which is around 1.3 million files!

Here’s a transcript of the error:

[root@morpheus log]# pwd
/home/morpheus/tools/birt-runtime-2_0_1/Report Engine/log
You have new mail in /var/spool/mail/root
[root@morpheus log]# rm *
-bash: /bin/rm: Argument list too long

Continue reading “When a List of Files is Too Long for a Typical “rm” Command”

RSYNC a File Through a Remote Firewall

One of my recent tasks was to set-up a new automatic backup script, which dumps out the MySQL database on the remote host at a regular time, at a later time, it is RSYNC’d from the backup server through a remote firewall. I must say that I was a little surprised, to discover that the finished script and the configuration that goes along with it, was actually quite simple and easily repeatable. I was able to replicate the process for three sites very quickly and will easily be able to scale it to many more when necessary.

SSH Tunneling and SSH Keys

In order to perform a process on a remote firewalled host, you need to first set up keys, to allow the trusted backup server to gain access to the intermediate host. You must also set up a key which allows the intermediate host to gain access to the firewalled host.

First, let’s generate a public key on the backup server, if we don’t already have one. Be sure to use an empty pass phrase since this is an unattended script.

[backup@lexx log]# ssh-keygen -t dsa
Generating public/private dsa key pair.
Enter file in which to save the key (/backup/.ssh/id_dsa): 
Enter passphrase (empty for no passphrase): 
Enter same passphrase again: 
Your identification has been saved in /backup/.ssh/id_dsa.
Your public key has been saved in /backup/.ssh/id_dsa.pub.
The key fingerprint is:
3d:48:9c:0f:46:dc:da:c3:a6:19:82:63:b1:18:91:62 backup@lexx

The default key will, by default, be located in ~/.ssh/id_dsa.pub Copy the contents of this file to the clipboard, you will need this to get the remote server to trust the backup server.

Logon to the remote external server via ssh. On this server we will configure it to trust the backup server.

[backup@lexx ~]# ssh user@remotehost.com
user@remotehost.com's password: 
Last login: Thu Jul 14 22:57:58 2011 from 69.73.94.214
[user@remotehost ~]# ls -al .ssh
total 28
drwx------  2 user user 4096 2011-07-14 22:05 .
drwxr-x--- 12 user user 4096 2011-07-14 21:54 ..
-rw-------  1 user user 3024 2011-07-14 21:57 authorized_keys2
-rw-------  1 user user  668 2010-10-27 23:52 id_dsa
-rw-r--r--  1 user user  605 2010-10-27 23:52 id_dsa.pub
-rw-r--r--  1 user user 5169 2010-10-21 13:01 known_hosts

If the authorized_keys2 or simmilarly named file does not yet exist, create it and open the file in your text editor of choice. Then paste the key you copied from the id_dsa.pub file on the backup server.

To make the remote server recognize the newly added key run the following:

[user@remotehost ~]# ssh-agent sh -c 'ssh-add < /dev/null && bash'

Now we can make sure that the key works as intended by running the following command, which will ssh into the server and execute the uptime command:

[backup@lexx ~]$ ssh user@remotehost.com uptime
 23:57:17 up 47 days,  4:11,  1 user,  load average: 0.54, 0.14, 0.04

Since we got the output of the uptime command without a login prompt, it means the key was created successfully.

Now we repeat the ssh key process, this time between the remotehost server and the firewalled server.

[user@remotehost ~]# ssh-keygen -t dsa
Generating public/private dsa key pair.
Enter file in which to save the key (/user/.ssh/id_dsa): 
Enter passphrase (empty for no passphrase): 
Enter same passphrase again: 
Your identification has been saved in /user/.ssh/id_dsa.
Your public key has been saved in /user/.ssh/id_dsa.pub.
The key fingerprint is:
3d:48:9c:0f:46:dd:df:c3:a6:19:82:63:b1:18:91:62 user@remotehost

Copy the information from the .ssh/id_dsa.pub of the remote external server to the firewalled server, then add to the authorized_keys file and run:

[user@firewalledserver ~]# ssh-agent sh -c 'ssh-add < /dev/null && bash'

Now you should be able to pass the rsync command all the way through the remote firewall, to the firewalled server from the backup server.

This can be tested by the following command, which tunnels through the firewall and executes the uptime command on the internal server:

[backup@lexx ~]$ ssh user@remotehost.com ssh user@firewalledserver uptime
 23:52:17 up 41 days,  4:12,  1 user,  load average: 0.50, 0.13, 0.03

RSYNC The Data From the Backup Server, Through The Firewall

Now that we’ve got all of our keys set-up, most of the work has been done. I’m assuming you have a cron job on the internal server which dumps the mysql database at a specific time. You should set-up your rsync command late enough, so that the cron job has had enough time to dump the database.

Here is the rsync command which puts you through the firewall to download the remote mysql database dump. The –z flag allows you to do this with compression, which can significantly speed up the process.

[backup@lexx ~]$ rsync -avz -e "ssh user@remotehost.com ssh" user@firewalledserver:/home/user/rsync-backup/mysqldump.sql /home/backup/

This will create a hidden file that will be named something like .mysqldump.sql.NvD8D, which stores the data until the sync is complete. After the sync is complete you will see a file named mysqldump.sql in /home/backup/ folder.

Just set up the necessary cron scripts to make sure everything happens at the right time, possibly put some logging in there so you can see what has happened and you’re done!

Here’s an example of what I did on the backup server, to call the backup script. It appends the output of both STDOUT and STDERR to the /var/log/remote_backuplog file each time it is run. It also runs the script as the backup user so the files it generates have the correct permissions for the backup user to access.

01 6 * * * backup /home/backup/run_backups.sh >> /var/log/remote_backuplog 2>&1

Here is what my rsync script run_backups.sh looks like.

#!/bin/bash
 
echo "running backups"
# print the date into the logfile
date
 
# backup server 1
echo "backing up server1"
ssh user@externalserver1 ssh user@internalserver1 ls -l /home/user/rsync-backup/mysqldump.sql
/usr/bin/rsync -avz -e "ssh user@externalserver1 ssh" user@internalserver1:/home/user/rsync-backup/mysqldump.sql /home/backup/server1/
 
# backup server 2
echo "backing up server2"
ssh user@externalserver2 ssh user@internalserver2 ls -l /home/user/rsync-backup/mysqldump.sql
/usr/bin/rsync -avz -e "ssh user@externalserver2 ssh" user@internalserver2:/home/user/rsync-backup/mysqldump.sql /home/backup/server2/
 
# backup server 3
echo "backing up server3"
ssh user@externalserver3 ssh user@internalserver3 ls -l /home/user/rsync-backup/mysqldump.sql
/usr/bin/rsync -avz -e "ssh user@externalserver3 ssh" user@internalserver3:/home/user/rsync-backup/mysqldump.sql /home/backup/server3/

Quick and Easy Regular Expression Command/Script to Run on Files in the Bash Shell

I often find it necessary to run regular expressions on, not just one file; but instead a range of files. There are perhaps dozens of ways this can be done, with varying levels of understanding necessary to do this.

The simplest way I have encountered utilizes the following syntax:

perl -pi -e "s/<find string>/<replace with string>/g" <files to replace in>

Here is an example where I replace the IP address in a range of report templates with a different IP address:

perl -pi -e "s/mysql:\/\/192.168.2.110/mysql:\/\/192.168.2.111/g" $reportTemplateLocation/*.rpt*

Basically, I am looking for a line which contains mysql://192.168.2.110, which I want to replace with mysql://192.168.2.111.

Here is an example of a bash script I call changeReportTemplateDatabase.sh, which I wrap around that command, to accomplish that same task with more elegance:

#!/bin/bash
#
# @(#)$Id$
#
# Point the report templates to a different database IP address.
reportTemplateLocation="/home/apphome/jboss-4.0.2/server/default/appResources/reportTemplates";
 
error()
{
    echo "$arg0: $*" 1>&2
    exit 1
}
usage()
{
        echo "Usage $0 -o <old-ip-address> -n <new-ip-address>";
}
 
vflag=0
oldip=
newip=
while getopts hvVo:n: flag
do
    case "$flag" in
    (h) help; exit 0;;
    (V) echo "$arg0: version 0.1 8/28/2010"; exit 0;;
    (v) vflag=1;;
    (o) oldip="$OPTARG";;
    (n) newip="$OPTARG";;
    (*) usage;;
    esac
done
shift $(expr $OPTIND - 1)
 
if [ "$oldip" = "" ]; then
        usage;
        exit 1;
fi
if [ "$newip" = "" ]; then
        usage;
        exit 1;
fi
 
echo "$0: Changing report templates to use the database at $newip from $oldip";
perl -pi -e "s/mysql:\/\/$oldip/mysql:\/\/$newip/g" $reportTemplateLocation/*.rpt*

Usage of the script is as simple as the command below. It will change every database reference on report templates in the directory referenced by the variable reportTemplateLocation to the new value.

./changeReportTemplateDatabase.sh  -o 192.168.2.110 -n 192.168.2.111

A further improvement, which may be useful to some, would be to make the directory a flag which can be edited at the command line.

How To Execute a Script After a Running Process Completes

Most people who are familiar with Linux, realize that there are ways of chaining processes to run one after another. Typically this is done by writing a script, or using && to daisy chain additional commands on command line.

There is, however, another way to do this; if you’ve already issued a command and want to add another command after the original has already started. This is especially useful if you’re unzipping, say a 15 gigabyte database dump, and you want to make sure that the import happens immediately after the import is complete.

Here’s an example of what would happen if I were entering the commands manually.

Macintosh:~ chriscase$ scp user@hostname.com:archive/database.sql.gz .
Macintosh:~ chriscase$ gunzip database.sql.gz
Macintosh:~ chriscase$ mysql -u dbusername -pdbpassword dbname &lt; database.sql

Since I’m not going to stay glued to the console for the entire duration of this process, I either need to write a script or figure out another technique, so I keep things moving along.

As it turns out, there is a very simple way to accomplish this, with the command wait. This command has the ability to wait until a specified process is complete before executing a command.

Here’s an example of how this could be used, if you wanted to add the last two processes after the scp from the above example had already begun.

Macintosh:~ chriscase$ scp user@hostname.com:archive/database.sql.gz .

Once the download is kicked off, you can kick it into the background by using [ctrl-z] which will pause the process and then issuing the command [bg]. This will put the paused process running into the background. Now, to chain the other processes afterward, you can do the following.

Macintosh:~ chriscase$ wait %1 && gunzip database.sql.gz && mysql -u dbusername -pdbpassword dbname < database.sql

The above code will wait to execute until the scp is done, then it will use gzip to unzip the file and mysql to import the database dump file. Now that you’ve done this, you can go off and do something else, confident that your database will be done importing in a few hours.

Sending Mail in Shell Scripts via an External Server with Nail

If you’ve ever tried sending email via the command line, using the mail utility, you may find that the method can be unreliable in some cases. The email messages are often intercepted by spam bots, filtered by security programs, etc. A more elegant and stable alternative, is to use your existing email server to send the message. Using the program nail makes this an easy task to do via the command line.

The following example shows you how to send a simple message with attachment. Here is the syntax for sending a message with nail.

echo "" | nail -s "" -a   ...

In order for nail to function, you must have the .mail configuration file in your path. Here is a sample .mail configuration file to get you started quickly.

set smtp=smtp://yourhost.com
set from="yourname@yourhost.com (Display Name)"
set smtp-auth=login
set smtp-auth-user=your_username
set smtp-auth-password=your_password

Bash Shell Script to Convert All MySQL Tables to use the Innodb Engine

The following, when run in the Bash shell, will convert the engine for all of your MySQL tables to InnoDB.

mysql -B -N -e "SHOW TABLES" -u  <username> --password=<password> <databasename> | while read table; \
do \
     echo "+ Converting Table $table"; \
     mysql -B -N -e "alter table $table engine=innodb" -u <username> --password=<password> <databasename>; \ 
done;

Often, if you have a heavily used database, you will want to consider the InnoDB engine. By default, the MySQL database engine is MyISAM. However, the InnoDB engine has many advantages; particularly for high utilization environments. With InnoDB data-integrity is maintained throughout the entire query process because it is transaction safe.

InnoDB also provides row-locking, as opposed to table-locking. With row-locking, while one query is busy updating or inserting a row, another query can update a different row at the same time. Hence InnoDB has superior features for multi-user concurrency and performance.