I was once perplexed by a bizarre performance issue, I encountered at seemingly random intervals, in an application I help to maintain. The application kept freezing up, without any log messages to use for diagnosis. This was very frustrating, because it meant the application server typically had to be restarted manually to restore service.

After a bit of research, I learned of thread blocking, as a potential performance issue. Being as I was fairly certain that the database was functioning within acceptable parameters and the server had ample CPU and memory to handle the load. I sought to determine if thread blocking was an issue.

I started by simply running a twiddle command to dump the threads, whenever this performance problem was reported. This showed that the BLOCKED threads were indeed the cause.

Here are a couple of commands to create a thread dump in an old JBoss 4.0.3 server, using the twiddle utility. This first command performs a thread dump as HTML, then the second command opens the HTML in the links command line browser.

/home/jboss/tools/jboss-4.0.2/bin/twiddle.sh invoke "jboss.system:type=ServerInfo" listThreadDump  > /tmp/tmp.html  
links /tmp/tmp.html

After I got an understanding of this, I wrote a command line script in bash, to run the command via a cron job and send an email with the thread dump whenever a certain number of BLOCKED threads are detected.

The script dumps the threads using twiddle, to a temp file, counts the number of BLOCKED threads, then if any are detected it fires off an email using nail, a great utility which sends email messages using your preferred SMTP server.

If you use this script, it would be a good idea to change the threshold for sending an email to a higher value, once you’ve determined the level that warrants concern on your system.

#!/bin/bash  

# Monitor JBoss Threads  
# Chris Case  
# December 4, 2011  
# Checks JBoss status every minute for ERROR level messages.  Sends user(s) a notification of status  

Host="192.168.1.103"  
customer="My Customer Name"  
emails="@" #multiple emails space separated  
status=0  

/home/jboss/tools/jboss-4.0.2/bin/twiddle.sh invoke "jboss.system:type=ServerInfo" listThreadDump  > /tmp/tmp.html  

errors_size=$(links /tmp/tmp.html --dump | grep "threadState:" | grep "BLOCKED" | wc -l)  

if [ $errors_size != "0" ]; then  
        status=1;  
        subject="$errors_size BLOCKED THREADS DETECTED FOR: $Host - $customer"  
        errors=$(cat /tmp/tmp.html)  
        echo $errors;  
fi  

if [ $status = 1 ]; then  
        for address in $emails; do  
                echo -e "See the attached text file for the Exception stack traces." | nail -s "$subject" -a /tmp/tmp.html -r "noreply@myserver.com" -S smtp="smtp://mail.myserver.com:587" -S smtp-auth-user="myusername" -S smtp-auth-password="mypassword" $address  
                echo "Found an issue, sent email to $address"  
        done  
fi