Fixing Performance Problems on Your JBoss Web Apps By Diagnosing Blocked Thread Issues

I was once per­plexed by a bizarre per­for­mance issue, I encoun­tered at seem­ing­ly ran­dom inter­vals, in an appli­ca­tion I help to main­tain. The appli­ca­tion kept freez­ing up, with­out any log mes­sages to use for diag­no­sis. This was very frus­trat­ing, because it meant the appli­ca­tion server typ­i­cal­ly had to be restart­ed man­u­al­ly to restore ser­vice.

After a bit of research, I learned of thread block­ing, as a poten­tial per­for­mance issue. Being as I was fair­ly cer­tain that the data­base was func­tion­ing with­in accept­able para­me­ters and the server had ample CPU and mem­o­ry to han­dle the load. I sought to deter­mine if thread block­ing was an issue.

I start­ed by sim­ply run­ning a twid­dle com­mand to dump the threads, when­ev­er this per­for­mance prob­lem was report­ed. This showed that the BLOCKED threads were indeed the cause.

Here are a cou­ple of com­mands to cre­ate a thread dump in an old JBoss 4.0.3 server, using the twid­dle util­i­ty. This first com­mand per­forms a thread dump as HTML, then the sec­ond com­mand opens the HTML in the links com­mand line browser.

/home/jboss/tools/jboss-4.0.2/bin/twiddle.sh invoke "jboss.system:type=ServerInfo" listThreadDump  > /tmp/tmp.html  
links /tmp/tmp.html

After I got an under­stand­ing of this, I wrote a com­mand line script in bash, to run the com­mand via a cron job and send an email with the thread dump when­ev­er a cer­tain num­ber of BLOCKED threads are detect­ed.

The script dumps the threads using twid­dle, to a temp file, counts the num­ber of BLOCKED threads, then if any are detect­ed it fires off an email using nail, a great util­i­ty which sends email mes­sages using your pre­ferred SMTP server.

If you use this script, it would be a good idea to change the thresh­old for send­ing an email to a high­er val­ue, once you’ve deter­mined the lev­el that war­rants con­cern on your sys­tem.

#!/bin/bash  
 
# Monitor JBoss Threads  
# Chris Case  
# December 4, 2011  
# Checks JBoss status every minute for ERROR level messages.  Sends user(s) a notification of status  
 
Host="192.168.1.103"  
customer="My Customer Name"  
emails="@" #multiple emails space separated  
status=0  
 
/home/jboss/tools/jboss-4.0.2/bin/twiddle.sh invoke "jboss.system:type=ServerInfo" listThreadDump  > /tmp/tmp.html  
 
errors_size=$(links /tmp/tmp.html --dump | grep "threadState:" | grep "BLOCKED" | wc -l)  
 
if [ $errors_size != "0" ]; then  
        status=1;  
        subject="$errors_size BLOCKED THREADS DETECTED FOR: $Host - $customer"  
        errors=$(cat /tmp/tmp.html)  
        echo $errors;  
fi  
 
if [ $status = 1 ]; then  
        for address in $emails; do  
                echo -e "See the attached text file for the Exception stack traces." | nail -s "$subject" -a /tmp/tmp.html -r "noreply@myserver.com" -S smtp="smtp://mail.myserver.com:587" -S smtp-auth-user="myusername" -S smtp-auth-password="mypassword" $address  
                echo "Found an issue, sent email to $address"  
        done  
fi

Leave a Reply

Your email address will not be published. Required fields are marked *