Fixing Performance Problems on Your JBoss Web Apps By Diagnosing Blocked Thread Issues

I was once per­plexed by a bizarre per­for­mance issue, I encoun­tered at seem­ing­ly ran­dom inter­vals, in an appli­ca­tion I help to main­tain. The appli­ca­tion kept freez­ing up, with­out any log mes­sages to use for diag­no­sis. This was very frus­trat­ing, because it meant the appli­ca­tion server typ­i­cal­ly had to be restart­ed man­u­al­ly to restore ser­vice.

After a bit of research, I learned of thread block­ing, as a poten­tial per­for­mance issue. Being as I was fair­ly cer­tain that the data­base was func­tion­ing with­in accept­able para­me­ters and the server had ample CPU and mem­o­ry to han­dle the load. I sought to deter­mine if thread block­ing was an issue.

I start­ed by sim­ply run­ning a twid­dle com­mand to dump the threads, when­ev­er this per­for­mance prob­lem was report­ed. This showed that the BLOCKED threads were indeed the cause. Con­tin­ue read­ing “Fix­ing Per­for­mance Prob­lems on Your JBoss Web Apps By Diag­nos­ing Blocked Thread Issues”

Monitoring Process Counts and Alerting Via Email

Below is a sim­ple script called monitor_jboss, which checks to see if jboss is run­ning and whether or not too many instances are cur­rent­ly run­ning. I found a need to write this script because we have some cron scripts which auto­mat­i­cal­ly restart JBoss each day and the JBoss shut­down script itself some­times fails to prop­er­ly shut down, caus­ing some quirky behav­ior.

If it deter­mi­nes that one of the fol­low­ing con­di­tions are true, it sends a short email to the address spec­i­fied in the vari­able email describ­ing the prob­lem.

  • JBoss is not run­ning at all
  • Jboss has more than max instances run­ning

This script is then placed in /etc/cron.d/cron.hourly/ where it will check the sys­tem once an hour and send an email as appro­pri­ate.

#!/bin/bash

# email addresses to send the message to
email="address@host.com"

# maximum number of concurrently running instances allowed
max=1

# determine the number of running instances
count_running_jbosses=$(ps aux | grep jboss | grep -v grep | grep -v monitor_jboss  | wc -l)

if [ $count_running_jbosses -eq "0" ]            # jboss isn't running
then
        message="JBoss Is Currently Not Running"
fi
if [ $count_running_jbosses >  $max ]           # too many jboss instances running
then
        message="JBoss Is Currently Running $count_running_jbosses instances; the maximum is $max"
fi

subject="JBOSS MONITORING ALERT FOR: $(hostname)"

echo "$message" | /bin/mail -s "$subject" "$email"