I was once perplexed by a bizarre performance issue, I encountered at seemingly random intervals, in an application I help to maintain. The application kept freezing up, without any log messages to use for diagnosis. This was very frustrating, because it meant the application server typically had to be restarted manually to restore service.
After a bit of research, I learned of thread blocking, as a potential performance issue. Being as I was fairly certain that the database was functioning within acceptable parameters and the server had ample CPU and memory to handle the load. I sought to determine if thread blocking was an issue.
I started by simply running a twiddle command to dump the threads, whenever this performance problem was reported. This showed that the BLOCKED threads were indeed the cause. Continue reading “Fixing Performance Problems on Your JBoss Web Apps By Diagnosing Blocked Thread Issues”
Below is a simple script called monitor_jboss, which checks to see if jboss is running and whether or not too many instances are currently running. I found a need to write this script because we have some cron scripts which automatically restart JBoss each day and the JBoss shutdown script itself sometimes fails to properly shut down, causing some quirky behavior.
If it determines that one of the following conditions are true, it sends a short email to the address specified in the variable email describing the problem.
- JBoss is not running at all
- Jboss has more than max instances running
This script is then placed in /etc/cron.d/cron.hourly/ where it will check the system once an hour and send an email as appropriate.
# email addresses to send the message to
# maximum number of concurrently running instances allowed
# determine the number of running instances
count_running_jbosses=$(ps aux | grep jboss | grep -v grep | grep -v monitor_jboss | wc -l)
if [ $count_running_jbosses -eq "0" ] # jboss isn't running
message="JBoss Is Currently Not Running"
if [ $count_running_jbosses > $max ] # too many jboss instances running
message="JBoss Is Currently Running $count_running_jbosses instances; the maximum is $max"
subject="JBOSS MONITORING ALERT FOR: $(hostname)"
echo "$message" | /bin/mail -s "$subject" "$email"