Copying Yesterday’s Exceptions with Stack Traces from Logs, Then Emailing To Administrators

When you have a java appli­ca­tion server which gen­er­ates a great deal of logs, it can be tricky to find the most impor­tant infor­ma­tion, espe­cial­ly if you have detailed log­ging. For­tu­nate­ly grep is capa­ble of doing this very well.

The fol­low­ing com­mand will gath­er all WARN, ERROR, FATAL, and Excep­tion stack traces. This com­mand can be very use­ful for Java log mon­i­tor­ing scripts.

cat /jboss-4.0.2/server/default/log/server.log | grep "ERROR\|FATAL\|Exception\|at.*\.java\:.*"

Understanding this expression

In this expres­sion ‘\|’ is used as an OR oper­a­tor to look for dif­fer­ent pat­terns. ‘WARN’, ‘ERROR’, and ‘FATAL’ pat­terns are used to fil­ter first line of the log event that can pos­si­bly con­tain an excep­tion at WARN, ERROR and FATAL log­ging lev­els. We lat­er fil­ter the first line of the stack trace with ‘Excep­tion’ as the first line of the stack trace usu­al­ly has the Excep­tion name fol­lowed by the excep­tion mes­sage e.g. ‘java.lang.NullPointerException’.

After this you will have the stack trace ele­ments which start with ‘at’ and end with pat­tern ‘(FileName.java:lineNo)’ e.g. at java.lang.Thread.run(Thread.java:595). The­se stack trace ele­ments are fil­tered with ‘at.*\.java\:.*’. All the­se pattern’s OR’ed togeth­er can fil­ter the com­plete stack trace in log at WARN, ERROR and FATAL log lev­el. Some false pos­i­tives may also get fil­tered out with this com­mand if the log com­ment has words like WARN, ERROR, FATAL, Excep­tion.

source: com­put­er tech­nol­o­gy roller

Filtering by Date: Yesterday’s Logs

If you want to fil­ter the log­files after a cer­tain date, the fol­low­ing com­mand is very use­ful. It gets the date for yes­ter­day, using the date for­mat yyyy-mm-dd, then is uses sed to print all of the lines after the spec­i­fied date. This is a good com­mand to run after mid­night, to retrieve the pre­vi­ous day’s logs.

cat /jboss-4.0.2/server/default/log/server.log | sed "1,/$(date --date='yesterday' '+%Y-%m-%d')/d"

Putting it All Together

Daily Log Monitor Script to Email Error Stack Traces to the Administrator

Here is a com­plete mon­i­tor­ing script I wrote, which emails me all of the pre­vi­ous days errors stack traces. I have it run­ning in cron.daily in order to reg­u­lar­ly send me the jboss error stack traces.

#!/bin/bash

# email addresses to send the message to
email="address@host.com"

# determine the number of running instances
errors=$(cat /jboss-4.0.2/server/default/log/server.log | sed "1,/$(date --date='yesterday' '+%Y-%m-%d')/d" | grep "ERROR\|FATAL\|Exception\|at.*\.java\:.*")

subject="JBOSS DAILY ERROR DIGEST FOR: $(hostname)"

echo "$errors" | /bin/mail -s "$subject" "$email"

Monitoring Process Counts and Alerting Via Email

Below is a sim­ple script called monitor_jboss, which checks to see if jboss is run­ning and whether or not too many instances are cur­rent­ly run­ning. I found a need to write this script because we have some cron scripts which auto­mat­i­cal­ly restart JBoss each day and the JBoss shut­down script itself some­times fails to prop­er­ly shut down, caus­ing some quirky behav­ior.

If it deter­mi­nes that one of the fol­low­ing con­di­tions are true, it sends a short email to the address spec­i­fied in the vari­able email describ­ing the prob­lem.

  • JBoss is not run­ning at all
  • Jboss has more than max instances run­ning

This script is then placed in /etc/cron.d/cron.hourly/ where it will check the sys­tem once an hour and send an email as appro­pri­ate.

#!/bin/bash

# email addresses to send the message to
email="address@host.com"

# maximum number of concurrently running instances allowed
max=1

# determine the number of running instances
count_running_jbosses=$(ps aux | grep jboss | grep -v grep | grep -v monitor_jboss  | wc -l)

if [ $count_running_jbosses -eq "0" ]            # jboss isn't running
then
        message="JBoss Is Currently Not Running"
fi
if [ $count_running_jbosses >  $max ]           # too many jboss instances running
then
        message="JBoss Is Currently Running $count_running_jbosses instances; the maximum is $max"
fi

subject="JBOSS MONITORING ALERT FOR: $(hostname)"

echo "$message" | /bin/mail -s "$subject" "$email"