When you have a java application server which generates a great deal of logs, it can be tricky to find the most important information, especially if you have detailed logging. Fortunately grep is capable of doing this very well.

The following command will gather all WARN, ERROR, FATAL, and Exception stack traces. This command can be very useful for Java log monitoring scripts.

cat /jboss-4.0.2/server/default/log/server.log | grep "ERROR\|FATAL\|Exception\|at.*\.java\:.*"

Understanding this expression

In this expression ‘\|’ is used as an OR operator to look for different patterns. ‘WARN’, ‘ERROR’, and ‘FATAL’ patterns are used to filter first line of the log event that can possibly contain an exception at WARN, ERROR and FATAL logging levels. We later filter the first line of the stack trace with ‘Exception’ as the first line of the stack trace usually has the Exception name followed by the exception message e.g. ‘java.lang.NullPointerException’.

After this you will have the stack trace elements which start with ‘at’ and end with pattern ‘(FileName.java:lineNo)’ e.g. at java.lang.Thread.run(Thread.java:595). These stack trace elements are filtered with ‘at.*\.java\:.*’. All these pattern’s OR’ed together can filter the complete stack trace in log at WARN, ERROR and FATAL log level. Some false positives may also get filtered out with this command if the log comment has words like WARN, ERROR, FATAL, Exception.

source: computer technology roller

Filtering by Date: Yesterday’s Logs

If you want to filter the logfiles after a certain date, the following command is very useful. It gets the date for yesterday, using the date format yyyy-mm-dd, then is uses sed to print all of the lines after the specified date. This is a good command to run after midnight, to retrieve the previous day’s logs.

cat /jboss-4.0.2/server/default/log/server.log | sed "1,/$(date --date='yesterday' '+%Y-%m-%d')/d"

Putting it All Together

Daily Log Monitor Script to Email Error Stack Traces to the Administrator

Here is a complete monitoring script I wrote, which emails me all of the previous days errors stack traces. I have it running in cron.daily in order to regularly send me the jboss error stack traces.

#!/bin/bash

# email addresses to send the message to
email="address@host.com"

# determine the number of running instances
errors=$(cat /jboss-4.0.2/server/default/log/server.log | sed "1,/$(date --date='yesterday' '+%Y-%m-%d')/d" | grep "ERROR\|FATAL\|Exception\|at.*\.java\:.*")

subject="JBOSS DAILY ERROR DIGEST FOR: $(hostname)"

echo "$errors" | /bin/mail -s "$subject" "$email"