When troubleshooting a systems or application problem zeroing in on the logs can be the best place to go to determine the root cause of an issue. Yet some sysadmins tend to overlook the logs or even leave them for last due to the massive amounts of information they tend to contain.

Even if your log directory has gigabytes of data and you haven’t a clue what to search for, below are some steps to get you started in the right direction.

GNU Grep the mighty tool

You can think of grep as a filter. Say you noticed an error flash by while your system was booting.

Where do you look? It is important to know that the majority of log files stored on a  Linux operating system can be found within /var/log

Lets take a look at this directory

ls -l /var/log

Screenshot from 2015-03-10 12:41:26

The boot.log file is a good place to check for messages regarding the boot process.

Wouldn’t it be great to be able to filter this file for only the useful information?

cat /var/log/boot.log | more

Screenshot from 2015-03-10 12:43:31

When troubleshooting systems what keywords tend to come to mind?

error, fail, incorrect, exception, quit, exceeded, timeout, panic, reject, overrun, dropped 

Wouldn’t it be great if you could just combine these all into one filter and parse the log file for any of those keywords? You sure can by simply separating each of the keywords with a regex pipe ‘|’.

Lets demonstrate how this is done:

cat /var/log/boot.log | egrep -i 'error|fail|incorrect|exception|quit|exceeded|timeout|panic|reject|overrun|dropped'

Here we can see that our keyword named ‘fail’ has matched some text that exists within the boot.log file. Congratulations now that you have identified what is failing you can focus on fixing it.

Screenshot from 2015-03-10 13:10:31

If you have multiple directories or log files that you need to grep through you can also grep recursively by adding a -r.

Here we are going to grep through all files within a single directory:

grep -i  'keyword' *.log

You can even pipe the results

grep -i  'keyword' *.log > /home/sysadmin/Desktop/results.txt

Happy grepping!