/var/log/messages log file in Linux (Red Hat and Centos)

**********Logs in Linux (Centos&RedHat) - PART1**********

Click here: Logs in Linux (Centos&RedHat) - PART2
Click here: Logs in Linux (Centos&RedHat) - PART3

There are different log files available on Red Hat and Centos Linux Servers for different information like for kernel activities, services running on the Linux server and the applications that are deployed on the server.

This article is dedicated for the log file which I like most and has plenty of information about the system and various issues is /var/log/messages especially from a enterprise infra maintenance of a Linux Server point of view.

After reading the details below you will surely come to know how and what to look for in /var/log/messages log file which will help you to fix various Issues in Linux.

/var/log/messages log file is basically a read only file for users as the data is written to this file by system itself.

 You can use tools such as more, less, head, tail or vi to view the contents of this file.

Example:

#tail -n 5 /var/log/messages

Feb 2 07:35:44 server1 cib: [22388]: info: cib_stats: Processed 7 operations (5714.00us average, 0% utilization) in the last 10min
Feb 2 07:37:29 server1 PowerPath: Management Component: Warning: Cannot retrieve devices from MPAPI.
Feb 2 07:45:44 server1 cib: [22388]: info: cib_stats: Processed 7 operations (2857.00us average, 0% utilization) in the last 10min
Feb 2 07:47:29 server1 PowerPath: Management Component: Warning: Cannot retrieve devices from MPAPI.
Feb 2 07:55:44 server1 cib: [22388]: info: cib_stats: Processed 8 operations (5000.00us average, 0% utilization) in the last 10min

This file mainly has details about server startup/shutdown logs, messages related to storage functionality which is attached to server, local file system related errors, network ports link status, server restart time and cluster related messages if your server is in cluster.

I would suggest one has to look into /var/log/messages for issues like abrupt system reboot, SAN file system hung, fsck errors of local file systems  and network connectivity issues etc..to find a  reason for the issue.

There will be bunch of logs in the file  but the trick is to match  the time at which issue was first noticed and time stamp in the log file.

Look for the logs before and after around  that time frame.

Trace from /var/log/messages for different issues:

1. PowerPath: Management Component: Warning: Cannot retrieve devices from MPAPI.
                                      OR
    kernel: sd 3:0:0:0: SCSI error: return code = 0x00010000

This message indicates an issue with EMC storage that may be attached to server or just recently removed incorrectly from server.

 Fix:

Check the status of SAN file systems using #df -kh command whether the FS is healthy and read/write operation is possible.
Also check for other messages related to this issue in /var/log/messages till we get a conclusion of the issue.

2. kernel:eth0: link status is down

This message shows that the network interface eth0 is currently down and needs action to fix the issue.

 Fix:

Refer my other post below to troubleshoot network issues on Linux servers.
http://linuxunixdatabase.blogspot.com/2017/02/linuxunix-network-troubleshooting.html

3. If you see something like file system is corrupted or has errors

Refer my other post below to troubleshoot the fs related errors

http://linuxunixdatabase.blogspot.com/2017/02/file-system-state-is-clean-with-errors.html

4. Feb 19 04:02:02 server1 syslogd 1.4.1: restart.
This message actually indicates syslogd restart time. In my experience, I have seen this message shows the time lines which is very close to the server reboot.

For exact time of Linux Server restart, use the below command:

[root@server1 ]# last | grep boot
reboot   system boot  2.6.18-308.el5   Sat Nov 19 09:38         (97+00:06)
reboot   system boot  2.6.18-308.el5   Sat Oct 22 10:49         (27+23:33)

You may see many other issues while you are working, please feel free to post the issues here. I can surely get something interesting for you accordingly.

Click here: Logs in Linux (Centos&RedHat) - PART2
Click here: Logs in Linux (Centos&RedHat) - PART3

HAPPY LINUX LEARNING :)