/var/log/messages log file in Linux (Red Hat and Centos)

**********Logs in Linux (Centos&RedHat) - PART1**********

Click here: Logs in Linux (Centos&RedHat) - PART2
Click here: Logs in Linux (Centos&RedHat) - PART3

There are different log files available on Red Hat and Centos Linux Servers for different information like for kernel activities, services running on the Linux server and the applications that are deployed on the server.

This article is dedicated for the log file which I like most and has plenty of information about the system and various issues is /var/log/messages especially from a enterprise infra maintenance of a Linux Server point of view.

After reading the details below you will surely come to know how and what to look for in /var/log/messages log file which will help you to fix various Issues in Linux.

/var/log/messages log file is basically a read only file for users as the data is written to this file by system itself.

 You can use tools such as more, less, head, tail or vi to view the contents of this file.

Example:

#tail -n 5 /var/log/messages

Feb 2 07:35:44 server1 cib: [22388]: info: cib_stats: Processed 7 operations (5714.00us average, 0% utilization) in the last 10min
Feb 2 07:37:29 server1 PowerPath: Management Component: Warning: Cannot retrieve devices from MPAPI.
Feb 2 07:45:44 server1 cib: [22388]: info: cib_stats: Processed 7 operations (2857.00us average, 0% utilization) in the last 10min
Feb 2 07:47:29 server1 PowerPath: Management Component: Warning: Cannot retrieve devices from MPAPI.
Feb 2 07:55:44 server1 cib: [22388]: info: cib_stats: Processed 8 operations (5000.00us average, 0% utilization) in the last 10min

This file mainly has details about server startup/shutdown logs, messages related to storage functionality which is attached to server, local file system related errors, network ports link status, server restart time and cluster related messages if your server is in cluster.

I would suggest one has to look into /var/log/messages for issues like abrupt system reboot, SAN file system hung, fsck errors of local file systems  and network connectivity issues etc..to find a  reason for the issue.

There will be bunch of logs in the file  but the trick is to match  the time at which issue was first noticed and time stamp in the log file.

Look for the logs before and after around  that time frame.

Trace from /var/log/messages for different issues:

1. PowerPath: Management Component: Warning: Cannot retrieve devices from MPAPI.
                                      OR
    kernel: sd 3:0:0:0: SCSI error: return code = 0x00010000

This message indicates an issue with EMC storage that may be attached to server or just recently removed incorrectly from server.

 Fix:

Check the status of SAN file systems using #df -kh command whether the FS is healthy and read/write operation is possible.
Also check for other messages related to this issue in /var/log/messages till we get a conclusion of the issue.

2. kernel:eth0: link status is down

This message shows that the network interface eth0 is currently down and needs action to fix the issue.

 Fix:

Refer my other post below to troubleshoot network issues on Linux servers.
http://linuxunixdatabase.blogspot.com/2017/02/linuxunix-network-troubleshooting.html

3. If you see something like file system is corrupted or has errors

Refer my other post below to troubleshoot the fs related errors

http://linuxunixdatabase.blogspot.com/2017/02/file-system-state-is-clean-with-errors.html

4. Feb 19 04:02:02 server1 syslogd 1.4.1: restart.
This message actually indicates syslogd restart time. In my experience, I have seen this message shows the time lines which is very close to the server reboot.

For exact time of Linux Server restart, use the below command:

[root@server1 ]# last | grep boot
reboot   system boot  2.6.18-308.el5   Sat Nov 19 09:38         (97+00:06)
reboot   system boot  2.6.18-308.el5   Sat Oct 22 10:49         (27+23:33)

You may see many other issues while you are working, please feel free to post the issues here. I can surely get something interesting for you accordingly.

Click here: Logs in Linux (Centos&RedHat) - PART2
Click here: Logs in Linux (Centos&RedHat) - PART3

HAPPY LINUX LEARNING :)

Removing existing LVM from your Linux System

GOAL:

To remove or clean up the existing LVs on Linux Server: 

I described the step by step process on how to remove the existing LVMs from your Linux Box.

This article best works with RedHat Linux and Centos Distributions.

1. We need to know what LVs are present on our Linux Box. Use the below commands to see LVs on your box. 

#lvs  OR  #lvdisplay 

Output is as below:

# lvs

  LV            VG              Attr      LSize    Pool Origin Data%  Move Log Cpy%Sync Convert
  linxback01.vol datavg01        -wi-ao--- 1000.00g
  linxback00.vol linxdatavg00 -wi-ao---    5.00t

# lvdisplay

  --- Logical volume ---
  LV Path                /dev/datavg01/linxback01.vol
  LV Name                linxback01.vol
  VG Name                datavg01
  LV UUID                2k8o6B-ZJuS-40dX-FZF0-Ym96-bpd7-UQ6fY6
  LV Write Access        read/write
  LV Creation host, time host1, 2015-12-18 09:24:11 -0500
  LV Status              available
  # open                 1
  LV Size                1000.00 GiB
  Current LE             256000
  Segments               1
  Allocation             inherit
  Read ahead sectors     auto
  - currently set to     256
  Block device           253:0

  --- Logical volume ---
  LV Path                /dev/linxdatavg00/linxback00.vol
  LV Name                linxback00.vol
  VG Name                linxdatavg00
  LV UUID                aqZsFS-djb4-v0Ws-Hzbv-1gIk-Vjmb-1hsHU7
  LV Write Access        read/write
  LV Creation host, time host1, 2014-08-26 17:00:28 -0400
  LV Status              available
  # open                 1
  LV Size                5.00 TiB
  Current LE             1310715
  Segments               5
  Allocation             inherit
  Read ahead sectors     auto
  - currently set to     256
  Block device           253:1

==>We also need to know what is the associated PV Name to the LV. Command is below: 

# pvs

  PV             VG              Fmt  Attr PSize    PFree
  /dev/emcpowerf03 datavg01        lvm2 a--  1024.00g 24.00g

2. I can go ahead and start the removal process of both the LVMs shown above.But  you also need to know on which file system these LVa are mounted.

How to check that? Here is the way...

#df -kh

Just check for linxback00.vol and linxback01.vol in the output to see which file systems are using this LVs.

Example:

/dev/mapper/linxdatavg00-linxback00.vol
                      5.0T  3.6T  1.1T  77% /Backup_FileSystem

So, the above entry in the df -kh output shows the LV: linxback00.vol is mounted on the Directory : /Backup_FileSystem
Before we remove the LV, we need to unmounts and delete associated File system.

3. Unmounting associated File system using the Commands below:

fuser -cu /Backup_FileSystem
umount /Backup_FileSystem

4. We are all set to remove associated LV now.

#lvremove

Example :#lvremove  linxback01.vol
 

5. Remove associated VG

#vgremove <VG name>

Example: #vgremove datavg01

6. Remove associated PV

pvremove <device name>

Example:#pvremove   /dev/emcpowerf03

7.Validate using below commands if the LV is cleaned as expected. Ideally there should not be any output for the below commands once LV clean up is done. These are all non-disruptive commands.

#vgscan

#lvs

#vgs

#pvs

Important Note:

Usually LVs are associated with external SAN. The above process cleans up your LVs but SAN will still be there on system. You need to Work with your storage team to get the LUNs unmounted from your Linux server as soon as you are done with the LV removal process.

HAPPY LINUX LEARNING AS ALWAYS :)

Feel free to ask any questions or start a discussion about this topic.

My other Posts are below:

File System State is clean with errors in Linux:

http://linuxunixdatabase.blogspot.com/2017/02/file-system-state-is-clean-with-errors.html

How to use IPERF to test interface/network throughput in Linux:

http://linuxunixdatabase.blogspot.com/2017/02/how-to-use-iperf-to-test.html

Linux/Unix Network Troubleshooting:

http://linuxunixdatabase.blogspot.com/2017/02/linuxunix-network-troubleshooting.html

Learning AWK and SED Tools for LINUX/UNIX
http://linuxunixdatabase.blogspot.com/2017/02/learning-awk-and-sed-tools-for-linuxunix.html