File System State is clean with errors in Linux

ISSUE: File system on device reported state clean with errors in Linux

SOLUTION:

 ==>This issue is usually reported in /var/log/messages or may be noticed at boot time and is really critical.

==>This issue can be dangerous when data consistency is considered on reported File System
==>File System errors can be noticed on root file system, SAN file systems or even cluster file systems.

Steps to fix the issue for Non-Root file systems:
===================================
Suppose say /dev/sda10 (/fs1) is showing file system state clean with errors.
you can see how /dev/sda10  and /fs1 are related in df -kh command output

Example:
/dev/sda10             7.8G  6.3G  1.2G  85% /fs1

1. Take backup of the data from /fs1 directory and place  the backup on a different system if possible and one copy locally.

Use the below command to tar up the data and to create a tar file of the same.        

#tar -cvzf /fs1_<date>.tar.gz /fs1/*

You will have a tar file something like /fs1_20150201.tar.gz (Just replace the date as per your requirement in the command)

2. #df -kh

if you see the file system /fs1 is mounted, then unmount it like below:

#umount /fs1

Just check the status of the errors before we try to fix the issue actually.

Below command will not fix the issue rather tells us how many errors are present on the FS.

3.#fsck -n /fs1

Output is like below:
#fsck -n /fs1
fsck 1.39 (29-May-2006)
e2fsck 1.39 (29-May-2006)
Warning: skipping journal recovery because doing a read-only filesystem check.
/fs1 contains a file system with errors, check forced.
Pass 1: Checking inodes, blocks, and sizes
Pass 2: Checking directory structure
Pass 3: Checking directory connectivity
Pass 4: Checking reference counts
Unattached zero-length inode 179868.  Clear? no
Unattached inode 179868
Connect to /lost+found? no
Pass 5: Checking group summary information
Block bitmap differences:  +386446
Fix? no
Free blocks count wrong for group #11 (2, counted=3).
Fix? no
Inode bitmap differences:  +179868
Fix? no
Free inodes count wrong for group #11 (15350, counted=15351).
Fix? no
Free inodes count wrong (2072117, counted=2072133).
Fix? no

/fs1: ********** WARNING: Filesystem still has errors **********
/fs1: 12747/2084864 files (10.0% non-contiguous), 1953016/4162197 blocks

4.#tune2fs -l /dev/sda10

tune2fs 1.39 (29-May-2006)
Filesystem volume name:   /fs1
Last mounted on:          <not available>
Filesystem UUID:          cf72df5e-24fa-4f40-92c6-244459r49e17
Filesystem magic number:  0xEF18
Filesystem revision #:    1 (dynamic)
Filesystem features:      has_journal resize_inode dir_index filetype needs_recovery sparse_super large_file
Default mount options:    (none)
Filesystem state:         Clean with errors
Errors behavior:          Continue
Filesystem OS type:       Linux
Inode count:              2081264
Block count:              4164597
Reserved block count:     208789
Free blocks:              2209181
Free inodes:              2074517
First block:              0
Block size:               4096
Fragment size:            4096
Reserved GDT blocks:      1016
Blocks per group:         32768
Fragments per group:      32768
Inodes per group:         16288
Inode blocks per group:   509
Filesystem created:       Mon Oct 26 02:50:22 2009
Last mount time:          Sat Feb 11 09:12:30 2017
Last write time:          Sat Feb 11 09:12:30 2017
Mount count:              68
Maximum mount count:      25
Last checked:             Wed May 29 20:38:40 2013
Check interval:           15552000 (6 months)
Next check after:         Mon Nov 25 19:38:40 2013
Reserved blocks uid:      0 (user root)
Reserved blocks gid:      0 (group root)
First inode:              11
Inode size:               128
Journal inode:            8
Default directory hash:   tea
Directory Hash Seed:      b16ea660-bcea-4a0a-8e9e-926b058e1928
Journal backup:           inode blocks
Things to note in the above command is File system state: clean   OR Not Clean

Check for any errors and note down the output
Looking at the above commands output, /fs1 is not clean and fsck has shown errors. So needs to be cleaned to avoid any data corruption.

5. Start the FSCK now to fix the above seen issues on /fs1 file system

Important Note: Running fsck -f -y on a mounted system will cause data loss. Be cautious.

#screen
#fsck -f -y /dev/sda10
This command may run for 4 to 5 Hrs or may be more. Monitor till the command is executed completely

6. Verify if the errors are gone on /fs1 using tune2fs

#tune2fs -l //dev/sda10
file system state should be clean.
#fsck -n /dev/sda10
check if any errors are still there.

7. If /fs1 is clean, mount it back and start using it as usual

#mount /fs1

=====

Running FSCK on root file system needs to be done through single user mode or maintenance mode in Linux as we cannot unmount root FS.

HAPPY LINUX LEARNING :)

Feel free to ask any questions or start a discussion about this topic.

My other Posts are below:

File System State is clean with errors in Linux:
http://linuxunixdatabase.blogspot.com/2017/02/file-system-state-is-clean-with-errors.html

How to use IPERF to test interface/network throughput in Linux:
http://linuxunixdatabase.blogspot.com/2017/02/how-to-use-iperf-to-test.html

Linux/Unix Network Troubleshooting:
http://linuxunixdatabase.blogspot.com/2017/02/linuxunix-network-troubleshooting.html

Removing existing LVM from your Linux System
http://linuxunixdatabase.blogspot.com/2017/02/removing-existing-lvm-from-your-linux.html

Learning AWK and SED Tools for LINUX/UNIX
http://linuxunixdatabase.blogspot.com/2017/02/learning-awk-and-sed-tools-for-linuxunix.html