Learning AWK and SED Tools for LINUX/UNIX

GOAL:

Learning AWK and SED Tools for LINUX/UNIX:

I have posted so far L2 level of Linux issues and the processes to fix them.

• In this post I would like to take something simple but still interesting.
• It is sometimes may be important to have some kind of scripting knowledge in hand for a better Linux/Unix system administration.
• Before I post something on scripting with examples, I would like to introduce two command line tools namely AWK and SED which are really powerful in LINUX/Unix world.
• So, this post has the information about most frequently used data filtering commands in Linux/Unix world, none other than AWK and SED.

What is AWK and why is it used for?

Awk is both a programming language and text processor that can be used to manipulate text data in desirable manner.

Exercise using AWK:

To exercise a scenario with AWK, let's take a file called myfile.txt which has the below information.
=======

root x 0 0 root /root /bin/bash

bin x 1 1 bin /bin /sbin/nologin

Albert x 2 2 daemon /sbin /sbin/nologin

Chin x 3 4 adm /var/adm /sbin/nologin

Neon x 4 7 lp /var/spool/lpd /sbin/nologin

sync x 5 0 sync /sbin /bin/sync

shutdown x 6 0 shutdown /sbin /sbin/shutdown

halt x 7 0 halt /sbin /sbin/halt

mail x 8 12 mail /var/spool/mail /sbin/nologin

=======

• In the above data, first column shows user name, and last column(7th column) shows login details like whether bash login is allowed or not.
• From the above output I Just need to know the user names and corresponding login details.

Here is how AWK is used to get the required output.

#cat myfile.txt | awk '{print $1" "$7}'
root /bin/bash

bin /sbin/nologin

Albert /sbin/nologin

Chin /sbin/nologin

Neon /sbin/nologin

sync /bin/sync

shutdown /sbin/shutdown

halt /sbin/halt

mail /sbin/nologin

If I want to separate the output using a colon, I can use the below command:

# cat myfile.txt | awk '{print $1" : "$7}'
root : /bin/bash

bin : /sbin/nologin

Albert : /sbin/nologin

Chin : /sbin/nologin

Neon : /sbin/nologin

sync : /bin/sync

shutdown : /sbin/shutdown

halt : /sbin/halt

mail : /sbin/nologin

We can separate the columns as per our convenience like below:
 #cat myfile.txt | awk '{print $1" ==> "$7}'

root ==> /bin/bash

bin ==> /sbin/nologin

Albert ==> /sbin/nologin

Chin ==> /sbin/nologin

Neon ==> /sbin/nologin

sync ==> /bin/sync

shutdown ==> /sbin/shutdown

halt ==> /sbin/halt

mail ==> /sbin/nologin

--------------

In this same article, I would like to introduce SED as well.

What is SED?

• SED is a stream editor. 
• SED is used to perform basic text transformations on an input stream like  a file or input from a pipeline. SED works by making only one pass over the input(s), and is consequently more efficient.  But it is sed’s ability to filter text in a pipeline which particularly distinguishes it from other types of editors.

Suppose say, you have the data in newfile.txt is something like below:

root:x:0:0:root:/root:/bin/bash

bin:x:1:1:bin:/bin:/sbin/nologin

Albert:x:2:2:daemon:/sbin:/sbin/nologin

Chin:x:3:4:adm:/var/adm:/sbin/nologin

Neon:x:4:7:lp:/var/spool/lpd:/sbin/nologin

sync:x:5:0:sync:/sbin:/bin/sync

shutdown:x:6:0:shutdown:/sbin:/sbin/shutdown

halt:x:7:0:halt:/sbin:/sbin/halt

mail:x:8:12:mail:/var/spool/mail:/sbin/nologin

  

The above data cannot be filtered by AWK as the columns are seperated by (:). Here we use SED to present the data to AWK in its readable limits.

Just to see try the above mentioned command on newfile.txt.

You have below output:

#cat myfile.txt | awk '{print $1" ==> "$7}'

root:x:0:0:root:/root:/bin/bash ==>

bin:x:1:1:bin:/bin:/sbin/nologin ==>

Albert:x:2:2:daemon:/sbin:/sbin/nologin ==>

Chin:x:3:4:adm:/var/adm:/sbin/nologin ==>

Neon:x:4:7:lp:/var/spool/lpd:/sbin/nologin ==>

sync:x:5:0:sync:/sbin:/bin/sync ==>

shutdown:x:6:0:shutdown:/sbin:/sbin/shutdown ==>

halt:x:7:0:halt:/sbin:/sbin/halt ==>

mail:x:8:12:mail:/var/spool/mail:/sbin/nologin ==>

But if you use SED in the middle, like shown below, we have a meaningful ouput like below:

# cat myfile.txt | sed 's/:/ /g' | awk '{print $1" ==> "$7}'

root ==> /bin/bash

bin ==> /sbin/nologin

Albert ==> /sbin/nologin

Chin ==> /sbin/nologin

Neon ==> /sbin/nologin

sync ==> /bin/sync

shutdown ==> /sbin/shutdown

halt ==> /sbin/halt

mail ==> /sbin/nologin

So SED allowed us to pass the output without delimiter to AWK which made AWK's life easy and thus ours :)

• You can implement SED on all delimiters and modify the data to required output.
• Using tail and head commands with AWK and SED makes your command or script more powerful.
• I hope this article gave a boost that you are looking to start as a first step towards scripting.
• Feel free to comment on my blog, ask questions and share to the people in need.

Last but not least, referring Man Pages for AWK and SED may help you with more options.

HAPPY LINUX LEARNING :)

You may be interested in my other posts below:

-----------------------------------------------------------

Please review or follow and share your comments if it helps.

File System State is clean with errors in Linux:
http://linuxunixdatabase.blogspot.com/2017/02/file-system-state-is-clean-with-errors.html

How to use IPERF to test interface/network throughput in Linux:
http://linuxunixdatabase.blogspot.com/2017/02/how-to-use-iperf-to-test.html

Linux/Unix Network Troubleshooting:
http://linuxunixdatabase.blogspot.com/2017/02/linuxunix-network-troubleshooting.html

Removing existing LVM from your Linux System

http://linuxunixdatabase.blogspot.com/2017/02/removing-existing-lvm-from-your-linux.html

Removing existing LVM from your Linux System

GOAL:

To remove or clean up the existing LVs on Linux Server: 

I described the step by step process on how to remove the existing LVMs from your Linux Box.

This article best works with RedHat Linux and Centos Distributions.

1. We need to know what LVs are present on our Linux Box. Use the below commands to see LVs on your box. 

#lvs  OR  #lvdisplay 

Output is as below:

# lvs

  LV            VG              Attr      LSize    Pool Origin Data%  Move Log Cpy%Sync Convert
  linxback01.vol datavg01        -wi-ao--- 1000.00g
  linxback00.vol linxdatavg00 -wi-ao---    5.00t

# lvdisplay

  --- Logical volume ---
  LV Path                /dev/datavg01/linxback01.vol
  LV Name                linxback01.vol
  VG Name                datavg01
  LV UUID                2k8o6B-ZJuS-40dX-FZF0-Ym96-bpd7-UQ6fY6
  LV Write Access        read/write
  LV Creation host, time host1, 2015-12-18 09:24:11 -0500
  LV Status              available
  # open                 1
  LV Size                1000.00 GiB
  Current LE             256000
  Segments               1
  Allocation             inherit
  Read ahead sectors     auto
  - currently set to     256
  Block device           253:0

  --- Logical volume ---
  LV Path                /dev/linxdatavg00/linxback00.vol
  LV Name                linxback00.vol
  VG Name                linxdatavg00
  LV UUID                aqZsFS-djb4-v0Ws-Hzbv-1gIk-Vjmb-1hsHU7
  LV Write Access        read/write
  LV Creation host, time host1, 2014-08-26 17:00:28 -0400
  LV Status              available
  # open                 1
  LV Size                5.00 TiB
  Current LE             1310715
  Segments               5
  Allocation             inherit
  Read ahead sectors     auto
  - currently set to     256
  Block device           253:1

==>We also need to know what is the associated PV Name to the LV. Command is below: 

# pvs

  PV             VG              Fmt  Attr PSize    PFree
  /dev/emcpowerf03 datavg01        lvm2 a--  1024.00g 24.00g

2. I can go ahead and start the removal process of both the LVMs shown above.But  you also need to know on which file system these LVa are mounted.

How to check that? Here is the way...

#df -kh

Just check for linxback00.vol and linxback01.vol in the output to see which file systems are using this LVs.

Example:

/dev/mapper/linxdatavg00-linxback00.vol
                      5.0T  3.6T  1.1T  77% /Backup_FileSystem

So, the above entry in the df -kh output shows the LV: linxback00.vol is mounted on the Directory : /Backup_FileSystem
Before we remove the LV, we need to unmounts and delete associated File system.

3. Unmounting associated File system using the Commands below:

fuser -cu /Backup_FileSystem
umount /Backup_FileSystem

4. We are all set to remove associated LV now.

#lvremove

Example :#lvremove  linxback01.vol
 

5. Remove associated VG

#vgremove <VG name>

Example: #vgremove datavg01

6. Remove associated PV

pvremove <device name>

Example:#pvremove   /dev/emcpowerf03

7.Validate using below commands if the LV is cleaned as expected. Ideally there should not be any output for the below commands once LV clean up is done. These are all non-disruptive commands.

#vgscan

#lvs

#vgs

#pvs

Important Note:

Usually LVs are associated with external SAN. The above process cleans up your LVs but SAN will still be there on system. You need to Work with your storage team to get the LUNs unmounted from your Linux server as soon as you are done with the LV removal process.

HAPPY LINUX LEARNING AS ALWAYS :)

Feel free to ask any questions or start a discussion about this topic.

My other Posts are below:

File System State is clean with errors in Linux:

http://linuxunixdatabase.blogspot.com/2017/02/file-system-state-is-clean-with-errors.html

How to use IPERF to test interface/network throughput in Linux:

http://linuxunixdatabase.blogspot.com/2017/02/how-to-use-iperf-to-test.html

Linux/Unix Network Troubleshooting:

http://linuxunixdatabase.blogspot.com/2017/02/linuxunix-network-troubleshooting.html

Learning AWK and SED Tools for LINUX/UNIX
http://linuxunixdatabase.blogspot.com/2017/02/learning-awk-and-sed-tools-for-linuxunix.html

File System State is clean with errors in Linux

ISSUE: File system on device reported state clean with errors in Linux

SOLUTION:

 ==>This issue is usually reported in /var/log/messages or may be noticed at boot time and is really critical.

==>This issue can be dangerous when data consistency is considered on reported File System
==>File System errors can be noticed on root file system, SAN file systems or even cluster file systems.

Steps to fix the issue for Non-Root file systems:
===================================
Suppose say /dev/sda10 (/fs1) is showing file system state clean with errors.
you can see how /dev/sda10  and /fs1 are related in df -kh command output

Example:
/dev/sda10             7.8G  6.3G  1.2G  85% /fs1

1. Take backup of the data from /fs1 directory and place  the backup on a different system if possible and one copy locally.

Use the below command to tar up the data and to create a tar file of the same.        

#tar -cvzf /fs1_<date>.tar.gz /fs1/*

You will have a tar file something like /fs1_20150201.tar.gz (Just replace the date as per your requirement in the command)

2. #df -kh

if you see the file system /fs1 is mounted, then unmount it like below:

#umount /fs1

Just check the status of the errors before we try to fix the issue actually.

Below command will not fix the issue rather tells us how many errors are present on the FS.

3.#fsck -n /fs1

Output is like below:
#fsck -n /fs1
fsck 1.39 (29-May-2006)
e2fsck 1.39 (29-May-2006)
Warning: skipping journal recovery because doing a read-only filesystem check.
/fs1 contains a file system with errors, check forced.
Pass 1: Checking inodes, blocks, and sizes
Pass 2: Checking directory structure
Pass 3: Checking directory connectivity
Pass 4: Checking reference counts
Unattached zero-length inode 179868.  Clear? no
Unattached inode 179868
Connect to /lost+found? no
Pass 5: Checking group summary information
Block bitmap differences:  +386446
Fix? no
Free blocks count wrong for group #11 (2, counted=3).
Fix? no
Inode bitmap differences:  +179868
Fix? no
Free inodes count wrong for group #11 (15350, counted=15351).
Fix? no
Free inodes count wrong (2072117, counted=2072133).
Fix? no

/fs1: ********** WARNING: Filesystem still has errors **********
/fs1: 12747/2084864 files (10.0% non-contiguous), 1953016/4162197 blocks

4.#tune2fs -l /dev/sda10

tune2fs 1.39 (29-May-2006)
Filesystem volume name:   /fs1
Last mounted on:          <not available>
Filesystem UUID:          cf72df5e-24fa-4f40-92c6-244459r49e17
Filesystem magic number:  0xEF18
Filesystem revision #:    1 (dynamic)
Filesystem features:      has_journal resize_inode dir_index filetype needs_recovery sparse_super large_file
Default mount options:    (none)
Filesystem state:         Clean with errors
Errors behavior:          Continue
Filesystem OS type:       Linux
Inode count:              2081264
Block count:              4164597
Reserved block count:     208789
Free blocks:              2209181
Free inodes:              2074517
First block:              0
Block size:               4096
Fragment size:            4096
Reserved GDT blocks:      1016
Blocks per group:         32768
Fragments per group:      32768
Inodes per group:         16288
Inode blocks per group:   509
Filesystem created:       Mon Oct 26 02:50:22 2009
Last mount time:          Sat Feb 11 09:12:30 2017
Last write time:          Sat Feb 11 09:12:30 2017
Mount count:              68
Maximum mount count:      25
Last checked:             Wed May 29 20:38:40 2013
Check interval:           15552000 (6 months)
Next check after:         Mon Nov 25 19:38:40 2013
Reserved blocks uid:      0 (user root)
Reserved blocks gid:      0 (group root)
First inode:              11
Inode size:               128
Journal inode:            8
Default directory hash:   tea
Directory Hash Seed:      b16ea660-bcea-4a0a-8e9e-926b058e1928
Journal backup:           inode blocks
Things to note in the above command is File system state: clean   OR Not Clean

Check for any errors and note down the output
Looking at the above commands output, /fs1 is not clean and fsck has shown errors. So needs to be cleaned to avoid any data corruption.

5. Start the FSCK now to fix the above seen issues on /fs1 file system

Important Note: Running fsck -f -y on a mounted system will cause data loss. Be cautious.

#screen
#fsck -f -y /dev/sda10
This command may run for 4 to 5 Hrs or may be more. Monitor till the command is executed completely

6. Verify if the errors are gone on /fs1 using tune2fs

#tune2fs -l //dev/sda10
file system state should be clean.
#fsck -n /dev/sda10
check if any errors are still there.

7. If /fs1 is clean, mount it back and start using it as usual

#mount /fs1

=====

Running FSCK on root file system needs to be done through single user mode or maintenance mode in Linux as we cannot unmount root FS.

HAPPY LINUX LEARNING :)

Feel free to ask any questions or start a discussion about this topic.

My other Posts are below:

File System State is clean with errors in Linux:
http://linuxunixdatabase.blogspot.com/2017/02/file-system-state-is-clean-with-errors.html

How to use IPERF to test interface/network throughput in Linux:
http://linuxunixdatabase.blogspot.com/2017/02/how-to-use-iperf-to-test.html

Linux/Unix Network Troubleshooting:
http://linuxunixdatabase.blogspot.com/2017/02/linuxunix-network-troubleshooting.html

Removing existing LVM from your Linux System
http://linuxunixdatabase.blogspot.com/2017/02/removing-existing-lvm-from-your-linux.html

Learning AWK and SED Tools for LINUX/UNIX
http://linuxunixdatabase.blogspot.com/2017/02/learning-awk-and-sed-tools-for-linuxunix.html

How to use IPERF to test interface/network throughput in Linux

SCEANARIO: I have installed a new NIC or set up a new network connection and wanted to test if the speed is up to the standard using Iperf command.

Process:

You must be a root user for this exercise.

Once the interface you wanted to test is ready on Server A from Server B,

1. Make Server A as a listening server using the command below:

#iperf -S OR #iperf3 -S

Output would be like below :

-----------------------------------------------------------
Server listening on 5201
-----------------------------------------------------------


2. Suppose say the IP of the interface you wanted to test is 192.168.1.25 which is assigned to eth0 on Server A, use the below command from Server B to test the speed of that interface.

#iperf3 -c 192.168.1.25 -i 2 -t 10

You see the output something like below:

Connecting to host 192.168.1.25, port 5201
[  5] local 192.168.1.5 port 44968 connected to 192.168.1.25 port 5201
[ ID] Interval       Transfer     Bandwidth
[  5] 0.00-2.00 sec  2.21 GBytes  9.48 Gbits/sec
[ ID] Interval       Transfer     Bandwidth
[  5] 2.00-4.00 sec  2.30 GBytes  9.86 Gbits/sec
[ ID] Interval       Transfer     Bandwidth
[  5] 4.00-6.00 sec  2.29 GBytes  9.83 Gbits/sec
[ ID] Interval       Transfer     Bandwidth
[  5] 6.00-8.00 sec  2.25 GBytes  9.67 Gbits/sec
[ ID] Interval       Transfer     Bandwidth
      Sent
[  5] 0.00-10.00 sec  11.3 GBytes  9.74 Gbits/sec
      Received
[  5] 0.00-10.00 sec  11.3 GBytes  9.74 Gbits/sec
iperf Done.

This output shows a 10Gig connection works fine as expected with good results.

Things to notice in the above output is:

1. See the bandwidth column which must be very close to your connections speed stats.
2. See the interval, I used every 2 Seconds in my command, that can be changed too.

What if, the expected speed is NOT observed on your network connection? Refer my other link below for network troubleshooting:

http://linuxunixdatabase.blogspot.in/2017/02/linuxunix-network-troubleshooting.html#!/2017/02/linuxunix-network-troubleshooting.html

OR

click below:

LUNIX UNIX NETWOTK TROUBLESHOOTING

Iperf is very handy command to check the connection throughput on Unix servers.

Refer Man page for more options after you get some basic idea on iperf usage

HAPPY LEARNING :)
Feel free to ask any questions or start a discussion about this topic.

My other Posts are below:

File System State is clean with errors in Linux:
http://linuxunixdatabase.blogspot.com/2017/02/file-system-state-is-clean-with-errors.html

How to use IPERF to test interface/network throughput in Linux:
http://linuxunixdatabase.blogspot.com/2017/02/how-to-use-iperf-to-test.html

Linux/Unix Network Troubleshooting:
http://linuxunixdatabase.blogspot.com/2017/02/linuxunix-network-troubleshooting.html

Removing existing LVM from your Linux System
http://linuxunixdatabase.blogspot.com/2017/02/removing-existing-lvm-from-your-linux.html

Learning AWK and SED Tools for LINUX/UNIX
http://linuxunixdatabase.blogspot.com/2017/02/learning-awk-and-sed-tools-for-linuxunix.html

Linux/Unix Network Troubleshooting

This document outlines the troubleshooting process of network interface related issues on a Linux/UNIX server

I do not post man page details here because every Linux distribution has man page details for all the commands.

I am taking a scenario here to discuss as to what can be done to fix a network interface issue on a Linux server.

Issue: Network connectivity is lost or throughput is ceased(getting very less speed comparatively) all of a sudden on my Linux server.

SOLUTION:

Things to do in sequential order:

You need to be a root user for all these steps.

Take output of #ifconfig command:

Example:

#ifconfig eth0

eth0      Link encap:Ethernet  HWaddr 01:89:A8:G8:4R:54

         inet addr:192.168.9.5  Bcast:192.168.99.255  Mask:255.255.255.0

         UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1

         RX packets:190458 errors:0 dropped:0 overruns:0 frame:0

         TX packets:86768 errors:0 dropped:0 overruns:0 carrier:0

         collisions:0 txqueuelen:150

         RX bytes:30701269 (29.3 Mb)  TX bytes:7878926 (1.9 Mb)

         Interrupt:9 Base address:0x5000

Check the details from the above output one by one as shown below:

1.Make sure this ip (192.168.9.5) does not have a duplicate entry in your network which means, any IP address has the capacity to server a single interface at a given time. It cannot be active on two or more devices at a time.

If you find a duplicate entry in network for this IP, make sure to drop the other connection and try the functionality of the interface. You can get in touch with your network team here to identify why this IP has a duplicate entry in the network and then to fix it.

2. Reset the problematic interface using below commands

#ifdown eth0

#ifup eth0

Check the functionality, proceed if the issue persists

3. Open the file /etc/sysconfig/network-scripts/ifcfg-eth0 and verify  the MAC address which is also called as Hardware address of the NIC (Network Interface Card) HWaddr 01:89:A8:G8:4R:54 is same as in the above output.

If you see any discrepancy, adjust the MAC address of the NIC to your interface (eth0 in this case).

There are different ways to find out the MAC address of an NIC in linux. You may need to google it which for the way which suits your case.

For an existing connection, ifconfig should be sufficient.

Check the functionality of the interface once the mac address is set appropriately.

4. If the issue still persists, next thing to check is RX packets and TX packets errors and drops.

RX packets/bytes ⇒ Receiving Data by server

TX packets/bytes ⇒ Sending Data by server

#Ethtool -S eth0

⇒ is the best command to see rx/tx or crc related errors on the interface

If you see any RX packets have errors or packets were dropped, this means data is becoming faulty even before it reaches the target server.

So, you need to consider resetting the switch port at this point and see if the errors or the dropped packets are still increasing.

If the errors continue to increase and drop packets are also increasing, first thing to suspect is NIC on your Linux server.

Check the NIC firmware version using #ethtool -i eth0 and see if it is up to date, if not go ahead and update the NIC firmware according to make of NIC.

Check the functionality, if the issue still persists, go ahead and replace your NIC and check the functionality

Same is true for TX errors and drops as well.

5. If the issue still persists, check the cable functionality with the help of your Data Center people and change the cable from external switch to your Linux server if found faulty.

6.If the issue persists, try host reboot and check the functionality

7. With the above steps, we isolated the issue on various aspects and we are clean from server point of view.

It's time to push  the issue to your Network Team and get it fixed from switch point of view.

=======

HOW TO TEST NETWORK SPEED: Use the link below

http://linuxunixdatabase.blogspot.in/2017/02/how-to-use-iperf-to-test.html#!/2017/02/how-to-use-iperf-to-test.html

OR
click on Using IPERF COMMAND LINUX
========

HAPPY LINUX LEARNING :)

Feel free to ask any questions or start a discussion about this topic.

My other Posts are below:

File System State is clean with errors in Linux:
http://linuxunixdatabase.blogspot.com/2017/02/file-system-state-is-clean-with-errors.html

How to use IPERF to test interface/network throughput in Linux:
http://linuxunixdatabase.blogspot.com/2017/02/how-to-use-iperf-to-test.html

Linux/Unix Network Troubleshooting:
http://linuxunixdatabase.blogspot.com/2017/02/linuxunix-network-troubleshooting.html

Removing existing LVM from your Linux System
http://linuxunixdatabase.blogspot.com/2017/02/removing-existing-lvm-from-your-linux.html

Learning AWK and SED Tools for LINUX/UNIX
http://linuxunixdatabase.blogspot.com/2017/02/learning-awk-and-sed-tools-for-linuxunix.html