Learning AWK and SED Tools for LINUX/UNIX

GOAL:

Learning AWK and SED Tools for LINUX/UNIX:

I have posted so far L2 level of Linux issues and the processes to fix them.

• In this post I would like to take something simple but still interesting.
• It is sometimes may be important to have some kind of scripting knowledge in hand for a better Linux/Unix system administration.
• Before I post something on scripting with examples, I would like to introduce two command line tools namely AWK and SED which are really powerful in LINUX/Unix world.
• So, this post has the information about most frequently used data filtering commands in Linux/Unix world, none other than AWK and SED.

What is AWK and why is it used for?

Awk is both a programming language and text processor that can be used to manipulate text data in desirable manner.

Exercise using AWK:

To exercise a scenario with AWK, let's take a file called myfile.txt which has the below information.
=======

root x 0 0 root /root /bin/bash

bin x 1 1 bin /bin /sbin/nologin

Albert x 2 2 daemon /sbin /sbin/nologin

Chin x 3 4 adm /var/adm /sbin/nologin

Neon x 4 7 lp /var/spool/lpd /sbin/nologin

sync x 5 0 sync /sbin /bin/sync

shutdown x 6 0 shutdown /sbin /sbin/shutdown

halt x 7 0 halt /sbin /sbin/halt

mail x 8 12 mail /var/spool/mail /sbin/nologin

=======

• In the above data, first column shows user name, and last column(7th column) shows login details like whether bash login is allowed or not.
• From the above output I Just need to know the user names and corresponding login details.

Here is how AWK is used to get the required output.

#cat myfile.txt | awk '{print $1" "$7}'
root /bin/bash

bin /sbin/nologin

Albert /sbin/nologin

Chin /sbin/nologin

Neon /sbin/nologin

sync /bin/sync

shutdown /sbin/shutdown

halt /sbin/halt

mail /sbin/nologin

If I want to separate the output using a colon, I can use the below command:

# cat myfile.txt | awk '{print $1" : "$7}'
root : /bin/bash

bin : /sbin/nologin

Albert : /sbin/nologin

Chin : /sbin/nologin

Neon : /sbin/nologin

sync : /bin/sync

shutdown : /sbin/shutdown

halt : /sbin/halt

mail : /sbin/nologin

We can separate the columns as per our convenience like below:
 #cat myfile.txt | awk '{print $1" ==> "$7}'

root ==> /bin/bash

bin ==> /sbin/nologin

Albert ==> /sbin/nologin

Chin ==> /sbin/nologin

Neon ==> /sbin/nologin

sync ==> /bin/sync

shutdown ==> /sbin/shutdown

halt ==> /sbin/halt

mail ==> /sbin/nologin

--------------

In this same article, I would like to introduce SED as well.

What is SED?

• SED is a stream editor. 
• SED is used to perform basic text transformations on an input stream like  a file or input from a pipeline. SED works by making only one pass over the input(s), and is consequently more efficient.  But it is sed’s ability to filter text in a pipeline which particularly distinguishes it from other types of editors.

Suppose say, you have the data in newfile.txt is something like below:

root:x:0:0:root:/root:/bin/bash

bin:x:1:1:bin:/bin:/sbin/nologin

Albert:x:2:2:daemon:/sbin:/sbin/nologin

Chin:x:3:4:adm:/var/adm:/sbin/nologin

Neon:x:4:7:lp:/var/spool/lpd:/sbin/nologin

sync:x:5:0:sync:/sbin:/bin/sync

shutdown:x:6:0:shutdown:/sbin:/sbin/shutdown

halt:x:7:0:halt:/sbin:/sbin/halt

mail:x:8:12:mail:/var/spool/mail:/sbin/nologin

  

The above data cannot be filtered by AWK as the columns are seperated by (:). Here we use SED to present the data to AWK in its readable limits.

Just to see try the above mentioned command on newfile.txt.

You have below output:

#cat myfile.txt | awk '{print $1" ==> "$7}'

root:x:0:0:root:/root:/bin/bash ==>

bin:x:1:1:bin:/bin:/sbin/nologin ==>

Albert:x:2:2:daemon:/sbin:/sbin/nologin ==>

Chin:x:3:4:adm:/var/adm:/sbin/nologin ==>

Neon:x:4:7:lp:/var/spool/lpd:/sbin/nologin ==>

sync:x:5:0:sync:/sbin:/bin/sync ==>

shutdown:x:6:0:shutdown:/sbin:/sbin/shutdown ==>

halt:x:7:0:halt:/sbin:/sbin/halt ==>

mail:x:8:12:mail:/var/spool/mail:/sbin/nologin ==>

But if you use SED in the middle, like shown below, we have a meaningful ouput like below:

# cat myfile.txt | sed 's/:/ /g' | awk '{print $1" ==> "$7}'

root ==> /bin/bash

bin ==> /sbin/nologin

Albert ==> /sbin/nologin

Chin ==> /sbin/nologin

Neon ==> /sbin/nologin

sync ==> /bin/sync

shutdown ==> /sbin/shutdown

halt ==> /sbin/halt

mail ==> /sbin/nologin

So SED allowed us to pass the output without delimiter to AWK which made AWK's life easy and thus ours :)

• You can implement SED on all delimiters and modify the data to required output.
• Using tail and head commands with AWK and SED makes your command or script more powerful.
• I hope this article gave a boost that you are looking to start as a first step towards scripting.
• Feel free to comment on my blog, ask questions and share to the people in need.

Last but not least, referring Man Pages for AWK and SED may help you with more options.

HAPPY LINUX LEARNING :)

You may be interested in my other posts below:

-----------------------------------------------------------

Please review or follow and share your comments if it helps.

File System State is clean with errors in Linux:
http://linuxunixdatabase.blogspot.com/2017/02/file-system-state-is-clean-with-errors.html

How to use IPERF to test interface/network throughput in Linux:
http://linuxunixdatabase.blogspot.com/2017/02/how-to-use-iperf-to-test.html

Linux/Unix Network Troubleshooting:
http://linuxunixdatabase.blogspot.com/2017/02/linuxunix-network-troubleshooting.html

Removing existing LVM from your Linux System

http://linuxunixdatabase.blogspot.com/2017/02/removing-existing-lvm-from-your-linux.html