Linux/Unix Network Troubleshooting

This document outlines the troubleshooting process of network interface related issues on a Linux/UNIX server

I do not post man page details here because every Linux distribution has man page details for all the commands.

I am taking a scenario here to discuss as to what can be done to fix a network interface issue on a Linux server.

Issue: Network connectivity is lost or throughput is ceased(getting very less speed comparatively) all of a sudden on my Linux server.

SOLUTION:

Things to do in sequential order:

You need to be a root user for all these steps.

Take output of #ifconfig command:

Example:

#ifconfig eth0

eth0      Link encap:Ethernet  HWaddr 01:89:A8:G8:4R:54

         inet addr:192.168.9.5  Bcast:192.168.99.255  Mask:255.255.255.0

         UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1

         RX packets:190458 errors:0 dropped:0 overruns:0 frame:0

         TX packets:86768 errors:0 dropped:0 overruns:0 carrier:0

         collisions:0 txqueuelen:150

         RX bytes:30701269 (29.3 Mb)  TX bytes:7878926 (1.9 Mb)

         Interrupt:9 Base address:0x5000

Check the details from the above output one by one as shown below:

1.Make sure this ip (192.168.9.5) does not have a duplicate entry in your network which means, any IP address has the capacity to server a single interface at a given time. It cannot be active on two or more devices at a time.

If you find a duplicate entry in network for this IP, make sure to drop the other connection and try the functionality of the interface. You can get in touch with your network team here to identify why this IP has a duplicate entry in the network and then to fix it.

2. Reset the problematic interface using below commands

#ifdown eth0

#ifup eth0

Check the functionality, proceed if the issue persists

3. Open the file /etc/sysconfig/network-scripts/ifcfg-eth0 and verify  the MAC address which is also called as Hardware address of the NIC (Network Interface Card) HWaddr 01:89:A8:G8:4R:54 is same as in the above output.

If you see any discrepancy, adjust the MAC address of the NIC to your interface (eth0 in this case).

There are different ways to find out the MAC address of an NIC in linux. You may need to google it which for the way which suits your case.

For an existing connection, ifconfig should be sufficient.

Check the functionality of the interface once the mac address is set appropriately.

4. If the issue still persists, next thing to check is RX packets and TX packets errors and drops.

RX packets/bytes ⇒ Receiving Data by server

TX packets/bytes ⇒ Sending Data by server

#Ethtool -S eth0

⇒ is the best command to see rx/tx or crc related errors on the interface

If you see any RX packets have errors or packets were dropped, this means data is becoming faulty even before it reaches the target server.

So, you need to consider resetting the switch port at this point and see if the errors or the dropped packets are still increasing.

If the errors continue to increase and drop packets are also increasing, first thing to suspect is NIC on your Linux server.

Check the NIC firmware version using #ethtool -i eth0 and see if it is up to date, if not go ahead and update the NIC firmware according to make of NIC.

Check the functionality, if the issue still persists, go ahead and replace your NIC and check the functionality

Same is true for TX errors and drops as well.

5. If the issue still persists, check the cable functionality with the help of your Data Center people and change the cable from external switch to your Linux server if found faulty.

6.If the issue persists, try host reboot and check the functionality

7. With the above steps, we isolated the issue on various aspects and we are clean from server point of view.

It's time to push  the issue to your Network Team and get it fixed from switch point of view.

=======

HOW TO TEST NETWORK SPEED: Use the link below

http://linuxunixdatabase.blogspot.in/2017/02/how-to-use-iperf-to-test.html#!/2017/02/how-to-use-iperf-to-test.html

OR
click on Using IPERF COMMAND LINUX
========

HAPPY LINUX LEARNING :)

Feel free to ask any questions or start a discussion about this topic.

My other Posts are below:

File System State is clean with errors in Linux:
http://linuxunixdatabase.blogspot.com/2017/02/file-system-state-is-clean-with-errors.html

How to use IPERF to test interface/network throughput in Linux:
http://linuxunixdatabase.blogspot.com/2017/02/how-to-use-iperf-to-test.html

Linux/Unix Network Troubleshooting:
http://linuxunixdatabase.blogspot.com/2017/02/linuxunix-network-troubleshooting.html

Removing existing LVM from your Linux System
http://linuxunixdatabase.blogspot.com/2017/02/removing-existing-lvm-from-your-linux.html

Learning AWK and SED Tools for LINUX/UNIX
http://linuxunixdatabase.blogspot.com/2017/02/learning-awk-and-sed-tools-for-linuxunix.html