iperf3 reports packet loss but no loss in tcpdump

[garykuok]

Hello,

I'm running iperf on two directly connected nodes, one being server and the other being client.
In order to calculate the metrics on a per packet basis, I use tcpdump to capture all packets generated in the test. To reduce the packet size, I only captured the header info (e.g., sequence number, timestamp, packet size, etc.).

Server: iperf3 -s -B 10.0.0.1
Client: iperf3 -c 10.0.0.1 -B 10.0.0.2 -b 256M -u -l 256 -t 30 # a UDP test
Network card: Intel I210 Gigabit NIC
System: Ubuntu 20.04 LTS, kernel version 5.4.0-90
In server's live report, it reports quite some packet loss (in total 15088/3749909 (0.4%)), but in tcpdump it actually manages to capture all packets without a single loss. I verified the number of packets in iperf and tcpdump are the same. Tested on iperf v3.11 and v3.9, they have similar condition.

Thus, I want to ask why would this behaviour happen? How is the algorithm of checking packet loss implemented? When comparing the packet, does iperf also verify if the content of the packet matches?

Best,
Chon Kit

[davidBar-On]

The lost packets are identified by the internal sequence number in the packet. E.g. if packet 12 was the last received and the next packet is 15, then 2 packets are added to the lost count (13, 14). If packet 13 is received later (out of order) the lost count will be reduced by 1.

It will help if you can add the following information:

• Add -J to the server, so the summary report will be Jason. It includes more information, including out of order count.
• Send the full output report from the server. It will help to see whether the reported lost packets are at certain test interval (e.g. the first or the last) or whether they spread all over the test.

[garykuok]

Hello,

thank you for your reply. In the attachment is the full JSON output of the server:
server.txt

From the report I see that the loss of packets are spread over the test; there's zero out_of_order packets count.

[davidBar-On]

Hi, there doesn't seem to be any out of order or duplicate packet. Can you explain how did you conclude that no packet was really lost? By counting the number of received packets per tcpdump on the server side compared to the number of sent packets by the client side? Other?

Also, it will help if you can verify that reported lost packet is not really lost per the iperf3 sequence number. As about one packet is lost for about 250 sent packets that may be possible to do. Evaluating the packets contents can be done by tcpdump -w and viewing the file using Wireshark.

The iperf3 sequence number is 32 bits by default, starting at byte (octet) offset 8 of the packet payload.

[garykuok]

Hello,
I use tcpdump to capture the first 54 bytes of each packet (ethernet 14 + IP 20 + UDP header 8 + 12 bytes for iperf seconds, iperf microseconds and iperf sequence number).

1. the total number of packets in tcpdump on both sides match the total packets on iperf's client side, no packet is dropped by tcpdump;
2. from the captured packets, I compared their iperf info (the last 12 bytes) and they all match. I calculated some metrics using tcpdump's timestamp instead. The iperf packet info acts as a matching criteria to make sure the packets are one to one corresponded.
3. I did a simple disjoint set check: one list is all iperf sequence numbers, one list is a generated list from one to the length of the iperf sequence list, also no packet mismatch was found.

I didn't capture the whole packet, as the content of the packet are some random characters (thus they are useless). As the first 12 bytes already give me enough information, I can conclude that at least tcpdump manages to capture all packets, but maybe some packets get lost when going from the server's tcpdump to iperf.

If this is the case, then iperf's report is not accurate...

Have a nice weekend!

[davidBar-On]

Hello Gary,
Sorry for the very late response to your impressive analysis.

I was able reproduce the problem (or a similar problem) on my computer - between Windows and WSL Linux. In my case the problem seem to be because of insufficient Kernel buffer size.

You can check whether this is also the problem in your case by:

1. Run netstat -s -u and note the UDP "receive buffer errors".
2. Run your test and note the number of lost packets reported by iperf3.
3. Run netstat -s -u again. The difference between the UDP "receive buffer errors" and the previous number should be the same (or close to) the number of lost packets reported by iperf3 (assuming tcpdump still shows tat no packet was lost).

If the numbers match, it may mean that the receive buffer size is not large enough and it should be increased. See for example the "UDP buffer size is insufficient" section here (I didn't try that).