Why only in tcp parallel mode it will retry a lot in same net traffic?

[littlebr369]

Hi everyone.
I have a 2.5G ehternet connected through a marvell chip.
When using 'iperf3 -c 192.168.0.142 -b 20M', the tcp retry is 0.
But if using 'iperf3 -c 192.168.0.142 -b 10M -P 2', the tcp retry is about 200 times.
The weird thing is using 'iperf3 -c 192.168.0.142 -b 20M -u ' or 'iperf3 -c 192.168.0.142 -b 10M -P 2 -u', it won't lost any packets.

Can someone help with this question?

[davidBar-On]

Hi @littlebr369, its hard to tell what may be the cause of the re-transmits without seeing the traffic log (I assume you are using Wireshark).

One possible cause is that network buffers are getting filled. iperf3 TCP default packet size is 128KB (iperf3 UDP default packet is much smaller). With the -P 2, the two streams send 256KB in certain times (128KB each). If this is the problem, then smaller packets will help. Can you try running the TCP test using smaller packets, e.g. -l 8K?

[littlebr369]

Thanks a lot~ In this way there is no retry again if -l 8K. But I wonder in actually use, if our message is larger than 128k, what we can do to prevent this from happening. I have try to modify the net.core.ipv4.mem and other setting which also can't fix it.

Besides, what is the iperf3 work mechanism? What's the interval of each 128KB packets?

[littlebr369]

Besides, I won't what is len represent? If our MTU is 9000 and set -b 20M, what frequency and split will the data be sent?

[bmah888]

-l just controls the amount of data that iperf3 tries to send in a single write socket call. Since you are using jumbo frames, TCP is going to chop up that 128K (or whatever) into 9K TCP segments. iperf3 does some 128K sends, waits some amoutn of time, and then repeats. This is a little bursty. But the actual packets going out on the wire are going to be paced by the TCP congestion control algorithm. If you're going to be sending 9000B packets at 20Mbps, then you have 20000000/(9000*8) = 278 packets per second (approximately).

[bmah888]

It's surprising to get packet loss for a couple of 10Mbps streams on a 2.5Gbps link.

TCP should be segmenting the 128KB sends into 1460B packets on the wire (or 9K if doing jumbo frames). The packets should be sitting in the send-side socket buffers...wonder if those are full? Maybe try experimenting with different values of -w (try 256K, 512K, etc.)?

Also what version of iperf3 and what operating system is this?

[littlebr369]

iperf3 is based on iperf 3.7 (cJSON 1.5.2), and our platform is Linux 5.10.104-tegra on two Nivida Jetson Orin which communication through Marvell switch chip. And the MTU is set to be 9000 otherwise the bandwidth cannot be filled.

I have tried that if -w is fixed at 80000 there is no retry when a couple of 10Mbps. But we can't fix the windows size because it can't adapt to changing traffic in practical applications.

[littlebr369]

Besides, we mainly have four orin(A,B,C,D). A and B is through marvell's 10G netport. C and D is through another marvell's 10G netport. These two marvell chip is connect through its 2.5G port.

There is no tcp retry between A and B or A and C if tcp is not parallel.
There is no udp lost between A and B or A and C no matter it is parallel or not.

Retry only happened between A and C if parallel. If change --len to 8k there is no retry unless the parallel is more than 20.

In fact, I wonder if it is something wrong with marvell's chip set. But there is no reson to affect tcp performance in link layer settings.

[bmah888]

-w just sets the maximum socket buffer size. It doesn't directly control the TCP congestion window, and all the TCP congestion control algorithms still work.

[littlebr369]

Thanks a lot for your help.

So if I have a data like 10MB, I can send it in linux function send() and it will be split into packets of MTU size at the IP layer. In iperf3, the -l 256k means I will first split the 10MB data into several 256k data and each will be sent in linux function send(). Between each send() will have a sleep time. And then each 256k data will be split into packets of MTU size at the IP layer.

Am I understanding this correctly? If right, what is the time gap set between each 256k data sent?

[davidBar-On]

Your understanding is generally correct. Note that TCP is a stream of bytes, so once send() writes the 256KB to the buffer, there is no split between the different 256KB. For example, with MTU 9KB and -l 10K, the second 9KB packet will be include 1KB from the first send() and 8KB from the second.

Regarding the time between the 256KB sends, iperf3 tries to split them uniformly, according the the target bit rate (-b). If target rate is not specified then iper3 will not wait between the sends (but other conditions will limit the throughput rate, such as the network throughput or CPU performance ).