Iperf3 with similar configuration generates different results from Iperf2

[mhypnaro]

iperf3 Server side: (unfortunately, I can not include the full logs as they contain customer confidential information)

2022-05-31 22:57:27,326 :: INFO :: execute 'screen -d -m sh -c "iperf-3.10/src/iperf3 -s -B 10.49.72.156 -p 5120 -1 -i 10 -f g --logfile /root/cycle_0_interface_eth0_server.txt"' on (serverA)
2022-05-31 22:57:27,439 :: INFO :: execute 'screen -d -m sh -c "iperf-3.10/src/iperf3 -s -B 10.49.72.161 -p 5121 -1 -i 10 -f g --logfile /root/cycle_0_interface_eth1_server.txt"' on (serverA)
2022-05-31 22:57:27,550 :: INFO :: execute 'screen -d -m sh -c "iperf-3.10/src/iperf3 -s -B 10.49.72.160 -p 5122 -1 -i 10 -f g --logfile /root/cycle_0_interface_eth2_server.txt"' on (serverA)
2022-05-31 22:57:27,660 :: INFO :: execute 'screen -d -m sh -c "iperf-3.10/src/iperf3 -s -B 10.49.72.165 -p 5123 -1 -i 10 -f g --logfile /root/cycle_0_interface_eth3_server.txt"' on (serverA)
2022-05-31 22:57:27,769 :: INFO :: execute 'screen -d -m sh -c "iperf-3.10/src/iperf3 -s -B 10.49.72.164 -p 5124 -1 -i 10 -f g --logfile /root/cycle_0_interface_eth4_server.txt"' on (serverA)
2022-05-31 22:57:27,877 :: INFO :: execute 'screen -d -m sh -c "iperf-3.10/src/iperf3 -s -B 10.49.72.168 -p 5125 -1 -i 10 -f g --logfile /root/cycle_0_interface_eth5_server.txt"' on (serverA)
2022-05-31 22:57:27,987 :: INFO :: execute 'screen -d -m sh -c "iperf-3.10/src/iperf3 -s -B 10.49.67.168 -p 5126 -1 -i 10 -f g --logfile /root/cycle_0_interface_eth6_server.txt"' on (serverA)
2022-05-31 22:57:28,095 :: INFO :: execute 'screen -d -m sh -c "iperf-3.10/src/iperf3 -s -B 10.49.67.167 -p 5127 -1 -i 10 -f g --logfile /root/cycle_0_interface_eth7_server.txt"' on (serverA)

iperf3 client side

2022-05-31 22:57:29,545 :: INFO :: execute 'screen -d -m sh -c "iperf-3.10/src/iperf3 -c 10.49.72.156 -p 5120 -B 10.49.67.86 -t 120 -P 4 -i 10 -f g --logfile /root/cycle_0_interface_eth0_client.txt"' on (serverB)
2022-05-31 22:57:29,655 :: INFO :: execute 'screen -d -m sh -c "iperf-3.10/src/iperf3 -c 10.49.72.161 -p 5121 -B 10.49.67.87 -t 120 -P 4 -i 10 -f g --logfile /root/cycle_0_interface_eth1_client.txt"' on (serverB)
2022-05-31 22:57:29,768 :: INFO :: execute 'screen -d -m sh -c "iperf-3.10/src/iperf3 -c 10.49.72.160 -p 5122 -B 10.49.67.88 -t 120 -P 4 -i 10 -f g --logfile /root/cycle_0_interface_eth2_client.txt"' on (serverB)
2022-05-31 22:57:29,882 :: INFO :: execute 'screen -d -m sh -c "iperf-3.10/src/iperf3 -c 10.49.72.165 -p 5123 -B 10.49.68.123 -t 120 -P 4 -i 10 -f g --logfile /root/cycle_0_interface_eth3_client.txt"' on (serverB)
2022-05-31 22:57:29,998 :: INFO :: execute 'screen -d -m sh -c "iperf-3.10/src/iperf3 -c 10.49.72.164 -p 5124 -B 10.49.72.166 -t 120 -P 4 -i 10 -f g --logfile /root/cycle_0_interface_eth4_client.txt"' on (serverB)
2022-05-31 22:57:30,111 :: INFO :: execute 'screen -d -m sh -c "iperf-3.10/src/iperf3 -c 10.49.72.168 -p 5125 -B 10.49.65.39 -t 120 -P 4 -i 10 -f g --logfile /root/cycle_0_interface_eth5_client.txt"' on (serverB)
2022-05-31 22:57:30,225 :: INFO :: execute 'screen -d -m sh -c "iperf-3.10/src/iperf3 -c 10.49.67.168 -p 5126 -B 10.49.67.164 -t 120 -P 4 -i 10 -f g --logfile /root/cycle_0_interface_eth6_client.txt"' on (serverB)
2022-05-31 22:57:30,343 :: INFO :: execute 'screen -d -m sh -c "iperf-3.10/src/iperf3 -c 10.49.67.167 -p 5127 -B 10.49.67.165 -t 120 -P 4 -i 10 -f g --logfile /root/cycle_0_interface_eth7_client.txt"' on (serverB)

iperf2 server side (iperf2 is already installed on the server, hence why we just run "iperf" without a path.)

2022-05-31 22:14:49,564 :: INFO :: execute 'screen -d -m sh -c "iperf -s -B 10.49.72.156 -p 5120 -1 -i 10 -f g &> /root/cycle_0_interface_eth0_server.txt"' on (serverA)
2022-05-31 22:14:49,676 :: INFO :: execute 'screen -d -m sh -c "iperf -s -B 10.49.72.161 -p 5121 -1 -i 10 -f g &> /root/cycle_0_interface_eth1_server.txt"' on (serverA)
2022-05-31 22:14:49,788 :: INFO :: execute 'screen -d -m sh -c "iperf -s -B 10.49.72.160 -p 5122 -1 -i 10 -f g &> /root/cycle_0_interface_eth2_server.txt"' on (serverA)
2022-05-31 22:14:49,898 :: INFO :: execute 'screen -d -m sh -c "iperf -s -B 10.49.72.165 -p 5123 -1 -i 10 -f g &> /root/cycle_0_interface_eth3_server.txt"' on (serverA)
2022-05-31 22:14:50,006 :: INFO :: execute 'screen -d -m sh -c "iperf -s -B 10.49.72.164 -p 5124 -1 -i 10 -f g &> /root/cycle_0_interface_eth4_server.txt"' on (serverA)
2022-05-31 22:14:50,114 :: INFO :: execute 'screen -d -m sh -c "iperf -s -B 10.49.72.168 -p 5125 -1 -i 10 -f g &> /root/cycle_0_interface_eth5_server.txt"' on (serverA)
2022-05-31 22:14:50,222 :: INFO :: execute 'screen -d -m sh -c "iperf -s -B 10.49.67.168 -p 5126 -1 -i 10 -f g &> /root/cycle_0_interface_eth6_server.txt"' on (serverA)
2022-05-31 22:14:50,331 :: INFO :: execute 'screen -d -m sh -c "iperf -s -B 10.49.67.167 -p 5127 -1 -i 10 -f g &> /root/cycle_0_interface_eth7_server.txt"' on (serverA)

iperf2 client side

2022-05-31 22:14:51,779 :: INFO :: execute 'screen -d -m sh -c "iperf -c 10.49.72.156 -p 5120 -B 10.49.67.86 -t 120 -P 4 -i 10 -f g &> /root/cycle_0_interface_eth0_client.txt"' on (serverB)
2022-05-31 22:14:51,889 :: INFO :: execute 'screen -d -m sh -c "iperf -c 10.49.72.161 -p 5121 -B 10.49.67.87 -t 120 -P 4 -i 10 -f g &> /root/cycle_0_interface_eth1_client.txt"' on (serverB)
2022-05-31 22:14:52,001 :: INFO :: execute 'screen -d -m sh -c "iperf -c 10.49.72.160 -p 5122 -B 10.49.67.88 -t 120 -P 4 -i 10 -f g &> /root/cycle_0_interface_eth2_client.txt"' on (serverB)
2022-05-31 22:14:52,118 :: INFO :: execute 'screen -d -m sh -c "iperf -c 10.49.72.165 -p 5123 -B 10.49.68.123 -t 120 -P 4 -i 10 -f g &> /root/cycle_0_interface_eth3_client.txt"' on (serverB)
2022-05-31 22:14:52,240 :: INFO :: execute 'screen -d -m sh -c "iperf -c 10.49.72.164 -p 5124 -B 10.49.72.166 -t 120 -P 4 -i 10 -f g &> /root/cycle_0_interface_eth4_client.txt"' on (serverB)
2022-05-31 22:14:52,361 :: INFO :: execute 'screen -d -m sh -c "iperf -c 10.49.72.168 -p 5125 -B 10.49.65.39 -t 120 -P 4 -i 10 -f g &> /root/cycle_0_interface_eth5_client.txt"' on (serverB)
2022-05-31 22:14:52,486 :: INFO :: execute 'screen -d -m sh -c "iperf -c 10.49.67.168 -p 5126 -B 10.49.67.164 -t 120 -P 4 -i 10 -f g &> /root/cycle_0_interface_eth6_client.txt"' on (serverB)
2022-05-31 22:14:52,598 :: INFO :: execute 'screen -d -m sh -c "iperf -c 10.49.67.167 -p 5127 -B 10.49.67.165 -t 120 -P 4 -i 10 -f g &> /root/cycle_0_interface_eth7_client.txt"' on (serverB)

Test Steps:

From a remote server...

1. Collect information for logging and test execution
2. Upload, unpack, and prepare iperf3 with ftp and tar -xf and (cd iperf-3.10/ && ./configure; make; make install) on both servers
3. Kill any existing iperf or screen processes on serverA
4. Start iperf3 for the server on serverA
5. Sleep for 1 second to avoid client starting before server
6. Kill any existing iperf or screen processes on serverB
7. Start iperd3 for the client on serverB
8. wait (duration + 20) seconds and then download logs (currently duration of test runs of the iperf scripts have been 120 seconds)

Log files

(Removed for customer privacy as the logs contain serial numbers we don't want shared.)

Notes

• We set up a route explicitly between each interface: serverA eth0 has a route to serverB eth0, serverA eth1 has a route to serverB eth1, etc.
• Configuration uses only qsfp cables for interfaces eth0 - eth5 (no breakout cables) and sfp cables for eth6 - eth7.
• We set MTU to 9216 on server and client
• Expected speed for all interfaces is >= 85% of 25G (21.25Gb/s or higher)
• When running iperf2 without -P 4, our speed is lower than expected
• --dualtest in iperf2 and --bidir in iperf3 produce equally underperforming results. Our customer is asking for 30 days of iperf logs, so we want to try and get it working with --bidir and iperf3, since the logs are much easier to read and we will be able to provide 30 days of forwards and reverse logs as opposed to 15 forward and 15 backward
• fec is off on ports 6 and 7, but enabled on the other ports. The other ports will not run with fec off as per the requirements of the NIC they are attached to.

Please provide insight on the following:

1. Possibilities why -P 4 is required to obtain full speed on iperf2
2. Possibilities why iperf3 is producing different results despite a seemingly identical configuration

[davidBar-On]

Not sure if I can help (especially regarding iper2), but can you please give some more information:

1. Run the command with -V (verbose) to see the packet/message length used by each.
2. What are the results when -P 4 is not used (both iperf2 and iperf3)?
3. I don't see the -P 4 in the iperf2 commands, but the test results show that it was used. Can you clarify?
4. Try using -l to set the packet/message length to the same value in both iper3/2. Do you still get different results?
5. Try setting iperf3 -l to 64K, 128K, 256K. Does any of the values improve performance?
6. Try setting "burst" value in iperf3 to 40 (done by -b /40). Do it change iperf3 performance?

[mhypnaro]

1. I'll update with these results, yes.
2. Speed on the qsfp cables drops to anywhere between 10 and 16 Gb/s.
3. Apologies, this is a result of testing many dozens of iperf commands and configurations. I meant to include it above. I'll update it when I am in the lab.
4. -l is never an option I've used before. I'll try setting it equal to the MTU.
5. I'll try this as well.
6. Does using burst mode leave the line idle for any period of time? Our goal is to have traffic passed at as close to line rate as possible for as long as possible (to verify the System is capable of performing for a long time without interruption).

[davidBar-On]

Does using burst mode leave the line idle for any period of time?

Using burst does not add idle time. Every time a message is sent, he number of burst message are sent. That saves some system calls and can improve throughput.

Our goal is to have traffic passed at as close to line rate as possible for as long as possible (to verify the System is capable of performing for a long time without interruption).

One other option that can improve iperf3 throughput (TCP only, not UDP) is -Z - "use a 'zero copy' method of sending data".

[mhypnaro]

Hello all,

I have learned some new information.

1. Iperf3 -P 4 was failing because of some issues with 2 of the NICs in each system. Now iperf3 -P 4 performs similarly to iperf2 -P 4. Update: I re-ran the test at a longer duration and it seems iperf3 is back to yielding lower results than iperf2. For now, I can use iperf2 to do my work, but this is still an issue. I can provide more information on this Monday when I am back in the lab.
2. Neither iperf3 --bidir nor iperf2 --dualtest (which are both the same thing) is capable of passing on the NICs I am concerned with testing. Since the NIC is only really rated for 25Gb/s on each interface, I suppose it makes sense that if we use both interfaces at the same time that it can not get 25 Gb/s on 4 parallel streams of traffic as opposed to the 2 that it is rated for. In other words, the NICs just can not keep up as far as I can tell. Is this understanding of how --bidir and --dualtest work correct? Do they effectively double the load on the interface, or is traffic traded off (every other packet for instance)?
3. -Z, -l of varying values (9k, 16k, 32, 64k, 128k, and 256k), and -b /40 did not yield improvements in my case.
4. iperf2 does not have a -V (verbose) option

Regardless, the original reason that this ticket was opened was more or less solved since 2 of the NICs in each system that I was originally testing with seem to be performing poorly (which will require more debugging unrelated to iperf3.

Thank you. The only thing I'd still like answered is my italicized question, since it can clarify how this program works for me.

Edit: Added update about failure.

[davidBar-On]

Thanks for the information about what helped and what didn't.

Iperf3 -P 4 was failing because of some issues with 2 of the NICs in each system.

Do you know/understand why the NICs issue affected iperf3 but not iperf2?

I can provide more information on this Monday when I am back in the lab.

If you can provide more information it will be helpful, as in principle there should be no degradation over time which is not seen only in iperf3 but not in iperf2. Can it be because of iperf3 issues such as memory leak?

Since the NIC is only really rated for 25Gb/s on each interface, I suppose it makes sense that if we use both interfaces at the same time that it can not get 25 Gb/s on 4 parallel streams of traffic as opposed to the 2 that it is rated for. In other words, the NICs just can not keep up as far as I can tell. Is this understanding of how --bidir and --dualtest work correct? Do they effectively double the load on the interface, or is traffic traded off (every other packet for instance)?

Yes, in general your understanding is correct - --bidir or -P2 does not double the load of the interface if the interface is fully loaded without using such options. However, note that because of different overheads one stream does not fully load the interface. In this case, adding streams may increase the total throughput (depending on the nature of the overheads). Also, there are interfaces where the throughput of each direction is independent - such as in cellular uplink and downlink channels. In this case, --bidir will increase the total throughput as each stream is sent on a different direction.

[mhypnaro]

Do you know/understand why the NICs issue affected iperf3 but not iperf2?

Sorry, no. These NICs are a bit of a black box to us. The vendor is pretty difficult to reach as they are in a different timezone.

If you can provide more information it will be helpful, as in principle there should be no degradation over time which is not seen only in iperf3 but not in iperf2. Can it be because of iperf3 issues such as memory leak?

All I can provide is more logs-- see attached. I'd love to understand what's going on as well. I'm not too familiar with how iperf3 works, but from a high level it looks like it uses normal socket system calls and C structures. Do you want me to try like, running Valgrind on it? Would that even work? It's been a while since I've used C.

Logs: Same commands as above except that iperf3 also includes -b /40 -Z -l 64K -V. Note the outputs from the newer version of iperf2 which say this a couple times. I don't know how to interpret these lines. The "connection reset by peer" looks like one variation of the output when an ssh connection dies with a broken pipe. Or not. Maybe someone has some input on what this could mean? Maybe iperf2 is doing some sort of error handling that iperf3 is not?

tcp write failed: Connection reset by peer
shutdown failed: Transport endpoint is not connected

(Removed for customer privacy as the logs contain serial numbers we don't want shared.)

Edit:

I also just noticed this warning from iperf2 which might be particularly helpful to consider. That said, these are $14,000 servers, so I don't think CPU is the issue. But I don't want to write it off either. It's more likely it is failing to understand the "underlying network devices" as we have 3 different types of NICs, 2 of which are proprietary to a customer with highly customized firmware, as well as a BMC (baseboard management controller) if that affects anything.

WARN: this test may have been CPU bound (1) (or may not be detecting the underlying network devices)

[davidBar-On]

I am not familiar with iperf2 design, but it seems that practically only one stream out of the 4 is active. It reports something about "cpu bound" for two streams and one stream is only not active. Do you understand what is this? Can it be related to the difference between iperf2 and iperf3, i.e. that iperf2 somehow knows to take advantage of the multi-cpu processors?

I also see that for iperf3, the report on the client side indicate that the server CPU is 100% loaded (e.g. CPU Utilization: local/sender 14.3% (0.5%u/13.8%s), remote/receiver 100.0% (0.5%u/99.5%s). This may also indicate that iperf2 may know to better utilize the cpu.

Can you check the CPU load during the test for both the server and client, both for iperf2 and iperf3?

Also, if possible, can you also check the memory utilization during the test of the client/server iperf2/3?

[mhypnaro]

Hello,

I have come across quite a few developments.

Developments:

1. Memory usage is not an issue in iperf3 nor in iperf2 for my use case.
2. When I wrote the code to use iperf2, I was actually editing my iperf3 script. I left the -1 option in on the server side of the iperf2 command. -1 for iperf2 has the following documentation which explains why it looks like only one stream at a time was active.

 -1, --singleclient
              set the server to process only one client at a time

3. My system has 2 12 core CPUs, each of which has 24 virtual cores for a total of 48 total virtual cores. When we are seeing errors about "CPU Bound" in iperf2, we can see from running top while the program is running that some of the iperf2 processes are going over 100% CPU usage (and therefore other virtual cores are picking them up.) In the one directional mode, the server is the side which is exceeding 100% of a single virtual core for 2 out of 4 processes running. In --dualtest mode, it is both sides which exceed this value.
4. On iperf3, some processes on either side are reaching 100% of a single virtual core but will not exceed it. This means that iperf3 is cpu bound in the sense that it doesn't seem to be multi-threaded like iperf2 is.

Screenshots:

Note the CPU usage for each process in the screenshot. Also note that the "idle" reading for the total cpu is 90 or greater for each side.

Iperf3 failing to use multiple virtual cores:

Iperf2 using multiple cores:

Conclusion:

This therefore resolves my questions and answers this discussion.

I'm not sure if iperf3 is intended to be multi-threaded or not, but if it is, this usage case demonstrates that it isn't performing as it should in some situations and may be worth discussion. I am aware of the -A (cpu affinity) option, and understand that this may be related, but to be perfectly honest I don't know how to use it and I don't know what CPU affinity really is.

I'm also still not entirely sure why we need iperf2 to use -P 4 to make it work, but that's another discussion.

Thank you @davidBar-On for your assistance.

[davidBar-On]

@mhypnaro, your analysis is important and helpful!

One comment in case you prefer using iperf3. I see that 4 iperf3 servers are running in parallel. Using the --affinity <cpu #> option it may be possible to bound each server to a different CPU.