liking the potential here seem to have found a potential issue with assert in evt_tls_feed_data

[ljackson]

if you test your libuv ./evt with openssl s_client or something like that it works fine however in trying to load test I get an assert and a program crash in debugging it appears the example code cannot handle when the stream is full right out of the gate?

repro with tcpkali util on ubuntu 16.04

tcpkali --workers 1 -c 1 -T 10s -1 "hello" -m "test" --ssl 127.0.0.1:8000

I added a bit of debug to on_tcp_read to see whats going on.... Seems like we need to feed another read/write to process the buffer and then consume the rest of the tls stream? Thx

Listening on 8000
on_tcp_read data len:65536 nrd:305
on_tcp_read data len:65536 nrd:318
on_tcp_read data len:65536 nrd:65536
evt: ../..//src/evt_tls.c:279: evt_tls_feed_data: Assertion `rv == sz' failed.

Program received signal SIGABRT, Aborted.
0x00007ffff7178428 in __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:54
54 ../sysdeps/unix/sysv/linux/raise.c: No such file or directory.
(gdb) bt
#0 0x00007ffff7178428 in __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:54
#1 0x00007ffff717a02a in __GI_abort () at abort.c:89
#2 0x00007ffff7170bd7 in __assert_fail_base (fmt=, assertion=assertion@entry=0x41fe2d "rv == sz", file=file@entry=0x41fcb0 "../..//src/evt_tls.c", line=line@entry=279,
function=function@entry=0x41ff40 <PRETTY_FUNCTION.15415> "evt_tls_feed_data") at assert.c:92
#3 0x00007ffff7170c82 in __GI___assert_fail (assertion=0x41fe2d "rv == sz", file=0x41fcb0 "../..//src/evt_tls.c", line=279, function=0x41ff40 <PRETTY_FUNCTION.15415> "evt_tls_feed_data") at assert.c:101
#4 0x0000000000404ebd in evt_tls_feed_data (c=0x6483c0, data=0x654a70, sz=65536) at ../..//src/evt_tls.c:279
#5 0x0000000000405553 in on_tcp_read (stream=0x648280, nrd=65536, data=0x7fffffffaff0) at uv_tls.c:75
#6 0x0000000000411057 in uv__read (stream=0x648280) at ../src/unix/stream.c:1178
#7 0x0000000000411312 in uv__stream_io (loop=0x6286c0 <default_loop_struct>, w=0x648308, events=1) at ../src/unix/stream.c:1241
#8 0x00000000004166d6 in uv__io_poll (loop=0x6286c0 <default_loop_struct>, timeout=-1) at ../src/unix/linux-core.c:345
#9 0x0000000000408034 in uv_run (loop=0x6286c0 <default_loop_struct>, mode=UV_RUN_DEFAULT) at ../src/unix/core.c:341
#10 0x000000000040442e in main () at test_tls.c:64
(gdb) frame 4
#4 0x0000000000404ebd in evt_tls_feed_data (c=0x6483c0, data=0x654a70, sz=65536) at ../..//src/evt_tls.c:279
279 assert( rv == sz);
(gdb) print rv
$1 = 17408

[ljackson]

one note on the test_tls.c the on_write callback I commented out the uv_tls_close to have the echo server continue otherwise I couldn't load test and this exposed the large frames issue. Thx!

[deleisha]

Thanks for reporting this issue and description.
Working on it

[deleisha]

I downloaded and compiled tcpkali on ubuntu 16.04. But it crashes with SIGSEGV. Hence, not trusting my tcpkali.
Hopefully, this commit should fixes the issue
@ljackson, Please check if it fixes the issue.

[ljackson]

apt get install tcpkali on 16.04 :) Will test the change thx.

[ljackson]

So looks better, the tcpkali test works well except it triggers a SIGPIPE which takes the process down. What is the best way to handle SIGPIPE so that the code doesn't go down after a disconnect...etc?

Program received signal SIGPIPE, Broken pipe.
0x00007ffff751d4a0 in __write_nocancel () at ../sysdeps/unix/syscall-template.S:84
84 ../sysdeps/unix/syscall-template.S: No such file or directory.
(gdb) bt
#0 0x00007ffff751d4a0 in __write_nocancel () at ../sysdeps/unix/syscall-template.S:84
#1 0x00000000004101c1 in uv__write (stream=0x7154e0) at ../src/unix/stream.c:826
#2 0x0000000000411945 in uv_write2 (req=0x7fffffff21b0, stream=0x7154e0, bufs=0x7fffffff22c0, nbufs=1, send_handle=0x0, cb=0x4119e7 <uv_try_write_cb>) at ../src/unix/stream.c:1398
#3 0x00000000004119e5 in uv_write (req=0x7fffffff21b0, handle=0x7154e0, bufs=0x7fffffff22c0, nbufs=1, cb=0x4119e7 <uv_try_write_cb>) at ../src/unix/stream.c:1423
#4 0x0000000000411a9c in uv_try_write (stream=0x7154e0, bufs=0x7fffffff22c0, nbufs=1) at ../src/unix/stream.c:1448
#5 0x0000000000405475 in uv_tls_writer (t=0x715620, bfr=0x92e220, sz=16413) at uv_tls.c:29
#6 0x0000000000404b5f in evt__send_pending (conn=0x715620) at ../..//src/evt_tls.c:222
#7 0x0000000000404db1 in evt__tls__op (conn=0x715620, op=EVT_TLS_OP_WRITE, buf=0x7fffffff64a0, sz=16384) at ../..//src/evt_tls.c:258
#8 0x000000000040509d in evt_tls_write (c=0x715620, msg=0x7fffffff64a0, str_len=16384, on_write=0x4059c9 <on_evt_write>) at ../..//src/evt_tls.c:330
#9 0x0000000000405ad7 in uv_tls_write (stream=0x7155e0, buf=0x7fffffff6440, cb=0x404196 <on_write>) at uv_tls.c:175
#10 0x00000000004041d7 in uv_rd_cb (strm=0x7155e0, nrd=16384, bfr=0x7fffffff6440) at test_tls.c:19
#11 0x000000000040575a in evt_on_rd (t=0x715620,
bfr=0x7fffffff64a0 "testtesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttest"..., sz=16384) at uv_tls.c:107
#12 0x0000000000404d49 in evt__tls__op (conn=0x715620, op=EVT_TLS_OP_READ, buf=0x0, sz=0) at ../..//src/evt_tls.c:250
#13 0x0000000000404f8c in evt_tls_feed_data (c=0x715620, data=0x843ce0, sz=65536) at ../..//src/evt_tls.c:299
#14 0x00000000004055e6 in on_tcp_read (stream=0x7154e0, nrd=65536, data=0x7fffffffa570) at uv_tls.c:75
#15 0x00000000004110ea in uv__read (stream=0x7154e0) at ../src/unix/stream.c:1178
#16 0x00000000004113a5 in uv__stream_io (loop=0x6286e0 <default_loop_struct>, w=0x715568, events=1) at ../src/unix/stream.c:1241
#17 0x0000000000416769 in uv__io_poll (loop=0x6286e0 <default_loop_struct>, timeout=-1) at ../src/unix/linux-core.c:345
#18 0x00000000004080c7 in uv_run (loop=0x6286e0 <default_loop_struct>, mode=UV_RUN_DEFAULT) at ../src/unix/core.c:341
#19 0x000000000040443e in main () at test_tls.c:64
(gdb) exit

[deleisha]

Looks like as tcpkali has closed the socket and libuv is trying to write. This need to be handled.
Thanks for quick update and testing it.

sudo apt-get install tcpkali returned "E: Unable to locate package tcpkali" for me.
Did you add any ubuntu PPA?

[ljackson]

actually I didn't I checked out master at 3c3175e41 i then just did ./configure and make/make install sorry thought it was apt get install ;) I do have the updated toolchain so that may explain a newer g++ ? I forget what the default is on 16.04 LTS

http://ppa.launchpad.net/ubuntu-toolchain-r/test/ubuntu

this is from the config log

Target: x86_64-linux-gnu
Configured with: ../src/configure -v --with-pkgversion='Ubuntu 5.4.1-2ubuntu116.04' --with-bugurl=file:///usr/share/doc/gcc-5/README.Bugs --enable-languages=c,ada,c++,java,go,d,fortran,objc,obj-c++ --prefix=/usr --p
rogram-suffix=-5 --enable-shared --enable-linker-build-id --libexecdir=/usr/lib --without-included-gettext --enable-threads=posix --libdir=/usr/lib --enable-nls --with-sysroot=/ --enable-clocale=gnu --enable-libstdcx
x-debug --enable-libstdcxx-time=yes --with-default-libstdcxx-abi=new --enable-gnu-unique-object --disable-vtable-verify --enable-libmpx --enable-plugin --with-system-zlib --disable-browser-plugin --enable-java-awt=gt
k --enable-gtk-cairo --with-java-home=/usr/lib/jvm/java-1.5.0-gcj-5-amd64/jre --enable-java-home --with-jvm-root-dir=/usr/lib/jvm/java-1.5.0-gcj-5-amd64 --with-jvm-jar-dir=/usr/lib/jvm-exports/java-1.5.0-gcj-5-amd64
--with-arch-directory=amd64 --with-ecj-jar=/usr/share/java/eclipse-ecj.jar --enable-objc-gc --enable-multiarch --disable-werror --with-arch-32=i686 --with-abi=m64 --with-multilib-list=m32,m64,mx32 --enable-multilib -
-with-tune=generic --enable-checking=release --build=x86_64-linux-gnu --host=x86_64-linux-gnu --target=x86_64-linux-gnu
Thread model: posix
gcc version 5.4.1 20160904 (Ubuntu 5.4.1-2ubuntu116.04)

[deleisha]

On SIGPIPE, libuv suggests to ignore it via signal. Check this

So, we will ignore it and libuv will report EPIPE, then we will close the connection.

[ljackson]

looks like this issue is resolved going to close... Now onto handling many many concurrent conns....

[deleisha]

The code for SIGPIPE is not committed yet. How does it resolved?
Looks like to me that tcpkali has problem too. It receives SIGSEGV on my system

[ljackson]

you are welcome to keep the issue open as it is your project afterall, I added SIGPIPE ignore locally and that covers the issue for me :)