Failing make check with adjoints example 5 with LIBMESH_BENCHMARK=1

[lindsayad]

I was told to report this to libMesh discussions 😄 This must only be triggered by LIBMESH_BENCHMARKS=1? Got this with:

make -j 64 check -C examples METHODS=opt LIBMESH_BENCHMARK=1 SUBDIRS='adjoints/adjoints_ex5'

Sensitivity of QoI 0 w.r.t parameter 0 is: 4.88045
Mismatch in sensitivity gold value!
Stack frames: 6
0: libMesh::print_trace(std::ostream&)
1: libMesh::MacroFunctions::report_error(char const*, int, char const*, char const*)
2: /home/lindad/projects/moose/libmesh/build/examples/adjoints/adjoints_ex5/.libs/example-devel(+0x21322) [0x55fb25112322]
3: /usr/lib/x86_64-linux-gnu/libc.so.6(+0x2dfd0) [0x7f5bcab00fd0]
4: __libc_start_main
5: /home/lindad/projects/moose/libmesh/build/examples/adjoints/adjoints_ex5/.libs/example-devel(+0x217a5) [0x55fb251127a5]
[0] ../../../../examples/adjoints/adjoints_ex5/adjoints_ex5.C, line 731, compiled Feb 28 2022 at 11:58:44
application called MPI_Abort(MPI_COMM_WORLD, 1) - process 0
[unset]: write_line error; fd=-1 buf=:cmd=abort exitcode=1
:
system msg for write_line failure : Bad file descriptor
FAIL: run.sh
===============================================================
1 of 1 test failed
Please report to https://github.com/libMesh/libmesh/discussions
===============================================================
make[3]: *** [Makefile:1549: check-TESTS] Error 1
make[3]: Leaving directory '/home/lindad/projects/moose/libmesh/build/examples/adjoints/adjoints_ex5'
make[2]: *** [Makefile:1676: check-am] Error 2
make[2]: Leaving directory '/home/lindad/projects/moose/libmesh/build/examples/adjoints/adjoints_ex5'
make[1]: *** [Makefile:1678: check] Error 2
make[1]: Leaving directory '/home/lindad/projects/moose/libmesh/build/examples/adjoints/adjoints_ex5'
make: *** [Makefile:742: check-recursive] Error 1
make: Leaving directory '/home/lindad/projects/moose/libmesh/build/examples'

[roystgnr]

Yeah, I can replicate this.

I don't actually understand it yet, though. That gold value predates the benchmark, it's not something that simply got added after the last time anyone ran in "benchmark mode". You'd expect to see some differences with a finer mesh but I don't see how you'd expect to see more differences now than back then. I'll see if I can bisect.

[lindsayad]

weird. Since I'm the one with reproducible failures, I'll work on a diagnosis

[roystgnr]

Do you at least get the gold result when running without LIBMESH_BENCHMARK?

[lindsayad]

yes

[lindsayad]

Opened #3179

[roystgnr]

Copying my shameful admission from Slack for future reference:

This fails deterministically for me with any refinement level other than the default. I couldn't "replicate" because I was looking at "make check" passes, which I get even when that test fails. What the hell. It wasn't dumb luck that the gold solution with float tolerance worked for extra refinement, because it wasn't working for extra refinement.

[roystgnr]

This was fixed in #3179