Fixes for opening an iteration

[franzpoeschel]

Commit isolated from #1237

1. When opening a new step in the Streaming API, don't flush the Series before doing that. It seems that flushing at this stage seems to confuse HDF5 and lead to rare deadlocks in parallel contexts, seen in PIConGPU. This PR is able to fix that behavior in the observed instance.
2. When reading a variable-based Series, make sure to open an IO step before doing anything. Otherwise, attributes from later iterations might leak into the early iteration (RandomAccess mode in BP4).

TODO:

• Testing

[ax3l]

restarting CI

[ax3l]

It looks like NVHPC compilers (formerly PGI) fail on the SerialIOTest for 2 test right now.

[franzpoeschel]

It looks like NVHPC compilers (formerly PGI) fail on the SerialIOTest for 2 test right now.

I can reproduce this locally, will have a look

[franzpoeschel]

The NVIDIA compilers have a lot of test failures for me locally..

[ax3l]

discussed today: likely a glitch that the read toggles into write mode at some point.
will be addressed with a separate PR.

[franzpoeschel]

When defining:

namespace std
{
inline string to_string(openPMD::FlushLevel const &level)
{
    switch (level)
    {
    case openPMD::FlushLevel::UserFlush:
        return "UserFlush";
    case openPMD::FlushLevel::InternalFlush:
        return "InternalFlush";
    case openPMD::FlushLevel::SkeletonOnly:
        return "SkeletonOnly";
    case openPMD::FlushLevel::CreateOrOpenFiles:
        return "CreateOrOpenFiles";
    }
}
} // namespace std

and adding to Iteration::flushGroupBased the lines

    std::cout << "Flushlevel is: " << std::endl;
    std::cout << std::to_string(flushParams.flushLevel) << std::endl;

the result is this

Flushlevel is: 
 _ xPpphh!@_!_pp؅ptime
                      P
                       `\pdte
                             !?
                               "p
imeUnitSI

         ?
          1usage informationqPI vectorЏ`ofstringsÝ =p+вBpвBp0P-l
                                                                --list-tests!"Bp-tp
                                                                                   --list-tags"Bp0P0-s  --success1"Bp0P-bp--break"Bp P0-e   --nothrowA"Bp(tabs, newlines)0P-i`
                                          --invisibles"Bp-o--outq"Bp(defaults to con0Pp-r0 
--reporter"Bp-n --name"Bp P@ -a --abort1"Bp P -x`--abortx"Bp-w!--warnq"Bp Pp!-d "
                                                                                 --durations`t durations for tests taking at least the given number of seconds%`"↑↑└␋┼↑␍┤⎼▒├␋⎺┼1◆B⎻▮P"↑°P#
                                                      ↑↑␋┼⎻┤├↑°␋┌␊◆B⎻▮P#↑##1◆B⎻⎻$B⎻@P#↑␌◆$      ↑↑⎽␊␌├␋⎺┼◆B⎻$┤├ ┴␊⎼␉⎺@P $↑┴$
                                                                                                                            ↑↑┴␊⎼␉⎺⎽
␋├≤D▮▮◆#◆B⎻⎽↑⎺┼┌≤!%@0%`Bp 0 &e order (defaults to decP0%--orderP@ Fecific seed for random numbers@`BPP˞P˞pPd@dH'``˞˞˞pP@'˞̞̞pЇ8( ̞""P̞̞̞pВ(̞͞͞p() ͞@͞͞͞p0 )͞  ͞ΞΞpp`filename8*output filenameΞPΞPΞp0 *name`Ξ%%Ξ+name(+
suite name Ϟ ϞpƝ+0ϞPϞϞϞp ĝĝ+
                            no. failuresϞϞООpĝp,
                                                warning name,enable warningsОPОPОp,yes|no`ООООpp`-secondsII0ўpўpўp`Pfilenameў""ўpP.ҞҞpҞpҞp.p..
L3pppPd@d3p@pP@p4pЇ4""0pppВ`5p5 ``p0 P6p  pp`filename6output filename00p0 @7name @%%pp7nameP7
suite namepƝ08880ppp ĝĝ8
                        no. failurespĝ 9
                                        warning name@9enable warnings00p9yes|no1@`pp:seconds1II`Pfilename2`;3@3@`A`Ap.p.x;
                                                                                                                          section na'time'|numberC&&CDDpPm@mH>yes|no DAborted (core dumped)

:D

I'll try passing by value instead of reference

[franzpoeschel]

I think that the last push should fix it. Looks like a compiler issue to me, but it works now.

However, I noticed that builds with MPI or ADIOS2 have lots of failure with NVC++, we should maybe try to extend our tests there and see if it can be fixed. The thing seems to do lots of very funny optimization that may or may not result in correct binaries.

[franzpoeschel]

NVHPC is now running fine again :)

[ax3l]

Looks like a compiler issue to me, but it works now.

Actually, in C++, adding anything in the namespace std that is not a (class) template specialization is undefined behavior, defined by the standard. (re: #1239 (comment))

To fix the UB in the code, rely on ADL and define the to_string overload in the openPMD namespace:

• https://stackoverflow.com/a/50555639/2719194
• https://en.cppreference.com/w/cpp/language/extending_std

It is undefined behavior to add declarations or definitions to namespace std or to any namespace nested within std, with a few exceptions noted below. [...]

[franzpoeschel]

Looks like a compiler issue to me, but it works now.

Actually, in C++, adding anything in the namespace std that is not a (class) template specialization is undefined behavior, defined by the standard. (re: #1239 (comment))

To fix the UB in the code, rely on ADL and define the to_string overload in the openPMD namespace:

* https://stackoverflow.com/a/50555639/2719194

* https://en.cppreference.com/w/cpp/language/extending_std

It is undefined behavior to add declarations or definitions to namespace std or to any namespace nested within std, with a few exceptions noted below. [...]

Ah yeah, true

On the other hand, it was just some temporary debugging output and the memory issue that I saw came from somewhere else

[franzpoeschel]

I think that the last push should fix it. Looks like a compiler issue to me, but it works now.

However, I noticed that builds with MPI or ADIOS2 have lots of failure with NVC++, we should maybe try to extend our tests there and see if it can be fixed. The thing seems to do lots of very funny optimization that may or may not result in correct binaries.

The Python bindings don't compile at all

[ax3l]

@franzpoeschel I think clang-tidy fails on this one?

[ax3l]

restarting CI

[ax3l]

@franzpoeschel one tiny suggestion inline for your consideration

[ax3l]

restarting CI since I merged a couple PRs in between

[ax3l]

Similar to #1253 (comment) - tricky to bkacport to release-0.14.5

[ax3l]

Backported via ax3l#5 to #1282 0.14.5