Why does build.yaml test-sim test only the rv64gc/lp64d bare-metal/newlib toolchain?

[TommyMurphyTM1234]

See here:

• riscv-gnu-toolchain/.github/workflows/build.yaml

  Line 87 in 6b13243

  test-sim:

And an example run here:

• https://github.com/riscv-collab/riscv-gnu-toolchain/actions/runs/6505010058/job/17667866866#step:6:1

Why does the test-sim test only the rv64gc/lp64d bare-metal/newlib toolchain?

Is this considered a sufficient test of the toolchain?

Why is Linux specified here:

• riscv-gnu-toolchain/.github/workflows/build.yaml

  Line 92 in 6b13243

  mode: [newlib, linux]

but then excluded here:

• riscv-gnu-toolchain/.github/workflows/build.yaml

  Line 95 in 6b13243

  exclude:

Is this perhaps some legacy hangover - e.g. from when multiple simulators were used for testing or the Linux toolchain was tested?

Should this seeming redundancy be removed - e.g.:

  test-sim:
    runs-on: ${{ matrix.os }}
    strategy:
      matrix:
        os:     [ubuntu-20.04]
        mode:   [newlib]
        target: [rv64gc-lp64d]
        sim:    [spike]

[cmuellner]

The test-sim job runs the regression test suite with Spike:

[...]
  test-sim:
[...]
        sim:    [spike]
[...]
          ./configure --prefix=/opt/riscv --with-arch=${TARGET_TUPLE[0]} --with-abi=${TARGET_TUPLE[1]} --with-sim=${{ matrix.sim }}
[...]

The build job runs the regression test suite with QEMU as defined in configure.ac:

AC_ARG_WITH(sim,
        [AS_HELP_STRING([--with-sim=qemu],
                [Sets the base RISC-V Simulator, defaults to qemu])],
        [],
        [with_sim=qemu]
        )

So, we have the following builds that are tested with QEMU:

      matrix:
        os:     [ubuntu-20.04, ubuntu-22.04]
        mode:   [newlib, linux, musl]
        target: [rv32gc-ilp32d, rv64gc-lp64d]
        compiler: [gcc, llvm]
        exclude:
          - mode: musl
            target: rv32gc-ilp32d
          - mode: musl
            compiler: llvm
[...]
      - name: make report
        if: |
          matrix.os == 'ubuntu-20.04'
          && (matrix.mode == 'linux' || matrix.mode == 'newlib')
          && matrix.compiler == 'gcc'
        run: |
          sudo make report-${{ matrix.mode }} -j $(nproc)
[...]

And for Spike we have:

      matrix:
        os:     [ubuntu-20.04]
        mode:   [newlib, linux]
        target: [rv64gc-lp64d]
        sim:    [spike]
        exclude:
          - sim: spike
            mode: linux

I think this is sufficient.

[TommyMurphyTM1234]

Thanks @cmuellner - couldn't/shouldn't the test-sim job be simplified so?

Should this seeming redundancy be removed - e.g.:

  test-sim:
    runs-on: ${{ matrix.os }}
    strategy:
      matrix:
        os:     [ubuntu-20.04]
        mode:   [newlib]
        target: [rv64gc-lp64d]
        sim:    [spike]

[cmuellner]

Thanks @cmuellner - couldn't/shouldn't the test-sim job be simplified so?

Should this seeming redundancy be removed - e.g.:

  test-sim:
    runs-on: ${{ matrix.os }}
    strategy:
      matrix:
        os:     [ubuntu-20.04]
        mode:   [newlib]
        target: [rv64gc-lp64d]
        sim:    [spike]

Yes, that would be a valid simplification.
I'm not sure if this is an accident, a leftover or intention ("we could run it, but it would take too long so we exclude it").

[TommyMurphyTM1234]

OK - thanks. I'll do a PR for this small change so.

[TommyMurphyTM1234]

Rather than open a separate issue...

Is there any reason why these jobs are run on Ubuntu 20.04 and not 22.04 (both are LTS releases)?

• test-sim:

  riscv-gnu-toolchain/.github/workflows/build.yaml

  Line 91 in 6b13243

  os: [ubuntu-20.04]

• build-multilibs:

  riscv-gnu-toolchain/.github/workflows/build.yaml

  Line 127 in 6b13243

  os: [ubuntu-20.04]

  currently disabled:

  riscv-gnu-toolchain/.github/workflows/build.yaml

  Line 123 in 6b13243

  if: ${{ false }} # Disable until multilib errors are triaged

[cmuellner]

Rather than open a separate issue...

Is there any reason why these jobs are run on Ubuntu 20.04 and not 22.04 (both are LTS releases)?

• test-sim:

  riscv-gnu-toolchain/.github/workflows/build.yaml

  Line 91 in 6b13243

  os: [ubuntu-20.04]

• build-multilibs:

  riscv-gnu-toolchain/.github/workflows/build.yaml

  Line 127 in 6b13243

  os: [ubuntu-20.04]

  
  currently disabled:

  riscv-gnu-toolchain/.github/workflows/build.yaml

  Line 123 in 6b13243

  if: ${{ false }} # Disable until multilib errors are triaged

Sounds like a good PR! ;)

[TommyMurphyTM1234]

Sounds like a good PR! ;)

Ok - I'll do that so. :-) Thanks.

[TommyMurphyTM1234]

Both issues mentioned above addressed by this PR:

• Switch from Ubuntu 20.04 LTS to 22.04 LTS for jobs #1353.