Sonata CI

[marnovandermaas]

This PR makes a few fixes to the Sonata board file and adds a script to run programs on the Sonata simulator.

Finally this PR is adding it to the CI, where all examples are run on the Sonata simulator and the UART output is checked against the model output.

This PR builds on top of adding the Sonata simulator and simulation boot stub to the development container: CHERIoT-Platform/devcontainer#19 and CHERIoT-Platform/devcontainer#21

[nwf]

Does the Sonata TB...

• understand enough of HTIF to gracefully exit,
• speak the Ibex "0x80 on the uart to exit" protocol,
• expect someone to drive the lowRISC simulation_ctrl verilog, or
• something else?

[marnovandermaas]

Unfortunately none of those, our current CI monitors the uart log file and based on the output there closes the simulation. I probably should change the script to do that instead of having a fixed timeout.

[davidchisnall]

It would be nice to reuse one of the many existing mechanisms, rather than invent a new one.

[marnovandermaas]

It would be nice to reuse one of the many existing mechanisms, rather than invent a new one.

The only existing solution I know of is our Sonata test runner script: https://github.com/lowRISC/sonata-system/blob/c4b717946aa64a06593feb95487fa498d457f920/util/test_runner.py#L176
Would you prefer me to use that script as opposed to the bash version I made? It essentially does the same thing, monitoring the UART script for a specific value.

[marnovandermaas]

Ok, I think I'm relatively happy with the state of this PR now. It runs all the examples on the Sonata simulator. It monitors the UART log and kills the simulator if it sees the model output match. There's also a timeout built in so the simulation doesn't take forever. The main question is whether you are happy with this mechanism I built here or whether you feel strongly enough to block this in favour of making changes to the simulator to add simulation control.

[davidchisnall]

I don't like cluttering the tree with model output in directories that users will explore. If we're going to do that, we need to make sure that all of the READMEs tell people not to look at those files.

Is there really no way of exiting the Sonata simulator from in the simulation? That seems like a core feature for a simulator or emulator. Is it very hard to add one? The Ibex-SAFE one is hacked in a little bit in the verilator wrapper for the UART: it just reads the top bit and exits if you write something to the UART with the top bit set. Not very clean, but it does work (until you forget about it and write binary data over the UART).

I also don't like the custom logic for running Sonata. We probably should have a sonata-sim board description that uses a different run script, so that we can just xmake run.

[marnovandermaas]

Thanks David, I can move the model output to this folder scripts/model_output/examples/. Would that be better or is there a better place?

For exiting the simulator, I don't think it's particularly hard but it would mean blocking this PR until I make that change in Sonata system.

I was trying to get xmake run to work. Is there a way to make the sonata simulator board description be the same as the main Sonata one but then with the following fields changed?
https://github.com/CHERIoT-Platform/cheriot-rtos/blob/main/sdk/boards/sonata-prerelease.json#L73-L74
It would be annoying having to update the simulator and the FPGA configuration separately.

[davidchisnall]

Thanks David, I can move the model output to this folder scripts/model_output/examples/. Would that be better or is there a better place?

I think that's fine. I want to restructure the scripts directory at some point and split the scripts that are part of the SDK from the ancillary ones, so anything in there is up for grabs.

For exiting the simulator, I don't think it's particularly hard but it would mean blocking this PR until I make that change in Sonata system.

Happy to merge this as a temporary fix, if that can come later.

I was trying to get xmake run to work. Is there a way to make the sonata simulator board description be the same as the main Sonata one but then with the following fields changed?

Not easily other than copy and paste currently. My plan at some point is to add something like json patch support in the build system. I'm pondering a simpler solution of look for a .jq file if there isn't a .json file and have that run and consume the output. That would let us have a file that is a small jq script that took a json file in the same directory and made some small changes. Unfortunately, we use some small JSON5 extensions (hex literals) and jq doesn't currently support them.

In particular, there's a bunch of stuff that's Ibex-specific that we should be able to share between a few places. Renode currently uses the sail JSON, but it should be possible to make it a 'sail + these bits' one.

[marnovandermaas]

Ok, I'm now happy with the state of this PR again. I've done the following things:

• Copied the Sonata prerelease board file into the simulator one and changed the simulator variables so that xmake run can use it.
• I added an argument to xmake config to enable model output checking with "testing-model-output".
• Those two things made it so I could remove the extra logic in the CI script and run Sonata like all the other boards.
• I moved the model output into the script directory and made them board dependent.

Some open things, which I don't think need to be dealt with before the merging:

• Using "expect" (suggestion from Wes) instead of the custom logic I built in simulator script.
• Sonata simulator exiting cleanly, I've opened an issue upstream for this: Simulation exit procedure lowRISC/sonata-system#365 Once that has been implemented we can update the script here.
• Reducing the duplication of the board files once the .jq method David mentioned is in.

[nwf]

LGTM. Thanks!

[davidchisnall]

Thanks!

[marnovandermaas]

Thanks for all your reviews, it's much better now than my first attempt :)