Support emulated NUMA for BCM2711 and BCM2712

[popcornmix]

From: Tvrtko Ursulin [email protected]

This series adds a very simple NUMA emulation implementation and enables
selecting it on arm64 platforms.

Obvious question is why? Short answer - it can bring a significant performance
uplift on Raspberry Pi 5.

Longer answer is that splitting the physical RAM into chunks, and utilising an
allocation policy such as interleaving, can enable the BCM2712 memory controller
to better utilise parallelism in physical memory chip organisation.

In more concrete numbers, testing with Geekbench 6 shows that splitting into four
emulated NUMA nodes can uplift the single core score of the benchmark by around
6%, and the multi-core by around 18%.

Code is quite simple and new functionality can be enabled using the new
NUMA_EMULATION Kconfig option and then at runtime using the existing (shared
with other platforms) numa=fake= kernel boot argument.

[popcornmix]

This PR also allows numa to be configured system wide (without directly using numactl) and enabled it by default through bootargs.

@pelwell I'm sure the bootargs change is better in a lower level file.

[pelwell]

I'm sure the bootargs change is better in a lower level file.

It isn't possible to put some command line settings in a common file and add to/modify it elsewhere, but if there are some devices that share a common command line then we can put it in a common file and override it in its entirety where necessary.

As it happens, of the BCM2711 devices, only CM4S has a different command line, and BCM2712 devices share the same one.

Is this something you are thinking to enable by default on BCM2711 and BCM2712?

[popcornmix]

Is this something you are thinking to enable by default on BCM2711 and BCM2712?

Yes - enabled by default on BCM2711 and BCM2712 (ideally on all distributions) is the goal.

[pelwell]

Bumped with a minor DT refactor.

[DTEAM-1]

Quick question. Can I use this script on a pi4 (64-Bit)?

[popcornmix]

Quick question. Can I use this script on a pi4 (64-Bit)?

There are benefits from NUMA on a Pi4, but not as great as those available on Pi5.
I would only recommend this if you know what you are doing (i.e. can restore/revert back if it doesn't work for you).

It is likely in the coming weeks that we will merge enough (both bootloader and kernel) so NUMA is easy to switch on to experiment with. Following some wider testing it will likely be enabled by default.

[popcornmix]

I've added a patch to disable numa if the settings are incompatible with cma settings (as otherwise cma heap creation fails and that leads to a hung boot with no output).

If numa is disabled due to cma, you'll see a message like:

[    0.000000] cma: CMA linux,cma [2000000-15ffffff] straddles range [0-fffffff]

and only a single numa region:

[    0.000000] NUMA: Faking a node at [mem 0x0000000000000000-0x000000007fffffff]

If numa is successful, say with 8 nodes, you would expect:

[    0.000000] NUMA: NODE_DATA [mem 0x0fffd2c0-0x0fffffff]
[    0.000000] NUMA: NODE_DATA [mem 0x1fffd2c0-0x1fffffff]
[    0.000000] NUMA: NODE_DATA [mem 0x2fffd2c0-0x2fffffff]
[    0.000000] NUMA: NODE_DATA [mem 0x3f7fd2c0-0x3f7fffff]
[    0.000000] NUMA: NODE_DATA [mem 0x4fffd2c0-0x4fffffff]
[    0.000000] NUMA: NODE_DATA [mem 0x5fffd2c0-0x5fffffff]
[    0.000000] NUMA: NODE_DATA [mem 0x6fffd2c0-0x6fffffff]
[    0.000000] NUMA: NODE_DATA [mem 0x7fe892c0-0x7fe8bfff]

The default cmdline adjustment is now numa_policy=interleave for 2711 and 2712, and additionally system_heap.max_order=0 iommu_dma_numa_policy=interleave for just 2712 (due to its better iommu support).

The key setting numa=fake=<n> is not set here, so we will boot with a single numa region and behaviour should be pretty much unchanged from before this PR.

But users can set it themselves, or if they have a new enough bootloader, it will set it optimally, based on whether the sdram is single or dual rank, and what the bank low setting is.

For the automatic setting, you currently need to set rpi-eeprom-config parameter SDRAM_BANKLOW to 3 (recommended for Pi4) or 1 (recommended for Pi5). e.g.

SDRAM_BANKLOW=1

You should then in /proc/cmdline a numa=fake=<n> setting, and hopefully sdram bandwidth limited processing tasks will run faster.

[popcornmix]

@pelwell I think this is ready. Any objections?