Talking to an FPGA over PCIe

[planetarian]

I've been working on a project that pairs a Pi with an FPGA, feeding commands to it and returning status data from it in real-time, using Circle. It's intended to be a highly-parallel setup, with a substantial number of endpoints within the FPGA that can be issued commands processed in parallel, so it can be moving quite a bit of command data.

To date, I've implemented a GPIO-based SPI-like parallel bus for communication between them, which is able to achieve higher throughput than the Pi's native SPI mechanisms. It consumes more GPIO pins than I'm comfortable with, however, and as the project expands, more and more data needs to be transferred over the communication bus. As such, the bus is becoming a bottleneck, and most of the FPGA's endpoints have to spend their time waiting as the communication bus can't send commands quickly enough to keep them busy.

So, I've been looking at alternative means of communication. I'm not really all that familiar with high-speed busses, as in the past, SPI and UART covered basically all my needs. Now though, FPGAs with PCIe-compatible transceivers are becoming quite affordable, and the particular FPGAs I've been working with -- the Xilinx XC7A100T and Gowin GW5AST-138 -- both have such transceivers built-in, and devkits commonly break them out to PCIe connectors. SiPEED is even about to release a Pi5 HAT+ that provides an easy connection to the Pi5's PCIe header for GW5AST FPGAs, along with a PCIe DMA demo, which may provide a handy reference for DIYers (see: https://github.com/sipeed/TangMega-138K-example/tree/main/pcie_dma_demo).

How hard would it be for a user to perform direct communication with generic devices such as FPGAs using the PCIe bus? Both Xilinx and Gowin transceiver-equipped FPGAs have Linux drivers provided by the vendors, but I don't know whether there's much application-specific stuff in them, or how hard it is to port such things. What kind of workload should one expect for porting such a driver for basic/generic PCIe communication to Circle? Is this a realistic solution, or am I in over my head?

[rsta2]

I think one needs a sound understanding of the affected technologies (PCIe, FPGA, Raspberry Pi, maybe Linux) to realize this. The starting point can be the driver in lib/bcmpciehostbridge.cpp, which is used for connecting the RP1 chip. Some parameters must be changed (e.g. PCIe I/O address, inbound and outbound memory addresses). It was a lot try-and-error, when I ported this driver from Linux for the Raspberry Pi 4 and 5. I have to note, that this driver uses the INTA interrupt pin on the RP1. For some unknown reason, I did not get the MSI / MSI-X interrupts working. I don't know, if the FPGA device needs this. I wouldn't say, that it's not doable, but probably not easy and takes some time.