Add swap to support pi3 series memory pressure #16
Conversation
* Create build-docker.yml * Upload artifacts
* Update build-docker.yml * Update build-docker.yml * Update build-docker.yml * Update build-docker.yml * Update build-docker.yml * Update build-docker.yml * Update build-docker.yml * Update build-docker.yml * Update build-docker.yml * Update build-docker.yml
* Add pigpiod package * remove NOOBS * start pigpiod on boot, install photonvision and run on boot * Add service unit file * Add whitespace, remove ExecStop in Service file
Update README.md
* Create build-docker.yml * Upload artifacts
* Update build-docker.yml * Update build-docker.yml * Update build-docker.yml * Update build-docker.yml * Update build-docker.yml * Update build-docker.yml * Update build-docker.yml * Update build-docker.yml * Update build-docker.yml * Update build-docker.yml
* Add pigpiod package * remove NOOBS * start pigpiod on boot, install photonvision and run on boot * Add service unit file * Add whitespace, remove ExecStop in Service file
- Supports bullseye for new raspberry pi boards - legacy camera enabled - added config-dev that enables cache for building - added userconf.txt so PI has initial user on boot - fixed files that had utf8 w/ bom which was preventing build
uses the img.xz extension since that is registered with the raspberry pi imager
Rebase of rpi-distro and photonvision changes, fixes Pi 4 image
should fix up the failures on uploading a new .jar
Increase max heap size to 0.5GB
We observed fantastic pipeline performance on a Pi4. Due to difficulty sourcing Pi4s, we did testing on a Pi3b+. During testing, we found large lags when the pipeline stages that recognize a target run. We also observed very slow GUI performance, and slow ssh connection. At first, we assumed that CPU was the issue. However, using htop to observe what was going on, the CPUs were fine. Memory pressure was the issue. The pi3b+ had exhausted all physical memory, and all 100M of the default swap. We changed swap to 700MB, and after that change, the pi3b+ performs pipelines just as fast as the Pi4. ( In fact a bit faster in our setup, but that's because the frame rate on our pi3 camera was faster. I'm curious about why the set value of 100 was installed, though. I believe the original value ( dynamic) might have allowed the 3b+ to work fine. Depending on why 100MB fixed was chosen, going back to dynamic might be the right answer We didn't do any testing to find out if this value is the minimal size to solve the problem, so more or less memory might work. If it would be
|
Just changed the base branch to arm64, which probably broke things. Sorry! Testing with dynamic would be helpful. I've personally never hit swap so we just never changed from the default 100M. You'll wanna cherry-pick your change I think |
|
@mcm001 ok let me re-test with dynamic. i'll also see if i can get an answer on how much swap was left after adding 700MB. Once that's done, i can make a new PR that only has one line change on it. What branch would you prefer I make a new PR on? |
|
I re targeted for arm64, that's where our raspi image is built from. The LL image is based on gloworm-arm64, and we should pull this change into there too (so a second PR) |
|
@mcm001 got it! will report back when i test ( should be tonight or tomorrow night ) |
|
@mcm001 very interesting findings. On a pi3b+ , enabling dynamic swap ( CONF_SWAPSIZE= ) actually allocated 1.8 GB of swap. Of course, performance is fine.
What's very odd ( see picture above) is that at this point, i dont see any swap usage at all-- which I didnt expect. I'm going to need to retry duplicating the issue I originally saw |
|
Weird! That's more in line with what I'd expect. And if we really were swapping hard, maybe there's a memory leak somewhere inside Photon itself? |
We observed fantastic pipeline performance on a Pi4.
Due to difficulty sourcing Pi4s, we did testing on a Pi3b+.
During testing, we found large lags when the pipeline stages that recognize a target run. We also observed very slow GUI performance, and slow ssh connection.
At first, we assumed that CPU was the issue. However, using htop to observe what was going on, the CPUs were fine. Memory pressure was the issue. The pi3b+ had exhausted all physical memory, and all 100M of the default swap.
We changed swap to 700MB, and after that change, the pi3b+ performs pipelines just as fast as the Pi4. ( In fact a bit faster in our setup, but that's because the frame rate on our pi3 camera was faster.
I'm curious about why the set value of 100 was installed, though. I believe the original value ( dynamic) might have allowed the 3b+ to work fine. Depending on why 100MB fixed was chosen, going back to dynamic might be the right answer We didn't do any testing to find out if this value is the minimal size to solve the problem, so more or less memory might work. If it would be