Child processes stuck cycling cpu ?

[alcohol]

I decided to give frankenphp a try locally in an existing project. We run everything in Docker locally as non-root, which was the first issue I ran into. Setting up Caddy to run as non-root is doable but requires some tinkering that is not very well documented. Once I had that up and running though, I ran into the following issue, which I have no clue how to debug..

Once frankenphp serves a request, there are child processes that remain stuck at using 100% cpu, even if no further requests are made at all. They just keep sucking up cpu cycles, making my laptop sound like a helicopter after a minute or so.

The project currently runs behind caddy with php-fpm. All I did was replace the caddy and fpm containers with a single frankenphp container. I'm not yet using worker mode, but I can try and see if that makes a difference or not.

1069707 rob         20   0 1481M  136M  105M S 311.3  0.4 15:45.00    ├─ frankenphp run --config /etc/caddy/Caddyfile --adapter caddyfile
1069770 rob         20   0 1481M  136M  105M S   0.0  0.4  0:00.00    │  ├─ frankenphp run --config /etc/caddy/Caddyfile --adapter caddyfile
1069771 rob         20   0 1481M  136M  105M S   0.0  0.4  0:00.00    │  ├─ frankenphp run --config /etc/caddy/Caddyfile --adapter caddyfile
1069772 rob         20   0 1481M  136M  105M S   0.0  0.4  0:00.00    │  ├─ frankenphp run --config /etc/caddy/Caddyfile --adapter caddyfile
1069773 rob         20   0 1481M  136M  105M S   0.0  0.4  0:00.00    │  ├─ frankenphp run --config /etc/caddy/Caddyfile --adapter caddyfile
1069774 rob         20   0 1481M  136M  105M S   0.0  0.4  0:00.00    │  ├─ frankenphp run --config /etc/caddy/Caddyfile --adapter caddyfile
1069782 rob         20   0 1481M  136M  105M S   0.0  0.4  0:00.03    │  ├─ frankenphp run --config /etc/caddy/Caddyfile --adapter caddyfile
1069783 rob         20   0 1481M  136M  105M S   0.0  0.4  0:00.02    │  ├─ frankenphp run --config /etc/caddy/Caddyfile --adapter caddyfile
1069784 rob         20   0 1481M  136M  105M S   0.0  0.4  0:00.01    │  ├─ frankenphp run --config /etc/caddy/Caddyfile --adapter caddyfile
1069785 rob         20   0 1481M  136M  105M S   0.0  0.4  0:00.00    │  ├─ frankenphp run --config /etc/caddy/Caddyfile --adapter caddyfile
1069789 rob         20   0 1481M  136M  105M S   0.0  0.4  0:00.03    │  ├─ frankenphp run --config /etc/caddy/Caddyfile --adapter caddyfile
1069795 rob         20   0 1481M  136M  105M S   0.0  0.4  0:00.00    │  ├─ frankenphp run --config /etc/caddy/Caddyfile --adapter caddyfile
1069796 rob         20   0 1481M  136M  105M S   0.0  0.4  0:00.00    │  ├─ frankenphp run --config /etc/caddy/Caddyfile --adapter caddyfile
1069797 rob         20   0 1481M  136M  105M S   0.0  0.4  0:00.01    │  ├─ frankenphp run --config /etc/caddy/Caddyfile --adapter caddyfile
1069801 rob         20   0 1481M  136M  105M S   0.0  0.4  0:09.99    │  ├─ frankenphp run --config /etc/caddy/Caddyfile --adapter caddyfile
1069802 rob         20   0 1481M  136M  105M R 103.8  0.4  5:20.80    │  ├─ frankenphp run --config /etc/caddy/Caddyfile --adapter caddyfile
1069803 rob         20   0 1481M  136M  105M R 104.4  0.4  5:06.13    │  ├─ frankenphp run --config /etc/caddy/Caddyfile --adapter caddyfile
1069804 rob         20   0 1481M  136M  105M R 103.8  0.4  5:07.77    │  └─ frankenphp run --config /etc/caddy/Caddyfile --adapter caddyfile

The last 3 have been stuck at basically >100% for 5 minutes now.

[withinboredom]

Hello, fellow Rob!

I'd recommend running in debug mode and grabbing a trace (which I just realized there is not really any good documentation on 🤔).

This pretty much sums up what is needed: https://mercure.rocks/docs/hub/debug (add debug to global part of the caddyfile).

Then go to http://localhost:2019/debug/ and dump the stack here so we can see what those threads are actually doing.

[alcohol]

I'm not running Mercure though (not yet). Should debug still be available? I enabled it in global caddy options, and I see the health check is regularly being hit by the docker daemon. But I am unable to access http://localhost:2019/debug/ -- I don't even see the request in the logs of the container. The port is forwarded afaik;

0.0.0.0:80->80/tcp, :::80->80/tcp, 0.0.0.0:443->443/tcp, :::443->443/tcp, 0.0.0.0:2019->2019/tcp, :::2019->2019/tcp

from host

$ curl -vvv http://localhost:2019/debug/
* Host localhost:2019 was resolved.
* IPv6: ::1
* IPv4: 127.0.0.1
*   Trying [::1]:2019...
* Connected to localhost (::1) port 2019
> GET /debug/ HTTP/1.1
> Host: localhost:2019
> User-Agent: curl/8.5.0
> Accept: */*
>
* Recv failure: Connection reset by peer
* Closing connection
curl: (56) Recv failure: Connection reset by peer

from inside container

/app $ curl -vvv http://localhost:2019/debug/
* Host localhost:2019 was resolved.
* IPv6: ::1
* IPv4: 127.0.0.1
*   Trying [::1]:2019...
* Immediate connect fail for ::1: Address not available
*   Trying 127.0.0.1:2019...
* Connected to localhost (127.0.0.1) port 2019
> GET /debug/ HTTP/1.1
> Host: localhost:2019
> User-Agent: curl/8.5.0
> Accept: */*
>
< HTTP/1.1 404 Not Found
< Content-Type: text/plain; charset=utf-8
< X-Content-Type-Options: nosniff
< Date: Fri, 15 Dec 2023 14:52:08 GMT
< Content-Length: 19
<
404 page not found
* Connection #0 to host localhost left intact

Maybe I should also note that I am using dunglas/frankenphp:latest-php8.3-alpine as a base image to extend from. Could it be there are no debug symbols included in this image?

[withinboredom]

Because the admin port allows anyone to take complete control of your caddy instance, it only listens on localhost, port forwarding won't work IIRC. So, you can only call it from within the container and not outside it.

[withinboredom]

Ah, I didn't see that you tried that as well 🤦. Is the admin port disabled?

[alcohol]

So, you can only call it from within the container and not outside it.

Yeah I tried that too.

[alcohol]

Ah, I didn't see that you tried that as well 🤦. Is the admin port disabled?

Nope, like I said, the healthcheck that is configured by default runs without any problems.

2023/12/15 14:46:14.408	DEBUG	admin.api	received request	{"method": "GET", "host": "localhost:2019", "uri": "/metrics", "remote_ip": "127.0.0.1", "remote_port": "42992", "headers": {"Accept":["*/*"],"User-Agent":["curl/8.5.0"]}}

[dunglas]

The admin port isn't exposed by default outside of the Docker container. You have either to expose it, or request it using curl from the inside of the container.

[dunglas]

A docs entry explaining how to run FrankenPHP as non-root with Docker is very welcome by the way!

[withinboredom]

I wonder if this is related to #366 and #442 will fix it?

[dunglas]

Closing as it looks resolved. Feel free to reopen if it's not the case!