Running EdgeX under Podman using libpod, not Docker Compose

[dmocek]

@jpwhitemn @bnevis-i @cloudxxx8

Moving the Slack discussion here:

To be clear about how we're running EdgeX, we are running it in Podman and all microservices are communicating with each other through the Podman network. We currently have no ports exposed to the host. The difference is we're not using Docker Compose to install and run the images/containers. We have a custom installer script that installs one microservice by calling the libpod API (socket) to install our "master" service. The master microservice makes socket calls to the libpod API to install and manage all other services. We currently have 12 microservices successfully running, including the EdgeX microservices (e.g. metadata) and the open-source microservices (e.g. consul), plus our own. We are now adding device services and trying to get one to install using our custom mechanism.

It turns out that @cloudxxx8 identified the issue, the network. We have a requirement for our master service to communicate with our cloud, which requires a proxy to be set when EdgeX is run in a customer's environment. The proxy is set in one of three locations during installation:

1. $HOME/.config/containers/containers.conf
2. /usr/share/containers/containers.conf
3. /etc/containers/containers.conf

and we set 3 things:

1. http_proxy/HTTP_PROXY
2. https_proxy/HTTPS_PROXY
3. no_proxy/NO_PROXY

The proxy is set and it allows the master container to communicate with the cloud. All other services only communicate over the Podman network, requiring to add them to the no_proxy/NO_PROXY setting. The device service we're installing cannot be reached unless it's added to the no_proxy/NO_PROXY setting and podman.socket is restarted, and (it looks like) all services that need to communicate with the device service are restarted. This is not ideal. Unfortunately, Podman uses the Golang no_proxy settings and it doesn't look like there's a way to apply a prefix to no_proxy (e.g. no_proxy=edgex-*), only a suffix (e.g. no_proxy=.edge). We may have to name hosts using a '.edge' suffix, which isn't ideal.

[cloudxxx8]

The standalone docker-compose can work normally with Podman according to our expierence.
https://docs.docker.com/compose/install/standalone/

It's easier to manage the container settings. FYI

[bnevis-i]

@dmocek,

One possible thing you could do about the proxy, though it is a bit contorted, is to send everything to a local proxy. The local proxy could have some logic in it that determines whether or not the target can be connected to locally. Connect to local targets directly, and remote targets can be forwarded to an upstream proxy.

I've never done something like this myself, but perhaps squid proxy can handle complex rules like that?