Internal Encryption on/off requires activator restart

[davidhadas]

Switching internal-encryption to true results in a non-functioning service.
/area networking

What version of Knative?

1.9
Using Kourier

Expected Behavior

Service should continue to serve ingress traffic.

Actual Behavior

Service stopped serving ingress traffic.
Clients receive 503

Steps to Reproduce the Problem

---- Attempt 1:

kn quickstart kind  --install-serving
...

Use kubectl edit cm config-network -n knative-serving and set internal-encryption to "true"

kn service create hello --image gcr.io/knative-samples/helloworld-go --port 8080 --env TARGET=World
...
19.014s Ingress has not yet been reconciled.
19.101s Waiting for load balancer to be ready
 ...

----- Attempts 2:

kn quickstart kind  --install-serving
...

kn service create hello --image gcr.io/knative-samples/helloworld-go --port 8080 --env TARGET=World
...
Service 'hello' created to latest revision 'hello-00001' is available at URL:
http://hello.default.127.0.0.1.sslip.io

curl http://hello.default.127.0.0.1.sslip.io -v
*   Trying 127.0.0.1:80...
...
< HTTP/1.1 200 OK
...
Hello World!
...

Use kubectl edit cm config-network -n knative-serving and set internal-encryption to "true"

curl http://hello.default.127.0.0.1.sslip.io -v
...
< HTTP/1.1 503 Service Unavailable
< content-length: 145
< content-type: text/plain
< date: Mon, 27 Feb 2023 23:30:17 GMT
< server: envoy
<
* Connection #0 to host hello.default.127.0.0.1.sslip.io left intact
upstream connect error or disconnect/reset before headers. reset reason: connection failure, transport failure reason: delayed connect error: 111

[davidhadas]

@ReToCode, @skonto fyi - not sure if this is kourier specific or not.

[davidhadas]

Reproduced on a regular Kubernetes cluster outside of Kind (again with Kourier)

[ReToCode]

@davidhadas, does it work if you restart activator first? We are still working on that: #13694. Also alpha is not quite there yet, see: #11906 (comment).

[davidhadas]

It solved the issue on both systems tested.
I will keep this open and indicate in #13694 that it fixes this one.

[dprotaso]

Closing dupe of #13694

[nak3]

I realized that this issue was linked to #13694 but this issue is for switching internal-encryption to true/false, right?

I think they should be handled separately. And actually not only internal-encryption but also some flags like mesh-compatibility-mode , enable-mesh-pod-addressability are also required to restart.