[TEST-ONLY] [WIP] debug kourier tls runtime failure

[skonto]

Fixes #

Proposed Changes

Release Note

[knative-prow]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: skonto

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [skonto]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 80.82%. Comparing base (19b9a09) to head (8df7745).
Report is 2 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff             @@
##             main   #15695      +/-   ##
==========================================
+ Coverage   80.78%   80.82%   +0.04%     
==========================================
  Files         222      222              
  Lines       18025    18027       +2     
==========================================
+ Hits        14561    14570       +9     
+ Misses       3092     3086       -6     
+ Partials      372      371       -1     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[skonto]

/retest

[skonto]

So far:

Activator:

2025-01-14T18:00:44.6007786Z     stream.go:304: E 17:59:59.628 activator-874694df9-mphrq [activator] [serving-tests/probe-runtime-http-get-yfsvnmvr-00001] error reverse proxying request; sockstat: sockets: used 260
2025-01-14T18:00:44.6008001Z         TCP: inuse 69 orphan 0 tw 28 alloc 1840 mem 193
2025-01-14T18:00:44.6008128Z         UDP: inuse 0 mem 0
2025-01-14T18:00:44.6008250Z         UDPLITE: inuse 0
2025-01-14T18:00:44.6008353Z         RAW: inuse 0
2025-01-14T18:00:44.6008516Z         FRAG: inuse 0 memory 0
2025-01-14T18:00:44.6008875Z          err=timed out dialing 10.96.177.28:8112 after 21.11s

Kourier gateway:

2025-01-14T17:59:41.733634585Z stdout F [2025-01-14T17:59:37.733Z] "GET /healthz HTTP/2" 200 - 0 0 1 1 "-" "Knative-Ingress-Probe" "c4d56a77-9b6f-4de7-ab89-b02c617301c2" "probe-runtime-http-get-yfsvnmvr.serving-tests" "10.244.3.8:8112"
..."internalkourier" "127.0.0.1:9901"
2025-01-14T18:00:21.738139469Z stdout F [2025-01-14T17:59:59.662Z] "GET /healthz/readiness HTTP/1.1" 502 - 0 49 20998 20997 "-" "Go-http-client/1.1" "77897c29-7698-454e-8a65-1f2bda9ad7b7" "probe-runtime-http-get-yfsvnmvr.serving-tests.example.com" "10.244.3.8:8112"

It seems certs are loaded and everything is up, including the targeted pod. Ingress probing works but when activator sends the request to the ksvc private service (10.96.177.28:8112) it fails. Something is off with the tls port or the TLS handshake because I dont see any logs at the QP side when presenting the certificate at the server side. So the connection proxied from the activator times out earlier with 502. Btw it never recovers as I a run a second request that fails as well.

[skonto]

Not reproducible any more 🤔

[skonto]

I am seeing this in cert manager's webhook log

2025-01-21T12:32:24.301181686Z stderr F I0121 12:32:24.301076       1 logs.go:59] http: TLS handshake error from 172.18.0.6:44136: EOF
2025-01-21T12:39:05.127018Z stderr F I0121 12:39:05.126912       1 logs.go:59] http: TLS handshake error from 172.18.0.6:4220: EOF
2025-01-21T12:40:36.752184067Z stderr F I0121 12:40:36.750348       1 logs.go:59] http: TLS handshake error from 172.18.0.6:37034: EOF
2025-01-21T12:41:13.682699565Z stderr F I0121 12:41:13.681779       1 logs.go:59] http: TLS handshake error from 172.18.0.6:5044: EOF

cert-manager/cert-manager#4594

[knative-prow]

@skonto: The following tests failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Required	Rerun command
unit-tests_serving_main	76c7838	link	true	/test unit-tests
build-tests_serving_main	76c7838	link	true	/test build-tests

Your PR dashboard.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.