Default endpoint based health checks

[maleck13]

What

Currently healthchecks have to be defined and use a HTTP request response model to check the health. This is good for checking the end to end connectivity. However it requires some effort from the developer and the platform engineer in order to get into a good state and it will not work for a wildcard listener. Ideally we would have a default health check that could be performed without the need for additional configuration.

I would like that the DNS operator on a given cluster automatically checked for available endpoints via an endpointslice for a given backend ref if that backend ref was of type Service. If there were no available endpoints across all services associated with a listener, it should remove those gateways addresses from the DNS response.

Note just as with the existing DNS health checks, we would still need to know how to avoid an NXDomain response

Additional Context

Each Listener on a gateway that results in a DNSRecord has at least one HTTPRoute attached. We can inspect the attached HTTPRoutes for a given listener and check each of their backend refs for available endpoints. If there are available endpoints for given listener then we publish or keep the record in place. If there are none we remove the the records for that gateway listener host. We can achieve this without need to make http requests to an application and instead just use the k8s API.

If we chose to go with this, it should be on by default but be something that can be disabled explicitly via the policy for a gateway or listener.

[maleck13]

@philbrookes interested in your thoughts on this