Keep original server names in HAproxy backends

[gdubicki]

Mostly to make the HAproxy logs more useful for tracing the requests.

Although it doesn't make much sense without #51 it works independently so it can be reviewed in the same way.

Also related to #45.

[aiharos]

Functionally looks good, but I'd like to hear from @ShimmerGlass too.

One thing I'm not entirely certain about is changing the generated server names to "disabled_server*" - if there is a possibility of these server names being retained (while the address information is updated), and thus creating confusion as to which ones are active or not - I'd suggest retaining the old non-informative "server_xyz" naming scheme.

[ShimmerGlass]

@aiharos @gdubicki, we cannot use the original server names, there is a mechanism that reuses server slots to avoid reloading haproxy each time an instance comes up or down. The name in the server slot must be stable even if the ip/port changes or the server goes down.

[pierresouchay]

cannot work as it is as node.Node might be twice in the list

[pierresouchay]

name would not be unique: the proxy might try to target several instances of the same service on 1 single node

[ShimmerGlass]

Closing this because I don't see a way to implement this right now.

[gdubicki]

One more thing:

Why is reloading HAproxy so bad, @ShimmerGlass and @pierresouchay ? Assuming that only some of the servers change and it happens only every few days, is that a problem?

I am asking because perhaps we could add an alternative mode to haproxy-consul-connect that WILL reload BUT will provide more features, like this one.

[ShimmerGlass]

@gdubicki, this would mean every server change triggers a reload. This is not so bad with a stable bare-metal architecture with few servers, however, it will be an issue for :

• Services with a large number of instances: more instances, more health check flapping (heath check changing constantly from passing to critical) more reload
• A single flapping instance would cause haproxies to reload. The typical Consul health check interval is 10s: reload every 10s, causing platform performance as a whole to degrade because of a single bad apple in the basket. More flapping instances and the problem quickly goes out of hand.
• Deployments by design cause all instances to flap in cascade, causing a lot of reloads, impacting client services

We need to decouple haproxy reloads from server health for a stable service.

I don't think an option for that would be a good idea. It is pretty low level and relates to an Haproxy implementation detail and it is hard for users to understand exactly how this can go bad. Instead, if observability is an issue, I propose we focus on prometheus metrics for example, or an HTML status page.