etcd server connection issues in client

[Dreamacro]

Hi,

I'm having trouble with etcd server connections in client. Since the server is self-hosted and used for cross-region and cross-continent access, network connectivity is very unstable. Sometimes the server needs to be restarted and client cannot reconnect.

I tried using with_keep_alive and with_connect_timeout but it didn't help.

The problem mainly occurs in lease_keep_alive. When the connection is interrupted or etcd is restarted, it cannot automatically reconnect. Sometimes it shows that the connection is refused and no longer retries.

What are some ways to handle client in this situation? How can I implement reasonable keep alive?

[davidli2010]

Have you tried keep_alive_while_idle? see #42

[Dreamacro]

I used ConnectOptions::new to create options, keep_alive_while_idle should be true by default

etcd-client/src/client.rs

Lines 816 to 828 in 3c46d0c

	pub const fn new() -> Self {
	ConnectOptions {
	user: None,
	keep_alive: None,
	keep_alive_while_idle: true,
	timeout: None,
	connect_timeout: None,
	#[cfg(feature = "tls")]
	tls: None,
	#[cfg(feature = "tls-openssl")]
	otls: None,
	}
	}

let opts = ConnectOptions::new()
  .with_user(cfg.etcd.username, cfg.etcd.password)
  .with_keep_alive(Duration::from_secs(20), Duration::from_secs(5));

BTW, client.watch also has this problem

[Dreamacro]

Currently, there are two types of parameters for etcd's auth_token, simple (the default) and jwt (recommended for production environments https://etcd.io/docs/v3.5/op-guide/configuration/#auth). These two modes require the functionality to refresh tokens in specific situations.

For simple, when the etcd server restarts, tonic returns an Unauthenticated status code, and the entire client cannot make any more requests after that.

For jwt, when the token's time limit exceeds the TTL (which is usually not very long), the entire client cannot make any more requests.

In the etcd-go client, there is some code for refreshing the token.

https://github.com/etcd-io/etcd/blob/53b48bbd5795210af2620ac757d9529b34a09e48/client/v3/retry_interceptor.go#L273-L281

Same as etcd-rs, etcd-client does not refresh the token automatically.

[davidli2010]

Currently, there are two types of parameters for etcd's auth_token, simple (the default) and jwt (recommended for production environments https://etcd.io/docs/v3.5/op-guide/configuration/#auth). These two modes require the functionality to refresh tokens in specific situations.

For simple, when the etcd server restarts, tonic returns an Unauthenticated status code, and the entire client cannot make any more requests after that.

For jwt, when the token's time limit exceeds the TTL (which is usually not very long), the entire client cannot make any more requests.

In the etcd-go client, there is some code for refreshing the token.

https://github.com/etcd-io/etcd/blob/53b48bbd5795210af2620ac757d9529b34a09e48/client/v3/retry_interceptor.go#L273-L281

Same as etcd-rs, etcd-client does not refresh the token automatically.

see #45

[davidli2010]

@Dreamacro please see #65