-
Notifications
You must be signed in to change notification settings - Fork 40
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
[PLAT-13974] Helm changes to support non-restart gflags workflow
Summary:
Added changes to helm chart to support non-restart gflags
Following is the mounted gflags file: `server.conf.template`
```
[root@ybvkumar-k8s-us-west1-a-vabq-yb-tserver-0 cores]# cat /opt/tserver/conf/server.conf.template
--fs_data_dirs=/mnt/disk0
--tserver_master_addrs=ybvkumar-k8s-us-west1-b-wabq-yb-master-0.ybvkumar-k8s-us-west1-b-wabq-yb-masters.yb-admin-vkumar-k8s.svc.cluster.local:7100,ybvkumar-k8s-us-west1-a-vabq-yb-master-0.ybvkumar-k8s-us-west1-a-vabq-yb-masters.yb-admin-vkumar-k8s.svc.cluster.local:7100,ybvkumar-k8s-us-west1-c-xabq-yb-master-0.ybvkumar-k8s-us-west1-c-xabq-yb-masters.yb-admin-vkumar-k8s.svc.cluster.local:7100
--enable_ysql=true
--pgsql_proxy_bind_address=0.0.0.0:5433
--metric_node_name=${HOSTNAME}
--memory_limit_hard_bytes=3400000000
--stderrthreshold=0
--max_log_size=256
--num_cpus=2
--undefok=num_cpus,enable_ysql
--use_node_hostname_for_local_tserver=true
--cql_proxy_bind_address=${HOSTNAME}.ybvkumar-k8s-us-west1-a-vabq-yb-tservers.${NAMESPACE}.svc.cluster.local
--rpc_bind_addresses=${HOSTNAME}.ybvkumar-k8s-us-west1-a-vabq-yb-tservers.${NAMESPACE}.svc.cluster.local
--server_broadcast_addresses=${HOSTNAME}.ybvkumar-k8s-us-west1-a-vabq-yb-tservers.${NAMESPACE}.svc.cluster.local:9100
--webserver_interface=0.0.0.0
--certs_for_cdc_dir=/mnt/disk0/yw-data/yugabyte-tls-producer
--placement_cloud=kubernetes
--placement_region=us-west1
--placement_uuid=50fae571-9681-41d3-b290-9581cc17988c
--placement_zone=us-west1-a
--start_redis_proxy=false
```
Following is the applied gflags file used in the process: `server.conf`
```
[root@ybvkumar-k8s-us-west1-a-vabq-yb-tserver-0 cores]# cat /home/yugabyte/tserver/conf/server.conf
--fs_data_dirs=/mnt/disk0
--tserver_master_addrs=ybvkumar-k8s-us-west1-b-wabq-yb-master-0.ybvkumar-k8s-us-west1-b-wabq-yb-masters.yb-admin-vkumar-k8s.svc.cluster.local:7100,ybvkumar-k8s-us-west1-a-vabq-yb-master-0.ybvkumar-k8s-us-west1-a-vabq-yb-masters.yb-admin-vkumar-k8s.svc.cluster.local:7100,ybvkumar-k8s-us-west1-c-xabq-yb-master-0.ybvkumar-k8s-us-west1-c-xabq-yb-masters.yb-admin-vkumar-k8s.svc.cluster.local:7100
--enable_ysql=true
--pgsql_proxy_bind_address=0.0.0.0:5433
--metric_node_name=ybvkumar-k8s-us-west1-a-vabq-yb-tserver-0
--memory_limit_hard_bytes=3400000000
--stderrthreshold=0
--max_log_size=256
--num_cpus=2
--undefok=num_cpus,enable_ysql
--use_node_hostname_for_local_tserver=true
--cql_proxy_bind_address=ybvkumar-k8s-us-west1-a-vabq-yb-tserver-0.ybvkumar-k8s-us-west1-a-vabq-yb-tservers.yb-admin-vkumar-k8s.svc.cluster.local
--rpc_bind_addresses=ybvkumar-k8s-us-west1-a-vabq-yb-tserver-0.ybvkumar-k8s-us-west1-a-vabq-yb-tservers.yb-admin-vkumar-k8s.svc.cluster.local
--server_broadcast_addresses=ybvkumar-k8s-us-west1-a-vabq-yb-tserver-0.ybvkumar-k8s-us-west1-a-vabq-yb-tservers.yb-admin-vkumar-k8s.svc.cluster.local:9100
--webserver_interface=0.0.0.0
--certs_for_cdc_dir=/mnt/disk0/yw-data/yugabyte-tls-producer
--placement_cloud=kubernetes
--placement_region=us-west1
--placement_uuid=50fae571-9681-41d3-b290-9581cc17988c
--placement_zone=us-west1-a
--start_redis_proxy=false
```
The process will now start as:
```
root 7 1 0 09:12 ? 00:00:00 python /home/yugabyte/tools/k8s_parent.py /home/yugabyte/bin/yb-tserver --flagfile /tmp/yugabyte/tserver
/conf/server.conf
root 31 7 1 09:12 ? 00:02:04 /home/yugabyte/bin/yb-tserver --flagfile /tmp/yugabyte/tserver/conf/server.conf
```
Gflags setup
- The gflags are created as per-zone secret.
- The secret will be mounted at `/opt/{master,tserver}/conf/server.conf.template`
- During runtime, the environment variables are used to replace the placeholder values. The final set of gflags after replacement are copied to `/home/yugabyte/{master,tserver}/conf/server.conf`
Modification and Upgrade
- Added a helm value per service:
```
tserver:
gflagsChecksum: "0"
master:
gflagsChecksum: "0"
```
- For upgrades which require restart, we will provide the above value as "0", such that helm will calculate the checksum of secret and restart pod if required.
- For upgrades which do not require restart, we will use the previous value of checksum, such that the secret modification does not lead to pod restart.
Test Plan:
Manually tested the changes for helm
- Tested modifying secret and verified there is pod restart if old checksum is supplied.
- Tested pods are restarted if gflags change and checksum provided is 0.
- Tested processes are up and running using new way of applying gflags.
Reviewers: anijhawan, sneelakantan, bgandhi, sanketh
Reviewed By: anijhawan, bgandhi
Subscribers: yugaware
Differential Revision: https://phorge.dev.yugabyte.com/D35040- Loading branch information
1 parent
50815b5
commit 692d04c
Showing
5 changed files
with
172 additions
and
109 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,57 @@ | ||
| {{- $root := . -}} | ||
| {{- range $service := $root.Values.Services }} | ||
| {{- if eq $service.name "yb-masters" }} | ||
| {{- $serviceValues := (dict "Service" $service "Values" $root.Values "Chart" $root.Chart "Release" $root.Release) -}} | ||
| --- | ||
| apiVersion: v1 | ||
| kind: Secret | ||
| metadata: | ||
| name: {{ include "yugabyte.fullname" $root }}-master-gflags | ||
| namespace: "{{ $root.Release.Namespace }}" | ||
| type: Opaque | ||
| stringData: | ||
| server.conf.template: | | ||
| {{- if not $root.Values.storage.ephemeral }} | ||
| --fs_data_dirs={{ template "yugabyte.fs_data_dirs" $root.Values.storage.master }} | ||
| {{- else }} | ||
| --fs_data_dirs=/var/yugabyte | ||
| {{- end }} | ||
| {{- if eq $root.Values.ip_version_support "v6_only" }} | ||
| --net_address_filter=ipv6_external,ipv6_non_link_local,ipv6_all,ipv4_external,ipv4_all | ||
| {{- end }} | ||
| {{- if $root.Values.isMultiAz }} | ||
| --master_addresses={{ $root.Values.masterAddresses }} | ||
| --replication_factor={{ $root.Values.replicas.totalMasters }} | ||
| {{- else }} | ||
| --master_addresses={{ template "yugabyte.master_addresses" $root }} | ||
| --replication_factor={{ $root.Values.replicas.master }} | ||
| {{- end }} | ||
| {{- if not $root.Values.disableYsql }} | ||
| --enable_ysql=true | ||
| {{- else }} | ||
| --enable_ysql=false | ||
| {{- end }} | ||
| {{- if $root.Values.tls.enabled }} | ||
| --certs_dir=/opt/certs/yugabyte | ||
| --use_node_to_node_encryption={{ $root.Values.tls.nodeToNode }} | ||
| --allow_insecure_connections={{ $root.Values.tls.insecure }} | ||
| {{- end }} | ||
| {{- if $root.Values.yugabytedUi.enabled }} | ||
| --master_enable_metrics_snapshotter={{ $root.Values.yugabytedUi.metricsSnapshotter.enabled }} | ||
| --metrics_snapshotter_tserver_metrics_whitelist={{ join "," $root.Values.yugabytedUi.metricsSnapshotter.whitelist }} | ||
| {{- end }} | ||
| --metric_node_name=${HOSTNAME} | ||
| --memory_limit_hard_bytes={{ template "yugabyte.memory_hard_limit" dict "size" $root.Values.resource.master.limits.memory "limitPercent" $root.Values.master.memoryLimitHardPercentage }} | ||
| --stderrthreshold=0 | ||
| --num_cpus={{ ceil $root.Values.resource.master.requests.cpu }} | ||
| --max_log_size=256 | ||
| --undefok=num_cpus,enable_ysql | ||
| --rpc_bind_addresses={{ include "yugabyte.rpc_bind_address" $serviceValues }} | ||
| --server_broadcast_addresses={{ include "yugabyte.server_broadcast_address" $serviceValues }} | ||
| --webserver_interface={{ include "yugabyte.webserver_interface" $serviceValues }} | ||
| {{- range $flag, $override := $root.Values.gflags.master }} | ||
| --{{ $flag }}={{ $override }} | ||
| {{- end }} | ||
| {{- end }} | ||
| {{- end }} | ||
| --- |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,73 @@ | ||
| {{- $root := . -}} | ||
| {{- range $service := $root.Values.Services }} | ||
| {{- if eq $service.name "yb-tservers" }} | ||
| {{- $serviceValues := (dict "Service" $service "Values" $root.Values "Chart" $root.Chart "Release" $root.Release) -}} | ||
| --- | ||
| apiVersion: v1 | ||
| kind: Secret | ||
| metadata: | ||
| name: {{ include "yugabyte.fullname" $root }}-tserver-gflags | ||
| namespace: "{{ $root.Release.Namespace }}" | ||
| type: Opaque | ||
| stringData: | ||
| server.conf.template: | | ||
| {{- if not $root.Values.storage.ephemeral }} | ||
| --fs_data_dirs={{ template "yugabyte.fs_data_dirs" $root.Values.storage.tserver }} | ||
| {{- else }} | ||
| --fs_data_dirs=/var/yugabyte | ||
| {{- end }} | ||
| {{- if eq $root.Values.ip_version_support "v6_only" }} | ||
| --net_address_filter=ipv6_external,ipv6_non_link_local,ipv6_all,ipv4_external,ipv4_all | ||
| {{- end }} | ||
| {{- if $root.Values.isMultiAz }} | ||
| --tserver_master_addrs={{ $root.Values.masterAddresses }} | ||
| {{- else }} | ||
| --tserver_master_addrs={{ template "yugabyte.master_addresses" $root }} | ||
| {{- end }} | ||
| {{- if $root.Values.authCredentials.ysql.password }} | ||
| --ysql_enable_auth=true | ||
| {{- if (include "yugabyte.tserver.readinessProbe" $root) }} | ||
| --ysql_hba_conf_csv="local all yugabyte trust" | ||
| {{- end }} | ||
| {{- end }} | ||
| {{- if or $root.Values.authCredentials.ycql.user $root.Values.authCredentials.ycql.password }} | ||
| --use_cassandra_authentication=true | ||
| {{- end }} | ||
| {{- if $root.Values.tls.enabled }} | ||
| --certs_dir=/opt/certs/yugabyte | ||
| --use_node_to_node_encryption={{ $root.Values.tls.nodeToNode }} | ||
| --allow_insecure_connections={{ $root.Values.tls.insecure }} | ||
| --use_client_to_server_encryption={{ $root.Values.tls.clientToServer }} | ||
| --certs_for_client_dir=/opt/certs/yugabyte | ||
| {{- if $root.Values.tserver.serverBroadcastAddress }} | ||
| --cert_node_filename={{ include "yugabyte.server_fqdn" $serviceValues }} | ||
| {{- end }} | ||
| {{- end }} | ||
| {{- if not $root.Values.disableYsql }} | ||
| --enable_ysql=true | ||
| --pgsql_proxy_bind_address={{ include "yugabyte.pgsql_proxy_bind_address" $serviceValues }} | ||
| {{- else }} | ||
| --enable_ysql=false | ||
| {{- end }} | ||
| {{- if $root.Values.yugabytedUi.enabled }} | ||
| --tserver_enable_metrics_snapshotter={{ $root.Values.yugabytedUi.metricsSnapshotter.enabled }} | ||
| --metrics_snapshotter_interval_ms={{ $root.Values.yugabytedUi.metricsSnapshotter.interval }} | ||
| --metrics_snapshotter_tserver_metrics_whitelist={{ join "," $root.Values.yugabytedUi.metricsSnapshotter.whitelist }} | ||
| {{- end }} | ||
| --metric_node_name=${HOSTNAME} | ||
| --memory_limit_hard_bytes={{ template "yugabyte.memory_hard_limit" dict "size" $root.Values.resource.tserver.limits.memory "limitPercent" $root.Values.tserver.memoryLimitHardPercentage }} | ||
| --stderrthreshold=0 | ||
| --max_log_size=256 | ||
| --num_cpus={{ ceil $root.Values.resource.tserver.requests.cpu }} | ||
| --undefok=num_cpus,enable_ysql | ||
| --use_node_hostname_for_local_tserver=true | ||
| --cql_proxy_bind_address={{ include "yugabyte.cql_proxy_bind_address" $serviceValues }} | ||
| --rpc_bind_addresses={{ include "yugabyte.rpc_bind_address" $serviceValues }} | ||
| --server_broadcast_addresses={{ $root.Values.tserver.serverBroadcastAddress | default (include "yugabyte.server_broadcast_address" $serviceValues) }} | ||
| --webserver_interface={{ include "yugabyte.webserver_interface" $serviceValues }} | ||
| {{- range $flag, $override := $root.Values.gflags.tserver }} | ||
| --{{ $flag }}={{ $override }} | ||
| {{- end }} | ||
| {{- end }} | ||
| {{- end }} | ||
| --- |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters