Abnormal disk usage for 2 server 2 worker cluster

[btrepp]

From

k3s-io/kine#105 (comment)

I have a 2 node, 2 worker mariadb backed cluster. It looks lik it should have 500kib/sec. https://rancher.com/docs/k3s/latest/en/installation/installation-requirements/resource-profiling/

I am experiencing 5mb/sec. I know this is more than the single node, but it seems like a large increase.

Is there any way I can inspect or debug this?. Worried that this might hurt the lifespan of the Ssd backing the database.

[brandond]

You can check to see which etcd keys have been written to most frequently by running a SQL query against the database:

SELECT COUNT(*), name FROM kine GROUP BY name ORDER BY COUNT(*) DESC LIMIT 25;

You shouldn't see the count for any of these being more than a few 10s, perhaps into the low 100s.

Are you seeing 5MB/sec in disk IO, or just network throughput against the server?

[btrepp]

COUNT(*);name
263;/registry/configmaps/kube-system/k3s
258;/registry/leases/kube-system/cloud-controller-manager
258;/registry/leases/kube-system/kube-controller-manager
257;/registry/services/endpoints/k8s1-system/cluster.local-k8s1-nfs-subdir-external-provisioner
257;/registry/leases/kube-system/kube-scheduler
208;/registry/leases/kube-system/cert-manager-cainjector-leader-election
207;/registry/configmaps/kube-system/cert-manager-cainjector-leader-election
62;/registry/masterleases/192.168.10.10
59;/registry/leases/kube-node-lease/agent2
59;/registry/leases/kube-node-lease/server1
56;/registry/leases/kube-node-lease/agent1
38;/registry/leases/kube-system/cert-manager-controller
37;/registry/configmaps/kube-system/cert-manager-controller
21;/registry/apiextensions.k8s.io/customresourcedefinitions/nodes.longhorn.io
13;/registry/minions/agent1

I seem to have quite a few things in the few hundres. So I wonder if something is unhappy. Some of those items (eg longhorn) have been removed.

Its around 3-5MB/sec in disk IO, reported from the NAS itself running mariaDB.

I may re-install the cluster from scratch to see if that helps. Perhaps my fiddling has caused something strange to happen. The kine table seems to have 4169 rows. Unsure if that's high or low. I don't feel like I'm running too many workloads.

EDIT: Completely reset the cluster to a new database. Only has coredns, metrics and argocd installed.
This still results in about 2MB/sec IO on the NAS and the query results in

COUNT(*);name
270;/registry/configmaps/kube-system/k3s
267;/registry/leases/kube-system/kube-controller-manager
267;/registry/leases/kube-system/cloud-controller-manager
266;/registry/leases/kube-system/kube-scheduler
59;/registry/masterleases/192.168.10.10
59;/registry/masterleases/192.168.10.11
57;/registry/leases/kube-node-lease/server1
57;/registry/leases/kube-node-lease/agent1
57;/registry/leases/kube-node-lease/agent2
56;/registry/leases/kube-node-lease/server2
6;/registry/argoproj.io/applications/argocd/k8s1
3;/registry/minions/server1
3;/registry/minions/agent1
3;/registry/minions/server2
3;/registry/minions/agent2
2;/registry/events/argocd/argocd-dex-server-cd5f6c7c5-2r8sr.16aa62e264fab220
2;/registry/events/kube-system/metrics-server-86cbb8457f-4zf58.16aa62d7e85f9717
2;/registry/events/argocd/argocd-repo-server-7dcd4b549b-bcbj8.16aa62e26516e6c2
2;/registry/events/kube-system/coredns-7448499f4d-ppmkg.16aa62d7e875a8fe
2;/registry/events/argocd/argocd-server-78b76c87b5-m7fmn.16aa62e2a09ccc24
2;/registry/events/argocd/argocd-redis-596b7c68bc-2f2qv.16aa62e2a0ac0312
2;/registry/events/argocd/argocd-application-controller-78b668dd8f-dv75k.16aa62e22a937aa0
1;/registry/events/kube-system/metrics-server-86cbb8457f-4zf58.16aa632fbcd86dc6
1;/registry/events/kube-system/coredns-7448499f4d-ppmkg.16aa62d1259f445a
1;/registry/events/default/server2.16aa62dc18fceed9

So I wonder if this is just 'normal' for kine behind mariadb?