The secret list page loads slowly

[niusmallnan]

Setup

• Rancher version: v2.6.6
• Browser type & version: Chrome on Mac

Describe the bug
If the number of these two kinds of secrets is large: svc-account token and TLS certificate.
When using global admin or cluster owner to view the secret list, the UI response time is slow.

To Reproduce
Create hundreds of secrets with the svc-account token or TLS certificate.

Result
Even if ns is specified on the UI, this API interface(/v1/secrets/) will obtain the data of all namespaces, and it will take a long time to download the data.

Expected Result
The UI loads quickly and improves the user experience.

Additional context
I try to think of some solutions.

Option One

Only request the specified ns data, which can avoid loading all secrets with a high probability.

/v1/secrets/ns1
/v1/secrets/ns2
/v1/secrets/ns3

Option II

Secret type filtering is supported on the UI, and only data of type Opaque is queried by default. This may require some enhancements to the steve API.

[richard-cox]

Thanks for raising this issue. Can you confirm

• The number of secrets in your system where you see this issue
• Whether the secrets have values that are large
• Whether creating 200 secrets as described reproduces the issue

There are a number of performance issues we are currently investigating and this information will greatly help

[niusmallnan]

@richard-cox Thanks for your reply.

The number of secrets in your system where you see this issue

I have seen from a community user environment that the data download size for the secret interface is about 30MB.
However, I haven't had a chance to see more detailed information on the number of secrets.

Whether the secrets have values that are large

Obviously, yes. However, this may not be their intention. Because the svc-token and tls certificate types in the secret store the x509 certificate, this data will be relatively large.

Whether creating 200 secrets as described reproduces the issue

I try to create 2k svc accounts, these will be created as secrets synchronously and saved as x509 certificates.

Referring to this script, 2k secrets can be generated in the test namespace.

#!/bin/bash

NS=test
NUM=2000

n=1
while [ $n -le $NUM ]; do
    kubectl create serviceaccount test$n -n $NS
    let n++
done

Check one of the secrets and you'll see an x509 certificate.

kubectl get sa test1 -o yaml -n test
apiVersion: v1
kind: ServiceAccount
metadata:
  creationTimestamp: "2022-07-20T07:25:14Z"
  name: test1
  namespace: test
  resourceVersion: "18581"
  uid: 51e11026-f83c-4c70-9831-c9a798d35925
secrets:
- name: test1-token-vm2zh
- 
kubectl get secret test1-token-vm2zh -o yaml -n test
apiVersion: v1
data:
  ca.crt: xxx
  namespace: dGVzdA==
  token: xxx
kind: Secret
metadata:
  annotations:
    kubernetes.io/service-account.name: test1
    kubernetes.io/service-account.uid: 51e11026-f83c-4c70-9831-c9a798d35925
  creationTimestamp: "2022-07-20T07:25:14Z"
  name: test1-token-vm2zh
  namespace: test
  resourceVersion: "18580"
  uid: 8a0fad8a-357d-4c6e-96d8-2d8180a60f00
type: kubernetes.io/service-account-token

Using a browser to access the secret list page, even if I specify the default namespace, I still need to wait for all the secret data to be downloaded.

In my network environment, this API request(/v1/secrets) completed the request in about 5s+:

• Waiting for server response: 1.25s
• Content download: 4.03s

[catherineluse]

Even if ns is specified on the UI, this API interface(/v1/secrets/) will obtain the data of all namespaces

The UI frequently uses findAll to get all resources of a certain type. https://github.com/rancher/dashboard/blob/master/shell/plugins/dashboard-store/actions.js#L78 One issue with the findAll is that it is not aware of active namespaces. I think it that before findAll fetches all of any resource type, it should check for selected namespaces, preferably using this active namespace cache. #6261 If any are selected, it should fetch the resources one namespace at a time.

Currently the work of filtering by namespace is done in ResourceTable. It would be more efficient if we just fetched less data with findAll.

[catherineluse]

Related issue #6515

[richard-cox]

Related issue #6483 (which is a workaround, more improvements to come)

[gaktive]

The related issues should help sort this out but there are also going to be backend fixes coming in later that should address this.

[richard-cox]

@niusmallnan You might be interested in a feature from 2.7.5 that filters resources via the values in the namespace / project filter in the header. This greatly reduces the number of resources fetched by the UI. To enable please go to burger menu top left --> Global Settings --> Performance --> Require Namespace / Project Filtering and check the box and hit Apply.

[richard-cox]

This falls in the general 'we shouldn't fetch all resources' sphere. There's a current workaround listed in #6400 (comment) and will be fully resolved via #8527