Kibana plugins directory in custom image is no longer accessible since ECK 2.16.0

[albundy83]

Hello,

with the new way to secure kibana (#7787), it seems that a new volume is always mounted /usr/share/kibana/plugins, cf commit https://github.com/elastic/cloud-on-k8s/pull/8086/files#diff-643daccc0cb377f39f56e15ac9dbcc083c7fe318a2e030ee931c7b96fd2baa81

But if you read the documentation, you should install plugins during image build: https://www.elastic.co/guide/en/cloud-on-k8s/current/k8s-kibana-plugins.html

But due to the mounted volume /usr/share/kibana/plugins, plugins are not available, they are "under" the mounted volume and not accessible.

Is it possible to disable the mount of /usr/share/kibana/plugins ?
I use the current latest operator version 2.16.0 version.

[pebrc]

Unfortunately currently the only way to disable the plugin volume mount is by disabling the default security context via set-default-security-context=false via the ECK operator configuration https://www.elastic.co/guide/en/cloud-on-k8s/current/k8s-operator-config.html This has the side-effect of also disabling the security context for Elasticsearch.

The problem we have is that Kibana writes into the plugins directory which makes it incompatible with readOnlyRootFilesystem: true in the security context.

You could potentially reinstate the security context manually through the podTemplate but leave in the case of Kibana readOnlyRootFilesystem: false

One option we could add to ECK would be an init container with a script that copies the content of /usr/share/kibana/plugins into the emptyDir volume that will shadow this directory. It is a pretty heavy weight approach that comes with its own set of drawbacks:

• slows down Kibana start up
• can cause evictions when filling up the kubelets filesystem with large plugins see Having Large Plugins in a Custom Image Causes Pods to be Evicted #6542 (this is for Elasticsearch though)

It would allows us however to keep the security context as is and still support plugin installation via custom container images.

[albundy83]

Hello,

To fix my issue, of course, I have added an initContainer in my Kibana on the same way that described here for elastic:

spec:
  podTemplate:
    spec:
      initContainers:
        - name: install-plugins
          command:
            - sh
            - -c
            - |
              /usr/share/kibana/bin/kibana-plugin install <my-plugin> || { echo "Plugin install failed with return code $?"; exit 1; }
              /usr/share/kibana/bin/kibana --optimize

Maybe with just more flexibility, we could have an option to mount or not the /usr/share/kibana/plugins volume ?