java.nio.file.AccessDeniedException: /usr/share/elasticsearch/data

[Slyke]

Fresh install of ECK and Elasticsearch. Bare-metal K8s. NFS mounted volume for PV on nodes.

I occasionally get pods that do this, which is often fixed with an initContainer. Example with Grafana:

      initContainers:
      - name: setup-perms
        image: busybox:1.35.0
        command: ["/bin/sh", "-c", "chown -R 472:472 /var/lib/grafana && chmod -R 770 /var/lib/grafana"]
        volumeMounts:
        - mountPath: /certs
          name: grafana-certs-claim
        - mountPath: /var/lib/grafana
          name: grafana-data-claim

But not sure if an initContainer can be used here.

Using version 2.16.0 for CRDs and Operator.

$ kubectl get -n elastic-system pods
NAME                 READY   STATUS             RESTARTS      AGE
elastic-operator-0   1/1     Running            0             3h52m
es-es-default-0      0/1     CrashLoopBackOff   7 (64s ago)   12m

Deployment:

apiVersion: elasticsearch.k8s.elastic.co/v1
kind: Elasticsearch
metadata:
  name: es
  namespace: elastic-system
spec:
  version: 8.17.0
  nodeSets:
  - name: default
    count: 1
    config:
      node.store.allow_mmap: false
    volumeClaimTemplates:
    - metadata:
        name: elasticsearch-data
        namespace: elastic-system
      spec:
        accessModes:
        - ReadWriteOnce
        resources:
          requests:
            storage: 1Gi
        storageClassName: ""

PV:

apiVersion: v1
kind: PersistentVolume
metadata:
  name: pv-elasticsearch
  namespace: elastic-system
spec:
  storageClassName: ""
  capacity:
    storage: 1Gi
  accessModes:
    - ReadWriteOnce
  persistentVolumeReclaimPolicy: Retain
  mountOptions:
    - hard
    - nfsvers=3
  nfs:
    server: 192.168.14.4
    path: "kubernetes/elastic-system/elasticsearch/data"
    readOnly: false

Error:

java.nio.file.AccessDeniedException: /usr/share/elasticsearch/data
        at java.base/sun.nio.fs.UnixException.translateToIOException(UnixException.java:90)
        at java.base/sun.nio.fs.UnixException.asIOException(UnixException.java:115)
        at java.base/sun.nio.fs.UnixFileSystemProvider.newDirectoryStream(UnixFileSystemProvider.java:502)
        at java.base/java.nio.file.Files.newDirectoryStream(Files.java:482)
        at java.base/java.nio.file.Files.list(Files.java:3796)
        at org.elasticsearch.xpack.security.cli.AutoConfigureNode.isDirEmpty(AutoConfigureNode.java:1153)
        at org.elasticsearch.xpack.security.cli.AutoConfigureNode.execute(AutoConfigureNode.java:167)
        at org.elasticsearch.server.cli.ServerCli.autoConfigureSecurity(ServerCli.java:185)
        at org.elasticsearch.server.cli.ServerCli.execute(ServerCli.java:91)
        at org.elasticsearch.common.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:55)
        at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:95)
        at org.elasticsearch.cli.Command.main(Command.java:52)
        at org.elasticsearch.launcher.CliToolLauncher.main(CliToolLauncher.java:65

[Slyke]

I tested this out with the docker image of Elasticsearch and could reproduce the error. It was fixed with an initContainer:

      initContainers:
      - name: perm-update
        image: busybox:1.36.1
        imagePullPolicy: IfNotPresent
        command: ["/bin/sh", "-c", "chmod -R 777 /usr/share/elasticsearch/logs; chown -R 1000:1000 /usr/share/elasticsearch/logs;chmod -R 777 /usr/share/elasticsearch/data; chown -R 1000:1000 /usr/share/elasticsearch/data;chmod -R 777 /usr/share/elasticsearch/config/certs; chown -R 1000:1000 /usr/share/elasticsearch/config/certs;chmod -R 777 /usr/share/elasticsearch; chown -R 1000:1000 /usr/share/elasticsearch; ls -ahl /usr/share/elasticsearch; ls -ahl /usr/share/elasticsearch/data; ls -ahl /usr/share/elasticsearch/config; ls -ahl /usr/share/elasticsearch/config/certs"]
        volumeMounts:
          - mountPath: /usr/share/elasticsearch/data
            name: elasticsearch-data
          - mountPath: /usr/share/elasticsearch/logs
            name: elasticsearch-logs
          - mountPath: /usr/share/elasticsearch/config/certs
            name: elasticsearch-certs

I did have to disabled:

  xpack.security.enabled: "false"
  xpack.security.http.ssl.enabled: "false"
  xpack.security.transport.ssl.enabled: "false"

As I was also getting this error:

java.nio.file.FileSystemException: /usr/share/elasticsearch/config/certs -> /usr/share/elasticsearch/config/certs.1734902782.orig: Device or resource busy
        at java.base/sun.nio.fs.UnixException.translateToIOException(UnixException.java:100)
        at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:106)
        at java.base/sun.nio.fs.UnixFileSystem.move(UnixFileSystem.java:875)
        at java.base/sun.nio.fs.UnixFileSystemProvider.move(UnixFileSystemProvider.java:310)
        at java.base/java.nio.file.Files.move(Files.java:1437)
        at org.elasticsearch.xpack.security.cli.AutoConfigureNode.moveDirectory(AutoConfigureNode.java:958)
        at org.elasticsearch.xpack.security.cli.AutoConfigureNode.execute(AutoConfigureNode.java:618)
        at org.elasticsearch.server.cli.ServerCli.autoConfigureSecurity(ServerCli.java:185)
        at org.elasticsearch.server.cli.ServerCli.execute(ServerCli.java:91)
        at org.elasticsearch.common.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:55)
        at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:95)