Azure, persistence and recreating the cluster

[klemen-df]

I created a new storage class exactly the same as instructed here (click!).

Our CRD looks like this (it's terraform but readable):

      "coreTemplate" = {
        "spec" = {
          "replicas" = "3"
          "volumeClaimTemplates" = {
            "storageClassName" = "azurefile-csi-nfs"
            "resources" = {
              "requests" = {
                "storage" = "20Mi"
              }
            }
            "accessModes" = ["ReadWriteOnce"]
          }

When applying this configuration (CRD) this is what happens on Azure:

• a new storage account is created ( Kind: FileStorage )
• a new NFS File share is created (size 100GiB) (per pod)
• new persistent volume is created (per pod)
• new persistent volume claim is created per pod
• each pod mounts the volume at '/opt/emqx/data'

Question(-s):

• Is new storage class created just because of nconnect?
• Is NFS mandatory? Any difference with SMB?

Great. So now data is persistent across pod reboot.

We use internal built-in database for users (password based).

The question is ; If we provide our own storage { volume (azure file share), pv, pvc }, destroy the cluster and then recreate it with the same shares, will it work?

The point is that we don't wish to loose the built in database (and it's content) if/when we destroy and then re-create the cluster.

[Rory-Z]

Hi, @KlemenDanfoss

Is NFS mandatory? Any difference with SMB?

The NFS is not mandatory, EMQX just need a storeClass, It doesn't care what storeClass is exactly

The question is ; If we provide our own storage { volume (azure file share), pv, pvc }, destroy the cluster and then recreate it with the same shares, will it work?

If destroy EMQX cluster, the PVC will still be there, can running kubectl get pvc to check it. And if recreate EMQX cluster, the PVC will it work. More info, please check https://docs.emqx.com/en/emqx-operator/latest/tasks/configure-emqx-persistence.html

[Rory-Z]

Because with the operator provisioning the share/pv/pvc they get deleted when I destroy the EMQX cluster.

No, the PV / PVC should not delete when destroy EMQX cluster. Could you please show the .metadata.ownerReferences of the operator provisioning the PV / PVC ?

[klemen-df]

Hi @Rory-Z

ownerReferences only exist:

• on the (core) pods (don't have any replicants):
• on the stateful set (emqx-core):

[Rory-Z]

Well, so strange, in Kubernetes, use ownerReferences to manage dependents, please check this: https://kubernetes.io/docs/concepts/overview/working-with-objects/owners-dependents/

So if PV / PVC didn't have ownerReference, it shouldn't have been deleted when the EMQX cluster was deleted.

Could you ask Azure about this? I'm not sure if they did anything

[klemen-df]

Ha,

it seems that if I use 'kubectl delete' it leaves the PVC therefore leaving the PV therefore leaving the underlying storage as is - persistent.

But, with using IaaC ( terraform ) I create / delete / create a kubernetes_manifest which also deletes the PVC, this also deletes the PV and the underlying storage - not persistent.

Need to investigate more.

[jacky-xbb]

Ha,

it seems that if I use 'kubectl delete' it leaves the PVC therefore leaving the PV therefore leaving the underlying storage as is - persistent.

But, with using IaaC ( terraform ) I create / delete / create a kubernetes_manifest which also deletes the PVC, this also deletes the PV and the underlying storage - not persistent.

Need to investigate more.

It could be related to the mechanism of terraform. You could double-check it by writing a demo about a pod associated a storage class, then destroy them with these two methods.

[klemen-df]

What about if I create an storage(azuredisk)/pv/pvc manually - how do I use them for, let's say, 3 core pods?

[Rory-Z]

Sorry, can not use exist PV / PVC for EMQX, the EMQX Operator just can use StoreClass.

[klemen-df]

Hm, the documentation says:

PV is a piece of storage in the cluster, which can be manually prepared according to requirements or can be dynamically created using storage classes ([StorageClass (opens new window)](https://kubernetes.io/docs/concepts/storage/storage-classes/)) preparation.

I understood this as it's possible to create our own PV/PVC and use it.

[Rory-Z]

PV is a piece of storage in the cluster, which can be manually prepared according to requirements or can be dynamically created using storage classes (StorageClass (opens new window)) preparation.

It just from Kubernetes document, not EMQX Operator, sorry for misleading you. @Gala-R Please fix this document.

[klemen-df]

Oh, ok.