From 46f93448237a7cc45bc0ddc313a3fb8806f9c03c Mon Sep 17 00:00:00 2001 From: Matt Richerson Date: Fri, 2 Aug 2024 13:20:02 -0500 Subject: [PATCH 1/3] Add NnfLustreMGT documentation Document how to create the NnfLustreMGT and ConfigMap for MGTs outside of NNF's control. Signed-off-by: Matt Richerson --- docs/guides/external-mgs/readme.md | 55 +++++++++++++++++++++++++++++- 1 file changed, 54 insertions(+), 1 deletion(-) diff --git a/docs/guides/external-mgs/readme.md b/docs/guides/external-mgs/readme.md index cc8a553..c058c0d 100644 --- a/docs/guides/external-mgs/readme.md +++ b/docs/guides/external-mgs/readme.md @@ -17,6 +17,7 @@ These three methods are not mutually exclusive on the system as a whole. Individ ## Configuration with an External MGT +### Storage Profile An existing MGT external to the NNF cluster can be used to manage the Lustre file systems on the NNF nodes. An advantage to this configuration is that the MGT can be highly available through multiple MGSs. A disadvantage is that there is only a single MGT. An MGT shared between more than a handful of Lustre file systems is not a common use case, so the Lustre code may prove less stable. The following yaml provides an example of what the `NnfStorageProfile` should contain to use an MGT on an external server. @@ -30,12 +31,64 @@ metadata: data: [...] lustreStorage: - externalMgs: 1.2.3.4@eth0 + externalMgs: 1.2.3.4@eth0:1.2.3.5@eth0 combinedMgtMdt: false standaloneMgtPoolName: "" [...] ``` +### NnfLustreMGT +A `NnfLustreMGT` resource tracks which fsnames have been used on the MGT to prevent fsname re-use. Any Lustre file systems that are created through the NNF software will request an fsname to use from a `NnfLustreMGT` resource. Every MGT must have a corresponding `NnfLustreMGT` resource. For MGTs that are hosted on NNF hardware, the `NnfLustreMGT` resources are created automatically. The NNF software also erases any no longer used fsnames from disk for any internally hosted MGTs. For an MGT hosted on an external node, an admin must create an `NnfLustreMGT`. This resource ensures that fsnames will be created in a sequential order without any fsname re-use. However, after an fsname is no longer in use by a file system, it will not be erased from the MGT disk. An admin may decide to periodically run the `lctl erase_lcfg [fsname]` command to remove fsnames that are no longer in use. + +Below is an example `NnfLustreMGT` resource. The `NnfLustreMGT` resource for external MGSs should be created in the `nnf-system` namespace. + +```yaml +apiVersion: nnf.cray.hpe.com/v1alpha1 +kind: NnfLustreMGT +metadata: + name: external-mgt + namespace: nnf-system +spec: + addresses: + - "1.2.3.4@eth0:1.2.3.5@eth0" + fsNameStart: "aaaaaaaa" + fsNameBlackList: + - "mylustre" + fsNameStartReference: + name: external-mgt + namespace: default + kind: ConfigMap +``` + +* `addresses` - This is a list of LNet addresses that could be used for this MGT. This should match any values that are used in the `externalMgs` field in the `NnfStorageProfiles`. +* `fsNameStart` - The first fsname to use. Subsequent fsnames will be incremented based on this starting fsname (e.g, `aaaaaaaa`, `aaaaaaab`, `aaaaaaac`). fsnames use lowercase letters `'a'`-`'z'`. +* `fsNameBlackList` - This is a list of fsnames that should not be given to any NNF Lustre file systems. If the MGT is hosting any non-NNF Lustre file systems, their fsnames should be included in this blacklist. +* `fsNameStartReference` - This is an optional ObjectReference to a `ConfigMap` that holds a starting fsname. If this field is specified, it takes precedence over the `fsNameStart` field in the spec. The `ConfigMap` will be updated to the next available fsname everytime an fsname is assigned to a new Lustre file system. + +### ConfigMap + +For external MGTs, the `fsNameStartReference` should be used to point to a `ConfigMap` in the default namespace. The `ConfigMap` should not be removed during an argocd undeploy/deploy. This allows the nnf-sos sofware to be undeployed (including any `NnfLustreMGT` resources), without having the fsname reset back to the `fsNameStart` value on a redeploy. The Configmap that is created should be left empty initially. + +### Argocd + +* An empty ConfigMap should be deployed with the `0-early-config` application. It should be created in the `default` namespace, and it can have any name. +* The argocd application for `0-early-config` should be updated to include the following under `ignoreDifferences`: +```yaml + - kind: ConfigMap + jsonPointers: + - /data +``` +* A yaml file for the `NnfLustreMGT` resource should be deployed with the `2-nnf-sos` application. It should be created in the `nnf-system` namespace, and it can have any name. The `ConfigMap` should be listed in the `fsNameStartReference` field. +* The argocd application for `2-nnf-sos` should be updated to include the following under `ignoreDifferences`: +```yaml + - group: nnf.cray.hpe.com + kind: NnfLustreMGT + jsonPointers: + - /spec/claimList +``` + +A separate `ConfigMap` and `NnfLustreMGT` is needed for every external Lustre MGT. + ## Configuration with Persistent Lustre The MGT from a persistent Lustre file system hosted on the NNF nodes can also be used as the MGT for other NNF Lustre file systems. This configuration has the advantage of not relying on any hardware outside of the cluster. However, there is no high availability, and a single MGT is still shared between all Lustre file systems created on the cluster. From 8961d69079d816b58a56be6855f3ee7347ff07ab Mon Sep 17 00:00:00 2001 From: Matt Richerson Date: Fri, 2 Aug 2024 14:54:18 -0500 Subject: [PATCH 2/3] review comments Signed-off-by: Matt Richerson --- docs/guides/external-mgs/readme.md | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-) diff --git a/docs/guides/external-mgs/readme.md b/docs/guides/external-mgs/readme.md index c058c0d..42e41bb 100644 --- a/docs/guides/external-mgs/readme.md +++ b/docs/guides/external-mgs/readme.md @@ -38,7 +38,8 @@ data: ``` ### NnfLustreMGT -A `NnfLustreMGT` resource tracks which fsnames have been used on the MGT to prevent fsname re-use. Any Lustre file systems that are created through the NNF software will request an fsname to use from a `NnfLustreMGT` resource. Every MGT must have a corresponding `NnfLustreMGT` resource. For MGTs that are hosted on NNF hardware, the `NnfLustreMGT` resources are created automatically. The NNF software also erases any no longer used fsnames from disk for any internally hosted MGTs. For an MGT hosted on an external node, an admin must create an `NnfLustreMGT`. This resource ensures that fsnames will be created in a sequential order without any fsname re-use. However, after an fsname is no longer in use by a file system, it will not be erased from the MGT disk. An admin may decide to periodically run the `lctl erase_lcfg [fsname]` command to remove fsnames that are no longer in use. + +A `NnfLustreMGT` resource tracks which fsnames have been used on the MGT to prevent fsname re-use. Any Lustre file systems that are created through the NNF software will request an fsname to use from a `NnfLustreMGT` resource. Every MGT must have a corresponding `NnfLustreMGT` resource. For MGTs that are hosted on NNF hardware, the `NnfLustreMGT` resources are created automatically. The NNF software also erases any unused fsnames from the MGT disk for any internally hosted MGTs. For a MGT hosted on an external node, an admin must create an `NnfLustreMGT` resource. This resource ensures that fsnames will be created in a sequential order without any fsname re-use. However, after an fsname is no longer in use by a file system, it will not be erased from the MGT disk. An admin may decide to periodically run the `lctl erase_lcfg [fsname]` command to remove fsnames that are no longer in use. Below is an example `NnfLustreMGT` resource. The `NnfLustreMGT` resource for external MGSs should be created in the `nnf-system` namespace. @@ -61,9 +62,9 @@ spec: ``` * `addresses` - This is a list of LNet addresses that could be used for this MGT. This should match any values that are used in the `externalMgs` field in the `NnfStorageProfiles`. -* `fsNameStart` - The first fsname to use. Subsequent fsnames will be incremented based on this starting fsname (e.g, `aaaaaaaa`, `aaaaaaab`, `aaaaaaac`). fsnames use lowercase letters `'a'`-`'z'`. +* `fsNameStart` - The first fsname to use. Subsequent fsnames will be incremented based on this starting fsname (e.g, `aaaaaaaa`, `aaaaaaab`, `aaaaaaac`). fsnames use lowercase letters `'a'`-`'z'`. `fsNameStart` should be exactly 8 characters long. * `fsNameBlackList` - This is a list of fsnames that should not be given to any NNF Lustre file systems. If the MGT is hosting any non-NNF Lustre file systems, their fsnames should be included in this blacklist. -* `fsNameStartReference` - This is an optional ObjectReference to a `ConfigMap` that holds a starting fsname. If this field is specified, it takes precedence over the `fsNameStart` field in the spec. The `ConfigMap` will be updated to the next available fsname everytime an fsname is assigned to a new Lustre file system. +* `fsNameStartReference` - This is an optional `ObjectReference` to a `ConfigMap` that holds a starting fsname. If this field is specified, it takes precedence over the `fsNameStart` field in the spec. The `ConfigMap` will be updated to the next available fsname everytime an fsname is assigned to a new Lustre file system. ### ConfigMap From d6851627e0cbff6dca82568d6dffd78e9a14f347 Mon Sep 17 00:00:00 2001 From: Matt Richerson Date: Tue, 20 Aug 2024 11:51:38 -0500 Subject: [PATCH 3/3] review comments Signed-off-by: Matt Richerson --- docs/guides/external-mgs/readme.md | 30 +++++++----------------------- 1 file changed, 7 insertions(+), 23 deletions(-) diff --git a/docs/guides/external-mgs/readme.md b/docs/guides/external-mgs/readme.md index 42e41bb..b8e0074 100644 --- a/docs/guides/external-mgs/readme.md +++ b/docs/guides/external-mgs/readme.md @@ -39,9 +39,11 @@ data: ### NnfLustreMGT -A `NnfLustreMGT` resource tracks which fsnames have been used on the MGT to prevent fsname re-use. Any Lustre file systems that are created through the NNF software will request an fsname to use from a `NnfLustreMGT` resource. Every MGT must have a corresponding `NnfLustreMGT` resource. For MGTs that are hosted on NNF hardware, the `NnfLustreMGT` resources are created automatically. The NNF software also erases any unused fsnames from the MGT disk for any internally hosted MGTs. For a MGT hosted on an external node, an admin must create an `NnfLustreMGT` resource. This resource ensures that fsnames will be created in a sequential order without any fsname re-use. However, after an fsname is no longer in use by a file system, it will not be erased from the MGT disk. An admin may decide to periodically run the `lctl erase_lcfg [fsname]` command to remove fsnames that are no longer in use. +A `NnfLustreMGT` resource tracks which fsnames have been used on the MGT to prevent fsname re-use. Any Lustre file systems that are created through the NNF software will request an fsname to use from a `NnfLustreMGT` resource. Every MGT must have a corresponding `NnfLustreMGT` resource. For MGTs that are hosted on NNF hardware, the `NnfLustreMGT` resources are created automatically. The NNF software also erases any unused fsnames from the MGT disk for any internally hosted MGTs. -Below is an example `NnfLustreMGT` resource. The `NnfLustreMGT` resource for external MGSs should be created in the `nnf-system` namespace. +For a MGT hosted on an external node, an admin must create an `NnfLustreMGT` resource. This resource ensures that fsnames will be created in a sequential order without any fsname re-use. However, after an fsname is no longer in use by a file system, it will not be erased from the MGT disk. An admin may decide to periodically run the `lctl erase_lcfg [fsname]` command to remove fsnames that are no longer in use. + +Below is an example `NnfLustreMGT` resource. The `NnfLustreMGT` resource for external MGSs must be created in the `nnf-system` namespace. ```yaml apiVersion: nnf.cray.hpe.com/v1alpha1 @@ -64,31 +66,13 @@ spec: * `addresses` - This is a list of LNet addresses that could be used for this MGT. This should match any values that are used in the `externalMgs` field in the `NnfStorageProfiles`. * `fsNameStart` - The first fsname to use. Subsequent fsnames will be incremented based on this starting fsname (e.g, `aaaaaaaa`, `aaaaaaab`, `aaaaaaac`). fsnames use lowercase letters `'a'`-`'z'`. `fsNameStart` should be exactly 8 characters long. * `fsNameBlackList` - This is a list of fsnames that should not be given to any NNF Lustre file systems. If the MGT is hosting any non-NNF Lustre file systems, their fsnames should be included in this blacklist. -* `fsNameStartReference` - This is an optional `ObjectReference` to a `ConfigMap` that holds a starting fsname. If this field is specified, it takes precedence over the `fsNameStart` field in the spec. The `ConfigMap` will be updated to the next available fsname everytime an fsname is assigned to a new Lustre file system. +* `fsNameStartReference` - This is an optional `ObjectReference` to a `ConfigMap` that holds a starting fsname. If this field is specified, it takes precedence over the `fsNameStart` field in the spec. The `ConfigMap` will be updated to the next available fsname every time an fsname is assigned to a new Lustre file system. ### ConfigMap -For external MGTs, the `fsNameStartReference` should be used to point to a `ConfigMap` in the default namespace. The `ConfigMap` should not be removed during an argocd undeploy/deploy. This allows the nnf-sos sofware to be undeployed (including any `NnfLustreMGT` resources), without having the fsname reset back to the `fsNameStart` value on a redeploy. The Configmap that is created should be left empty initially. - -### Argocd - -* An empty ConfigMap should be deployed with the `0-early-config` application. It should be created in the `default` namespace, and it can have any name. -* The argocd application for `0-early-config` should be updated to include the following under `ignoreDifferences`: -```yaml - - kind: ConfigMap - jsonPointers: - - /data -``` -* A yaml file for the `NnfLustreMGT` resource should be deployed with the `2-nnf-sos` application. It should be created in the `nnf-system` namespace, and it can have any name. The `ConfigMap` should be listed in the `fsNameStartReference` field. -* The argocd application for `2-nnf-sos` should be updated to include the following under `ignoreDifferences`: -```yaml - - group: nnf.cray.hpe.com - kind: NnfLustreMGT - jsonPointers: - - /spec/claimList -``` +For external MGTs, the `fsNameStartReference` should be used to point to a `ConfigMap` in the `default` namespace. The `ConfigMap` should be left empty initially. The `ConfigMap` is used to hold the value of the next available fsname, and it should not be deleted or modified while a `NnfLustreMGT` resource is referencing it. Removing the `ConfigMap` will cause the Rabbit software to lose track of which fsnames have already been used on the MGT. This is undesireable unless the external MGT is no longer being used by Rabbit software or if an admin has erased all previously used fsnames with the `lctl erase_lcfg [fsname]` command. -A separate `ConfigMap` and `NnfLustreMGT` is needed for every external Lustre MGT. +When using the `ConfigMap`, the nnf-sos software may be undeployed and redeployed without losing track of the next fsname value. During an undeploy, the `NnfLustreMGT` resource will be removed. During a deploy, the `NnfLustreMGT` resource will read the fsname value from the `ConfigMap` if it is present. The value in the `ConfigMap` will override the fsname in the `fsNameStart` field. ## Configuration with Persistent Lustre