Relationship betw cray.nnf.node.drain taint and Storage resource.

[roehrich-hpe]

No description provided.

[roehrich-hpe]

See NearNodeFlash/nnf-sos#341

[jameshcorbett]

🎉

[jameshcorbett]

After reading this, my understanding is that disabling and draining a rabbit node is the same from Flux's perspective, because applying the taint causes the storage to go to .status.state == Disabled, which is the only thing Flux checks. Can you provide some guidance on when you would want to drain vs disable?

[roehrich-hpe]

After reading this, my understanding is that disabling and draining a rabbit node is the same from Flux's perspective, because applying the taint causes the storage to go to .status.state == Disabled, which is the only thing Flux checks. Can you provide some guidance on when you would want to drain vs disable?

Today I changed the PR so that status.state==Drained when the drain taint is applied.

The update to the doc today begins by describing how to disable a node, by setting spec.state=Disabled. It explains that any jobs currently active will be unaffected, but that the WLM won't schedule more jobs on it. This "Disabled" mechanism is likely to be the most-often-used action.

Then it goes into the drain part, describing this drain taint. This would be used only after you have removed the jobs from that rabbit (preferably) and have some reason to also remove the NNF software from it. I'd like to have this happen before a rabbit is powered off and pulled out of the cabinet, but we haven't been doing this and the cluster seems to be doing fine anyway, other than that it leaves the "Terminating" pods behind (harmless, annoying noise).

If an admin used this taint before power-off it would mean you won't have "Terminating" pods laying around for that rabbit. And after a new/same rabbit is put back in its place, the NNF software wouldn't jump back on it unless the taint has been removed. The taint can be removed at any time, from immediately after the node is powered off up to some time after the new/same rabbit is powered back on.

@behlendorf, ^

[jameshcorbett]

This would be used only after you have removed the jobs from that rabbit (preferably) and have some reason to also remove the NNF software from it. I'd like to have this happen before a rabbit is powered off and pulled out of the cabinet, but we haven't been doing this and the cluster seems to be doing fine anyway, other than that it leaves the "Terminating" pods behind (harmless, annoying noise).

If an admin used this taint before power-off it would mean you won't have "Terminating" pods laying around for that rabbit. And after a new/same rabbit is put back in its place, the NNF software wouldn't jump back on it unless the taint has been removed. The taint can be removed at any time, from immediately after the node is powered off up to some time after the new/same rabbit is powered back on.

I wonder if you might add this to the doc? It seems very helpful.