Issue while testing most recent version (Directory creation step fails for some users)

[stvoutsin]

Tried testing a branch that is based on the most recent in main, and the test user imports seemed to fail at the Ceph / ssh creation steps

Some of the errors from the import script logs:

"Error mounting /user/Reyesfan: 2024-02-05 15:29:16.108 7f0bf8641c00 -1 auth: error parsing file /etc/ceph/ceph.client.iris-gaia-red-user-Reyesfan-rw.keyring: buffer::malformed_input: error setting modifier for [client.iris-gaia-red-user-Reyesfan-rw] type=key val=null\n2024-02-05 15:29:16.108 7f0bf8641c00 -1 auth: failed to load /etc/ceph/ceph.client.iris-gaia-red-user-Reyesfan-rw.keyring: (5) Input/output error\nmount error 22 = Invalid argument\n"

"debug": {
    "script": "create-ceph-share.sh",
    "result": "FAIL",
    "messages": ["PASS: Share [iris-gaia-red-user-Drizzbinson] created [27c99972-fad3-4399-aae3-81c462ab7029][creating]","PASS: Share [iris-gaia-red-user-Drizzbinson][27c99972-fad3-4399-aae3-81c462ab7029] status [creating]","PASS: Share [iris-gaia-red-user-Drizzbinson][27c99972-fad3-4399-aae3-81c462ab7029] status [available]","PASS: Share [iris-gaia-red-user-Drizzbinson][27c99972-fad3-4399-aae3-81c462ab7029] [ro] access created","PASS: Share [iris-gaia-red-user-Drizzbinson][27c99972-fad3-4399-aae3-81c462ab7029] [rw] access created","FAIL: Ansible mount playbook failed"]
    }

"messages": ["PASS: adduser [Surli] done","FAIL: mkdir [/home/Surli/.ssh] failed","mkdir: cannot create directory ‘/home/Surli/.ssh’: No such file or directory","PASS: updated public keys for [Surli] (sed)","FAIL: chown [/home/Surli] failed","chown: cannot access '/home/Surli': No such file or directory","FAIL: chmod [/home/Surli] failed","chmod: cannot access '/home/Surli': No such file or directory","FAIL: chown [/home/Surli/.ssh] failed","chown: cannot access '/home/Surli/.ssh': No such file or directory","FAIL: chmod [/home/Surli/.ssh] failed","chmod: cannot access '/home/Surli/.ssh': No such file or directory"]
}

Looks like the linux user directory step, and the Ceph user directory creation step failed for some of the users:

ls -al /user/
total 8
drwxr-xr-x.  6 root root 4096 Feb  5 15:37 .
dr-xr-xr-x. 21 root root 4096 Feb  5 15:32 ..
drwxrwxrwx.  2 root root    0 Feb  5 15:32 Florelight
drwxrwxrwx.  2 root root    0 Feb  5 15:32 Nelia
drwxrwxrwx.  2 root root    0 Feb  5 15:36 Surli
drwxrwxrwx.  2 root root    0 Feb  5 15:35 Thozzt

ls -al /home/
total 16
drwxr-xr-x.  7 root        root          4096 Feb  5 15:37 .
dr-xr-xr-x. 21 root        root          4096 Feb  5 15:32 ..
drwxr-x---.  3 Drizzbinson Drizzbinson      1 Feb  5 15:37 Drizzbinson
drwxr-x---.  3 Evison      Evison           1 Feb  5 15:30 Evison
drwxr-x---.  3 Florelight  Florelight       1 Feb  5 15:32 Florelight
drwxr-x---.  3 Reyesfan    Reyesfan         1 Feb  5 15:29 Reyesfan
drwxr-x---.  9 fedora      zeppelinusers 4096 Feb  5 15:49 fedora

[Zarquan]

Probably caused by this change I made yesterday.
#1273

I relocated the create-user-tools and create-ceph-share scripts.
Look for a file not found earlier in the sequence ?

[Zarquan]

Nope, thinking about it. If it was because I moved the scripts, it would fail for all of them, not for some.
Probably not this then.

[Zarquan]

Check to see if the shares were created in Openstack, and it was just the mounts that failed, or did the Manila openstack share create command fail ?

[stvoutsin]

Seems like the Ceph shares are created (they are listed in Openstack), but the mount step fails.
Also note that for some of the test users, the linux home directory create steps fails instead.

[stvoutsin]

Just created two new test users on the same system, and both were successful (all directories created fine).
So seems like an intermittent issue. I'll try a fresh deploy again tomorrow