Gluster dispersed deployement and things to take care about.

[DrGamepad]

I was thinking about deploying a HA cluster myself and stumbled across gluster during a GNU/Linux talk about a year ago. Today, I have a setup of three Odroid HC4 and 6x4TB HDDs.
My idea was to setup gluster in dispersed variant with two bricks per node, one brick per HDD. The exact command I used was:

gluster volume create gv0 disperse 6 redundancy 1 gluster{1..3}:/export/sda1/brick/ gluster{1..3}:/export/sdb1/brick/, which resulted in the following information:

volume create: gv0: failed: Multiple bricks of a disperse volume are present on the same server. This setup is not optimal. Bricks should be on different nodes to have best fault tolerant configuration. Use 'force' at the end of the command if you want to override this behavior.

With force the volume was created and I got a nice and fully operational result.

root@gluster1:~# gluster volume status
Status of volume: gv0
Gluster process                             TCP Port  RDMA Port  Online  Pid
------------------------------------------------------------------------------
Brick gluster1:/export/sda1/brick           49152     0          Y       12873
Brick gluster2:/export/sda1/brick           49152     0          Y       56062
Brick gluster3:/export/sda1/brick           49152     0          Y       55121
Brick gluster1:/export/sdb1/brick           49153     0          Y       12890
Brick gluster2:/export/sdb1/brick           49153     0          Y       56079
Brick gluster3:/export/sdb1/brick           49153     0          Y       55138
Self-heal Daemon on localhost               N/A       N/A        Y       12907
Self-heal Daemon on gluster3                N/A       N/A        Y       55155
Self-heal Daemon on gluster2                N/A       N/A        Y       56096

Task Status of Volume gv0
------------------------------------------------------------------------------
There are no active volume tasks

root@server:~# df -h /gluster
Dateisystem    Größe Benutzt Verf. Verw% Eingehängt auf
gluster2:/gv0    19T    368G   18T    2% /gluster

My OS died, due to faulty segments on the microSD card and one whole node needed to be replaced and that is where my trouble began and ended up in deploying a whole new setup...

My questions so far:

• Even though it says that this setup is not optimal, it is functional (I guess?). What are the flaws and drawbacks I will probably run into, if I decide to go on with this setup?
• What is the correct way to replace a node, if it fails again and I don't want to start all over again?
• If this setup is a major example on "DYI done wrong!", what are the alternatives with this hardware setup?

I will thankfully take any kind of expertise, advice, kind shake awakes and guesses on how to improve my knowledge so far.

[edrock200]

I'm no expert but in my research and setup I ran into the same issue you did trying to create my dispersed set. The issue with disperse 6 replicate 1 is you are only tolerant of one brick failure. With 3 nodes, if any one node goes down you will lose 2 bricks, meaning the entire volume would go down. It would probably let you do what your trying to do if you did dispurse 3 replicate 1, meaning 2 bricks of redundancy and it would create 2 subvolumes, but would present as one volume. In this setup, the one failed node would not have taken down your volume. You could try dispurse 6 replicate 2 and make it one volume but I think it will still bark about not being optimal.