Multiple Directory Search Path, Combining Multiple Pillar Git Repos

[nickgarber]

With Pillarstack, files are only searched relative to the directory of the stack.cfg file.

GitStack now supports defining multiple git pillar repos as pillarstack targets, but they are processed sequentially, as distinct invocation/passes.

This feature requests the option to include other pillar directories into search path used in a single stack.cfg pass.

I'd like to be able to break pillar configuration into multiple git repos that overlay on each other into a single tree that could be passed to pillarstack.

This would allow files in different directory trees to be processed by precedence within the same pass.

This would allow for greater modularity, which would support greater re-use.

[mrichar1]

I was just coming here to raise this same issue.

We operate a delegated setup where we do this using the standard git ext_pillar, ansd use the mountpoint option to merge other repos in at specific directories. it would be useful to have a single 'top' stack repo which contained stack.cfg, and then multiple other repos that contained the yaml files. It would look something like:

stack.cfg:

# sourced from the 'top' repo
defaults.yml
# sourced from the relevant dept repo, mounted in a subdir
{{ __grains__['dept'] }}/defaults.yml 
{{ __grains__['dept'] }}/nodes/{{ minion_id }}.yml

top.git

defaults.yml

finance.git

defaults.yml
nodes/finance-server.yml

admin.git

defaults.yml
nodes/admin-server.yml

[nickgarber]

Hi @amendlik, thanks for the project!

What do you think of this feature request?

This would enable a breakthrough in our ability to modularize and publish much more of our work, which is something I'm keen to do.

If you're not opposed to the idea, would you accept a pull request?

[amendlik]

@nickgarber: I like the idea very much and would be happy to take a PR for this. Before you put too much time into it, can you propose a configuration format (preferably one that is backwards compatible)? How do we know which repo is the "top" repo?

[nickgarber]

Sweet!

Here's what comes to mind for me...

ext_pillar:
  - gitstack: 
    - master https://github.com/org/myrepo.git:
      ## perhaps some config to control overlay file merge handling, (not sure about these yet)
      #- permit_submount_to_shadow_merge: false
      #- permit_using_stack_cfg: false
      ## these configs would remain unchanged
      - stack:
          pillar:environment:
            dev: _stack/stack.cfg
            prod: _stack/stack.cfg
          grains:custom:grain:
            value:
              - _stack/stack1.cfg
              - _stack/stack2.cfg
          opts:custom:opt:
            value: _stack/stack0.cfg
      ## 'submount' is the first term that comes to mind, but maybe there's something better?
      - submounts:
        - master https://github.com/org/myrepo2.git:
          - root: pillarstack_config.d
          - mountpoint: org2
        - master https://github.com/org/myrepo3.git
          ## 'root' would default to the top directory of the repo
          ## 'mointpoint' would default to the top level of the merge directory
  - git: 
    - master https://github.com/org/myrepo.git
    - master https://github.com/org/myrepo2.git
    - master https://github.com/org/myrepo3.git

[amendlik]

@nickgarber: This all looks very good to me

[nickgarber]

As I look into this my idea of the ideal approach has changed a bit so wanted to check in.

Originally I thought of using symlinks in the checkout at the git_pillar cache, though quickly realized that would be impractical and barbaric.

Next I thought that perhaps the right approach would be to have gitstack perform a "first-found wins" ordered search through candidate directories and only pass the paths that it new to exist. This still seems feasible, but there's one final method thats caught my attention that I'm learning about now...

The mountpoint option is mentioned in the https://github.com/saltstack/salt/blob/develop/salt/pillar/git_pillar.py file, so I continued reading in https://github.com/saltstack/salt/blob/develop/salt/utils/gitfs.py, (the latter is imported from the former), and where mountpoint is frequently mentioned.

It was in the salt/utils/gitfs.py file that I saw linkdirs mentioned, so I'm wrapping my head around that concept in order to understand if it would be a more full featured solution. Also reading through the info here, https://docs.saltstack.com/en/latest/ref/pillar/all/salt.pillar.git_pillar.html#how-multiple-remotes-are-handled .

Based on what I come away understanding about creating and using the features provided in salt/utils/gitfs.py from within salt/pillar/gitstack.py, I may favor option 2 (with a single unified view) or options 3 (with greater re-use of existing feature code).

It feels like option 3 is the more promising way to go.

Thoughts?
Cheers :)

[nickgarber]

I've struggled to make progress on this issue. I'm not giving up, but want to communicate that I'm open to assistance and guidance.