Gitman ignores groups on nested gitman install runs #222
Comments
|
Thanks for the report!
By this do you mean that |
|
Yes - sure! Here it is (some output is obfuscated): The config looks like this: location: .gitman
sources:
- name: module-production
repo: [email protected]:username/gitman-subsub.git
rev: 2.0.0
link: modules/module0
- name: module-sandbox
repo: [email protected]:username/gitman-subsub.git
rev: master
link: modules/module0
groups:
- name: production
members:
- module-production
- name: sandbox
members:
- module-sandbox
|
|
Does |
|
Yes, sure: location: .gitman
sources:
# Production sources
- name: module-production
link: modules/module1
repo: [email protected]:username/gitman-submodule.git
rev: 1.0.0
# Dev sources
- name: module-sandbox
link: modules/module1
repo: [email protected]:username/gitman-submodule.git
rev: master
groups:
- name: production
members:
- module-production
- name: sandbox
members:
- module-sandbox |
|
Thanks, that's helpful! You're correct that groups are only processed once at the top level, so I believe this issue is a request to pass that information to each nested level. FWIW, I'm not sure if installing different versions of the same dependency is the intended use case for groups. You should be able to accomplish what you're trying to do with locked sources: https://gitman.readthedocs.io/en/latest/use-cases/branch-tracking/ where your "dev" setup is |
|
Hi Jace
Thank you for your answer. You’re right that I can used locked_sources for that simple example. Truth is that I have 3 environments - production, preprod and prod - which requires the group functionality, together with different named versions of the same sources. You could probably argue that I could/should use locked_sources for both preprod and prod, but it would be nice to differentiate those two, until I feel confident that the version is ready for production.
Do you have any idea if it can be implemented..? I took a look at the code yesterday, but I can’t seem to find where it descends into the sub repo and calls itself again..
(As a side note: I use this for a Infrastructure As Code repo, where the sources that Gilman is managing, is the terraform modules and sub modules used for managing the infrastructure. So I can have situations where I’m doing a new release of the infrastructure, but still need to be able to ensure that the production environment is running the current version (be able to run terraform plan without seeing any changes). Unfortunately, our in-house tool is having the same exact ‘bug’ (not honoring the groups in the sub repos) :-( - together with other bugs ).
…Sent from my iPad
On 20 Aug 2020, at 16.46, Jace Browning ***@***.***> wrote:
Thanks, that's helpful! You're correct that groups are only processed once at the top level, so I believe this issue is a request to pass that information to each nested level.
FWIW, I'm not sure if installing different versions of the same dependency is the intended use case for groups.
You should be able to accomplish what you're trying to do with locked sources: https://gitman.readthedocs.io/en/latest/use-cases/branch-tracking/
location: .gitman
sources:
- name: module-production
repo: ***@***.***:username/gitman-subsub.git
rev: master
link: modules/module0
sources_locked:
- name: module-production
repo: ***@***.***:username/gitman-subsub.git
rev: 2.0.0
link: modules/module0
where your "dev" setup is gitman update --skip-lock and your "production" setup is gitman install.
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub, or unsubscribe.
|
|
This is the spot where gitman/gitman/models/config.py Line 87 in 5acc0e4 This is the spot where we recurse into nested repositories: gitman/gitman/models/config.py Line 117 in 5acc0e4 You'll notice that Another thing to consider is how to handle nested |
|
Hi Jace
Thank you for the pointers - I’ll have a look at it.
Gitman files without groups: Maybe just treat that as a normal gitman install, and install the repos listed under sources...?
… On 21 Aug 2020, at 16.55, Jace Browning ***@***.***> wrote:
This is the spot where "production" is translated to ["tf-module-production"]: https://github.com/jacebrowning/gitman/blob/5acc0e4b1d5c15683213563901acd0349a1fa155/gitman/models/config.py#L87
This is the spot where we recurse into nested repositories: https://github.com/jacebrowning/gitman/blob/5acc0e4b1d5c15683213563901acd0349a1fa155/gitman/models/config.py#L117
You'll notice that names are not forwarded to recursive installation calls. To achieve the behavior you're looking for, we would need to determine that "production" is the name of a group (and not just the name of a single source), then forward that as *names to all recursive calls. In hindsight, having a separate --groups argument may have been a better design to keep these lists of names separate.
Another thing to consider is how to handle nested gitman configs that don't have groups defined, but are passed a group name.
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub, or unsubscribe.
|
|
We have a similar requirement solved by using seperate branches in the top level repository (like dev and prod) to maintain seperate gitman configs because of our many repositories and parallel dev enviroments. Switching between environments is just accomblished by switching between branches. Furthermore we realized that the recursive dependency resolution increases the complexity of the config management at a certain amount of nested dependency levels. We have a very complex dependency graph with more than 80 repositories. We resolve all product related dependencies over a global gitman config. Each developer and also our build system needs to use this product repository to build all or parts of the product. This helps to avoid workspace clashes. In a release phase we create a new release branch in this toplevel repository and pining each of the dependencies one by one. Maybe this approach is also helpful for your application. |
|
@daniel-brosche Thank you for your input - although I have a hard time figuring out what you are doing. Can you maybe paint the picture a little bit clearer with a simple example..? |
|
Maybe this helps, I have tried to illustrate it with an abstract example. The product repository here is Mostly we continously build the mainline (master) of all repositories during the common development phase.
In this global gitman.yml there are groups defined which represents sub-workspaces for the main developer domains. In the release phase I create a release branch in the product repository and reference tags over time. At the end of stabilization only tags are referenced in this product repository release branch.
For very complex features I mix these approaches and create a distinct development environment that reference feature branches and tags.
Over this way, I can ensure a stable feature environment where only the related feature repositories are mutable over time (e.g. 75% of all repositories are immutable and 25% are mutable). This is also very helpful to exchange feature related workspaces between developers and buildsystem to build reproducible feature related builds. Some parts of this worklow can also be achieved by locking sources but in my experience this approach is more explicit and pretty easy in terms of config management. |
|
This is almost how I do it. I have a directory structure like this one:
I was hoping that groups could do that for me, and it works fine in the top level. But when fx If gitman supported sending the groupname to the subsequent Unfortunately I'm not a python programmer, so I'm not exactly sure how to solve this in the code, but I'll try to take a stab at it. I might end up using a dedicated wrapper tool like |
|
Just a quick thought: Would it be possible/nice to structure the config as follows, instead of the separate groups section?: location: .gitman
sources:
default:
# Production sources
- name: tf-module-production
link: tf-module
repo: git@githost:tf-module.git
rev: 1.2.0
dev:
# Dev sources
- name: tf-module-dev
link: tf-module
repo: git@githost:tf-module.git
rev: masterThen always have a default "group" or section, which get's installed when just running |
According to my described approach, there is no nested gitman.yml and then I would arrange it like this: In the master Branch of tv-life: location: .gitman
sources:
- name: module0
repo: [email protected]:username/gitman-subsub.git
rev: master
link: modules/module0
- name: module1
link: modules/module1
repo: [email protected]:username/gitman-submodule.git
rev: masterIn the branch v1.0.0 location: .gitman
sources:
- name: module0
repo: [email protected]:username/gitman-subsub.git
rev: 2.0.0
link: modules/module0
- name: module1
link: modules/module1
repo: [email protected]:username/gitman-submodule.git
rev: 1.0.0You could also change the gitman location to I do import everything in defined |
|
I've come up with an alternative approach that I like a bit better. First we add a gitman config file for each environment: gitman-production.yml Then a simple Makefile: .PHONY: install clean uninstall
all: .gitman.yml
@echo "Preparing for deploying to ${ENV}..."
gitman install
.gitman.yml::
@ln -sf gitman-${ENV}.yml .gitman.yml
clean:
gitman uninstall --force
-rm -f .gitman.yml
uninstall:
gitman uninstall
-rm -f .gitman.ymlAll it does is symlinking the gitman-${ENV}.yml file to .gitman.yml and run gitman install (besides the clean and uninstall targets) In all subrepos we add a similar structure (if it has dependencies on anything, of course): gitman-production.yml Each gitman-env.yml file looks like this (with multiple different repos in different revisions of course): location: .gitman
sources:
- name: testhest
repo: [email protected]:username/gitman-test.git
rev: master
link: testhest
scripts:
- ENV=${ENV} makeAs you can see it runs make in the end, with the $ENV var preprended. This makes the sub repos’ git install command use the correct gitman config file. The .gitman.yml (which is just a symlink) should be ignored in .gitignore. So to sum it up: With a setup like this, we have a workflow looking like this (deploying to production):
|
We have just decided to switch from an inhouse developed, almost identical tool, to gitman. But I've stumbled across what seems to be a bug, during my initial testing:
Given a gitman.yml file that looks like this:
I expect that I end up with a directory looking like this, if I do a
gitman install dev:And if I do a
gitman install production:This works - so far so good.
But if my tf-module contains a gitman.yml file with a similar structure and I run
gitman install production, gitman actually installs all the sources, linking for each source, and it seems like I end up with the dev sources (so I end up with the master branches). I guess that gitman doesn't honour the groupname in the subsequent gitman install calls...?The text was updated successfully, but these errors were encountered: