feat: git Access, AccessMethod through BlobAccess

[jakobmoellerdev]

What this PR does / why we need it:

This is an extremely early (WIP) draft of implementing ctf archives in directory format tracked via git repositories, ultimately serving me as a playground to get familiar with OCM and also to play around with its feature set. So we do not need to merge this if there is no interest or disagreement on the implementation.

This PR has developed into the development of a pure access method and input method that can be used to work with arbitrary git repositories. Examples for a component constructor could be:

name: test.de/x
version: %s
provider:
  name: ocm
resources:
- name: hello-world-access
  type: git
  version: 0.0.1
  access:
    type: git
    commit: "7fd1a60b01f91b314f59955a4e4d4e80d8edf11d"
    ref: refs/heads/master
    repository: https://github.com/octocat/Hello-World.git
- name: hello-world-input
  type: git
  version: 0.0.1
  input:
    type: git
    ref: refs/heads/master
    commit: "7fd1a60b01f91b314f59955a4e4d4e80d8edf11d"
    repository: https://github.com/octocat/Hello-World.git

Not only does this allow using git repositories for storing blobs, it also attempts to add git downloader functionality for a generic git access method based on in memory cloning and I plan to add a OCM implementation on it as well. While the download is available, this is not laid out to the CLI, library only.

Currently the implementation is written so that any modification to the CTF in the work directory also triggers a dedicated commit, and any lookup triggers a refres/pull cycle on git.

For testing I use file:// URLs which makes it quite easy to verify E2E in the test suites.

TODO:

• Implement Access Credential mapping
• Make Git Client thread-safe, optionally in-memory and allow more customization (currently hard bount to one target ref that I am creating or syncing against, see tests for examples)
• Expand Git Options - Authentication support for repo and client

Which issue(s) this PR fixes:

As this is my own "experimentation" issue, it doesnt necessarily fix anything, it might as well be moved to a plugin for example or be scrapped alltogether

Creates an access method for arbitrary git repositories using git-go.

[Skarlso]

Hello.

So this work happens to be implementing this issue open-component-model/ocm-project#239.

I'll assign it to you then.

[mandelsoft]

Hi @jakobmoellerdev, concerning your idea of git backed CTF repositories, if I understand your approach correctly,
every change will create a new commit. I think this approach misses the need for a semantical bracket for a set of changes.
It is more like creating a commit for every changed line in a source file, which does not look like a good idea.
A useful bracket (which is already supported) could be the open/close of a CTF. The typical workflow is to open a CTF, apply some changes (for example with the ocm add cv command) and then close it again. So a more useful point in time to create a new commit should be the Close operation of a CTF.

[jakobmoellerdev]

Hi @mandelsoft, thanks for taking a look.

I think the semantical equivalent to a git push would be to close the repository. In your description that would be

1. Open CTF -> Git Clone / Checkout
2. Apply Changes (e.g. add cv) -> Write/Update Files in CTF, Commit atomically
3. Close CTF -> Git Push

Of course we can change the flow to

1. Open CTF -> Git Clone / Checkout
2. Apply Changes (e.g. add cv) -> Write/Update Files in CTF,
3. Close CTF -> Git Commit all changes, Git Push

If you are concerned with that. In my view they are equivalent and the atomic commits are much more transparent when a change was made. However Im totally fine with your take as well, it shouldnt be too hard to change.

I think we could change it so that there is one commit that has all the current commit messages in the commit body. Then we will have 1 commit. WDYT?

[jakobmoellerdev]

this is ready for a first pass review. We might change the way the commits are generated (see discussion above) but generally i would still like some input as its my first take on a larger change in this codebase

[jakobmoellerdev]

Just a quick update on this PR: I didnt get around to it yet, but I will probably remove the repo code because its unlikely to be used (as it all started for me to get into the OCM codebase). The AccessMethod etc can stay in and are still valid though.

[Skarlso]

I have a few questions, but generally, this is working. :D

[Skarlso]

sap/git-access-test/test.ca on ☁️   [email protected]
❯ l
total 4
drwxr-x--- 3 skarlso staff   96 Dec  6 15:12 blobs/
-rwxr-x--- 1 skarlso staff 1115 Dec  6 15:11 component-descriptor.yaml*
sap/git-access-test/test.ca on ☁️   [email protected]
➜ cd blobs/
git-access-test/test.ca/blobs on ☁️   [email protected]
➜ tar xzf sha256.c3bdcc7317c2afe8a4ef373af21404eece1563db4a51aaab70372455c12c3624
git-access-test/test.ca/blobs on ☁️   [email protected]
➜ cat README
Hello World!
git-access-test/test.ca/blobs on ☁️   [email protected]

[Skarlso]

There are a few more tests to do as well.

[Skarlso]

Amazing job!! 👍

outdated