[pull] main from bacalhau-project:main #5
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
![pull[bot]](https://avatars.githubusercontent.com/in/12910?s=60&v=4){kind=small}
This PR switches staging to use NATS transport layer instead of Libp2p. It has been baking for couple of days with no issues reported by the canaries. Keep in mind that this is working with no additional changes related to auth because of the bug reported [here](https://github.com/bacalhau-project/expanso-planning/issues/518), where requester nodes can join a network even without auth keys. When we fix that issue, we will need to pre-provision the auth key instead of letting the requester node auto-generate it, or reuse the terraform modules using for marketplace Closes bacalhau-project/expanso-planning#521 <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit - **Chores** - Updated stage environment configuration to support a new network type. <!-- end of auto-generated comment: release notes by coderabbit.ai -->
This opens up the necessary firewall port to allow in requests for local publisher content at each compute node, for the demo network. It also tells the requester node to default to the local publisher if the incoming job does not specify a publisher to use.
When exiting devstack, there are often errors logged that are unnecessary, or in some cases delay when the process actuall exits. This commit checks some obvious context cancellation errors and gives the code a chance to exit without logging and continuing. Should also fix #3484
### Summary This PR bumps the repo version from v2 to v3, where we introduce the following changes: 1. Change the execution store and job store default paths to no longer include the node name in the path. To clarify, v2 paths look like `~/.bacalhau/QmUBgU7xHKK44RuTHgrvnJfoSdZJS4fddT197iyTF5qjEV-compute/executions.db` and `QmUBgU7xHKK44RuTHgrvnJfoSdZJS4fddT197iyTF5qjEV-requester/jobs.db`, whereas in v3 they are `~/.baclahau/compute_store/executions.db` and `~/.bacalhau/orchestrator_store/jobs.db`. This change is needed to allow users to change our node names to longer depend on libp2p keys. 1. Persists the execution store and job store paths in `config.yaml` so that users won't lose their state if we change our default path names in the future 2. Adds a new `Node.Name` config, generates a node name still based on libp2p, and persist it. This will simplify an upcoming PR that allows users to define their own node name through the cli 3. Introduce migration scripts that will migrate v2 repos by creating `config.yaml` if it doesn't exist, persist the store paths, and rename old path names. No such migration will happen if the user provided their own `config.yaml` with store paths configured. ### `bacalhau config show` I've also introduced `bacalhau config show` command to print out the current configuration which includes configs in `config.yaml`, env variables and default values. ### Few issues encountered 1. The repo is initialized in the `RootCmd` before any flags are registered. This means only default configurations are used when initializing the repo and any env variables or flags will be ignored. For example, if I run `bacalhau serve --compute-execution-store-path my-awesome-path` for the very first time, then bacalhau will still initialize the repo with default value, create `~/.bacalhau/compute-store` directory, and create `config.yaml` with `~/.bacalhau/compute-store` instead of my provided one. It is difficult to do things differently with our current setup as we initialize the repo in any command, but these flags are only available for `serve` command 2. We use global `viper` instance to set our config. I had to reset the config in my tests before each run. It might be safer to not use the global instance 4. I've tried creating a repo per node in devstack, but couldn't get that to work because of our global configs. Configurations of the first repo initialized are also passed to the second repo's config, and we have test cases that fetch info of the very first repo initialized, such as https://github.com/bacalhau-project/bacalhau/blob/badd1ce5fd9508bd615f9a8e08bd8ce44d2eceb7/pkg/test/devstack/timeout_test.go#L112 <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit - **New Features** - Introduced new files to enhance configuration management, including displaying default and current settings. - Added functionality for handling private keys within the libp2p network. - Implemented a new approach for repository migration, including updating repository versions and handling specific configurations. - Enhanced CLI commands for better configuration setup and display. - **Refactor** - Updated the logic flow in configuration setup, removing unnecessary imports and functions. - Simplified the initialization of repository and node configuration, including passing context parameters where needed. - Adjusted error messaging for clearer debugging. - Streamlined setup and teardown processes in tests. - **Bug Fixes** - Fixed exclusion patterns in pre-commit hooks to correctly ignore specified directories. - **Chores** - Updated imports and removed unused code across multiple files for cleaner codebase management. - **Documentation** - Added summaries and comments for clarity in understanding the purpose and impact of changes. <!-- end of auto-generated comment: release notes by coderabbit.ai -->
Enable users to set their own nodeID. Users can either set their node
name manually, or chose any of the following providers to automatically
set the name:
- `puuid`: (default) generate a node name using `n-{uuid}` pattern, such
as `n-f1bab231-68ad-4c72-bab6-580cd49bf521`
- `uuid`: generate uuid as a node name
- `hostname`: use the hostname as the node id, but replacing any `.`
with `-` to be compatible with nats
- `aws`: use the EC2 instance name, if the node is deployed on aws
- `gcp`: use the VM's id, if the node is deployed on gcp
### Persisted Node Name
These providers will only be called if no existing node name is found in
`config.yaml`, cli `--name` flag or env variables. Once a node name is
generated, it will be persisted in `config.yaml`
### Examples
```
# set the node name manually
bacalhau serve --name my-custom-name
# use a puuid as the node name (default)
bacalhau serve
# use hostname as the node name
bacalhau serve --name-provider hostname
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
- **New Features**
- Added node name configuration options, supporting various providers
like hostname, AWS, GCP, UUID, and PUUID.
- Introduced a new naming convention for nodes and job executions to
enhance readability and uniqueness.
- **Refactor**
- Updated ID generation across the application to use `ShortUUID` and
`ShortNodeID` for job IDs, node IDs, and execution IDs, improving
consistency and clarity in displays and logs.
- **Tests**
- Added comprehensive testing for new node naming strategies and ID
generation methods.
- **Bug Fixes**
- Fixed potential subscription issues in NATS subjects by restricting
characters in node IDs.
- **Documentation**
- Updated internal documentation to reflect changes in ID generation and
node naming conventions.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
Recently the `slices.SortFunc` package was updated, and the signature of SortFunc was changed to expect a cmp() int style return value, rather than a boolean. This PR updates the exp/slices package (and as a result, others) fixing the issue with sorting in various places. In particular `model/network` and `models/network` logic was unclear, and also suffered from the CompactFunc only performing a single run over the domain list. The newly introduced `util/cmp#Compare` type provides a temporary version of the functions that will be available to us in the builtin cmp package, once we upgrade, which is now feasible. Fixes #3498
Bumps [github.com/multiformats/go-multiaddr](https://github.com/multiformats/go-multiaddr) from 0.9.0 to 0.12.2. --- Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Ross Jones <[email protected]>
- previously this field was never set. It's now set based on the evaluationID in the plan when an execution is created.
Bumps [ip](https://github.com/indutny/node-ip) from 2.0.0 to 2.0.1. <details> <summary>Commits</summary> <ul> <li><a href="https://github.com/indutny/node-ip/commit/3b0994a74eca51df01f08c40d6a65ba0e1845d04"><code>3b0994a</code></a> 2.0.1</li> <li><a href="https://github.com/indutny/node-ip/commit/32f468f1245574785ec080705737a579be1223aa"><code>32f468f</code></a> lib: fixed CVE-2023-42282 and added unit test</li> <li>See full diff in <a href="https://github.com/indutny/node-ip/compare/v2.0.0...v2.0.1">compare view</a></li> </ul> </details> <br /> [](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores) Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@dependabot rebase`. [//]: # (dependabot-automerge-start) [//]: # (dependabot-automerge-end) --- <details> <summary>Dependabot commands and options</summary> <br /> You can trigger Dependabot actions by commenting on this PR: - `@dependabot rebase` will rebase this PR - `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it - `@dependabot merge` will merge this PR after your CI passes on it - `@dependabot squash and merge` will squash and merge this PR after your CI passes on it - `@dependabot cancel merge` will cancel a previously requested merge and block automerging - `@dependabot reopen` will reopen this PR if it is closed - `@dependabot close` will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually - `@dependabot show <dependency name> ignore conditions` will show all of the ignore conditions of the specified dependency - `@dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself) You can disable automated security fix PRs for this repo from the [Security Alerts page](https://github.com/bacalhau-project/bacalhau/network/alerts). </details> --------- Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: David Aronchick <[email protected]>
This allows clients to detect when their token is invalid and re-run authentication to get a new one.
This commit allows the Bacalhau CLI to automatically remove tokens that the server says are not valid, e.g. because they have expired or because the server no longer recognises the signing authority. In particular, this process is transparent to the rest of the system - the authentication happens as part of a normal API call, and the API call doesn't complete until the authentication flow is done. Naturally this is an opt-in for anyone using the Bacalhau codebase as a library because this might not be desired. This involved splitting the transport parts of the API out from the domain-specific parts so that the authentication code could monitor for invalid auth tokens and invoke the authentication flow.
We've skipped over this for a while because no single Dependabot upgrade was able to resolve the entire compat issue. We now need to upgrade this to fix this issue: aws/aws-sdk-go-v2#2370
Bumps [cryptography](https://github.com/pyca/cryptography) from 42.0.2 to 42.0.4. <details> <summary>Changelog</summary> <p><em>Sourced from <a href="https://github.com/pyca/cryptography/blob/main/CHANGELOG.rst">cryptography's changelog</a>.</em></p> <blockquote> <p>42.0.4 - 2024-02-20</p> <pre><code> * Fixed a null-pointer-dereference and segfault that could occur when creating a PKCS#12 bundle. Credit to **Alexander-Programming** for reporting the issue. **CVE-2024-26130** * Fixed ASN.1 encoding for PKCS7/SMIME signed messages. The fields ``SMIMECapabilities`` and ``SignatureAlgorithmIdentifier`` should now be correctly encoded according to the definitions in :rfc:`2633` :rfc:`3370`. <p>.. _v42-0-3:</p> <p>42.0.3 - 2024-02-15 </code></pre></p> <ul> <li>Fixed an initialization issue that caused key loading failures for some users.</li> </ul> <p>.. _v42-0-2:</p> </blockquote> </details> <details> <summary>Commits</summary> <ul> <li><a href="https://github.com/pyca/cryptography/commit/fe18470f7d05f963e7267e34fdf985d81ea6ceea"><code>fe18470</code></a> Bump for 42.0.4 release (<a href="https://redirect.github.com/pyca/cryptography/issues/10445">#10445</a>)</li> <li><a href="https://github.com/pyca/cryptography/commit/aaa2dd06ed470695de818405a982d4c459869803"><code>aaa2dd0</code></a> Fix ASN.1 issues in PKCS#7 and S/MIME signing (<a href="https://redirect.github.com/pyca/cryptography/issues/10373">#10373</a>) (<a href="https://redirect.github.com/pyca/cryptography/issues/10442">#10442</a>)</li> <li><a href="https://github.com/pyca/cryptography/commit/7a4d012991061974da5d9cb7614de65eac94f49b"><code>7a4d012</code></a> Fixes <a href="https://redirect.github.com/pyca/cryptography/issues/10422">#10422</a> -- don't crash when a PKCS#12 key and cert don't match (<a href="https://redirect.github.com/pyca/cryptography/issues/10423">#10423</a>) ...</li> <li><a href="https://github.com/pyca/cryptography/commit/df314bb182bdfd661333969a94325e4680d785f6"><code>df314bb</code></a> backport actions m1 switch to 42.0.x (<a href="https://redirect.github.com/pyca/cryptography/issues/10415">#10415</a>)</li> <li><a href="https://github.com/pyca/cryptography/commit/c49a7a5271178c6e8ef36fa1c499f62c63ec19b9"><code>c49a7a5</code></a> changelog and version bump for 42.0.3 (<a href="https://redirect.github.com/pyca/cryptography/issues/10396">#10396</a>)</li> <li><a href="https://github.com/pyca/cryptography/commit/396bcf64c5be826ec00e7d7f45838c858c049cbc"><code>396bcf6</code></a> fix provider loading take two (<a href="https://redirect.github.com/pyca/cryptography/issues/10390">#10390</a>) (<a href="https://redirect.github.com/pyca/cryptography/issues/10395">#10395</a>)</li> <li><a href="https://github.com/pyca/cryptography/commit/0e0e46f5f73f477b8ee9682738c42129d5d60177"><code>0e0e46f</code></a> backport: initialize openssl's legacy provider in rust (<a href="https://redirect.github.com/pyca/cryptography/issues/10323">#10323</a>) (<a href="https://redirect.github.com/pyca/cryptography/issues/10333">#10333</a>)</li> <li>See full diff in <a href="https://github.com/pyca/cryptography/compare/42.0.2...42.0.4">compare view</a></li> </ul> </details> <br /> [](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores) Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@dependabot rebase`. [//]: # (dependabot-automerge-start) [//]: # (dependabot-automerge-end) --- <details> <summary>Dependabot commands and options</summary> <br /> You can trigger Dependabot actions by commenting on this PR: - `@dependabot rebase` will rebase this PR - `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it - `@dependabot merge` will merge this PR after your CI passes on it - `@dependabot squash and merge` will squash and merge this PR after your CI passes on it - `@dependabot cancel merge` will cancel a previously requested merge and block automerging - `@dependabot reopen` will reopen this PR if it is closed - `@dependabot close` will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually - `@dependabot show <dependency name> ignore conditions` will show all of the ignore conditions of the specified dependency - `@dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself) You can disable automated security fix PRs for this repo from the [Security Alerts page](https://github.com/bacalhau-project/bacalhau/network/alerts). </details> --------- Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: David Aronchick <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
See Commits and Changes for more details.
Created by
pull[bot]
Can you help keep this open source service alive? 💖 Please sponsor : )