[pull] main from bacalhau-project:main #8
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
![pull[bot]](https://avatars.githubusercontent.com/in/12910?s=60&v=4){kind=small}
For some reason, `config.yaml` file in staging and demo stack have
`JobStore` and `ExecutorStore` entries, but with empty paths, and look
like:
```
...
Compute:
ExecutionStore:
Type: BoltDB
Path: ""
...
Requester:
JobStore:
Type: BoltDB
Path: ""
...
```
The v2 -> v3 migrator will skip migrating if the store configs are not
empty. This PR fixes the migrator by checking if the path is empty
instead of if the entry is empty
Currently if you want to build the bacalhau binary, you also have to build the webui and plugins as well, even if you've no intention of using them. This change adds: ### make binary Just builds the bacalhau binary (assuming webui is available already for embedding). ### make binary-web Builds the webui ready for embedding in the binary ### make build-bacalhau The existing step now calls binary-web and then binary. ### make build Does the same as always
Implements a NATS KV backed store for Node Information to persist node information across requester node restarts. Currently we only have an in-memory store, and as a result we lose all knowledge of any compute nodes when we restart the requester. The new persistent store is the first step in providing more effective node information and being able to track compute nodes as they become inaccessible to the cluster for short periods of time. The KV Store depends on NATS JetStream and so there have been changes to the set up of the NAT Server, and a first pass at using a test NATS server in Bacalhau tests (cf https://github.com/bacalhau-project/bacalhau/blob/1c1662c595dfacbcbdb124bd4bd4cef3c6bdd6de/pkg/routing/kvstore/kvstore_test.go#L33-L40) TIme metrics and spans are available for both stores (inmemory for libp2p and kvstore for NATS) although it is expected that the inmemory store eventually be removed. These changes did involve modifications to how the network transport performed pubsub for adding newly discovered node, only because the transport depends on the store, but this particular store depends on the transport and so construction is now a two-step process. ## Summary by CodeRabbit - **New Features** - Introduced network storage directory configuration for enhanced data management. - Enhanced node discovery process for improved reliability and efficiency. - Implemented new logging and node information management features in transport layers. - Added key-value store functionality for robust node information storage and retrieval. - Introduced metrics and tracing for node management operations, enhancing monitoring and debugging capabilities. - **Bug Fixes** - Adjusted node discovery logic to ensure accurate node selection and information retrieval. - **Refactor** - Unified node information storage interfaces and methods across different components for consistency and maintainability. - Streamlined configuration and initialization processes for network and node components.
We shouldn't print auth tokens unless absolutely necessary. Now the system will hide the auth token if the user supplied it explicitly. Resolves #3490.
Adds a new node approval field to the node_info structure where we currently track information about nodes in the cluster. The approval field currently defaults to unknown, and is not yet used for anything. This will be used after: * Persistent node store can pass filters to list() * There is a registration endpoint * The CLI can list/approve/reject nodes.
Currently the V1 CLI commands that we want to move to V2 (docker run, wasm run) make use of the `pkg/job` package to build a job ready for submission. The files here make use of pkg/model. To make it clearer what needs to change in the CLI this PR moves `pkg/job` to `pkg/legacyjob`. This leaves pkg/job open to re-implementation following a similar but not identical approach (builders with options). This aims to make it easier to move over the job creation commands _without_ forcing the implementor to change both, and all the tests in the same PR. Now it should be possible for docker run to move over to v2 independently of the wasm run command. ## Summary by CodeRabbit - **Refactor** - Updated various components and tests to use `legacy_job` instead of `jobutils` for job verification and handling, enhancing clarity and consistency across the application. - **Chores** - Renamed the `job` package to `legacy_job` across multiple files to better reflect its purpose and usage. - **Tests** - Adjusted tests to align with the renaming from `job` to `legacy_job`, ensuring consistent test execution and results.
Bumps [github.com/nats-io/nats-server/v2](https://github.com/nats-io/nats-server) from 2.10.7 to 2.10.11. <details> <summary>Release notes</summary> <p><em>Sourced from <a href="https://github.com/nats-io/nats-server/releases">github.com/nats-io/nats-server/v2's releases</a>.</em></p> <blockquote> <h2>Release v2.10.11</h2> <h2>Changelog</h2> <p>Refer to the <a href="https://docs.nats.io/release-notes/whats_new/whats_new_210">2.10 Upgrade Guide</a> for backwards compatibility notes with 2.9.x.</p> <h3>Go Version</h3> <ul> <li>1.21.7</li> </ul> <h3>Dependencies</h3> <ul> <li>github.com/nats-io/nats.go v1.33.0</li> </ul> <h3>Improved</h3> <p>JetStream</p> <ul> <li>Optimize replaying messages when they are at the end of the stream (<a href="https://redirect.github.com/nats-io/nats-server/issues/5083">#5083</a>) Thanks to <a href="https://github.com/david-wakeo"><code>@david-wakeo</code></a> for the report!</li> <li>Optimize replaying messages with start sequence by time when there are large gaps between the time sequence and first sequence (<a href="https://redirect.github.com/nats-io/nats-server/issues/5088">#5088</a>) Thanks again to <a href="https://github.com/david-wakeo"><code>@david-wakeo</code></a> for the report!</li> <li>Reduce possible contention for NRG step-down (<a href="https://redirect.github.com/nats-io/nats-server/issues/4990">#4990</a>)</li> <li>Reduce log-level to debug for non-actionable reallocations warnings (<a href="https://redirect.github.com/nats-io/nats-server/issues/5085">#5085</a>)</li> <li>Improved placement of streams in larger clusters when created in rapid succession (<a href="https://redirect.github.com/nats-io/nats-server/issues/5079">#5079</a>) Thanks to <a href="https://github.com/kohlisid"><code>@kohlisid</code></a> for the report!</li> <li>Improved filtered consumer retrieval time for large subject space streams with wildcards (<a href="https://redirect.github.com/nats-io/nats-server/issues/5089">#5089</a>)</li> </ul> <h3>Fixed</h3> <p>JetStream</p> <ul> <li>Fixed a bug that could cause consumers to not match filtered subjects with wildcards correctly (<a href="https://redirect.github.com/nats-io/nats-server/issues/5080">#5080</a>)</li> <li>Fixed a bug that could not allow keys/subjects to be found after restarts after extended downtime (<a href="https://redirect.github.com/nats-io/nats-server/issues/5054">#5054</a>)</li> <li>Fixed a bug that would not properly update all consumer state when a filtered purge of the parent stream was narrower then the consumer’s filtered subject space (<a href="https://redirect.github.com/nats-io/nats-server/issues/5075">#5075</a>)</li> </ul> <p>MQTT</p> <ul> <li>Fixed an issue with retained messages not working properly if the server name had a ‘.’ (<a href="https://redirect.github.com/nats-io/nats-server/issues/5048">#5048</a>)</li> </ul> <h3>Complete Changes</h3> <p><a href="https://github.com/nats-io/nats-server/compare/v2.10.10...v2.10.11">https://github.com/nats-io/nats-server/compare/v2.10.10...v2.10.11</a></p> <h2>Release v2.10.10</h2> <h2>Changelog</h2> <p>Refer to the <a href="https://docs.nats.io/release-notes/whats_new/whats_new_210">2.10 Upgrade Guide</a> for backwards compatibility notes with 2.9.x.</p> <h3>Go Version</h3> <ul> <li>1.21.6</li> </ul> <h3>Dependencies</h3> <ul> <li>github.com/klauspost/compress v1.17.5</li> <li>github.com/nats-io/nats.go v1.32.0</li> </ul> <h3>Added</h3> <ul> <li>Add <code>ping_interval</code> cluster route option to configure differently than client connections (<a href="https://redirect.github.com/nats-io/nats-server/issues/5029">#5029</a>)</li> </ul> <h3>Improved</h3> <p>JetStream</p> <!-- raw HTML omitted --> </blockquote> <p>... (truncated)</p> </details> <details> <summary>Commits</summary> <ul> <li><a href="https://github.com/nats-io/nats-server/commit/5fc6051c33438553b3dc828ab856915011bd1945"><code>5fc6051</code></a> Release v2.10.11 (<a href="https://redirect.github.com/nats-io/nats-server/issues/5095">#5095</a>)</li> <li><a href="https://github.com/nats-io/nats-server/commit/045b8482fc731e45e36f0db5817a02ce3c13445e"><code>045b848</code></a> Fixed incorrect mqtt-test reference (mqttex tests failure)</li> <li><a href="https://github.com/nats-io/nats-server/commit/66b3722403a720c07b9042428595581f8936d1ae"><code>66b3722</code></a> Bump version to 2.10.11-RC.4 [ci skip]</li> <li><a href="https://github.com/nats-io/nats-server/commit/54eafdfe889b384df0357bc6cc34cf94e3c0b760"><code>54eafdf</code></a> Cherry picks for v2.10.11 release (<a href="https://redirect.github.com/nats-io/nats-server/issues/5090">#5090</a>)</li> <li><a href="https://github.com/nats-io/nats-server/commit/835840bb347bd0e846ffdb0af7d75152b60ad7df"><code>835840b</code></a> When doing firstMatching we were picking wrong on linearScans. Also did not o...</li> <li><a href="https://github.com/nats-io/nats-server/commit/78875579d5f43fa2e6946f6d955d7d1b0d6a9a3e"><code>7887557</code></a> When a filtered consumer starts by time, there could be a large gap between t...</li> <li><a href="https://github.com/nats-io/nats-server/commit/cf1b8e4ca163e7f13541b9dc3ba224aeeb0eecb3"><code>cf1b8e4</code></a> Bump version to 2.10.11-RC.3 [ci skip]</li> <li><a href="https://github.com/nats-io/nats-server/commit/11f580841647fcbc77f7f7f8b54bd2483d8df0f8"><code>11f5808</code></a> Cherry picks for Release v2.10.11 (<a href="https://redirect.github.com/nats-io/nats-server/issues/5084">#5084</a>)</li> <li><a href="https://github.com/nats-io/nats-server/commit/82ea3952b7a1935c959e5dac2edd9fa7678be379"><code>82ea395</code></a> Update go client to 1.33.0</li> <li><a href="https://github.com/nats-io/nats-server/commit/b9763c19ff4f5ed4e1ccbcb279c0bc2d8e8c55c9"><code>b9763c1</code></a> Move reallocation warnings on psim to debug.</li> <li>Additional commits viewable in <a href="https://github.com/nats-io/nats-server/compare/v2.10.7...v2.10.11">compare view</a></li> </ul> </details> <br /> [](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores) Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@dependabot rebase`. [//]: # (dependabot-automerge-start) [//]: # (dependabot-automerge-end) --- <details> <summary>Dependabot commands and options</summary> <br /> You can trigger Dependabot actions by commenting on this PR: - `@dependabot rebase` will rebase this PR - `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it - `@dependabot merge` will merge this PR after your CI passes on it - `@dependabot squash and merge` will squash and merge this PR after your CI passes on it - `@dependabot cancel merge` will cancel a previously requested merge and block automerging - `@dependabot reopen` will reopen this PR if it is closed - `@dependabot close` will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually - `@dependabot show <dependency name> ignore conditions` will show all of the ignore conditions of the specified dependency - `@dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself) </details> --------- Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
See Commits and Changes for more details.
Created by
pull[bot]
Can you help keep this open source service alive? 💖 Please sponsor : )