[pull] main from bacalhau-project:main #11
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
![pull[bot]](https://avatars.githubusercontent.com/in/12910?s=60&v=4){kind=small}
- iff the publisher is set - closes #3863 Co-authored-by: frrist <[email protected]>
## What are we doing here ? - With this PR, we are making sure we do locking correctly. - In addition, we come up with a good way to cancel the stream. We store a `cancel` function as part of stream. This cancel function is attached to the context which gets passed on to the engine, which serves the stream (currently only the log stream). If a heartbeat response is received from the consumer stating that it is no longer interested in this stream, then we invoke this `cancel` function. There might be scenarios where few logs will make it through after context cancelation due to race conditions. This should be fine and won't result in `nil` pointers or writing to a closed writer as we only close the writer after source channel is closed.
- provids clear errors when a job contains: - un-settable fields - invalid field types - uknow fields - closes #3719
Removing flyte - it's out of date, and needs rewriting. Non blocking for 1.4.
Remove docs. Also removed unused experimental directory which has visualization tool for p2p network when compute nodes used to connect to peer with each other.
- maintains backwards compatibility with old clients - closes #3839
Co-authored-by: frrist <[email protected]>
- closes #4114 When this value is set to `-1` the client will never stop attempting to reconnect to the server and thus never remove the server from its connection pool: https://docs.nats.io/using-nats/developer/connecting/reconnect/max + https://github.com/nats-io/nats.go/blob/main/nats.go#L1674: setting maxReconnect to -1 when only a single server exists in the pool means that the server will never be removed from the pool. Since we only have one server (at the moment) this seems like reasonable behavior. But in the future I expect more advanced re-connection logic will be required. Co-authored-by: frrist <[email protected]>
Improves visibility of queued jobs by adding a dedicated `Queued` state
Forgot a word in README
Improve stability and fix few tests, including: - Make IPFS daemon offline to stabilize IPFS related tests - Fix `get_test.go` which was checking for wrong error message. Not sure how they were passing before! - Fix `TestDockerRunSuite/TestRun_BadExecutables/bad-image-bad-executable` who was checking for node error details in the output message, but we weren't printing it in the new print path. Again not sure how they were passing! - Fix tracing tests race condition when checking for span closure Actually printing node details when there is an error is not test related and should be merged before 1.4
Some changes in the upcoming release are not compatible with existing clients, such as #4115 This PR prints a human readable message asking users to update their client
- Revert recent circleci changes since v1.3.2 to unblock release pipeline - Fix not updating job state from Queued to Running. It was only moving from Queued to Completed when the job was done - Disable webui in demo network - Deploy v1.4.0-rc9 to staging - Deploy to-be-released v1.4.0 to demo
There was a bug in how we disabled web-ui previously. This has already been deployed and verified
closes #4140 part of #3832 --------- Co-authored-by: frrist <[email protected]>
Today we only normalize and validate requests at the client side, but
only validate on the server side without normalizing first. This causes
inconsistencies and issues around optional fields, such as publishers.
## Example before the fix:
```
# hello.json
{
"Job": {
"Type": "batch",
"Count": 1,
"Tasks": [
{
"Name": "main",
"Engine": {
"Type": "docker",
"Params": {
"Image": "ubuntu",
"Parameters": ["echo", "hello"]
}
}
}
]
}
}
```
```
curl -X PUT localhost:1234/api/v1/orchestrator/jobs -H "Content-Type: application/json" -d @hello.json
{
"error": "code=400, message=task main validation failed: publisher validation failed: nil spec config",
"message": "task main validation failed: publisher validation failed: nil spec config"
}
```
## Example after the fix
```
→ curl -X PUT localhost:1234/api/v1/orchestrator/jobs -H "Content-Type: application/json" -d @hello.json
{
"JobID": "j-26b215f9-582c-4374-80cb-fd079e73fc37",
"EvaluationID": "36fd6f65-64fd-421b-a9ba-aaf86d876704",
"Warnings": null
}
```
```
→ bacalhau job describe j-26b215f9-582c-4374-80cb-fd079e73fc37
ID = j-26b215f9-582c-4374-80cb-fd079e73fc37
Name = j-26b215f9-582c-4374-80cb-fd079e73fc37
Namespace = default
Type = batch
State = Completed
Count = 1
Created Time = 2024-06-25 09:14:40
Modified Time = 2024-06-25 09:14:43
Version = 0
Summary
Completed = 1
Job History
TIME REV. STATE TOPIC EVENT
2024-06-25 11:14:40 1 Pending Submission Job submitted
2024-06-25 11:14:40 2 Running
2024-06-25 11:14:43 3 Completed
Executions
ID NODE ID STATE DESIRED REV. CREATED MODIFIED COMMENT
e-ad7cc59d n-6e8998ad Completed Stopped 6 14s ago 12s ago Accepted job
Execution e-ad7cc59d History
TIME REV. STATE TOPIC EVENT
2024-06-25 11:14:40 1 New
2024-06-25 11:14:40 2 AskForBid
2024-06-25 11:14:40 3 AskForBidAccepted Requesting Node Accepted job
2024-06-25 11:14:40 4 AskForBidAccepted
2024-06-25 11:14:40 5 BidAccepted
2024-06-25 11:14:43 6 Completed
Standard Output
hello
```
```
→ bacalhau job describe j-26b215f9-582c-4374-80cb-fd079e73fc37 --output json --pretty | jq ".Job"
{
"ID": "j-26b215f9-582c-4374-80cb-fd079e73fc37",
"Name": "j-26b215f9-582c-4374-80cb-fd079e73fc37",
"Namespace": "default",
"Type": "batch",
"Priority": 0,
"Count": 1,
"Constraints": [],
"Meta": {
"bacalhau.org/requester.id": "n-6e8998ad-e0b9-43e6-9ccb-47f05a780f4a"
},
"Labels": {},
"Tasks": [
{
"Name": "main",
"Engine": {
"Type": "docker",
"Params": {
"Image": "ubuntu",
"Parameters": [
"echo",
"hello"
]
}
},
"Publisher": {
"Type": ""
},
"Resources": {},
"Network": {
"Type": "None"
},
"Timeouts": {
"TotalTimeout": 1800
}
}
],
"State": {
"StateType": "Completed"
},
"Version": 0,
"Revision": 3,
"CreateTime": 1719306880611865000,
"ModifyTime": 1719306883073224000
}
```
Closes #3950
With this change, we are adding tests for interaction between the producer and consumer client. The test scenarion being added is where we kill the consumer client and make sure that producer client kills the stream as well
For some reason, timed-out executions were marked as cancelled instead of failed, which is wrong. Also this resulted in compute node calling `OnCancelComplete` on the requester node, which is a noop. This means the requester node will only mark the execution as failed when the housekeeper kicks in which has a buffer of 2 minutes, instead of as soon as the failure is reported by the compute node. Previously, this job will be marked as failed after 2-2:30 minutes: ``` bacalhau docker --timeout 10 run ubuntu sleep 120 ``` With this change it will marked as failed in ~10 seconds
## Example API and CLI usage when on <= 1.3.2 ``` → curl 127.0.0.1:1234/api/v1/requester/nodes "This endpoint is deprecated. See the migration guide at https://docs.bacalhau.org/v/v.1.4.0/references/cli-reference/command-migration for more information" ``` ## Example CLI Usage when on 1.4 ``` Walid-MacBook-5: ~/ProtocolLabs/workspace/bacalhau (main) ✗ [bacalhau]→ bacalhau create job.yaml Command "create" is deprecated, Please use `job run` to create jobs. See the migration guide at https://docs.bacalhau.org/v/v.1.4.0/references/cli-reference/command-migration for more information. Walid-MacBook-5: ~/ProtocolLabs/workspace/bacalhau (main) ✗ [bacalhau]→ echo hi | bacalhau create Command "create" is deprecated, Please use `job run` to create jobs. See the migration guide at https://docs.bacalhau.org/v/v.1.4.0/references/cli-reference/command-migration for more information. Walid-MacBook-5: ~/ProtocolLabs/workspace/bacalhau (main) ✗ [bacalhau]→ bacalhau id Command "id" is deprecated, Please use `agent node` to inspect bacalhau nodes. See the migration guide at https://docs.bacalhau.org/v/v.1.4.0/references/cli-reference/command-migration for more information. Walid-MacBook-5: ~/ProtocolLabs/workspace/bacalhau (main) ✗ [bacalhau]→ bacalhau get Command "get" is deprecated, Please use `job get` to download results of a job. See the migration guide at https://docs.bacalhau.org/v/v.1.4.0/references/cli-reference/command-migration for more information. Walid-MacBook-5: ~/ProtocolLabs/workspace/bacalhau (main) ✗ [bacalhau]→ bacalhau list Command "list" is deprecated, Please use `job list` to list jobs. See the migration guide at https://docs.bacalhau.org/v/v.1.4.0/references/cli-reference/command-migration for more information. Walid-MacBook-5: ~/ProtocolLabs/workspace/bacalhau (main) ✗ [bacalhau]→ bacalhau logs Command "logs" is deprecated, Please use `job logs` to follow job logs. See the migration guide at https://docs.bacalhau.org/v/v.1.4.0/references/cli-reference/command-migration for more information. Walid-MacBook-5: ~/ProtocolLabs/workspace/bacalhau (main) ✗ [bacalhau]→ bacalhau validate Command "validate" is deprecated, Please use `job validate` to validate jobs. See the migration guide at https://docs.bacalhau.org/v/v.1.4.0/references/cli-reference/command-migration for more information. ```
Closes #4146, #4141, #4119, and #4143 --------- Authored-by: frrist <[email protected]>
Co-authored-by: frrist <[email protected]>
## Problem When nodes restart or lose state, the current sequence number synchronization can lead to message gaps or duplicates. This occurs because nodes unconditionally trust each other's sequence numbers during handshake, without considering local state recovery scenarios. ## Solution This PR implements a "trust your own state" approach for sequence number synchronization during handshakes. Each node relies on its local state to determine its starting point, while using the handshake to inform the other party of its position. ### Changes in Handshake Flow - **Orchestrator Behavior** - Uses its local knowledge of the compute node's last received sequence number - Starts streaming from 0 if no prior state exists, regardless of compute node's reported position - Tracks compute node's progress through heartbeat updates - **Compute Node Behavior** - Starts from its local checkpoint, preserved across restarts - Ignores orchestrator's suggested sequence position if local state exists - Continues reporting processed sequence numbers via heartbeats <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit ## Release Notes - **New Features** - Introduced new methods for managing node sequence numbers and state handling. - Added functionality to ensure message publishing starts correctly after node restarts. - Enhanced dispatcher state management with a new structured format. - **Bug Fixes** - Improved error handling for sequence number resolution and checkpointing processes. - **Tests** - Added new tests to verify the behavior of the data plane and dispatcher during various scenarios. - **Documentation** - Updated comments and documentation to reflect changes in methods and logic. <!-- end of auto-generated comment: release notes by coderabbit.ai -->
When compute nodes shut down, they should notify the orchestrator with their final sequence numbers. While this info is already shared in heartbeats, a dedicated shutdown message confirms intentional shutdown vs connection failure and guarantees the orchestrator receives latest sequence numbers. This lets the orchestrator clean up node state immediately rather than waiting for missing heartbeats. Changes: - Added `sendShutdownNotification` method to ControlPlane for notifying orchestrator before shutdown - Modified Stop() to send notification if node is connected and context isn't cancelled - Added test cases covering successful notification, skipped notifications (when disconnected/cancelled), and error handling <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit - **New Features** - Introduced constants for shutdown notice request and response message types. - Added structures for handling shutdown notice requests and responses. - Implemented a method for processing shutdown notifications in the node manager. - Enhanced control plane to send shutdown notifications to the orchestrator. - Updated the compute manager to handle shutdown requests from nodes. - **Bug Fixes** - Improved error handling and state management during shutdown operations. - **Tests** - Added comprehensive tests for shutdown functionality in node manager and control plane. - Enhanced mock responder to simulate shutdown notifications for testing purposes. <!-- end of auto-generated comment: release notes by coderabbit.ai -->
<!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit ## Release Notes - **New Features** - Enhanced error handling and cleanup processes for connection management. - State reset functionality added to the dispatcher during start and stop operations. - New recovery mechanisms for graceful stopping of the recovery process. - **Bug Fixes** - Improved error reporting during connection cleanup and state management. - **Tests** - Added new test methods to validate dispatcher state management and recovery processes. - Enhanced existing tests for better error handling and recovery logic validation. <!-- end of auto-generated comment: release notes by coderabbit.ai -->
When orchestrator restarts, compute nodes wait for 5 failed heartbeats (~75s) before attempting to reconnect, even though orchestrator immediately returns "Handshake required" errors. Modify compute nodes to detect this specific error and trigger immediate reconnection, rather than waiting for the heartbeat failure threshold. <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit - **New Features** - Enhanced error handling for heartbeat operations, specifically addressing handshake requirements. - New boolean field `HandshakeRequired` added to track handshake necessity. - **Bug Fixes** - Improved robustness of connection health monitoring by incorporating handshake checks. - **Tests** - Added tests for new handshake handling scenarios in both `ControlPlaneTestSuite` and `ConnectionManagerTestSuite`. - Enhanced coverage for `HealthTracker` functionality regarding handshake states. - **Documentation** - Updated comments in connection health checks for clarity on new criteria. <!-- end of auto-generated comment: release notes by coderabbit.ai -->
<!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit - **New Features** - Enhanced logic for determining the starting iterator in the watcher, improving error handling for various iterator types. - **Bug Fixes** - Improved handling of scenarios where the requested sequence number is unavailable, preventing potential issues during event processing. - **Tests** - Expanded test cases for the watcher to cover additional iterator behaviors and error handling scenarios, enhancing overall test coverage. <!-- end of auto-generated comment: release notes by coderabbit.ai -->
<!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit - **New Features** - Introduced JSON serialization and deserialization for error handling. - Added standardized error handling mechanisms using the `bacerrors` package. - Implemented new error reporting functions for connection management. - **Bug Fixes** - Enhanced error handling logic in various components, improving robustness and clarity. - **Tests** - Added comprehensive unit tests for JSON marshalling and unmarshalling of error structures. - **Chores** - Updated import statements across multiple files to include the `bacerrors` package. <!-- end of auto-generated comment: release notes by coderabbit.ai -->
Only update LastComputeSeqNum if greater than 0, as zero can indicate either no messages processed yet or a connection that has just been established. This preserves the existing sequence number in those cases. <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit - **New Features** - Enhanced node management with improved health monitoring and state persistence. - Added conditional logic to prevent overwriting sequence numbers with zero. - Improved handling of disconnected nodes to retain existing connection states. - **Bug Fixes** - Adjusted control flow and error handling for better robustness in node state updates. - **Documentation** - Added logging statements for improved visibility into state transitions and error conditions. <!-- end of auto-generated comment: release notes by coderabbit.ai -->
[dist.ipfs.tech](https://dist.ipfs.tech) is down and had to switch to deploying from github instead <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit - **New Features** - Updated `bacalhau_version` to `"v1.6.0"` in production and staging configurations. - Enhanced the installation script for IPFS to download from a new source, targeting the Kubo project. - **Bug Fixes** - Addressed versioning issues in the IPFS installation process. <!-- end of auto-generated comment: release notes by coderabbit.ai -->
<!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit - **Documentation** - Expanded documentation for the NCL Protocol, including detailed descriptions of sequence number management, connection lifecycle, and failure recovery processes. - Added new sections and clarified existing ones to enhance understanding of message sequencing and connection management. - Updated structures in the documentation to include comments and new fields for better clarity on their usage. <!-- end of auto-generated comment: release notes by coderabbit.ai -->
<!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit - **Bug Fixes** - Enhanced error handling in tests for concurrent heartbeats. - Improved assertions and validations for expected states and error messages. - **New Features** - Added new test cases for shutdown notice handling and resource management. - Included concurrency tests for simultaneous operations. - Verified lifecycle management with checks for idempotency. - Ensured state persistence during manager stop and context cancellation. <!-- end of auto-generated comment: release notes by coderabbit.ai -->
<!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit - **New Features** - Introduced a new linter for improved logging practices. - **Bug Fixes** - Removed unnecessary `fmt` package usage to streamline output. <!-- end of auto-generated comment: release notes by coderabbit.ai -->
<!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit - **New Features** - Enhanced display of job execution IDs and node information with improved formatting. - Updated job history representation with a more precise column naming convention. - **Bug Fixes** - Streamlined handling of execution IDs in job history display. - **Chores** - Removed obsolete `HistoryTime` column for cleaner output. <!-- end of auto-generated comment: release notes by coderabbit.ai -->
<!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit - **New Features** - Enhanced encapsulation by updating node ID retrieval methods across various components. - **Bug Fixes** - Improved robustness in heartbeat handling and node information updates in tests. - **Tests** - Updated test cases to reflect changes in node ID handling, ensuring accurate validation of control plane behavior. <!-- end of auto-generated comment: release notes by coderabbit.ai -->
Smaller job resource requirements so that jobs can run on smaller EC2 instances by default. This PR also introduce the following changes: - Increase allocated capacity from 70% to 80% - Unify the library used for printing human readable bytes <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit - **New Features** - Enhanced memory and disk capacity outputs with improved human-readable formatting. - Updated default configuration settings for resource allocation in Bacalhau nodes. - Introduced new API endpoints for orchestrator nodes and version retrieval. - **Bug Fixes** - Adjusted memory allocation settings for batch, daemon, and service jobs to optimize resource management. - Improved error handling and response schemas for various API endpoints. <!-- end of auto-generated comment: release notes by coderabbit.ai -->
) Bumps [github.com/ipld/go-car/v2](https://github.com/ipld/go-car) from 2.13.1 to 2.14.2. <details> <summary>Release notes</summary> <p><em>Sourced from <a href="https://github.com/ipld/go-car/releases">github.com/ipld/go-car/v2's releases</a>.</em></p> <blockquote> <h2>v2.14.2</h2> <h2>What's Changed</h2> <ul> <li>chore(deps): bump github.com/ipld/go-car/v2 from 2.14.0 to 2.14.1 in /cmd by <a href="https://github.com/dependabot"><code>@dependabot</code></a> in <a href="https://redirect.github.com/ipld/go-car/pull/539">ipld/go-car#539</a></li> <li>chore(deps): bump github.com/rogpeppe/go-internal from 1.12.0 to 1.13.1 in /cmd by <a href="https://github.com/dependabot"><code>@dependabot</code></a> in <a href="https://redirect.github.com/ipld/go-car/pull/538">ipld/go-car#538</a></li> <li>chore(deps): bump github.com/ipfs/go-unixfsnode from 1.9.0 to 1.9.1 in /v2 by <a href="https://github.com/dependabot"><code>@dependabot</code></a> in <a href="https://redirect.github.com/ipld/go-car/pull/530">ipld/go-car#530</a></li> <li>fix: goreleaser v2 compat, trigger release-binaries with workflow_run by <a href="https://github.com/rvagg"><code>@rvagg</code></a> in <a href="https://redirect.github.com/ipld/go-car/pull/540">ipld/go-car#540</a></li> </ul> <p><strong>Full Changelog</strong>: <a href="https://github.com/ipld/go-car/compare/v2.14.1...v2.14.2">https://github.com/ipld/go-car/compare/v2.14.1...v2.14.2</a></p> <h2>v2.14.1</h2> <h2>What's Changed</h2> <ul> <li>chore(deps): bump github.com/ipld/go-car/v2 from 2.13.1 to 2.14.0 in /cmd by <a href="https://github.com/dependabot"><code>@dependabot</code></a> in <a href="https://redirect.github.com/ipld/go-car/pull/533">ipld/go-car#533</a></li> <li>chore: downgrade golang.org/x/exp to avoid specific go versioning by <a href="https://github.com/rvagg"><code>@rvagg</code></a> in <a href="https://redirect.github.com/ipld/go-car/pull/534">ipld/go-car#534</a></li> </ul> <p><strong>Full Changelog</strong>: <a href="https://github.com/ipld/go-car/compare/v2.14.0...v2.14.1">https://github.com/ipld/go-car/compare/v2.14.0...v2.14.1</a></p> <h2>v2.14.0</h2> <h2>What's Changed</h2> <ul> <li>chore(deps): bump goreleaser/goreleaser-action from 4 to 5 by <a href="https://github.com/dependabot"><code>@dependabot</code></a> in <a href="https://redirect.github.com/ipld/go-car/pull/500">ipld/go-car#500</a></li> <li>chore(deps): bump github.com/ipld/go-car/v2 from 2.11.0 to 2.13.1 in /cmd by <a href="https://github.com/dependabot"><code>@dependabot</code></a> in <a href="https://redirect.github.com/ipld/go-car/pull/499">ipld/go-car#499</a></li> <li>chore(deps): bump github.com/ipfs/go-unixfsnode from 1.8.0 to 1.8.1 in /v2 by <a href="https://github.com/dependabot"><code>@dependabot</code></a> in <a href="https://redirect.github.com/ipld/go-car/pull/501">ipld/go-car#501</a></li> <li>chore(deps): bump github.com/ipfs/go-unixfsnode from 1.8.0 to 1.8.1 in /cmd by <a href="https://github.com/dependabot"><code>@dependabot</code></a> in <a href="https://redirect.github.com/ipld/go-car/pull/502">ipld/go-car#502</a></li> <li>chore(deps): bump github.com/ipfs/go-unixfsnode from 1.8.1 to 1.9.0 in /v2 by <a href="https://github.com/dependabot"><code>@dependabot</code></a> in <a href="https://redirect.github.com/ipld/go-car/pull/503">ipld/go-car#503</a></li> <li>chore(deps): bump github.com/ipfs/go-unixfsnode from 1.8.1 to 1.9.0 in /cmd by <a href="https://github.com/dependabot"><code>@dependabot</code></a> in <a href="https://redirect.github.com/ipld/go-car/pull/504">ipld/go-car#504</a></li> <li>chore(deps): bump github.com/urfave/cli/v2 from 2.25.7 to 2.26.0 in /cmd by <a href="https://github.com/dependabot"><code>@dependabot</code></a> in <a href="https://redirect.github.com/ipld/go-car/pull/505">ipld/go-car#505</a></li> <li>chore(deps): bump actions/setup-go from 4 to 5 by <a href="https://github.com/dependabot"><code>@dependabot</code></a> in <a href="https://redirect.github.com/ipld/go-car/pull/506">ipld/go-car#506</a></li> <li>chore(deps): bump golang.org/x/crypto from 0.6.0 to 0.17.0 by <a href="https://github.com/dependabot"><code>@dependabot</code></a> in <a href="https://redirect.github.com/ipld/go-car/pull/508">ipld/go-car#508</a></li> <li>chore(deps): bump golang.org/x/crypto from 0.7.0 to 0.17.0 in /cmd by <a href="https://github.com/dependabot"><code>@dependabot</code></a> in <a href="https://redirect.github.com/ipld/go-car/pull/509">ipld/go-car#509</a></li> <li>chore(deps): bump github.com/rogpeppe/go-internal from 1.11.0 to 1.12.0 in /cmd by <a href="https://github.com/dependabot"><code>@dependabot</code></a> in <a href="https://redirect.github.com/ipld/go-car/pull/507">ipld/go-car#507</a></li> <li>chore(deps): bump golang.org/x/crypto from 0.6.0 to 0.17.0 in /v2 by <a href="https://github.com/dependabot"><code>@dependabot</code></a> in <a href="https://redirect.github.com/ipld/go-car/pull/510">ipld/go-car#510</a></li> <li>chore(deps): bump github.com/urfave/cli/v2 from 2.26.0 to 2.27.1 in /cmd by <a href="https://github.com/dependabot"><code>@dependabot</code></a> in <a href="https://redirect.github.com/ipld/go-car/pull/512">ipld/go-car#512</a></li> <li>Add a <code>car ls --unixfs-blocks</code> to render two-column output by <a href="https://github.com/willscott"><code>@willscott</code></a> in <a href="https://redirect.github.com/ipld/go-car/pull/514">ipld/go-car#514</a></li> <li>chore(deps): bump github.com/stretchr/testify from 1.8.4 to 1.9.0 in /v2 by <a href="https://github.com/dependabot"><code>@dependabot</code></a> in <a href="https://redirect.github.com/ipld/go-car/pull/515">ipld/go-car#515</a></li> <li>chore(deps): bump github.com/stretchr/testify from 1.8.4 to 1.9.0 by <a href="https://github.com/dependabot"><code>@dependabot</code></a> in <a href="https://redirect.github.com/ipld/go-car/pull/516">ipld/go-car#516</a></li> <li>ci: uci/update-go by <a href="https://github.com/web3-bot"><code>@web3-bot</code></a> in <a href="https://redirect.github.com/ipld/go-car/pull/517">ipld/go-car#517</a></li> <li>chore(deps): bump google.golang.org/protobuf from 1.30.0 to 1.33.0 in /cmd by <a href="https://github.com/dependabot"><code>@dependabot</code></a> in <a href="https://redirect.github.com/ipld/go-car/pull/520">ipld/go-car#520</a></li> <li>chore(deps): bump google.golang.org/protobuf from 1.28.1 to 1.33.0 by <a href="https://github.com/dependabot"><code>@dependabot</code></a> in <a href="https://redirect.github.com/ipld/go-car/pull/519">ipld/go-car#519</a></li> <li>chore(deps): bump google.golang.org/protobuf from 1.28.1 to 1.33.0 in /v2 by <a href="https://github.com/dependabot"><code>@dependabot</code></a> in <a href="https://redirect.github.com/ipld/go-car/pull/518">ipld/go-car#518</a></li> <li>ci: uci/copy-templates by <a href="https://github.com/web3-bot"><code>@web3-bot</code></a> in <a href="https://redirect.github.com/ipld/go-car/pull/521">ipld/go-car#521</a></li> <li>chore(deps): bump github.com/urfave/cli/v2 from 2.27.1 to 2.27.2 in /cmd by <a href="https://github.com/dependabot"><code>@dependabot</code></a> in <a href="https://redirect.github.com/ipld/go-car/pull/522">ipld/go-car#522</a></li> <li>Re-factor cmd functions to library by <a href="https://github.com/willscott"><code>@willscott</code></a> in <a href="https://redirect.github.com/ipld/go-car/pull/524">ipld/go-car#524</a></li> <li>chore(deps): bump goreleaser/goreleaser-action from 5 to 6 by <a href="https://github.com/dependabot"><code>@dependabot</code></a> in <a href="https://redirect.github.com/ipld/go-car/pull/525">ipld/go-car#525</a></li> <li>chore(deps): bump github.com/urfave/cli/v2 from 2.27.2 to 2.27.3 in /cmd by <a href="https://github.com/dependabot"><code>@dependabot</code></a> in <a href="https://redirect.github.com/ipld/go-car/pull/526">ipld/go-car#526</a></li> <li>chore(deps): bump github.com/urfave/cli/v2 from 2.27.3 to 2.27.4 in /cmd by <a href="https://github.com/dependabot"><code>@dependabot</code></a> in <a href="https://redirect.github.com/ipld/go-car/pull/527">ipld/go-car#527</a></li> <li>ci: uci/update-go by <a href="https://github.com/web3-bot"><code>@web3-bot</code></a> in <a href="https://redirect.github.com/ipld/go-car/pull/528">ipld/go-car#528</a></li> <li>chore(deps): bump github.com/ipfs/go-unixfsnode from 1.9.0 to 1.9.1 in /cmd by <a href="https://github.com/dependabot"><code>@dependabot</code></a> in <a href="https://redirect.github.com/ipld/go-car/pull/529">ipld/go-car#529</a></li> <li>fix(cmd): properly pick up --inverse and --cid-file args by <a href="https://github.com/rvagg"><code>@rvagg</code></a> in <a href="https://redirect.github.com/ipld/go-car/pull/531">ipld/go-car#531</a></li> </ul> <!-- raw HTML omitted --> </blockquote> <p>... (truncated)</p> </details> <details> <summary>Commits</summary> <ul> <li><a href="https://github.com/ipld/go-car/commit/40add1a048591fa33225e158e7aedbbd51258618"><code>40add1a</code></a> v2.14.2 bump</li> <li><a href="https://github.com/ipld/go-car/commit/0f4efe384683fdc0d2318c2c9346ba86967e8ec0"><code>0f4efe3</code></a> fix: goreleaser v2 compat, trigger release-binaries with workflow_run</li> <li><a href="https://github.com/ipld/go-car/commit/ed334d8aa743b7225682adbf06bb76e3ffbf6d5c"><code>ed334d8</code></a> chore(deps): bump github.com/ipfs/go-unixfsnode in /v2</li> <li><a href="https://github.com/ipld/go-car/commit/e2353edea31231cc1d6ea3bb68988b954c535c9b"><code>e2353ed</code></a> chore(deps): bump github.com/rogpeppe/go-internal in /cmd</li> <li><a href="https://github.com/ipld/go-car/commit/f27e000345a22a132a316b63a4bc4507090d806b"><code>f27e000</code></a> chore(deps): bump github.com/ipld/go-car/v2 in /cmd</li> <li><a href="https://github.com/ipld/go-car/commit/92d3f4e5cc6eadb2223959365be208b73d190ac7"><code>92d3f4e</code></a> v2.14.1 bump</li> <li><a href="https://github.com/ipld/go-car/commit/960bbfefac37fbbb231cafa00c02cee89c02fec7"><code>960bbfe</code></a> chore: downgrade golang.org/x/exp to avoid specific go versioning</li> <li><a href="https://github.com/ipld/go-car/commit/57a13711934af657d2407354351ad7345dbb004b"><code>57a1371</code></a> Merge pull request <a href="https://redirect.github.com/ipld/go-car/issues/533">#533</a> from ipld/dependabot/go_modules/cmd/github.com/ipld/g...</li> <li><a href="https://github.com/ipld/go-car/commit/c10b0c6e1dba7f6b749dbd8006f73f308fbb4050"><code>c10b0c6</code></a> chore(deps): bump github.com/ipld/go-car/v2 in /cmd</li> <li><a href="https://github.com/ipld/go-car/commit/f7e4bb740ddf5bf0d28e5b8dfbd953db0af14988"><code>f7e4bb7</code></a> chore: update fuzz to Go 1.22</li> <li>Additional commits viewable in <a href="https://github.com/ipld/go-car/compare/v2.13.1...v2.14.2">compare view</a></li> </ul> </details> <br /> [](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores) Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@dependabot rebase`. [//]: # (dependabot-automerge-start) [//]: # (dependabot-automerge-end) --- <details> <summary>Dependabot commands and options</summary> <br /> You can trigger Dependabot actions by commenting on this PR: - `@dependabot rebase` will rebase this PR - `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it - `@dependabot merge` will merge this PR after your CI passes on it - `@dependabot squash and merge` will squash and merge this PR after your CI passes on it - `@dependabot cancel merge` will cancel a previously requested merge and block automerging - `@dependabot reopen` will reopen this PR if it is closed - `@dependabot close` will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually - `@dependabot show <dependency name> ignore conditions` will show all of the ignore conditions of the specified dependency - `@dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself) </details> Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Jamil Shamy <[email protected]>
Bumps [nanoid](https://github.com/ai/nanoid) from 3.3.7 to 3.3.8. <details> <summary>Changelog</summary> <p><em>Sourced from <a href="https://github.com/ai/nanoid/blob/main/CHANGELOG.md">nanoid's changelog</a>.</em></p> <blockquote> <h2>3.3.8</h2> <ul> <li>Fixed a way to break Nano ID by passing non-integer size (by <a href="https://github.com/myndzi"><code>@myndzi</code></a>).</li> </ul> </blockquote> </details> <details> <summary>Commits</summary> <ul> <li><a href="https://github.com/ai/nanoid/commit/3044cd5e73f4cf31795f61f6e6b961c8c0a5c744"><code>3044cd5</code></a> Release 3.3.8 version</li> <li><a href="https://github.com/ai/nanoid/commit/4fe34959c34e5b3573889ed4f24fe91d1d3e7231"><code>4fe3495</code></a> Update size limit</li> <li><a href="https://github.com/ai/nanoid/commit/d643045f40d6dc8afa000a644d857da1436ed08c"><code>d643045</code></a> Fix pool pollution, infinite loop (<a href="https://redirect.github.com/ai/nanoid/issues/510">#510</a>)</li> <li>See full diff in <a href="https://github.com/ai/nanoid/compare/3.3.7...3.3.8">compare view</a></li> </ul> </details> <br /> [](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores) Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@dependabot rebase`. [//]: # (dependabot-automerge-start) [//]: # (dependabot-automerge-end) --- <details> <summary>Dependabot commands and options</summary> <br /> You can trigger Dependabot actions by commenting on this PR: - `@dependabot rebase` will rebase this PR - `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it - `@dependabot merge` will merge this PR after your CI passes on it - `@dependabot squash and merge` will squash and merge this PR after your CI passes on it - `@dependabot cancel merge` will cancel a previously requested merge and block automerging - `@dependabot reopen` will reopen this PR if it is closed - `@dependabot close` will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually - `@dependabot show <dependency name> ignore conditions` will show all of the ignore conditions of the specified dependency - `@dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself) You can disable automated security fix PRs for this repo from the [Security Alerts page](https://github.com/bacalhau-project/bacalhau/network/alerts). </details> Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Jamil Shamy <[email protected]>
…integration (#4760) Bumps [golang.org/x/crypto](https://github.com/golang/crypto) from 0.27.0 to 0.31.0. <details> <summary>Commits</summary> <ul> <li><a href="https://github.com/golang/crypto/commit/b4f1988a35dee11ec3e05d6bf3e90b695fbd8909"><code>b4f1988</code></a> ssh: make the public key cache a 1-entry FIFO cache</li> <li><a href="https://github.com/golang/crypto/commit/7042ebcbe097f305ba3a93f9a22b4befa4b83d29"><code>7042ebc</code></a> openpgp/clearsign: just use rand.Reader in tests</li> <li><a href="https://github.com/golang/crypto/commit/3e90321ac7bcee3d924ed63ed3ad97be2079cb56"><code>3e90321</code></a> go.mod: update golang.org/x dependencies</li> <li><a href="https://github.com/golang/crypto/commit/8c4e668694ccbaa1be4785da7e7a40f2ef93152b"><code>8c4e668</code></a> x509roots/fallback: update bundle</li> <li><a href="https://github.com/golang/crypto/commit/6018723c74059e3b91c84268b212c2f6cdab1f64"><code>6018723</code></a> go.mod: update golang.org/x dependencies</li> <li><a href="https://github.com/golang/crypto/commit/71ed71b4faf97caafd1863fed003e9ac311f10ee"><code>71ed71b</code></a> README: don't recommend go get</li> <li><a href="https://github.com/golang/crypto/commit/750a45fe5e473d5afa193e9088f3d135e64eca26"><code>750a45f</code></a> sha3: add MarshalBinary, AppendBinary, and UnmarshalBinary</li> <li><a href="https://github.com/golang/crypto/commit/36b172546bd03a74c79e109ec84c599b672ea9e4"><code>36b1725</code></a> sha3: avoid trailing permutation</li> <li><a href="https://github.com/golang/crypto/commit/80ea76eb17c0c52f5d5d04e833d6aeb6b062d81d"><code>80ea76e</code></a> sha3: fix padding for long cSHAKE parameters</li> <li><a href="https://github.com/golang/crypto/commit/c17aa50fbd32393e5d52fa65ca51cbfff0a75aea"><code>c17aa50</code></a> sha3: avoid buffer copy</li> <li>Additional commits viewable in <a href="https://github.com/golang/crypto/compare/v0.27.0...v0.31.0">compare view</a></li> </ul> </details> <br /> [](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores) Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@dependabot rebase`. [//]: # (dependabot-automerge-start) [//]: # (dependabot-automerge-end) --- <details> <summary>Dependabot commands and options</summary> <br /> You can trigger Dependabot actions by commenting on this PR: - `@dependabot rebase` will rebase this PR - `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it - `@dependabot merge` will merge this PR after your CI passes on it - `@dependabot squash and merge` will squash and merge this PR after your CI passes on it - `@dependabot cancel merge` will cancel a previously requested merge and block automerging - `@dependabot reopen` will reopen this PR if it is closed - `@dependabot close` will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually - `@dependabot show <dependency name> ignore conditions` will show all of the ignore conditions of the specified dependency - `@dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself) You can disable automated security fix PRs for this repo from the [Security Alerts page](https://github.com/bacalhau-project/bacalhau/network/alerts). </details> Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Jamil Shamy <[email protected]>
Bumps [next](https://github.com/vercel/next.js) from 14.2.10 to 14.2.15. <details> <summary>Commits</summary> <ul> <li><a href="https://github.com/vercel/next.js/commit/737c29e49f9532cce3da8e25d88bc82cf5f5fe9a"><code>737c29e</code></a> v14.2.15</li> <li><a href="https://github.com/vercel/next.js/commit/1c8234eb20bc8afd396b89999a00f06b61d72d7b"><code>1c8234e</code></a> backport: [i18n] Routing fix <a href="https://redirect.github.com/vercel/next.js/issues/70761">#70761</a> (<a href="https://redirect.github.com/vercel/next.js/issues/70976">#70976</a>)</li> <li><a href="https://github.com/vercel/next.js/commit/bb3f58011885369404fe2d6abe5d40b867183fce"><code>bb3f580</code></a> fix: Do not omit alt on getImgProps return type, ImgProps (<a href="https://redirect.github.com/vercel/next.js/issues/70818">#70818</a>)</li> <li><a href="https://github.com/vercel/next.js/commit/cb8113bb0dec432470ac82a741670920c8c51d35"><code>cb8113b</code></a> fix(next/image): handle undefined <code>images.localPatterns</code> config in `images-ma...</li> <li><a href="https://github.com/vercel/next.js/commit/9c1191abccb08b984369e1b488d1ed99f4f9f7f6"><code>9c1191a</code></a> fix missing helpers in <code>next-test-utils.ts</code></li> <li><a href="https://github.com/vercel/next.js/commit/0de418351392e302c9c4538f06d352b1c157a445"><code>0de4183</code></a> fix lint</li> <li><a href="https://github.com/vercel/next.js/commit/e03c4ff4ff53d7ad94dde19c45ed2dd4d94d91f0"><code>e03c4ff</code></a> feat(next/image): add <code>images.localPatterns</code> config (<a href="https://redirect.github.com/vercel/next.js/issues/70802">#70802</a>)</li> <li><a href="https://github.com/vercel/next.js/commit/540ea2d892b974bd3141faa05824b6118a634c1b"><code>540ea2d</code></a> backport: support breadcrumb style catch-all parallel routes (<a href="https://redirect.github.com/vercel/next.js/issues/65063">#65063</a>) (<a href="https://redirect.github.com/vercel/next.js/issues/70794">#70794</a>)</li> <li><a href="https://github.com/vercel/next.js/commit/0d0448be0345cfa314dab274662bccdf5b8384a3"><code>0d0448b</code></a> Fix client reference access causing metadata missing (<a href="https://redirect.github.com/vercel/next.js/issues/70732">#70732</a>) (<a href="https://redirect.github.com/vercel/next.js/issues/70788">#70788</a>)</li> <li><a href="https://github.com/vercel/next.js/commit/381d1f9a06292758cc94bee7bc702045cff94010"><code>381d1f9</code></a> feat(next/image): add support for <code>decoding</code> prop (<a href="https://redirect.github.com/vercel/next.js/issues/70678">#70678</a>)</li> <li>Additional commits viewable in <a href="https://github.com/vercel/next.js/compare/v14.2.10...v14.2.15">compare view</a></li> </ul> </details> <br /> [](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores) Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@dependabot rebase`. [//]: # (dependabot-automerge-start) [//]: # (dependabot-automerge-end) --- <details> <summary>Dependabot commands and options</summary> <br /> You can trigger Dependabot actions by commenting on this PR: - `@dependabot rebase` will rebase this PR - `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it - `@dependabot merge` will merge this PR after your CI passes on it - `@dependabot squash and merge` will squash and merge this PR after your CI passes on it - `@dependabot cancel merge` will cancel a previously requested merge and block automerging - `@dependabot reopen` will reopen this PR if it is closed - `@dependabot close` will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually - `@dependabot show <dependency name> ignore conditions` will show all of the ignore conditions of the specified dependency - `@dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself) You can disable automated security fix PRs for this repo from the [Security Alerts page](https://github.com/bacalhau-project/bacalhau/network/alerts). </details> Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Jamil Shamy <[email protected]>
…4791) Bumps [jinja2](https://github.com/pallets/jinja) from 3.1.3 to 3.1.5. <details> <summary>Release notes</summary> <p><em>Sourced from <a href="https://github.com/pallets/jinja/releases">jinja2's releases</a>.</em></p> <blockquote> <h2>3.1.5</h2> <p>This is the Jinja 3.1.5 security fix release, which fixes security issues and bugs but does not otherwise change behavior and should not result in breaking changes compared to the latest feature release.</p> <p>PyPI: <a href="https://pypi.org/project/Jinja2/3.1.5/">https://pypi.org/project/Jinja2/3.1.5/</a> Changes: <a href="https://jinja.palletsprojects.com/changes/#version-3-1-5">https://jinja.palletsprojects.com/changes/#version-3-1-5</a> Milestone: <a href="https://github.com/pallets/jinja/milestone/16?closed=1">https://github.com/pallets/jinja/milestone/16?closed=1</a></p> <ul> <li>The sandboxed environment handles indirect calls to <code>str.format</code>, such as by passing a stored reference to a filter that calls its argument. <a href="https://github.com/pallets/jinja/security/advisories/GHSA-q2x7-8rv6-6q7h">GHSA-q2x7-8rv6-6q7h</a></li> <li>Escape template name before formatting it into error messages, to avoid issues with names that contain f-string syntax. <a href="https://redirect.github.com/pallets/jinja/issues/1792">#1792</a>, <a href="https://github.com/pallets/jinja/security/advisories/GHSA-gmj6-6f8f-6699">GHSA-gmj6-6f8f-6699</a></li> <li>Sandbox does not allow <code>clear</code> and <code>pop</code> on known mutable sequence types. <a href="https://redirect.github.com/pallets/jinja/issues/2032">#2032</a></li> <li>Calling sync <code>render</code> for an async template uses <code>asyncio.run</code>. <a href="https://redirect.github.com/pallets/jinja/issues/1952">#1952</a></li> <li>Avoid unclosed <code>auto_aiter</code> warnings. <a href="https://redirect.github.com/pallets/jinja/issues/1960">#1960</a></li> <li>Return an <code>aclose</code>-able <code>AsyncGenerator</code> from <code>Template.generate_async</code>. <a href="https://redirect.github.com/pallets/jinja/issues/1960">#1960</a></li> <li>Avoid leaving <code>root_render_func()</code> unclosed in <code>Template.generate_async</code>. <a href="https://redirect.github.com/pallets/jinja/issues/1960">#1960</a></li> <li>Avoid leaving async generators unclosed in blocks, includes and extends. <a href="https://redirect.github.com/pallets/jinja/issues/1960">#1960</a></li> <li>The runtime uses the correct <code>concat</code> function for the current environment when calling block references. <a href="https://redirect.github.com/pallets/jinja/issues/1701">#1701</a></li> <li>Make <code>|unique</code> async-aware, allowing it to be used after another async-aware filter. <a href="https://redirect.github.com/pallets/jinja/issues/1781">#1781</a></li> <li><code>|int</code> filter handles <code>OverflowError</code> from scientific notation. <a href="https://redirect.github.com/pallets/jinja/issues/1921">#1921</a></li> <li>Make compiling deterministic for tuple unpacking in a <code>{% set ... %}</code> call. <a href="https://redirect.github.com/pallets/jinja/issues/2021">#2021</a></li> <li>Fix dunder protocol (<code>copy</code>/<code>pickle</code>/etc) interaction with <code>Undefined</code> objects. <a href="https://redirect.github.com/pallets/jinja/issues/2025">#2025</a></li> <li>Fix <code>copy</code>/<code>pickle</code> support for the internal <code>missing</code> object. <a href="https://redirect.github.com/pallets/jinja/issues/2027">#2027</a></li> <li><code>Environment.overlay(enable_async)</code> is applied correctly. <a href="https://redirect.github.com/pallets/jinja/issues/2061">#2061</a></li> <li>The error message from <code>FileSystemLoader</code> includes the paths that were searched. <a href="https://redirect.github.com/pallets/jinja/issues/1661">#1661</a></li> <li><code>PackageLoader</code> shows a clearer error message when the package does not contain the templates directory. <a href="https://redirect.github.com/pallets/jinja/issues/1705">#1705</a></li> <li>Improve annotations for methods returning copies. <a href="https://redirect.github.com/pallets/jinja/issues/1880">#1880</a></li> <li><code>urlize</code> does not add <code>mailto:</code> to values like <code>@A@b</code>. <a href="https://redirect.github.com/pallets/jinja/issues/1870">#1870</a></li> <li>Tests decorated with <code>@pass_context</code> can be used with the <code>|select</code> filter. <a href="https://redirect.github.com/pallets/jinja/issues/1624">#1624</a></li> <li>Using <code>set</code> for multiple assignment (<code>a, b = 1, 2</code>) does not fail when the target is a namespace attribute. <a href="https://redirect.github.com/pallets/jinja/issues/1413">#1413</a></li> <li>Using <code>set</code> in all branches of <code>{% if %}{% elif %}{% else %}</code> blocks does not cause the variable to be considered initially undefined. <a href="https://redirect.github.com/pallets/jinja/issues/1253">#1253</a></li> </ul> <h2>3.1.4</h2> <p>This is the Jinja 3.1.4 security release, which fixes security issues and bugs but does not otherwise change behavior and should not result in breaking changes.</p> <p>PyPI: <a href="https://pypi.org/project/Jinja2/3.1.4/">https://pypi.org/project/Jinja2/3.1.4/</a> Changes: <a href="https://jinja.palletsprojects.com/en/3.1.x/changes/#version-3-1-4">https://jinja.palletsprojects.com/en/3.1.x/changes/#version-3-1-4</a></p> <ul> <li>The <code>xmlattr</code> filter does not allow keys with <code>/</code> solidus, <code>></code> greater-than sign, or <code>=</code> equals sign, in addition to disallowing spaces. Regardless of any validation done by Jinja, user input should never be used as keys to this filter, or must be separately validated first. GHSA-h75v-3vvj-5mfj</li> </ul> </blockquote> </details> <details> <summary>Changelog</summary> <p><em>Sourced from <a href="https://github.com/pallets/jinja/blob/main/CHANGES.rst">jinja2's changelog</a>.</em></p> <blockquote> <h2>Version 3.1.5</h2> <p>Released 2024-12-21</p> <ul> <li>The sandboxed environment handles indirect calls to <code>str.format</code>, such as by passing a stored reference to a filter that calls its argument. :ghsa:<code>q2x7-8rv6-6q7h</code></li> <li>Escape template name before formatting it into error messages, to avoid issues with names that contain f-string syntax. :issue:<code>1792</code>, :ghsa:<code>gmj6-6f8f-6699</code></li> <li>Sandbox does not allow <code>clear</code> and <code>pop</code> on known mutable sequence types. :issue:<code>2032</code></li> <li>Calling sync <code>render</code> for an async template uses <code>asyncio.run</code>. :pr:<code>1952</code></li> <li>Avoid unclosed <code>auto_aiter</code> warnings. :pr:<code>1960</code></li> <li>Return an <code>aclose</code>-able <code>AsyncGenerator</code> from <code>Template.generate_async</code>. :pr:<code>1960</code></li> <li>Avoid leaving <code>root_render_func()</code> unclosed in <code>Template.generate_async</code>. :pr:<code>1960</code></li> <li>Avoid leaving async generators unclosed in blocks, includes and extends. :pr:<code>1960</code></li> <li>The runtime uses the correct <code>concat</code> function for the current environment when calling block references. :issue:<code>1701</code></li> <li>Make <code>|unique</code> async-aware, allowing it to be used after another async-aware filter. :issue:<code>1781</code></li> <li><code>|int</code> filter handles <code>OverflowError</code> from scientific notation. :issue:<code>1921</code></li> <li>Make compiling deterministic for tuple unpacking in a <code>{% set ... %}</code> call. :issue:<code>2021</code></li> <li>Fix dunder protocol (<code>copy</code>/<code>pickle</code>/etc) interaction with <code>Undefined</code> objects. :issue:<code>2025</code></li> <li>Fix <code>copy</code>/<code>pickle</code> support for the internal <code>missing</code> object. :issue:<code>2027</code></li> <li><code>Environment.overlay(enable_async)</code> is applied correctly. :pr:<code>2061</code></li> <li>The error message from <code>FileSystemLoader</code> includes the paths that were searched. :issue:<code>1661</code></li> <li><code>PackageLoader</code> shows a clearer error message when the package does not contain the templates directory. :issue:<code>1705</code></li> <li>Improve annotations for methods returning copies. :pr:<code>1880</code></li> <li><code>urlize</code> does not add <code>mailto:</code> to values like <code>@A@b</code>. :pr:<code>1870</code></li> <li>Tests decorated with <code>@pass_context`` can be used with the ``|select`` filter. :issue:</code>1624`</li> <li>Using <code>set</code> for multiple assignment (<code>a, b = 1, 2</code>) does not fail when the target is a namespace attribute. :issue:<code>1413</code></li> <li>Using <code>set</code> in all branches of <code>{% if %}{% elif %}{% else %}</code> blocks does not cause the variable to be considered initially undefined. :issue:<code>1253</code></li> </ul> <!-- raw HTML omitted --> </blockquote> <p>... (truncated)</p> </details> <details> <summary>Commits</summary> <ul> <li><a href="https://github.com/pallets/jinja/commit/877f6e51be8e1765b06d911cfaa9033775f051d1"><code>877f6e5</code></a> release version 3.1.5</li> <li><a href="https://github.com/pallets/jinja/commit/8d588592653b052f957b720e1fc93196e06f207f"><code>8d58859</code></a> remove test pypi</li> <li><a href="https://github.com/pallets/jinja/commit/eda8fe86fd716dfce24910294e9f1fc81fbc740c"><code>eda8fe8</code></a> update dev dependencies</li> <li><a href="https://github.com/pallets/jinja/commit/c8fdce1e0333f1122b244b03a48535fdd7b03d91"><code>c8fdce1</code></a> Fix bug involving calling set on a template parameter within all branches of ...</li> <li><a href="https://github.com/pallets/jinja/commit/66587ce989e5a478e0bb165371fa2b9d42b7040f"><code>66587ce</code></a> Fix bug where set would sometimes fail within if</li> <li><a href="https://github.com/pallets/jinja/commit/fbc3a696c729d177340cc089531de7e2e5b6f065"><code>fbc3a69</code></a> Add support for namespaces in tuple parsing (<a href="https://redirect.github.com/pallets/jinja/issues/1664">#1664</a>)</li> <li><a href="https://github.com/pallets/jinja/commit/b8f4831d41e6a7cb5c40d42f074ffd92d2daccfc"><code>b8f4831</code></a> more comments about nsref assignment</li> <li><a href="https://github.com/pallets/jinja/commit/ee832194cd9f55f75e5a51359b709d535efe957f"><code>ee83219</code></a> Add support for namespaces in tuple assignment</li> <li><a href="https://github.com/pallets/jinja/commit/1d55cddbb28e433779511f28f13a2d8c4ec45826"><code>1d55cdd</code></a> Triple quotes in docs (<a href="https://redirect.github.com/pallets/jinja/issues/2064">#2064</a>)</li> <li><a href="https://github.com/pallets/jinja/commit/8a8eafc6b992ba177f1d3dd483f8465f18a11116"><code>8a8eafc</code></a> edit block assignment section</li> <li>Additional commits viewable in <a href="https://github.com/pallets/jinja/compare/3.1.3...3.1.5">compare view</a></li> </ul> </details> <br /> [](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores) Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@dependabot rebase`. [//]: # (dependabot-automerge-start) [//]: # (dependabot-automerge-end) --- <details> <summary>Dependabot commands and options</summary> <br /> You can trigger Dependabot actions by commenting on this PR: - `@dependabot rebase` will rebase this PR - `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it - `@dependabot merge` will merge this PR after your CI passes on it - `@dependabot squash and merge` will squash and merge this PR after your CI passes on it - `@dependabot cancel merge` will cancel a previously requested merge and block automerging - `@dependabot reopen` will reopen this PR if it is closed - `@dependabot close` will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually - `@dependabot show <dependency name> ignore conditions` will show all of the ignore conditions of the specified dependency - `@dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself) You can disable automated security fix PRs for this repo from the [Security Alerts page](https://github.com/bacalhau-project/bacalhau/network/alerts). </details> Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Jamil Shamy <[email protected]>
<!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit - **Chores** - Updated the `.gitignore` file to include new entries for ignoring `.flox` and all files in the `webui/compiled` directory. <!-- end of auto-generated comment: release notes by coderabbit.ai --> Co-authored-by: Jamil Shamy <[email protected]>
# Add License Validation using JWT and JWKS
This PR introduces a robust license validation system using JSON Web
Tokens (JWT) and JSON Web Key Sets (JWKS). The implementation provides
cryptographic verification of license tokens using public key
cryptography, specifically supporting both RSA and ECDSA algorithms.
## Overview
The license validation system is built on two modern standards:
1. JWT (JSON Web Tokens) for the license token format
2. JWKS (JSON Web Key Sets) for public key distribution
License tokens are cryptographically signed JWTs that contain claims
about the license, such as:
- Features enabled
- Usage limitations
- Expiration dates
- License metadata
- Customer ID
- License ID
Example License Data:
```json
{
"product": "Bacalhau",
"license_id": "license_1",
"license_type": "prod_tier_1",
"license_version": "v_1",
"customer_id": "customer_1",
"capabilities": {"tier_1": "no"},
"metadata": {"something1": "something1_value"}
}
```
The validation system uses JWKS to maintain a set of trusted public keys
that can verify these license tokens. Each key in the JWKS has a unique
key ID (kid), allowing for key rotation and multiple signing
authorities.
## Implementation Details
- Uses `github.com/golang-jwt/jwt/v5` for JWT handling
- Uses `github.com/MicahParks/keyfunc/v3` for JWKS processing
- Support for both RSA and ECDSA signing algorithms
- Custom claims validation for license-specific requirements
Example usage:
```go
// Initialize validator with embedded JWKS
validator, err := license.NewLicenseValidatorFromJSON(embeddedJWKS)
if err != nil {
log.Fatalf("Failed to create validator: %v", err)
}
// Validate a license token
claims, err := validator.ValidateToken(licenseToken)
if err != nil {
log.Fatalf("License validation failed: %v", err)
}
// Access license features and limitations
fmt.Printf("Licensed capabilities: %v\n", claims.Capabilities)
```
## Future Extensibility
While the current implementation embeds the JWKS in the binary, the
system is designed to be easily extended to fetch JWKS from a well-known
URL endpoint. This follows the standard practice used in OAuth2 and
OpenID Connect where JWKs are typically available at
`/.well-known/jwks.json`.
This future enhancement would enable:
- Dynamic key rotation
- Real-time key revocation
- Multiple signing authorities
- Zero-downtime key updates
## Security Considerations
- Uses industry-standard cryptographic algorithms (RSA/ECDSA)
- No secret keys stored in the binary (only public keys)
- Support for key rotation via key IDs
- Validation of all JWT claims including expiration and issuance time
- Protection against common JWT attacks through proper validation
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
## Release Notes
- **New Features**
- Enhanced license validation with support for JSON Web Token (JWT) and
JSON Web Key Sets (JWKS)
- Added robust token verification mechanisms with detailed claim
validation
- **Dependencies**
- Updated JWT library to latest version
- Added new dependencies for improved token and key management
- **Documentation**
- Added comprehensive README instructions for license validation and
token generation
- Included code examples for initializing license validators and
handling token verification
- **Tests**
- Introduced extensive unit tests for license validation functionality
- Covered various scenarios including token expiration, claim
validation, and error handling
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
This PR introduces the foundation for partitioned execution support in bacalhau's orchestrator, allowing the scheduler to track and maintain partition assignments for parallel executions. This is the first step towards full partitioned execution support, focusing only on the orchestrator's ability to track partitions. ## Changes - Added `PartitionIndex` field to `Execution` model to track which partition (0 to job.Count-1) an execution belongs to - Modified scheduler to: - Assign unique partition indices during scheduling - Maintain partition assignments across retries/rescheduling - Handle approval/rejection independently per partition - No changes to compute execution or engine - this purely adds orchestrator-side tracking ## Feature Details When a job specifies Count > 1, the scheduler will now: 1. Create N partitions (0 to N-1) based on job.Count 2. Track which execution belongs to which partition 3. Ensure exactly one active execution per partition 4. Preserve partition assignments during rescheduling/retries ## Future Work Next steps will include: 1. Exposing partition information to compute nodes via environment variables 2. Modifying engine to populate partition environment ## Testing Added comprehensive test coverage for partition tracking: 1. Verified partition assignment and maintenance 2. Tested per-partition scheduling decisions 3. Validated failure recovery with partition preservation 4. Confirmed batch vs service job partition handling The PR sets up the foundation for partition-aware execution, which will be built upon in future PRs that modify the compute side. <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit ## Release Notes - **New Features** - Added support for job execution partitioning across the scheduler - Introduced partition index tracking for job executions - **Improvements** - Enhanced job scheduling logic to handle parallel and partitioned job executions - Improved execution management with more granular tracking of job states - Updated test suites to validate new partitioning capabilities - **Internal Changes** - Refactored scheduler components to support more flexible execution tracking - Modified execution state management methods <!-- end of auto-generated comment: release notes by coderabbit.ai --> --------- Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com>
This PR introduces two key features: ## 1. Enhanced Execution Environment Variables Added support for passing rich job metadata to execution engines via environment variables including: - `BACALHAU_PARTITION_INDEX`: Current partition index (0 to N-1) - `BACALHAU_PARTITION_COUNT`: Total number of partitions - `BACALHAU_JOB_ID`: Unique job identifier - `BACALHAU_JOB_NAME`: User-provided job name - `BACALHAU_JOB_NAMESPACE`: Job namespace - `BACALHAU_JOB_TYPE`: Job type (Batch/Service) - `BACALHAU_EXECUTION_ID`: Unique execution identifier - `BACALHAU_NODE_ID`: ID of executing compute node This allows jobs to: - Be partition-aware and handle their specific partition's work - Access their execution context - Track node assignment ## 2. Test Suite for Partition Scheduling Added comprehensive test suite that validates: - Environment variable propagation to executors - Partition scheduling behavior: - Unique partition indices - Node distribution - Retry behavior - Service job continuous execution <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit Here are the release notes for this pull request: **New Features** - Added environment variable management for job executions - Enhanced support for system and task-level environment variables - Improved job partitioning and execution context handling **Bug Fixes** - Fixed potential nil slice access in job task retrieval - Added validation for environment variable naming conventions **Improvements** - Streamlined executor and job handling interfaces - Added utility functions for environment variable manipulation - Enhanced test coverage for job execution scenarios **Technical Enhancements** - Refactored execution context management - Improved error handling in task and job validation - Added robust environment variable sanitization and merging capabilities <!-- end of auto-generated comment: release notes by coderabbit.ai -->
This PR makes execution information available to storage providers in preparation for enabling partitioned inputs https://linear.app/expanso/issue/ENG-520/partitioned-s3-input-source <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit ## Release Notes - **Refactor** - Updated storage preparation methods across multiple packages to include execution context. - Modified method signatures to support more comprehensive input handling. - Enhanced flexibility in the storage preparation process. - **Testing** - Updated test suites to incorporate mock execution contexts. - Improved test coverage for storage-related functionality. These changes represent a significant architectural refinement in how storage and execution contexts are managed throughout the system, focusing on more robust and context-aware storage preparation mechanisms. <!-- end of auto-generated comment: release notes by coderabbit.ai -->
This PR introduces configurable partitioning strategies for S3 input
sources, enabling distributed job executions to efficiently process
subsets of S3 objects. When a job is created with multiple executions (N
> 1), each execution is assigned a unique partition index (0 to N-1) and
will only process its designated subset of objects based on the
configured partitioning strategy.
## Motivation
- Enable parallel processing of large S3 datasets across multiple job
executions
- Allow users to control how objects are distributed based on their data
organization patterns
- Provide deterministic object distribution for reproducible results
## Features
- Multiple partitioning strategies:
- `none`: No partitioning, all objects available to all executions
(default)
- `object`: Partition by complete object key using consistent hashing
- `regex`: Partition using regex pattern matches from object keys
- `substring`: Partition based on a specific portion of object keys
- `date`: Partition based on dates found in object keys
- Hash-based partitioning using FNV-1a ensures:
- Deterministic assignment of objects to partitions
- Distribution based on the chosen strategy and input data patterns
- Robust handling of edge cases:
- Fallback to partition 0 for unmatched objects
- Proper handling of directories and empty paths
- Unicode support for substring partitioning
## Example Usage
Basic object partitioning:
```yaml
source:
type: s3
params:
bucket: mybucket
key: data/*
partition:
type: object
```
Regex partitioning with capture groups:
```yaml
source:
type: s3
params:
bucket: mybucket
key: data/*
partition:
type: regex
pattern: "data/(\d{4})/(\d{2})/.*\.csv"
```
Date-based partitioning:
```yaml
source:
type: s3
params:
bucket: mybucket
key: logs/*
partition:
type: date
dateFormat: "2006-01-02"
```
## Testing
- Unit tests covering all partitioning strategies
- Integration tests with actual S3 storage
- Edge case handling and error scenarios
- Distribution analysis with various input patterns
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
Based on the comprehensive summary of changes, here are the release
notes:
## Release Notes
- **New Features**
- Added S3 Object Partitioning system with support for multiple
partitioning strategies (Object, Regex, Substring, Date)
- Enhanced storage and compute modules to support execution-level
context
- **Improvements**
- Refined method signatures across multiple packages to include
execution context
- Updated error handling and message formatting in various storage and
compute modules
- Improved flexibility in resource calculation and bidding strategies
- **Bug Fixes**
- Updated volume size calculation methods to handle more complex input
scenarios
- Enhanced validation for storage and partitioning configurations
- **Documentation**
- Added comprehensive documentation for S3 Object Partitioning system
- Improved inline documentation for new features and method changes
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
<!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit - **Refactor** - Removed environment variable tracking for `NODE_ID`, `JOB_NAME`, and `JOB_NAMESPACE` in job execution. - Updated test cases to remove node-specific verifications and focus on partition-specific outputs. - Adjusted expected outputs in tests to reflect changes in environment variable priorities. <!-- end of auto-generated comment: release notes by coderabbit.ai -->
) This PR removes the dependency on kubectl's template package by providing a simplified implementation for CLI help text formatting. Binary size reduced from `92M` to `81M`. The new implementation: - Removes i18n support to reduce complexity - Drops markdown processing (blackfriday) dependency - Keeps the core formatting functionality for command help text and examples - Maintains heredoc support for clean multiline strings - Includes comprehensive tests The result is a lighter, more focused package that handles just what we need for CLI help formatting while removing a heavy dependency. Original inspiration from kubectl is credited in the package documentation. <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit - **Refactor** - Removed internationalization (i18n) support from CLI commands - Updated template handling to use a custom local package - Simplified string definitions for command descriptions and examples - Reduced external dependencies related to Kubernetes libraries - **Chores** - Cleaned up module dependencies in `go.mod` - Removed unused Kubernetes-related packages - **New Features** - Introduced a new `templates` utility package for CLI text formatting <!-- end of auto-generated comment: release notes by coderabbit.ai -->
<!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit - **Documentation** - Updated terminology from "consistent hashing" to "deterministic hashing" in S3 object partitioning documentation - Added guidance on date format verification for timezone scenarios in best practices section <!-- end of auto-generated comment: release notes by coderabbit.ai -->
# Add `bacalhau license inspect` Command
## Summary
Added a new CLI command `bacalhau license inspect` that allows users to
inspect and validate Bacalhau license files. The command supports both
offline validation and multiple output formats (table, JSON, YAML).
## Features
- New `bacalhau license inspect` command with the following
capabilities:
- Validates license file authenticity using RSA public key verification
- Displays license details including product name, license ID, customer
ID, validity period, and capabilities
- Supports offline validation using embedded JWKS public keys
- Multiple output formats: table (default), JSON, and YAML
- Includes metadata field in JSON/YAML output formats
## Implementation Details
- Added `inspect.go` implementing the license inspection command
- Integrated with existing license validation framework
- Added `NewOfflineLicenseValidator` with hardcoded JWKS verification
keys for offline validation
- Comprehensive test coverage including:
- Unit tests for various license scenarios
- Integration tests for CLI functionality
- Tests for different output formats
- Invalid license handling
## Usage Examples
```bash
# Basic inspection
bacalhau license inspect license.json
# JSON output
bacalhau license inspect license.json --output=json
# YAML output
bacalhau license inspect license.json --output=yaml
```
## Example output
```
Product = Bacalhau
License ID = e66d1f3a-a8d8-4d57-8f14-00722844afe2
Customer ID = test-customer-id-123
Valid Until = 2045-07-28
Version = v1
Capabilities = max_nodes=1
Metadata = {}
```
## Test Coverage
- Unit tests covering:
- Valid/invalid license validation
- Various output formats
- Error handling scenarios
- Offline validation
- Integration tests verifying CLI functionality
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
## Release Notes
- **New Features**
- Added a new `license` command to the CLI for inspecting and validating
local license information.
- Introduced functionality to inspect license details with support for
JSON and YAML output formats.
- Added new test files for various license scenarios, including valid
and invalid licenses.
- **Testing**
- Enhanced test coverage for license validation and inspection,
including offline validation scenarios.
- Added integration tests for local license validation scenarios.
- **Improvements**
- Implemented offline license validation.
- Refined error messaging for license-related operations.
- **Configuration**
- Updated configuration files to include new settings for orchestrator
and API.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
See Commits and Changes for more details.
Created by
pull[bot]
Can you help keep this open source service alive? 💖 Please sponsor : )