chore: Merge Orchestrator history #471

notlesh · 2025-01-21T23:15:46Z

This PR pulls in the entire madara-orchestrator git history into this project.

Because the entire history was pulled in (using git merge --allow-unrelated-histories) the PR is absolutely massive (and probably difficult to review). My process was basically this:

git merge --allow-unrelated-histories (that is, pull in the entire codebase history)
Move files (crates under ~~crates/orchestrator~~ orchestrator/crates/ and project files to a temp dir)
Update Cargo.toml to build (note that orchestrator is not a member of default-members, disagree if you want!)
Fight dependency conflicts

Status 🚧

So far this builds and madara appears to be unaffected. The project files for madara-orchestrator were kept, but were moved (temporarily) to old_orchestrator_files/. This means that things like CI will not currently work.

TODO:

Migrate orchestrator workflows (Orchestrator CI Merge #472)
Ensure minimal build and test in github workflows (filter by path, etc.)
Update READMEs
Test (and fix) orchestrator build/docker/etc
Clean up and remove old_orchestrator_files/
Remove orchestrator git submodules (This should be trivial, it was previously done)

Repository forks 🍴

This required forking several repositories in order to avoid version conflicts. Each fork uses the same madara-monorepo branch name. These repos were forked:

Init

Change the external_id field to be an enum instead of a regular String

refactor to workspace

* update: added verification status to verify job * update: added verification time * update: JOB_METADATA_PROCESSING_COMPLETED_AT

* update: added operation_job_status

* chore: readme udpated: * chore: readme updated * chore(readme): re-added a few sections that were missed earlier * chore(readme): linting * chore: linting and formatting * chore: linting and formatting * chore: linting and formatting * chore: script added for dummy contract deployment, readme udpated to use dummy sharp service * update: aws_event_bridge rework * update: e2e * chore: linting-formatting and readme updated * chore: readme misinformation fixed * update: verification job timeout and polling changed to work for 25 hours * chore: comment added for older python version needed * refactor: event bridge type added * chore: linting and formatting of toml files * update: time delay for txn finality in state update increased to 30 mins --------- Co-authored-by: mohiiit <[email protected]> Co-authored-by: Heemank Verma <[email protected]>

* update: fixed block_state collection

* refactor: error handling added for the storage functions * chore: removed expect * chore: changelog * refactor: http-client refactored to handle panic better --------- Co-authored-by: mohiiit <[email protected]>

Trantorian1

Given we are moving to a monorepo, I feel like we need to rethink what files we have at the root of the project. We probably want to move any configuration files and READMEs to either crates/madara or crates/orchestrator and keep only common logic at the root.

We also need to think about how we want to run the CI for each project so that this doesn't take for ever on simple changes.

Trantorian1 · 2025-01-22T08:23:11Z

.env.example

Shouldn't this be moved to crates/orchestrator if it is related to that?

Trantorian1 · 2025-01-22T08:23:38Z

.env.test

Trantorian1 · 2025-01-22T08:24:44Z

pathfinder

Why is this here?

Trantorian1 · 2025-01-22T08:25:38Z

scripts/artifacts/README.md

Are we sure we want to strore artefacts under scripts?

Trantorian1 · 2025-01-22T08:29:31Z

.github/workflows/rust-build.yml

We probably want to figure out a way to compile only SNOS or Madara if we change only one of those no? At least I feel like testing Madara against SNOS should be a separate and last step in the CI if all other steps pass. And make it so that if changes are made to SNOS we do not run e2e on Madara.

Trantorian1 · 2025-01-22T08:30:09Z

.github/workflows/e2e-test.yml

I feel like we should tests Madara and SNOS in different CI actions

Trantorian1 · 2025-01-22T08:33:38Z

bootstrap_input.json

Imo this should also be under crates/ortchestrator. Given we are making this a monorepo we probably also want to move some Madara-specific files such as the README under crates/madara

Trantorian1 · 2025-01-22T08:36:17Z

Cargo.toml

@@ -139,7 +153,7 @@ mc-devnet = { path = "crates/madara/client/devnet" }
 m-cairo-test-contracts = { path = "crates/madara/cairo-test-contracts" }

 # Starknet dependencies
-cairo-vm = "=1.0.1"
+cairo-vm = { git = "https://github.com/Moonsong-Labs/cairo-vm", branch = "madara-monorepo" }


Instead of using forks I feel like we should be importing two separate versions of the same crate under an alias so that madara and the orchestrator can both have their own version. This feels more maintainable long term as far as supporting new features is concerned, and should also make it easier to merge both into a single dependency in the future.

apoorvsadana and others added 30 commits March 5, 2024 10:56

Initial commit

5490788

init

e320c80

add workflows

3e9b41c

cliipy fix

06e7964

more clippy fixes

6e14ef0

more clippy fixes

c81d279

remove .idea

1ee102b

fix ci

9062d21

add .DS_store to ignore

b04bfdd

remove coverage

eaf571f

Merge pull request #1 from karnotxyz/init

eea4f83

Init

change the external_id field to be an enum instead of a regular string

4a948da

Merge pull request #3 from nils-mathieu/free_external_id

b420e1d

Change the external_id field to be an enum instead of a regular String

refactor to workspace

1768f47

taplo

5dd26f8

fix typo

119e5bb

Merge pull request #4 from karnotxyz/orchestrator_refactor

3fe3733

refactor to workspace

initial commit for review

5051bec

move tests to dev dependencies

9dcd974

set up modules for testing each crate

e8eaeda

add previous unti tests

8c4a854

restructure common fixtures and helper function, add rstest cases

0c34cbd

write tests for config, and try out da, database and starknetclient

fb815f7

fix database test, and write test to check config init multiple times

f6c4c10

write job tests, create new fixtures

5fa41a8

write da_job tests, process_job remaining

f6c73c8

implement mocks for config

99b6f9d

mock starknet_client

8b3874e

Try out more approaches for mock_client

b2ca7e9

temp commit only for starknet-provider

52ec7fe

heemankv and others added 16 commits December 15, 2024 13:42

Fix/instrumentations (#192)

9e67649

* update: added verification status to verify job * update: added verification time * update: JOB_METADATA_PROCESSING_COMPLETED_AT

update: added operation_job_status (#195)

07c38a3

* update: added operation_job_status

added: grafana dashboard (#196)

302542a

Fix/instrumentation (#200)

15b321b

* update: fixed block_state collection

Expect removed from various places (#197)

cb8d82f

* refactor: error handling added for the storage functions * chore: removed expect * chore: changelog * refactor: http-client refactored to handle panic better --------- Co-authored-by: mohiiit <[email protected]>

fixed: instrumentation (#203)

3ec2873

Merge madara-orchestrator history into this repo

b9f0947

Move orchestrator crates into crates/orchestrator/

8e6bfe2

Incremental: merge orchestrator Cargo.toml into ours

8c7fb53

Use nightly rust compiler channel to work around SNOS issue

59df910

Incremental: bump Cargo.lock

3d453a2

Use consistent zaun

45b1a75

Bump Cargo.lock

1e2eede

Update dependencies to use git instead of local paths

b7c683f

Magic Cargo.lock

0d7df60

Trantorian1 assigned notlesh Jan 22, 2025

Trantorian1 requested changes Jan 22, 2025

View reviewed changes

Trantorian1 added docs Documentation changes or improvements feature Request for new feature or enhancement infrastructure CI/CD, deployment and infrastructure changes labels Jan 22, 2025

Trantorian1 requested review from HermanObst, antiyro and jbcaron January 22, 2025 08:41

notlesh added 5 commits January 22, 2025 17:07

Merge branch 'main' into notlesh/merge-in-orchestrator-repo

bd1b71b

Remove orchestrator's madara submodule

032cf5a

Move crates/madara to madara/crates

5cd7213

Move crates/orchestrator to orchestrator/crates

65f81f5

Move some orchestrator project files

454b33c

notlesh mentioned this pull request Jan 22, 2025

Orchestrator CI Merge #472

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

chore: Merge Orchestrator history #471

chore: Merge Orchestrator history #471

notlesh commented Jan 21, 2025 •

edited

Loading

Trantorian1 left a comment

Trantorian1 Jan 22, 2025

Trantorian1 Jan 22, 2025

Trantorian1 Jan 22, 2025

Trantorian1 Jan 22, 2025

Trantorian1 Jan 22, 2025 •

edited

Loading

Trantorian1 Jan 22, 2025

Trantorian1 Jan 22, 2025

Trantorian1 Jan 22, 2025

chore: Merge Orchestrator history #471

Are you sure you want to change the base?

chore: Merge Orchestrator history #471

Conversation

notlesh commented Jan 21, 2025 • edited Loading

Status 🚧

Repository forks 🍴

Trantorian1 left a comment

Choose a reason for hiding this comment

Trantorian1 Jan 22, 2025

Choose a reason for hiding this comment

Trantorian1 Jan 22, 2025

Choose a reason for hiding this comment

Trantorian1 Jan 22, 2025

Choose a reason for hiding this comment

Trantorian1 Jan 22, 2025

Choose a reason for hiding this comment

Trantorian1 Jan 22, 2025 • edited Loading

Choose a reason for hiding this comment

Trantorian1 Jan 22, 2025

Choose a reason for hiding this comment

Trantorian1 Jan 22, 2025

Choose a reason for hiding this comment

Trantorian1 Jan 22, 2025

Choose a reason for hiding this comment

notlesh commented Jan 21, 2025 •

edited

Loading

Trantorian1 Jan 22, 2025 •

edited

Loading