Assembler: enable vendoring of compiled libraries (fixes #1435) #1643

paracetamolo · 2025-01-27T14:58:51Z

Before this MR, the only ways to link a library during assembly are:

Add the module sources. The library is effectively recompiled as part of the new library/program
Add a compiled library. In this case all references to this library are compiled as external nodes

This MR adds the possibility to link a compiled library during assembly but have its MAST forest be merged in the resulting library/program. Two desirable properties:

No external nodes are added for procedures that are present in vendored libraries, which improves performance.
No unused procedures are present in the resulting program/library. If the vendored library contains extra unused procedures they are removed during assembly.

bobbinth

Looks good! Thank you. Not a full review yet, but I left a few comments inline - one of them describing a potential alternative approach.

assembly/src/assembler/mod.rs

paracetamolo · 2025-01-30T14:07:18Z

I pushed a new version that uses the approach mentioned by @bobbinth in the comment above. At assembly time, we merge all the vendored libraries collected into a single MAST forest that is passed to the builder. On a call to ensure_external the builder first checks into the vendored mast if the procedure is present and it that case it will copy the its subtree.

bobbinth

Looks good! Thank you! Not a full review, but I left some comments inline.

Overall, it feels like this approach should work better than the previous one.

bobbinth · 2025-01-31T04:36:05Z

assembly/src/assembler/mod.rs

+    pub fn add_vendored_library(&mut self, library: impl AsRef<Library>) -> Result<(), Report> {
+        self.add_library(&library)?;
+        self.vendored_libraries
+            .insert(*library.as_ref().digest(), library.as_ref().clone());
+        Ok(())
+    }
+
+    pub fn with_vendored_library(mut self, library: impl AsRef<Library>) -> Result<Self, Report> {
+        self.add_vendored_library(library)?;
+        Ok(self)
+    }


Let's add doc comments to these functions.

I think add_library and add_vendored_library are a bit confusing names, especially the former. Would it make sense to rename them dynamically_link and statically_link ?

I don't mind renaming, but I'm not sure dynamically_link() and statically_link() are better options. Curious what others think though.

assembly/src/assembler/mod.rs

assembly/src/assembler/mast_forest_builder.rs

core/src/mast/mod.rs

paracetamolo · 2025-01-31T10:55:59Z

Are there more interesting tests that we could run? Right now there is a single test checking that a used procedure is inlined and an unused one is deleted.

bobbinth

Looks good! Thank you. I reviewed pretty much all non-test and on-CLI code and left some comments inline.

bobbinth · 2025-02-01T06:55:10Z

assembly/src/assembler/mast_forest_builder.rs

+    pub fn new<'a>(
+        vendored_libraries: impl IntoIterator<Item = &'a Library>,
+    ) -> Result<Self, Report> {


Let's add doc comments to this function.

Also, nit: I'd move it above the build() function (e.g., above line 79).

bobbinth · 2025-02-01T07:07:01Z

core/src/mast/mod.rs

+/// Iterates over all the nodes a root depends on, in pre-order.
+/// The iteration can include other roots in the same forest.
+pub struct RootIterator<'a> {
+    forest: &'a MastForest,
+    discovered: Vec<MastNodeId>,
+    unvisited: Vec<MastNodeId>,
+}
+impl<'a> RootIterator<'a> {
+    pub fn new(root: &MastNodeId, forest: &'a MastForest) -> Self {
+        let discovered = vec![];
+        let unvisited = vec![*root];
+        RootIterator { forest, discovered, unvisited }
+    }
+}
+impl Iterator for RootIterator<'_> {
+    type Item = MastNodeId;
+    fn next(&mut self) -> Option<MastNodeId> {
+        while let Some(id) = self.unvisited.pop() {
+            let mut children = self.forest[id].children();
+            if children.is_empty() {
+                return Some(id);
+            };
+            self.discovered.push(id);
+            self.unvisited.append(&mut children);
+        }
+        self.discovered.pop()
+    }
+}


nit: it feels a bit weird to have this defined between MastForest impl blocks. I would probably move down to line 657 and add a section separator there.

Also, RootIterator feels like we are iterating over roots. Maybe it RootSubtreeIterator or something similar would be more clear?

bobbinth · 2025-02-01T07:09:07Z

core/src/mast/mod.rs

+    pub fn remap(self, remapping: &Remapping) -> Self {
+        *remapping.get(&self).unwrap_or(&self)
+    }


Let's add doc comments to this function.

bobbinth · 2025-02-01T07:15:37Z

core/src/mast/node/mod.rs

+        }
+    }
+
+    pub fn children(&self) -> Vec<MastNodeId> {


This will result on a vector allocation on every invocation of this method. Looking at how this method is used, a more efficient alternative could be:

pub fn append_children_to(&self, target: &mut Vec<MastNodeId>) { ... }

bobbinth · 2025-02-01T07:15:54Z

core/src/mast/node/mod.rs

@@ -143,6 +143,34 @@ impl MastNode {
        }
    }

+    pub fn remap(&mut self, remapping: &BTreeMap<MastNodeId, MastNodeId>) {


Let's add doc comments to this function.

bobbinth · 2025-02-01T07:19:49Z

core/src/mast/mod.rs

+            let mut children = self.forest[id].children();
+            if children.is_empty() {
+                return Some(id);
+            };
+            self.discovered.push(id);
+            self.unvisited.append(&mut children);


Related to the previous comment, we could re-write this as:

let node = self.forest[id]; if !node.has_children() { return Some(id); } else { self.discovered.push(id); node.append_children_to(&mut self.unvisited); }

bobbinth · 2025-02-01T07:22:07Z

core/src/mast/mod.rs

    pub fn remove_nodes(
        &mut self,
        nodes_to_remove: &BTreeSet<MastNodeId>,
-    ) -> Option<BTreeMap<MastNodeId, MastNodeId>> {
+    ) -> BTreeMap<MastNodeId, MastNodeId> {


Is this change needed for this PR?

bobbinth · 2025-02-01T07:28:15Z

assembly/src/assembler/mod.rs

+    /// Adds a compiled library that can be used to copy procedures during assembly instead of
+    /// introducing external nodes.
+    pub fn add_vendored_library(&mut self, library: impl AsRef<Library>) -> Result<(), Report> {


I would write this as:

/// Adds a compiled library procedures from which will be vendored into the assembled code. /// /// Vendoring in this context means that when a procedure from this library is invoked from the /// assembled code, the entire procedure MAST will be copied into the assembled code. Thus, /// when the resulting code is executed on the VM, the vendored library does not need to be /// provided to the VM to resolve external calls.

We should also update the description of add_library() procedure above.

bobbinth · 2025-02-01T07:29:00Z

assembly/src/assembler/mod.rs

+    /// See [`Self::add_vendored_library`]
+    pub fn with_vendored_library(mut self, library: impl AsRef<Library>) -> Result<Self, Report> {


I would write this as:

/// Adds a compiled library procedures from which will be vendored into the assembled code. /// /// See [`Self::add_vendored_library`]

bobbinth · 2025-02-01T07:40:43Z

assembly/src/assembler/mast_forest_builder.rs

+            for old_id in RootIterator::new(&root_id, &self.vendored_mast.clone()) {
+                let mut node = self.vendored_mast[old_id].clone();
+                node.remap(&self.vendored_remapping);
+                let new_id = self.ensure_node(node)?;
+                self.vendored_remapping.insert(old_id, new_id);
+            }


I think this will copy the node + its decorators. But where do we copy the advice map data from the vendored libraries?

bobbinth · 2025-02-01T07:44:27Z

@plafer, @bitwalker - could you also take a look at this PR?

paracetamolo self-assigned this Jan 27, 2025

paracetamolo changed the title ~~Marco vendoring~~ Assembler: enable vendoring of compiled libraries (fixes #1435) Jan 27, 2025

paracetamolo added the assembly Related to Miden assembly label Jan 27, 2025

paracetamolo force-pushed the marco-vendoring branch from 129d91e to 437a846 Compare January 27, 2025 15:08

paracetamolo mentioned this pull request Jan 27, 2025

Implement support for vendoring procedures in the assembler #1435

Open

bobbinth requested review from plafer and bitwalker January 28, 2025 06:13

bobbinth reviewed Jan 28, 2025

View reviewed changes

assembly/src/assembler/mod.rs Outdated Show resolved Hide resolved

assembly/src/assembler/mod.rs Outdated Show resolved Hide resolved

assembly/src/assembler/mod.rs Outdated Show resolved Hide resolved

paracetamolo force-pushed the marco-vendoring branch 3 times, most recently from f73998a to 806a91e Compare January 30, 2025 14:03

bobbinth reviewed Jan 31, 2025

View reviewed changes

paracetamolo force-pushed the marco-vendoring branch from d45855f to 825057c Compare January 31, 2025 10:01

paracetamolo closed this Jan 31, 2025

paracetamolo force-pushed the marco-vendoring branch from 825057c to ea39c05 Compare January 31, 2025 10:02

paracetamolo reopened this Jan 31, 2025

paracetamolo added 13 commits January 31, 2025 11:08

fix(cli): add support for non-vendored libraries to run command

9e3c498

refactor(Assembler): minor leftovers from PR #1606

77422d9

refactor(Assembler): remove_nodes do not return option

43d7366

feat(Mast): add iteration by root and retain_roots

cb2d8e6

feat(Mast): add remap to Node

5eca9a3

feat(Assembler): add support for vendoring compiled libraries

2552d53

fixup: avoid a clone

97830f9

fixup: inline function take_subtree_from_vendored_if_present

387c16f

fixup: remove unused function

5f2d502

fixup: use Arc to avoid copying the whole mast

d7bab43

fixup: new constructor for MastForestBuilder

dd1346d

fixup: docstrings

75940fc

fixup: changelog

00a4327

paracetamolo force-pushed the marco-vendoring branch from 606ea88 to 00a4327 Compare January 31, 2025 10:08

bobbinth reviewed Feb 1, 2025

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Assembler: enable vendoring of compiled libraries (fixes #1435) #1643

Assembler: enable vendoring of compiled libraries (fixes #1435) #1643

paracetamolo commented Jan 27, 2025 •

edited

Loading

bobbinth left a comment

paracetamolo commented Jan 30, 2025

bobbinth left a comment

bobbinth Jan 31, 2025

paracetamolo Jan 31, 2025

bobbinth Feb 1, 2025

paracetamolo commented Jan 31, 2025

bobbinth left a comment

bobbinth Feb 1, 2025

bobbinth Feb 1, 2025

bobbinth Feb 1, 2025

bobbinth Feb 1, 2025

bobbinth Feb 1, 2025

bobbinth Feb 1, 2025

bobbinth Feb 1, 2025

bobbinth Feb 1, 2025

bobbinth Feb 1, 2025

bobbinth Feb 1, 2025

bobbinth commented Feb 1, 2025

		/// See [`Self::add_vendored_library`]
		pub fn with_vendored_library(mut self, library: impl AsRef<Library>) -> Result<Self, Report> {

Assembler: enable vendoring of compiled libraries (fixes #1435) #1643

Are you sure you want to change the base?

Assembler: enable vendoring of compiled libraries (fixes #1435) #1643

Conversation

paracetamolo commented Jan 27, 2025 • edited Loading

bobbinth left a comment

Choose a reason for hiding this comment

paracetamolo commented Jan 30, 2025

bobbinth left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

paracetamolo commented Jan 31, 2025

bobbinth left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

bobbinth commented Feb 1, 2025

paracetamolo commented Jan 27, 2025 •

edited

Loading