Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add documentation for the Indexed Merkle Map API #1088

Open
wants to merge 2 commits into
base: main
Choose a base branch
from
+270 −3
Open

Add documentation for the Indexed Merkle Map API #1088

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
872e59d
Add documentation for the Indexed Merkle Map API
Shigoto-dev19 Jan 10, 2025
aa74239
Add note about height
Shigoto-dev19 Jan 14, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
268 changes: 268 additions & 0 deletions docs/zkapps/o1js/indexed-merkle-map.mdx
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,268 @@
---
title: Indexed Merkle Map
hide_title: true
description:
A comprehensive guide on how to use Indexed Merkle Map to reference off-chain data in zkApps on Mina.
- zkapp
- o1js
- merkle map
- merkle tree
- zkapps
- mina blockchain
- storage
- off-chain data
- blockchain technology
- data structures
---

:::experimental

The Indexed Merkle Map API is currently an experimental feature.

:::

# Indexed Merkle Map

Similar to a Merkle Tree, a Merkle Map allows referencing off-chain data by storing a single hash, also known as the root.

A Merkle Map is a wrapper around a [Merkle Tree](/zkapps/o1js/merkle-tree). Both data structures are analogous, but instead of using an index to set a leaf in a tree, a Merkle Map uses a key in a map.

## Design

The Indexed Merkle Map is an improved version of the [MerkleMap](/zkapps/tutorials/common-types-and-functions#merkle-map), offering enhanced efficiency and usability:
45930 marked this conversation as resolved.
Show resolved Hide resolved

- **Reduced Constraints:** Uses 4-8x fewer constraints than `MerkleMap`.
- **Support for Dummy Updates:** Handles dummy updates for keys like `0` and `-1`.
- **Provable Code Integration:** Unlike `MerkleTree` and `MerkleMap`, the high-level API of `IndexedMerkleMap` is usable within provable code.
- **Comprehensive Methods:** Comes with a rich set of methods that enable a wide range of operations.

:::note

The `Indexed Merkle Map` can have a height of at most `52`, whereas the `Merkle Map` has a larger height fixed at `256`.
:::

## Utilizing Indexed Merkle Map

### Prerequisites

The `IndexedMerkleMap` API is accessible within the `Experimental` namespace. To use the API, import `Experimental` from o1js version 1.5.0 or higher.

```ts
import { Experimental } from 'o1js';

const { IndexedMerkleMap } = Experimental;
```

### Instantiating an Indexed Merkle Map

Given a height, you can instantiate an Indexed Merkle Map by extending the base class.
The height determines the capacity of the map; the maximum number of leaf nodes it can contain.

```ts
const height = 31;
class IndexedMerkleMap31 extends IndexedMerkleMap(height) {}
```

In this example, `IndexedMerkleMap31` is a Merkle map capable of holding up to 2<sup>(31−1)</sup> leaves; approximately 1 billion entries.

### Utilizing IndexedMerkleMap in a smart contract

Developers can integrate the `IndexedMerkleMap` into their zkApps by passing a Merkle map as a method parameter and invoking its methods as needed.

This is possible because the `IndexedMerkleMap` can be used within provable code.

```ts
class MyContract extends SmartContract {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would rather write new examples using zkProgram since we have more confidence that it will not be changed soon. SmartContract is more likely to be modified by the new API.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Regardless of documentation maintainability, I think that demonstrating a SmartContract example for the Indexed Merkle Map API provides greater benefits for developers and serves as more effective documentation.

When someone wants to use the Indexed Merkle Map API, they will likely integrate it directly within a smart contract, as it works seamlessly in provable code.

In contrast, a ZkProgram example might lead readers to mistakenly believe that the API should be used with a ZkProgram. This misunderstanding could imply that verifying a zkProgram proof for Indexed Merkle Map updates is straightforward, when in fact it is only practical in advanced scenarios such as recursive reducers.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This misunderstanding could imply that verifying a zkProgram proof for Indexed Merkle Map updates is straightforward, when in fact it is only practical in advanced scenarios such as recursive reducers.

Maybe I don't understand this myself. What's wrong with this example:

import { Experimental, Field, SelfProof, ZkProgram } from 'o1js';

const height = 5;
class MyMerkleMap extends Experimental.IndexedMerkleMap(height) {}

const RecursiveProgram = ZkProgram({
  name: 'RecursiveProgram',
  publicOutput: MyMerkleMap,
  methods: {
    init: {
      privateInputs: [],
      method: async () => {
        const map = new MyMerkleMap();
        return { publicOutput: map };
      },
    },
    insert: {
      privateInputs: [SelfProof, Field, Field],
      method: async (
        proof: SelfProof<undefined, MyMerkleMap>,
        key: Field,
        value: Field
      ) => {
        proof.verify();
        const newMap = proof.publicOutput.clone();
        newMap.insert(key, value);
        return { publicOutput: newMap };
      },
    },
    update: {
      privateInputs: [SelfProof, Field, Field],
      method: async (
        proof: SelfProof<undefined, MyMerkleMap>,
        key: Field,
        value: Field
      ) => {
        proof.verify();
        const newMap = proof.publicOutput.clone();
        newMap.update(key, value);
        return { publicOutput: newMap };
      },
    },
  },
});

await RecursiveProgram.compile();

class RecursiveProof extends ZkProgram.Proof(RecursiveProgram) {}

const p = await RecursiveProgram.init();
const p2 = await RecursiveProgram.insert(p.proof, Field(1), Field(2));
const p3 = await RecursiveProgram.update(p2.proof, Field(1), Field(3));

console.log(p3.proof.publicOutput.get(Field(1))); // 3

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There's nothing wrong with the example but I think whoever wants to use the Indexed MM would use it directly within a zkApp as it's more practical and for that, I think a SmartContract example would be a better guide for the reader and less confusing.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think whoever wants to use the Indexed MM would use it directly within a zkApp as it's more practical

I'm not sold on this. Do you actually believe indexed merkle map is a usable solution for zkapps today? To me, your example is misleading because it teases the existence of a map API that works on Mina, but doesn't address the same old concurrent update or state storage issues that exist for every zkapp. In production, this example would not work with multiple users accessing it at once, and I want to stop creating new examples that break down at scale.

I know that this is a touchy issue in the community right now as protokit devs are looking for L1 alternatives to the StateMap so that they can deploy on mainnet: (discord example). IMO there is absolutely no alternative that currently works, and I don't want to beat around the bush with that fact and pretend this is a viable alternative. Do you disagree with this?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fair enough, I don't disagree!

I see your point here! But that's also applicable to many primitives in the documentation. I was trying to keep it consistent with the current o1js docs as I was documenting the Indexed Merkle Map API.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

My goal with example code in the docs is for developers to grasp the concept and then take it to build one step further. My problem with the pattern we have in most of our existing documentation is that it only works to a certain point, and that's what we show. When someone tries to take the next step, they hit a wall.

What I like about ZkProgram is that the final product can be a Mina ZkApp, or it could live on protokit or zeko. And before deploying it to any network, a developer can change different variables, combine it with other ZkPrograms, and extend it in all kinds of ways locally.

@state(Field) mapRoot = State<Field>();

init() {
super.init();
this.mapRoot.set(new IndexedMerkleMap31().root);
}

@method async insertNewLeaf(
newKey: Field,
newValue: Field,
indexedMerkleMap: IndexedMerkleMap31
) {
// Validate the integrity of the Merkle Map input
const currentRoot = this.mapRoot.getAndRequireEquals();
currentRoot.assertEquals(
indexedMerkleMap.root,
'Off-chain Indexed Merkle Map is out of sync!'
);

// Insert a new key-value pair
indexedMerkleMap = indexedMerkleMap.clone();
indexedMerkleMap.insert(newKey, newValue);

// Update the on-chain map root
const newMapRoot = indexedMerkleMap.root;
this.mapRoot.set(newMapRoot);
}

@method async updateExistingLeaf(
key: Field,
newValue: Field,
indexedMerkleMap: IndexedMerkleMap31
) {
// Validate integrity of the Merkle Map input
const currentRoot = this.mapRoot.getAndRequireEquals();
currentRoot.assertEquals(
indexedMerkleMap.root,
'Off-chain Indexed Merkle Map is out of sync!'
);

indexedMerkleMap = indexedMerkleMap.clone();

/**
* Proves that the key exists
* Updates an existing leaf
* Returns the previous value
*/
indexedMerkleMap.update(key, newValue);

// Update the on-chain map root
const newMapRoot = indexedMerkleMap.root;
this.mapRoot.set(newMapRoot);
}
}
```

:warning: Direct modification of a method input can lead to errors. To avoid this, perform changes on a cloned Merkle Map rather than altering the map passed as a method input.

### Interacting with a smart contract utilizing an indexed Merkle Map

To interact with a zkapp that utilizes an `IndexedMerkleMap`, instantiate a map instance and pass it as an argument to the contract’s method within a Mina transaction.

In addition, ensure you synchronize your off-chain map with on-chain updates to maintain data integrity and prepare for subsequent interactions.

```ts
// Instantiate a new Indexed Merkle Map
indexedMerkleMap = new IndexedMerkleMap31();

const insertTx = await Mina.transaction(userPubKey, async () => {
await contract.insertNewLeaf(Field(1), Field(1234), indexedMerkleMap);
});

await insertTx.prove();
await insertTx.sign([userKey]).send();

// Synchornize the off-chain Indexed Merkle Map to match the on-chain state root
indexedMerkleMap.insert(Field(1), Field(1234));

console.log(
indexedMerkleMap.root.toBigInt() === contract.mapRoot.get().toBigInt()
);
console.log(indexedMerkleMap.data.get().sortedLeaves);

const updateTx = await Mina.transaction(userPubKey, async () => {
await contract.updateExistingLeaf(Field(1), Field(5678), indexedMerkleMap);
});

await updateTx.prove();
await updateTx.sign([userKey]).send();

// Synchronize the off-chain Indexed Merkle Map to match the on-chain state root
indexedMerkleMap.update(Field(1), Field(5678));

console.log(
indexedMerkleMap.root.toBigInt() === contract.mapRoot.get().toBigInt()
);
console.log(indexedMerkleMap.data.get().sortedLeaves);
```

## Indexed Merkle Map - API reference
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I feel strongly that this should not be included here. This definitely belongs in the Typedoc reference, so we should work on generating that correctly instead of manually adding auto-generated type declarations here.


The Indexed Merkle Map API provides a comprehensive set of methods that enable developers to perform a wide range of operations within provable code.

```ts
/**
* Clone the entire Merkle map.
*
* This method is provable.
*/
clone(): IndexedMerkleMapBase;
/**
* Overwrite the entire Merkle map with another one.
*
* This method is provable.
*/
overwrite(other: IndexedMerkleMapBase): void;
/**
* Overwrite the entire Merkle map with another one, if the condition is true.
*
* This method is provable.
*/
overwriteIf(condition: Bool | boolean, other: IndexedMerkleMapBase): void;
/**
* Insert a new leaf `(key, value)`.
*
* Proves that `key` doesn't exist yet.
*/
insert(key: Field | bigint, value: Field | bigint): void;
/**
* Update an existing leaf `(key, value)`.
*
* Proves that the `key` exists.
*
* Returns the previous value.
*/
update(key: Field | bigint, value: Field | bigint): Field;
/**
* Perform _either_ an insertion or update, depending on whether the key exists.
*
* Note: This method is handling both the `insert()` and `update()` case at the same time, so you
* can use it if you don't know whether the key exists or not.
*
* However, this comes at an efficiency cost, so prefer to use `insert()` or `update()` if you know whether the key exists.
*
* Returns the previous value, as an option (which is `None` if the key didn't exist before).
*/
set(key: Field | bigint, value: Field | bigint): Option<Field, bigint>;
/**
* Perform an insertion or update, if the enabling condition is true.
*
* If the condition is false, we instead set the 0 key to the value 0.
* This is the initial value and for typical uses of `IndexedMerkleMap`, it is guaranteed to be a no-op because the 0 key is never used.
*
* **Warning**: Only use this method if you are sure that the 0 key is not used in your application.
* Otherwise, you might accidentally overwrite a valid key-value pair.
*/
setIf(condition: Bool | boolean, key: Field | bigint, value: Field | bigint): Option<import("./field.js").Field, bigint>;
/**
* Get a value from a key.
*
* Proves that the key already exists in the map yet and fails otherwise.
*/
get(key: Field | bigint): Field;
/**
* Get a value from a key.
*
* Returns an option which is `None` if the key doesn't exist. (In that case, the option's value is unconstrained.)
*
* Note that this is more flexible than `get()` and allows you to handle the case where the key doesn't exist.
* However, it uses about twice as many constraints for that reason.
*/
getOption(key: Field | bigint): Option<Field, bigint>;
/**
* Prove that the given key exists in the map.
*/
assertIncluded(key: Field | bigint, message?: string): void;
/**
* Prove that the given key does not exist in the map.
*/
assertNotIncluded(key: Field | bigint, message?: string): void;
/**
* Check whether the given key exists in the map.
*/
isIncluded(key: Field | bigint): Bool;
```

## Additional Resources

For more details and examples, please refer to the following GitHub resources:

- [Indexed Merkle Tree: o1js PR#1666](https://github.com/o1-labs/o1js/pull/1666)
- [IndexedMerkleMap: Support 0 and -1 Keys: o1js PR#1671](https://github.com/o1-labs/o1js/pull/1671)
- [Mastermind zkApp Example Using Indexed Merkle Map](https://github.com/o1-labs-XT/mastermind-zkApp/tree/level3)
5 changes: 2 additions & 3 deletions sidebars.js
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -75,6 +75,7 @@ module.exports = {
'zkapps/o1js/bitwise-operations',
'zkapps/o1js/foreign-fields',
'zkapps/o1js/merkle-tree',
'zkapps/o1js/indexed-merkle-map',
'zkapps/o1js/keccak',
'zkapps/o1js/ecdsa',
'zkapps/o1js/sha256',
Expand Down Expand Up @@ -150,9 +151,7 @@ module.exports = {
type: 'doc',
id: 'zkapps/front-end-integration-guides/angular',
},
items: [
'zkapps/front-end-integration-guides/angular',
],
items: ['zkapps/front-end-integration-guides/angular'],
},
{
type: 'category',
Expand Down