Feat: Verification with MerkleDAG

[hopeyen]

Problem statement

To use File CID directly, we need to add additional interaction between the data provider and consumers.

Expectation proposal

Data Structure for MerkleDAG

• A HashMap of File CID to the corresponding DAG
• A recursive structure to represent DAG. A block, identified by block hash, also contains a list of blocks, potentially an enum that is a parent block, or data content (leave blocks)
• Function to return DAG without leave blocks
• Function to return leave blocks by File CID and position

File Hasher

• Create MerkleDAG for files, matching the same schemes as IPFS files
• Validate a DAG from root to all children blocks
• Validate specific data content to a specific block of the DAG

File Service

• Expose a function or a dedicated endpoint for providing DAG to clients for a File CID (serialize in json object)
• Returns specific data content for a request that contains a File CID and a range

Downloader

• Given a File CID, request for the DAG to available file services
• Once receiving a DAG, de-serialize it and validate top to bottom without leave blocks
• For each request, predetermine the position, data range, and the correct hash from DAG
• For each content response, validate the content with respect to the correct hash

Make sure that File CID created matches exactly with IPFS CID v0

Alternative considerations
Reference existing implementation