Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Missing perfectly divisible deposits #1821

Open
shawn-zil opened this issue Nov 14, 2024 · 9 comments · May be fixed by #1915
Open

Missing perfectly divisible deposits #1821

shawn-zil opened this issue Nov 14, 2024 · 9 comments · May be fixed by #1915
Assignees
Labels
Agate Required for mainnet launch bug Something isn't working

Comments

@shawn-zil
Copy link
Contributor

Running z2 deployer deposit on a large number of Validators doesn't always stake every node e.g. running it successfully for 35 nodes, may result in only 33 validators - but there are no errors.

[2024-11-14T06:20:01Z INFO ] SUCCESS: zq2-perftest-validator-ase1-0-0348
[2024-11-14T06:20:01Z INFO ] SUCCESS: zq2-perftest-validator-ase1-1-0348
[2024-11-14T06:20:01Z INFO ] SUCCESS: zq2-perftest-validator-ase1-10-0348
[2024-11-14T06:20:01Z INFO ] SUCCESS: zq2-perftest-validator-ase1-11-0348
[2024-11-14T06:20:01Z INFO ] SUCCESS: zq2-perftest-validator-ase1-12-0348
[2024-11-14T06:20:01Z INFO ] SUCCESS: zq2-perftest-validator-ase1-13-0348
[2024-11-14T06:20:01Z INFO ] SUCCESS: zq2-perftest-validator-ase1-2-0348
[2024-11-14T06:20:01Z INFO ] SUCCESS: zq2-perftest-validator-ase1-3-0348
[2024-11-14T06:20:01Z INFO ] SUCCESS: zq2-perftest-validator-ase1-4-0348
[2024-11-14T06:20:01Z INFO ] SUCCESS: zq2-perftest-validator-ase1-5-0348
[2024-11-14T06:20:01Z INFO ] SUCCESS: zq2-perftest-validator-ase1-6-0348
[2024-11-14T06:20:01Z INFO ] SUCCESS: zq2-perftest-validator-ase1-7-0348
[2024-11-14T06:20:01Z INFO ] SUCCESS: zq2-perftest-validator-ase1-8-0348
[2024-11-14T06:20:01Z INFO ] SUCCESS: zq2-perftest-validator-ase1-9-0348
[2024-11-14T06:20:01Z INFO ] SUCCESS: zq2-perftest-validator-ewe1-0-0348
[2024-11-14T06:20:01Z INFO ] SUCCESS: zq2-perftest-validator-ewe1-1-0348
[2024-11-14T06:20:01Z INFO ] SUCCESS: zq2-perftest-validator-ewe1-2-0348
[2024-11-14T06:20:01Z INFO ] SUCCESS: zq2-perftest-validator-ewe1-3-0348
[2024-11-14T06:20:01Z INFO ] SUCCESS: zq2-perftest-validator-ewe1-4-0348
[2024-11-14T06:20:01Z INFO ] SUCCESS: zq2-perftest-validator-ewe1-5-0348
[2024-11-14T06:20:01Z INFO ] SUCCESS: zq2-perftest-validator-ewe1-6-0348
[2024-11-14T06:20:01Z INFO ] SUCCESS: zq2-perftest-validator-ewe1-7-0348
[2024-11-14T06:20:01Z INFO ] SUCCESS: zq2-perftest-validator-ewe1-8-0348
[2024-11-14T06:20:01Z INFO ] SUCCESS: zq2-perftest-validator-ewe1-9-0348
[2024-11-14T06:20:01Z INFO ] SUCCESS: zq2-perftest-validator-ewe2-0-0348
[2024-11-14T06:20:01Z INFO ] SUCCESS: zq2-perftest-validator-ewe2-1-0348
[2024-11-14T06:20:01Z INFO ] SUCCESS: zq2-perftest-validator-ewe2-2-0348
[2024-11-14T06:20:01Z INFO ] SUCCESS: zq2-perftest-validator-ewe2-3-0348
[2024-11-14T06:20:01Z INFO ] SUCCESS: zq2-perftest-validator-ewe2-4-0348
[2024-11-14T06:20:01Z INFO ] SUCCESS: zq2-perftest-validator-ewe2-5-0348
[2024-11-14T06:20:01Z INFO ] SUCCESS: zq2-perftest-validator-ewe2-6-0348
[2024-11-14T06:20:01Z INFO ] SUCCESS: zq2-perftest-validator-ewe2-7-0348
[2024-11-14T06:20:01Z INFO ] SUCCESS: zq2-perftest-validator-ewe2-8-0348
[2024-11-14T06:20:01Z INFO ] SUCCESS: zq2-perftest-validator-ewe2-9-0348

then a second deposit

[2024-11-14T06:32:51Z INFO ] SUCCESS: zq2-perftest-validator-ase1-12-0348
[2024-11-14T06:32:51Z INFO ] SUCCESS: zq2-perftest-validator-ewe1-4-0348
[2024-11-14T06:32:51Z ERROR] FAILURE: zq2-perftest-validator-ase1-0-0348
[2024-11-14T06:32:51Z ERROR] FAILURE: zq2-perftest-validator-ase1-1-0348
[2024-11-14T06:32:51Z ERROR] FAILURE: zq2-perftest-validator-ase1-10-0348
[2024-11-14T06:32:51Z ERROR] FAILURE: zq2-perftest-validator-ase1-11-0348
[2024-11-14T06:32:51Z ERROR] FAILURE: zq2-perftest-validator-ase1-13-0348
[2024-11-14T06:32:51Z ERROR] FAILURE: zq2-perftest-validator-ase1-2-0348
[2024-11-14T06:32:51Z ERROR] FAILURE: zq2-perftest-validator-ase1-3-0348
[2024-11-14T06:32:51Z ERROR] FAILURE: zq2-perftest-validator-ase1-4-0348
[2024-11-14T06:32:51Z ERROR] FAILURE: zq2-perftest-validator-ase1-5-0348
[2024-11-14T06:32:51Z ERROR] FAILURE: zq2-perftest-validator-ase1-6-0348
[2024-11-14T06:32:51Z ERROR] FAILURE: zq2-perftest-validator-ase1-7-0348
[2024-11-14T06:32:51Z ERROR] FAILURE: zq2-perftest-validator-ase1-8-0348
[2024-11-14T06:32:51Z ERROR] FAILURE: zq2-perftest-validator-ase1-9-0348
[2024-11-14T06:32:51Z ERROR] FAILURE: zq2-perftest-validator-ewe1-0-0348
[2024-11-14T06:32:51Z ERROR] FAILURE: zq2-perftest-validator-ewe1-1-0348
[2024-11-14T06:32:51Z ERROR] FAILURE: zq2-perftest-validator-ewe1-2-0348
[2024-11-14T06:32:51Z ERROR] FAILURE: zq2-perftest-validator-ewe1-3-0348
[2024-11-14T06:32:51Z ERROR] FAILURE: zq2-perftest-validator-ewe1-5-0348
[2024-11-14T06:32:51Z ERROR] FAILURE: zq2-perftest-validator-ewe1-6-0348
[2024-11-14T06:32:51Z ERROR] FAILURE: zq2-perftest-validator-ewe1-7-0348
[2024-11-14T06:32:51Z ERROR] FAILURE: zq2-perftest-validator-ewe1-8-0348
[2024-11-14T06:32:51Z ERROR] FAILURE: zq2-perftest-validator-ewe1-9-0348
[2024-11-14T06:32:51Z ERROR] FAILURE: zq2-perftest-validator-ewe2-0-0348
[2024-11-14T06:32:51Z ERROR] FAILURE: zq2-perftest-validator-ewe2-1-0348
[2024-11-14T06:32:51Z ERROR] FAILURE: zq2-perftest-validator-ewe2-2-0348
[2024-11-14T06:32:51Z ERROR] FAILURE: zq2-perftest-validator-ewe2-3-0348
[2024-11-14T06:32:51Z ERROR] FAILURE: zq2-perftest-validator-ewe2-4-0348
[2024-11-14T06:32:51Z ERROR] FAILURE: zq2-perftest-validator-ewe2-5-0348
[2024-11-14T06:32:51Z ERROR] FAILURE: zq2-perftest-validator-ewe2-6-0348
[2024-11-14T06:32:51Z ERROR] FAILURE: zq2-perftest-validator-ewe2-7-0348
[2024-11-14T06:32:51Z ERROR] FAILURE: zq2-perftest-validator-ewe2-8-0348
[2024-11-14T06:32:51Z ERROR] FAILURE: zq2-perftest-validator-ewe2-9-0348

Could this be due to the timing of Epochs?

@shawn-zil
Copy link
Contributor Author

Check if #1852 has fixed this issue.

@DrZoltanFazekas
Copy link
Contributor

DrZoltanFazekas commented Nov 22, 2024

#1852 won't fix that. After the introduction of epochs, deposits are activated in the epoch after next i.e. it takes in best case 3601 in worst case 7199 subsequent blocks for them to become effective. If some of the 35 deposit transactions were included in blocks in epoch N and the rest of them in epoch N+1, the deposits that were submitted during epoch N+1 will only take effect a whole epoch i.e. 3600 blocks later. If this is what happened, then it's expected behavior.

@shawn-zil
Copy link
Contributor Author

shawn-zil commented Nov 29, 2024

I've encountered this issue several times in perftest, including today. So I think I'll investigate this to see what's up.

Validator zq2-perftest-validator-ase1-2-0348:                                                                                                                                                                                                                 
Deposit: add 20 M $ZIL to 12D3KooWPC4i3WPHKXaQPfYCvnxtdysKKwRvnzL7H18d7psHdMzk                                                                                                                                                                                
Sent tx: {"hash":"0x52b261a7f4c0c8c18fd1f00fda122c27b725686aaf4e40e71d972f06e22a381f","nonce":"0x2","blockHash":"0xcb0ca1bcfd03f7c7267f2a12127b8d297d247c3d67ca90c482a423aa41f7f8be","blockNumber":"0x592","transactionIndex":"0x0","from":"0x54b15e228dddbcb5
62d1573651e57f160765c3d9","to":"0x000000000000000000005a494c4445504f534954","value":"0x108b2a2c28029094000000","gasPrice":"0x454b7a4e100","gas":"0x140a22","input":"0xe12cf4cb00000000000000000000000000000000000000000000000000000000000000800000000000000000
0000000000000000000000000000000000000000000000e000000000000000000000000000000000000000000000000000000000000001400000000000000000000000006edd2f32683112a208474cecdda741f02dc632ea0000000000000000000000000000000000000000000000000000000000000030a7c3005e470a11
6869cde037c7fc7016300ac3f280eef8b18a5e7adc844ed611abbdd1f223867df6ed5d8e70a62ab244000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000026002408011220c6b6f721c97a7f22f0d45c77815e3892b4f176519461eebc16b928ba5e180a85
00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000060ae28acbb657d51afdc3947617c554058dd027864bed772d486ae1a15b7d6eece670abd9e730a0d37b6fe943909c71e1f04a43e6e52a77a3db938c3e1d4cd1345c09c56ce57
b981a1aabba951520a92133413f2baca19c6b111b3d8c88ac56ef0","v":"0x0","r":"0xbb98db054d2f95431980da350efaa85976e89062c2f2b3b9c06a3a044c829d0e","s":"0x49e3acefd9d133c33d3a12420c7fdb768afb21e382ae445fef822e55f704e9d","type":"0x0","chainId":"0x82bd"}           
                                                                                                                                                                                                                                                              
Tx receipt: {"transactionHash":"0x52b261a7f4c0c8c18fd1f00fda122c27b725686aaf4e40e71d972f06e22a381f","transactionIndex":"0x0","blockHash":"0xcb0ca1bcfd03f7c7267f2a12127b8d297d247c3d67ca90c482a423aa41f7f8be","blockNumber":"0x592","from":"0x54b15e228dddbcb5
62d1573651e57f160765c3d9","to":"0x000000000000000000005a494c4445504f534954","cumulativeGasUsed":"0x13895c","gasUsed":"0x13895c","contractAddress":null,"logs":[{"address":"0x000000000000000000005a494c4445504f534954","topics":["0xc758b38fca30d8a2d8b0de67b5
fc116c2cdc671f466eda1eaa9dc0543785bd2a"],"data":"0x000000000000000000000000000000000000000000000000000000000000006000000000000000000000000000000000000000000000000000000000000005c4000000000000000000000000000000000000000000108b2a2c2802909400000000000000000
00000000000000000000000000000000000000000000000000030a7c3005e470a116869cde037c7fc7016300ac3f280eef8b18a5e7adc844ed611abbdd1f223867df6ed5d8e70a62ab24400000000000000000000000000000000","blockHash":"0xcb0ca1bcfd03f7c7267f2a12127b8d297d247c3d67ca90c482a423aa
41f7f8be","blockNumber":"0x592","transactionHash":"0x52b261a7f4c0c8c18fd1f00fda122c27b725686aaf4e40e71d972f06e22a381f","transactionIndex":"0x0","logIndex":"0x0","removed":false}],"status":"0x1","logsBloom":"0x000000000000000000000000000000000000000000000
00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000004000000000000000000000000000000000000000000000000000000000000000000200000000000000
000000000000000000000000000008000004010000000000000000000000000000000000000000000800000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000","type":"0x0","effectiveGasPrice":"0x454
b7a4e100","r":"0xbb98db054d2f95431980da350efaa85976e89062c2f2b3b9c06a3a044c829d0e","s":"0x49e3acefd9d133c33d3a12420c7fdb768afb21e382ae445fef822e55f704e9d","v":"0x0"}                                                                                         

Validator zq2-perftest-validator-ase1-3-0348:                                                                                                                                                                                                                 
Deposit: add 20 M $ZIL to 12D3KooWS4n5idrrhZeA5m6WKtE95ygdYnCwTJwiFDt5aNzUrYyR                                                                                                                                                                                
Sent tx: {"hash":"0x20827851a6111510a4c241fd6be1afa3a8ae1ebc5230de52d75afd39cfba1b4b","nonce":"0x3","blockHash":"0xcccdbd37ff38c4b4eee23fd9bb7662984f82c0f07cd0d7d83c3674758fe941cc","blockNumber":"0x5a0","transactionIndex":"0x0","from":"0x54b15e228dddbcb5
62d1573651e57f160765c3d9","to":"0x000000000000000000005a494c4445504f534954","value":"0x108b2a2c28029094000000","gasPrice":"0x454b7a4e100","gas":"0x140a2d","input":"0xe12cf4cb00000000000000000000000000000000000000000000000000000000000000800000000000000000
0000000000000000000000000000000000000000000000e000000000000000000000000000000000000000000000000000000000000001400000000000000000000000006e30e28a14449ffaf8ccee1de4d6105d641573600000000000000000000000000000000000000000000000000000000000000030a30d1b1da25f70
d38417d3660815bca8d04e8308c325909f7cb76d086399c2cdbf672b769a867d9e2f9cc9e6e0c556c9000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000026002408011220f16c293b1c05e3cf221cb517d45a7916b761bb99c18d0c4fbf746d1a8be316dc
00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000060af1f89807eea9a9bb412a73ce9cc88a5a467b14b096f9ab29610e0685a40b5710785da30eab57b65dc516d9f42908f610ece339a2b6926fe736b9612e8cbfce4e05e446ec9
9126611f512c80a21d5028520d97eeaa7d8f34fe019edf3b07117e","v":"0x0","r":"0x6d0abeb5406e32a6061b8bb75d226217bfcc95f5f5e12dcb66fa68552b6892a","s":"0x469c57994c7786b06f3256e8dbbeecd47f2ff87facbca80bd74211a99ccd0d","type":"0x0","chainId":"0x82bd"}             
                                                                                                                                                                                                                                                              
Tx receipt: {"transactionHash":"0x20827851a6111510a4c241fd6be1afa3a8ae1ebc5230de52d75afd39cfba1b4b","transactionIndex":"0x0","blockHash":"0xcccdbd37ff38c4b4eee23fd9bb7662984f82c0f07cd0d7d83c3674758fe941cc","blockNumber":"0x5a0","from":"0x54b15e228dddbcb5
62d1573651e57f160765c3d9","to":"0x000000000000000000005a494c4445504f534954","cumulativeGasUsed":"0x140a2d","gasUsed":"0x140a2d","contractAddress":null,"logs":[],"status":"0x0","logsBloom":"0x000000000000000000000000000000000000000000000000000000000000000
00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000
000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000","type":"0x0","effectiveGasPrice":"0x454b7a4e100","r":"0x6
d0abeb5406e32a6061b8bb75d226217bfcc95f5f5e12dcb66fa68552b6892a","s":"0x469c57994c7786b06f3256e8dbbeecd47f2ff87facbca80bd74211a99ccd0d","v":"0x0"}   

Validator zq2-perftest-validator-ase1-4-0348:                                                                                                                                                                                                                 
Deposit: add 20 M $ZIL to 12D3KooWAhrqTRMag5iTa1adM3SxFogrkHxXCoHv2fEaeCbJEr2n
Sent tx: {"hash":"0x3c9af0393aca288df40bf8e4bbbf98e57c3d7959b060d3a0a85348a3c77bff4c","nonce":"0x4","blockHash":"0x730dc27de2de8d574d2c2c43faf8e1b380a42020eac9e482d3ac0eaf77146009","blockNumber":"0x5ad","transactionIndex":"0x0","from":"0x54b15e228dddbcb5
62d1573651e57f160765c3d9","to":"0x000000000000000000005a494c4445504f534954","value":"0x108b2a2c28029094000000","gasPrice":"0x454b7a4e100","gas":"0x1a3a83","input":"0xe12cf4cb00000000000000000000000000000000000000000000000000000000000000800000000000000000
0000000000000000000000000000000000000000000000e0000000000000000000000000000000000000000000000000000000000000014000000000000000000000000000ce019031389b1d2cde39973edcff2a26d4c3ba000000000000000000000000000000000000000000000000000000000000003083417fd2848f16
5b13f2435f6640bb60ea89500cc538947cde49ae8a9fa1fa5c37512b1f4c11bf084d184967718126ce0000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000260024080112200d3109203441685454f54784551ab17c0a76da64dc49296fb066b6c97267e1e3
00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000060a82d61558d625ecfc4e08f0ebbf9a82604f6a1f6f02917513b7b8e027001490ce4cbfadd76f36688c4c40fb7346f427312e7d3a9f564c70c682c1b924206c268f7c73c14cf
5bf0268493cc9e0f744a6e469db0c8eee977cd0f397876f90d6430","v":"0x1","r":"0x5f49bd8df839269cf6e3d6871a62b182b95f4d82328c1ac70e6fb33e7c2f5463","s":"0x2d11748eb2c768da50a53e1f4e854ef1e5afeacd2feb00993aa6a57d10af1f9","type":"0x0","chainId":"0x82bd"}           
                                                                                                                                                                                                                                                              
Tx receipt: {"transactionHash":"0x3c9af0393aca288df40bf8e4bbbf98e57c3d7959b060d3a0a85348a3c77bff4c","transactionIndex":"0x0","blockHash":"0x730dc27de2de8d574d2c2c43faf8e1b380a42020eac9e482d3ac0eaf77146009","blockNumber":"0x5ad","from":"0x54b15e228dddbcb5
62d1573651e57f160765c3d9","to":"0x000000000000000000005a494c4445504f534954","cumulativeGasUsed":"0x19c637","gasUsed":"0x19c637","contractAddress":null,"logs":[{"address":"0x000000000000000000005a494c4445504f534954","topics":["0xc758b38fca30d8a2d8b0de67b5
fc116c2cdc671f466eda1eaa9dc0543785bd2a"],"data":"0x000000000000000000000000000000000000000000000000000000000000006000000000000000000000000000000000000000000000000000000000000005e8000000000000000000000000000000000000000000108b2a2c2802909400000000000000000
0000000000000000000000000000000000000000000000000003083417fd2848f165b13f2435f6640bb60ea89500cc538947cde49ae8a9fa1fa5c37512b1f4c11bf084d184967718126ce00000000000000000000000000000000","blockHash":"0x730dc27de2de8d574d2c2c43faf8e1b380a42020eac9e482d3ac0eaf
77146009","blockNumber":"0x5ad","transactionHash":"0x3c9af0393aca288df40bf8e4bbbf98e57c3d7959b060d3a0a85348a3c77bff4c","transactionIndex":"0x0","logIndex":"0x0","removed":false}],"status":"0x1","logsBloom":"0x000000000000000000000000000000000000000000000
00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000004000000000000000000000000000000000000000000000000000000000000000000200000000000000
000000000000000000000000000008000004010000000000000000000000000000000000000000000800000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000","type":"0x0","effectiveGasPrice":"0x454
b7a4e100","r":"0x5f49bd8df839269cf6e3d6871a62b182b95f4d82328c1ac70e6fb33e7c2f5463","s":"0x2d11748eb2c768da50a53e1f4e854ef1e5afeacd2feb00993aa6a57d10af1f9","v":"0x1"}                                                                                         


[2024-11-29T06:49:32Z INFO ] SUCCESS: zq2-perftest-validator-ase1-0-0348
[2024-11-29T06:49:32Z INFO ] SUCCESS: zq2-perftest-validator-ase1-1-0348
[2024-11-29T06:49:32Z INFO ] SUCCESS: zq2-perftest-validator-ase1-2-0348
[2024-11-29T06:49:32Z INFO ] SUCCESS: zq2-perftest-validator-ase1-3-0348
[2024-11-29T06:49:32Z INFO ] SUCCESS: zq2-perftest-validator-ase1-4-0348
[2024-11-29T06:49:32Z INFO ] SUCCESS: zq2-perftest-validator-ase1-5-0348
[2024-11-29T06:49:32Z INFO ] SUCCESS: zq2-perftest-validator-ase1-6-0348
[2024-11-29T06:49:32Z INFO ] SUCCESS: zq2-perftest-validator-ase1-7-0348
[2024-11-29T06:49:32Z INFO ] SUCCESS: zq2-perftest-validator-ase1-8-0348
[2024-11-29T06:49:32Z INFO ] SUCCESS: zq2-perftest-validator-ase1-9-0348
Validator zq2-perftest-validator-ase1-3-0348:
Deposit: add 20 M $ZIL to 12D3KooWS4n5idrrhZeA5m6WKtE95ygdYnCwTJwiFDt5aNzUrYyR
Sent tx: {"hash":"0x54bd2f081fc3df324ae8b7a68fa503d61a1c6f602ac57faf0696102b34614522","nonce":"0xa","blockHash":"0x523ff41906eaf8170ca71728c29f7a65c65aac7473a12f8db0e4ca8134db275d","blockNumber":"0xdc5","transactionIndex":"0x0","from":"0x54b15e228dddbcb5
62d1573651e57f160765c3d9","to":"0x000000000000000000005a494c4445504f534954","value":"0x108b2a2c28029094000000","gasPrice":"0x454b7a4e100","gas":"0x236b4d","input":"0xe12cf4cb00000000000000000000000000000000000000000000000000000000000000800000000000000000
0000000000000000000000000000000000000000000000e000000000000000000000000000000000000000000000000000000000000001400000000000000000000000006e30e28a14449ffaf8ccee1de4d6105d641573600000000000000000000000000000000000000000000000000000000000000030a30d1b1da25f70
d38417d3660815bca8d04e8308c325909f7cb76d086399c2cdbf672b769a867d9e2f9cc9e6e0c556c9000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000026002408011220f16c293b1c05e3cf221cb517d45a7916b761bb99c18d0c4fbf746d1a8be316dc
00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000060af1f89807eea9a9bb412a73ce9cc88a5a467b14b096f9ab29610e0685a40b5710785da30eab57b65dc516d9f42908f610ece339a2b6926fe736b9612e8cbfce4e05e446ec9
9126611f512c80a21d5028520d97eeaa7d8f34fe019edf3b07117e","v":"0x0","r":"0xe087b16aabbf43235533a9bddcd71371faff25f0b79e85494dfa8b7f6a518116","s":"0xa301cfbbc8e6df2e1364f6525dd78e38b467ccb8c1123979365631fece57678","type":"0x0","chainId":"0x82bd"}

Tx receipt: {"transactionHash":"0x54bd2f081fc3df324ae8b7a68fa503d61a1c6f602ac57faf0696102b34614522","transactionIndex":"0x0","blockHash":"0x523ff41906eaf8170ca71728c29f7a65c65aac7473a12f8db0e4ca8134db275d","blockNumber":"0xdc5","from":"0x54b15e228dddbcb5
62d1573651e57f160765c3d9","to":"0x000000000000000000005a494c4445504f534954","cumulativeGasUsed":"0x21dac0","gasUsed":"0x21dac0","contractAddress":null,"logs":[{"address":"0x000000000000000000005a494c4445504f534954","topics":["0xc758b38fca30d8a2d8b0de67b5
fc116c2cdc671f466eda1eaa9dc0543785bd2a"],"data":"0x00000000000000000000000000000000000000000000000000000000000000600000000000000000000000000000000000000000000000000000000000000dec000000000000000000000000000000000000000000108b2a2c2802909400000000000000000
00000000000000000000000000000000000000000000000000030a30d1b1da25f70d38417d3660815bca8d04e8308c325909f7cb76d086399c2cdbf672b769a867d9e2f9cc9e6e0c556c900000000000000000000000000000000","blockHash":"0x523ff41906eaf8170ca71728c29f7a65c65aac7473a12f8db0e4ca81
34db275d","blockNumber":"0xdc5","transactionHash":"0x54bd2f081fc3df324ae8b7a68fa503d61a1c6f602ac57faf0696102b34614522","transactionIndex":"0x0","logIndex":"0x0","removed":false}],"status":"0x1","logsBloom":"0x000000000000000000000000000000000000000000000
00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000004000000000000000000000000000000000000000000000000000000000000000000200000000000000
000000000000000000000000000008000004010000000000000000000000000000000000000000000800000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000","type":"0x0","effectiveGasPrice":"0x454
b7a4e100","r":"0xe087b16aabbf43235533a9bddcd71371faff25f0b79e85494dfa8b7f6a518116","s":"0xa301cfbbc8e6df2e1364f6525dd78e38b467ccb8c1123979365631fece57678","v":"0x0"}


[2024-11-29T07:25:32Z INFO ] SUCCESS: zq2-perftest-validator-ase1-3-0348
[2024-11-29T07:25:32Z ERROR] FAILURE: zq2-perftest-validator-ase1-0-0348
[2024-11-29T07:25:32Z ERROR] FAILURE: zq2-perftest-validator-ase1-1-0348
[2024-11-29T07:25:32Z ERROR] FAILURE: zq2-perftest-validator-ase1-2-0348
[2024-11-29T07:25:32Z ERROR] FAILURE: zq2-perftest-validator-ase1-4-0348
[2024-11-29T07:25:32Z ERROR] FAILURE: zq2-perftest-validator-ase1-5-0348
[2024-11-29T07:25:32Z ERROR] FAILURE: zq2-perftest-validator-ase1-6-0348
[2024-11-29T07:25:32Z ERROR] FAILURE: zq2-perftest-validator-ase1-7-0348
[2024-11-29T07:25:32Z ERROR] FAILURE: zq2-perftest-validator-ase1-8-0348
[2024-11-29T07:25:32Z ERROR] FAILURE: zq2-perftest-validator-ase1-9-0348

Note that the missing deposit has a block number of 0x5a0 (1440) which is perfectly divisible by 36, the blocks/epoch value. There may be an off-by-one error in the deposit.sol contract.

@shawn-zil shawn-zil self-assigned this Nov 29, 2024
@shawn-zil
Copy link
Contributor Author

shawn-zil commented Nov 29, 2024

Just stashing here the block numbers: 0x577 0x584 0x592 0x5a0 0x5ad 0x5ba 0x5c8 0x5d5 0x5e2 0x5ef. 0x5a0 is the only one divisible by 36 blocks/epoch - the configured value in perftest

And the block 0xdc5 is >> 0x5a0. So, it's past many epochs.

@shawn-zil shawn-zil added the bug Something isn't working label Dec 2, 2024
@DrZoltanFazekas DrZoltanFazekas added the Agate Required for mainnet launch label Dec 2, 2024
@shawn-zil shawn-zil changed the title Missing deposits Missing perfectly divisible deposits Dec 9, 2024
@shawn-zil
Copy link
Contributor Author

shawn-zil commented Dec 9, 2024

Confirmed that the issue still plagues the network post #1946 deposit upgrade changes.

@shawn-zil
Copy link
Contributor Author

Confirmed that issue disappears when LRU cache is removed - commit d4a2330 does not exhibit this issue.

@shawn-zil shawn-zil linked a pull request Dec 11, 2024 that will close this issue
@shawn-zil
Copy link
Contributor Author

Confirmed that this issue is not present in #1915

@JamesHinshelwood
Copy link
Contributor

JamesHinshelwood commented Dec 11, 2024

#1915 removes the state trie cache and allocates that space to the redb cache, since I found the performance was the same and it makes the code simpler.

I am curious if we can identify why the cache causes this bug though? Maybe we can run a version which reads from both the cache and the DB and logs if they are different?

@shawn-zil
Copy link
Contributor Author

I came about this in a reversed way. After looking thru both the Solidity deposit contract and Rust code and not finding any obvious logical errors, I began to work backwards on how the issue could have happened. One hypothesis I had was that the state being presented to the EVM was somehow inconsistent. Since removing the caching layer eliminated the issue, I chalked it up to some inconsistency in caching. Not sure if it might be an issue on our part or the external crate.

@shawn-zil shawn-zil removed their assignment Dec 13, 2024
@shawn-zil shawn-zil self-assigned this Dec 20, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Agate Required for mainnet launch bug Something isn't working
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants