Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

lake compaction scheduler might introduce too many tasks on BE/CN while sending task rpc partially succeed #54071

Open
wxl24life opened this issue Dec 18, 2024 · 0 comments · May be fixed by #54832 or #54908
Open

lake compaction scheduler might introduce too many tasks on BE/CN while sending task rpc partially succeed #54071

wxl24life opened this issue Dec 18, 2024 · 0 comments · May be fixed by #54832 or #54908
Labels
type/bug Something isn't working

Comments

@wxl24life
Copy link
Contributor

wxl24life commented Dec 18, 2024 •
edited
Loading

Steps to reproduce the behavior (Required)

kill some CN node while compaction scheduler was busy

Expected behavior (Required)

Real behavior (Required)

MySQL [(none)]> select BE_ID, count(*) from information_schema.be_cloud_native_compactions group by BE_ID;
+--------+----------+
| BE_ID  | count(*) |
+--------+----------+
| 926056 |       912 |
| 926049 |       711 |
| 926048 |       610 |
| 926052 |        699 |
| 926051 |       710 |
+--------+----------+
10 rows in set (0.02 sec)
2024-12-18 19:14:30.790+08:00 ERROR (COMPACTION_DISPATCH|102) [CompactionScheduler.startCompaction():324] Unable to validate object
java.lang.RuntimeException: Unable to validate object
	at com.baidu.jprotobuf.pbrpc.transport.ChannelPool.getChannel(ChannelPool.java:86) ~[jprotobuf-rpc-core-4.2.1.jar:?]
	at com.baidu.jprotobuf.pbrpc.transport.RpcChannel.getConnection(RpcChannel.java:73) ~[jprotobuf-rpc-core-4.2.1.jar:?]
	at com.baidu.jprotobuf.pbrpc.client.ProtobufRpcProxy.invoke(ProtobufRpcProxy.java:499) ~[jprotobuf-rpc-core-4.2.1.jar:?]
	at com.sun.proxy.$Proxy43.compact(Unknown Source) ~[?:?]
	at com.starrocks.lake.compaction.CompactionTask.sendRequest(CompactionTask.java:115) ~[starrocks-fe.jar:?]
	at com.starrocks.lake.compaction.CompactionScheduler.startCompaction(CompactionScheduler.java:319) ~[starrocks-fe.jar:?]
	at com.starrocks.lake.compaction.CompactionScheduler.schedule(CompactionScheduler.java:213) ~[starrocks-fe.jar:?]
	at com.starrocks.lake.compaction.CompactionScheduler.runOneCycle(CompactionScheduler.java:113) ~[starrocks-fe.jar:?]
	at com.starrocks.common.util.Daemon.run(Daemon.java:107) ~[starrocks-fe.jar:?]
Caused by: java.util.NoSuchElementException: Unable to validate object
	at org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:506) ~[commons-pool2-2.3.jar:2.3]
	at org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:363) ~[commons-pool2-2.3.jar:2.3]
	at com.baidu.jprotobuf.pbrpc.transport.ChannelPool.getChannel(ChannelPool.java:80) ~[jprotobuf-rpc-core-4.2.1.jar:?]

StarRocks version (Required)

  • 3.x.x
@wxl24life wxl24life added the type/bug Something isn't working label Dec 18, 2024
@wxl24life wxl24life changed the title lake compaction scheduler might introduce too many tasks on BE/CN while sending task rpc partially failed lake compaction scheduler might introduce too many tasks on BE/CN while sending task rpc partially succeed Dec 18, 2024
This was referenced Jan 8, 2025
Open
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
type/bug Something isn't working
Projects
None yet
Milestone
No milestone
1 participant
@wxl24life