Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Workers don't get new IP package from frontier #26

Open
MichaelRoeder opened this issue Oct 6, 2017 · 0 comments
Open

Workers don't get new IP package from frontier #26

MichaelRoeder opened this issue Oct 6, 2017 · 0 comments

Comments

@MichaelRoeder
Copy link
Member

Problem

During the 3rd run, the workers were throwing exceptions because the frontier stopped answering, e.g.,

{"log":"2017-10-02 13:07:46,583 [main] ERROR [o.h.c.r.RabbitRpcClient       ] - \u003cException while sending query. Returning null.\u003e\n","stream":"stdout","time":"2017-10-02T13:07:46.585655178Z"}
{"log":"java.util.concurrent.TimeoutException: Timeout waiting for task.\n","stream":"stdout","time":"2017-10-02T13:07:46.585709808Z"}
{"log":"\u0009at com.google.common.util.concurrent.AbstractFuture$Sync.get(AbstractFuture.java:269)\n","stream":"stdout","time":"2017-10-02T13:07:46.585717681Z"}
{"log":"\u0009at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:96)\n","stream":"stdout","time":"2017-10-02T13:07:46.585725113Z"}
{"log":"\u0009at org.hobbit.core.rabbit.RabbitRpcClient.request(RabbitRpcClient.java:156)\n","stream":"stdout","time":"2017-10-02T13:07:46.585732453Z"}
{"log":"\u0009at org.aksw.simba.squirrel.components.WorkerComponent.getNextUris(WorkerComponent.java:83)\n","stream":"stdout","time":"2017-10-02T13:07:46.585739529Z"}
{"log":"\u0009at org.aksw.simba.squirrel.worker.impl.WorkerImpl.run(WorkerImpl.java:80)\n","stream":"stdout","time":"2017-10-02T13:07:46.585746485Z"}
{"log":"\u0009at org.aksw.simba.squirrel.components.WorkerComponent.run(WorkerComponent.java:71)\n","stream":"stdout","time":"2017-10-02T13:07:46.585753514Z"}
{"log":"\u0009at org.hobbit.core.run.ComponentStarter.main(ComponentStarter.java:67)\n","stream":"stdout","time":"2017-10-02T13:07:46.585760435Z"}
{"log":"2017-10-02 13:17:48,586 [main] ERROR [o.h.c.r.RabbitRpcClient       ] - \u003cException while sending query. Returning null.\u003e\n","stream":"stdout","time":"2017-10-02T13:17:48.587168887Z"}
{"log":"java.util.concurrent.TimeoutException: Timeout waiting for task.\n","stream":"stdout","time":"2017-10-02T13:17:48.587222859Z"}
{"log":"\u0009at com.google.common.util.concurrent.AbstractFuture$Sync.get(AbstractFuture.java:269)\n","stream":"stdout","time":"2017-10-02T13:17:48.587230644Z"}
{"log":"\u0009at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:96)\n","stream":"stdout","time":"2017-10-02T13:17:48.587238083Z"}
{"log":"\u0009at org.hobbit.core.rabbit.RabbitRpcClient.request(RabbitRpcClient.java:156)\n","stream":"stdout","time":"2017-10-02T13:17:48.587245243Z"}
{"log":"\u0009at org.aksw.simba.squirrel.components.WorkerComponent.getNextUris(WorkerComponent.java:83)\n","stream":"stdout","time":"2017-10-02T13:17:48.587252323Z"}
{"log":"\u0009at org.aksw.simba.squirrel.worker.impl.WorkerImpl.run(WorkerImpl.java:80)\n","stream":"stdout","time":"2017-10-02T13:17:48.587259355Z"}
{"log":"\u0009at org.aksw.simba.squirrel.components.WorkerComponent.run(WorkerComponent.java:71)\n","stream":"stdout","time":"2017-10-02T13:17:48.587266352Z"}
{"log":"\u0009at org.hobbit.core.run.ComponentStarter.main(ComponentStarter.java:67)\n","stream":"stdout","time":"2017-10-02T13:17:48.587272977Z"}

This could be caused by an empty queue in the frontier. But this should be answered in a different way. It could be helpful if the frontier regularly logs its status (queue length, number of IPs blocked, etc.).

@gsjunior86 gsjunior86 removed the bug label Feb 26, 2019
MichaelRoeder pushed a commit that referenced this issue Jun 11, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants