Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Sometimes client hangs in startup, when a required plugin fails to load. #8122

Open
ak88 opened this issue Jan 29, 2025 · 1 comment · May be fixed by #8145
Open

Sometimes client hangs in startup, when a required plugin fails to load. #8122

ak88 opened this issue Jan 29, 2025 · 1 comment · May be fixed by #8145
Assignees
@ak88

Comments

@ak88
Copy link
Contributor

ak88 commented Jan 29, 2025 •
edited
Loading

Description
There is a race condition, that sometimes causes the client to hang on startup, when a required plugin fails.

Steps to Reproduce
In order to replicate the behavior, please provide a detailed list of steps:

  1. Setup a required plugin (MustInitialize == true)
  2. Make it fail to start (throw exception etc.)
  3. repeat until the client hangs and does not progress.
@ak88 ak88 self-assigned this Jan 29, 2025
@ak88
Copy link
Contributor Author

ak88 commented Jan 29, 2025

stack trace when client hangs:

2025-01-28 14:57:46.0895|ERROR|Init.Steps.InitializePlugins|6|Failed to initialize plugin Clef signer by Nethermind Nethermind.Network.NetworkingException: Remote signer at http://172.17.0.1:8550/ did not respond.
---> System.Net.Http.HttpRequestException: Connection refused (172.17.0.1:8550)
---> System.Net.Sockets.SocketException (111): Connection refused
at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.CreateException(SocketError error, Boolean forAsyncThrow)
at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.ConnectAsync(Socket socket, Boolean saeaCancelable)
at System.Net.Sockets.Socket.ConnectAsync(EndPoint remoteEP, CancellationToken cancellationToken)
at System.Net.Http.HttpConnectionPool.ConnectToTcpHostAsync(String host, Int32 port, HttpRequestMessage initialRequest, Boolean async, CancellationToken cancellationToken)
at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[TStateMachine](TStateMachine& stateMachine)
at System.Net.Http.HttpConnectionPool.ConnectToTcpHostAsync(String host, Int32 port, HttpRequestMessage initialRequest, Boolean async, CancellationToken cancellationToken)
at System.Net.Http.HttpConnectionPool.ConnectAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[TStateMachine](TStateMachine& stateMachine)
at System.Net.Http.HttpConnectionPool.ConnectAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
at System.Net.Http.HttpConnectionPool.CreateHttp11ConnectionAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[TStateMachine](TStateMachine& stateMachine)
at System.Net.Http.HttpConnectionPool.CreateHttp11ConnectionAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
at System.Net.Http.HttpConnectionPool.InjectNewHttp11ConnectionAsync(QueueItem queueItem)
at System.Runtime.CompilerServices.AsyncTaskMethodBuilder1.AsyncStateMachineBox1.ExecutionContextCallback(Object s)
at System.Threading.ExecutionContext.RunFromThreadPoolDispatchLoop(Thread threadPoolThread, ExecutionContext executionContext, ContextCallback callback, Object state)
at System.Runtime.CompilerServices.AsyncTaskMethodBuilder1.AsyncStateMachineBox1.MoveNext(Thread threadPoolThread)
at System.Runtime.CompilerServices.AsyncTaskMethodBuilder1.AsyncStateMachineBox1.ExecuteFromThreadPool(Thread threadPoolThread)
at System.Threading.ThreadPoolWorkQueue.Dispatch()
at System.Threading.PortableThreadPool.WorkerThread.WorkerThreadStart()

--- End of stack trace from previous location ---

at System.Net.Http.HttpConnectionPool.ConnectToTcpHostAsync(String host, Int32 port, HttpRequestMessage initialRequest, Boolean async, CancellationToken cancellationToken)
--- End of inner exception stack trace ---
at System.Net.Http.HttpConnectionPool.ConnectToTcpHostAsync(String host, Int32 port, HttpRequestMessage initialRequest, Boolean async, CancellationToken cancellationToken)
at System.Net.Http.HttpConnectionPool.ConnectAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
at System.Net.Http.HttpConnectionPool.CreateHttp11ConnectionAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
at System.Net.Http.HttpConnectionPool.InjectNewHttp11ConnectionAsync(QueueItem queueItem)
at System.Threading.Tasks.TaskCompletionSourceWithCancellation`1.WaitWithCancellationAsync(CancellationToken cancellationToken)
at System.Net.Http.HttpConnectionPool.SendWithVersionDetectionAndRetryAsync(HttpRequestMessage request, Boolean async, Boolean doRequestAuth, CancellationToken cancellationToken)
at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
at System.Net.Http.HttpClient.g__Core|83_0(HttpRequestMessage request, HttpCompletionOption completionOption, CancellationTokenSource cts, Boolean disposeCts, CancellationTokenSource pendingRequestsCts, CancellationToken originalCancellationToken)
at Nethermind.JsonRpc.Client.BasicJsonRpcClient.Post[T](String method, Object[] parameters) in /src/Nethermind/Nethermind.JsonRpc/Client/BasicJsonRpcClient.cs:line 51
at Nethermind.ExternalSigner.Plugin.ClefSigner.GetSignerAddress(IJsonRpcClient rpcClient, Address blockAuthorAccount) in /src/Nethermind/Nethermind.ExternalSigner.Plugin/ClefSigner.cs:line 98
at Nethermind.ExternalSigner.Plugin.ClefSigner.Create(IJsonRpcClient jsonRpcClient, Address blockAuthorAccount) in /src/Nethermind/Nethermind.ExternalSigner.Plugin/ClefSigner.cs:line 29
at Nethermind.ExternalSigner.Plugin.ClefSignerPlugin.SetupExternalSigner(Uri urlSigner, String blockAuthorAccount) in /src/Nethermind/Nethermind.ExternalSigner.Plugin/ClefSignerPlugin.cs:line 59
--- End of inner exception stack trace ---
at Nethermind.ExternalSigner.Plugin.ClefSignerPlugin.SetupExternalSigner(Uri urlSigner, String blockAuthorAccount) in /src/Nethermind/Nethermind.ExternalSigner.Plugin/ClefSignerPlugin.cs:line 63
at Nethermind.ExternalSigner.Plugin.ClefSignerPlugin.Init(INethermindApi nethermindApi) in /src/Nethermind/Nethermind.ExternalSigner.Plugin/ClefSignerPlugin.cs:line 43
at Nethermind.Init.Steps.InitializePlugins.Execute(CancellationToken cancellationToken) in /src/Nethermind/Nethermind.Init/Steps/InitializePlugins.cs:line 34

2025-01-28 14:57:46.0895|ERROR|Init.Steps.EthereumStepsManager|6|Step InitializePlugins failed after 76ms Nethermind.Network.NetworkingException: Remote signer at http://172.17.0.1:8550/ did not respond.
---> System.Net.Http.HttpRequestException: Connection refused (172.17.0.1:8550)
---> System.Net.Sockets.SocketException (111): Connection refused
at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.CreateException(SocketError error, Boolean forAsyncThrow)
at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.ConnectAsync(Socket socket, Boolean saeaCancelable)
at System.Net.Sockets.Socket.ConnectAsync(EndPoint remoteEP, CancellationToken cancellationToken)
at System.Net.Http.HttpConnectionPool.ConnectToTcpHostAsync(String host, Int32 port, HttpRequestMessage initialRequest, Boolean async, CancellationToken cancellationToken)
at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[TStateMachine](TStateMachine& stateMachine)
at System.Net.Http.HttpConnectionPool.ConnectToTcpHostAsync(String host, Int32 port, HttpRequestMessage initialRequest, Boolean async, CancellationToken cancellationToken)
at System.Net.Http.HttpConnectionPool.ConnectAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[TStateMachine](TStateMachine& stateMachine)
at System.Net.Http.HttpConnectionPool.ConnectAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
at System.Net.Http.HttpConnectionPool.CreateHttp11ConnectionAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[TStateMachine](TStateMachine& stateMachine)
at System.Net.Http.HttpConnectionPool.CreateHttp11ConnectionAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
at System.Net.Http.HttpConnectionPool.InjectNewHttp11ConnectionAsync(QueueItem queueItem)
at System.Runtime.CompilerServices.AsyncTaskMethodBuilder1.AsyncStateMachineBox1.ExecutionContextCallback(Object s)
at System.Threading.ExecutionContext.RunFromThreadPoolDispatchLoop(Thread threadPoolThread, ExecutionContext executionContext, ContextCallback callback, Object state)
at System.Runtime.CompilerServices.AsyncTaskMethodBuilder1.AsyncStateMachineBox1.MoveNext(Thread threadPoolThread)
at System.Runtime.CompilerServices.AsyncTaskMethodBuilder1.AsyncStateMachineBox1.ExecuteFromThreadPool(Thread threadPoolThread)
at System.Threading.ThreadPoolWorkQueue.Dispatch()
at System.Threading.PortableThreadPool.WorkerThread.WorkerThreadStart()
--- End of stack trace from previous location ---
at System.Net.Http.HttpConnectionPool.ConnectToTcpHostAsync(String host, Int32 port, HttpRequestMessage initialRequest, Boolean async, CancellationToken cancellationToken)
--- End of inner exception stack trace ---
at System.Net.Http.HttpConnectionPool.ConnectToTcpHostAsync(String host, Int32 port, HttpRequestMessage initialRequest, Boolean async, CancellationToken cancellationToken)
at System.Net.Http.HttpConnectionPool.ConnectAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
at System.Net.Http.HttpConnectionPool.CreateHttp11ConnectionAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
at System.Net.Http.HttpConnectionPool.InjectNewHttp11ConnectionAsync(QueueItem queueItem)
at System.Threading.Tasks.TaskCompletionSourceWithCancellation`1.WaitWithCancellationAsync(CancellationToken cancellationToken)
at System.Net.Http.HttpConnectionPool.SendWithVersionDetectionAndRetryAsync(HttpRequestMessage request, Boolean async, Boolean doRequestAuth, CancellationToken cancellationToken)
at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
at System.Net.Http.HttpClient.g__Core|83_0(HttpRequestMessage request, HttpCompletionOption completionOption, CancellationTokenSource cts, Boolean disposeCts, CancellationTokenSource pendingRequestsCts, CancellationToken originalCancellationToken)
at Nethermind.JsonRpc.Client.BasicJsonRpcClient.Post[T](String method, Object[] parameters) in /src/Nethermind/Nethermind.JsonRpc/Client/BasicJsonRpcClient.cs:line 51
at Nethermind.ExternalSigner.Plugin.ClefSigner.GetSignerAddress(IJsonRpcClient rpcClient, Address blockAuthorAccount) in /src/Nethermind/Nethermind.ExternalSigner.Plugin/ClefSigner.cs:line 98
at Nethermind.ExternalSigner.Plugin.ClefSigner.Create(IJsonRpcClient jsonRpcClient, Address blockAuthorAccount) in /src/Nethermind/Nethermind.ExternalSigner.Plugin/ClefSigner.cs:line 29
at Nethermind.ExternalSigner.Plugin.ClefSignerPlugin.SetupExternalSigner(Uri urlSigner, String blockAuthorAccount) in /src/Nethermind/Nethermind.ExternalSigner.Plugin/ClefSignerPlugin.cs:line 59
--- End of inner exception stack trace ---
at Nethermind.ExternalSigner.Plugin.ClefSignerPlugin.SetupExternalSigner(Uri urlSigner, String blockAuthorAccount) in /src/Nethermind/Nethermind.ExternalSigner.Plugin/ClefSignerPlugin.cs:line 63
at Nethermind.ExternalSigner.Plugin.ClefSignerPlugin.Init(INethermindApi nethermindApi) in /src/Nethermind/Nethermind.ExternalSigner.Plugin/ClefSignerPlugin.cs:line 43
at Nethermind.Init.Steps.InitializePlugins.Execute(CancellationToken cancellationToken) in /src/Nethermind/Nethermind.Init/Steps/InitializePlugins.cs:line 34
at Nethermind.Init.Steps.EthereumStepsManager.ExecuteStep(IStep step, StepInfo stepInfo, CancellationToken cancellationToken) in /src/Nethermind/Nethermind.Init/Steps/EthereumStepsManager.cs:line 154

@ak88 ak88 linked a pull request Jan 31, 2025 that will close this issue
Open
16 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@ak88 ak88

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Fix/initsteps race condition NethermindEth/nethermind
1 participant
@ak88