Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

a running workflow turn postponed due parallelism limit #14123

Open
3 of 4 tasks
tczhao opened this issue Jan 24, 2025 · 3 comments
Open
3 of 4 tasks

a running workflow turn postponed due parallelism limit #14123

tczhao opened this issue Jan 24, 2025 · 3 comments
Labels
area/controller Controller issues, panics type/bug

Comments

@tczhao
Copy link
Member

tczhao commented Jan 24, 2025

Pre-requisites

  • I have double-checked my configuration
  • I have tested with the :latest image tag (i.e. quay.io/argoproj/workflow-controller:latest) and can confirm the issue still exists on :latest. If not, I have explained why, in detail, in my description below.
  • I have searched existing issues and could not find a match for this bug
  • I'd like to contribute the fix myself (see contributing guide)

What happened? What did you expect to happen?

Workflow was running but all of sudden unable to proceed,
looking through controller log we found

"Jan 20, 2025 @ 14:17:31.998","time=""2025-01-20T08:47:31.998Z"" level=info msg=""Processing workflow"" namespace=default workflow=hive-1698426951-cron-1737275400","6aa8356f-0e1f-4842-b7f1-8035a97e8c95"
"Jan 20, 2025 @ 14:17:52.598","time=""2025-01-20T08:47:52.598Z"" level=info msg=""Workflow processing has been postponed due to max parallelism limit"" key=default/hive-1698426951-cron-1737275400","6aa8356f-0e1f-4842-b7f1-8035a97e8c95"

We have configured parallelism limit to 10, but only reach max 8 in the past 24hrs before the issue happen
no controller restart in the past 24hr.
We are not sure what is the root cause, and this has happened only once so far.

We can mitigate this issue by allowing workflow to progress if its in parallelism mutex or having running state.
Since workflows that are running should already be in parallelism mutex and it should keep running
line

argo-workflows/workflow/controller/controller.go

Line 869 in 7305c18

if !(woc.GetShutdownStrategy().Enabled() && woc.GetShutdownStrategy() == wfv1.ShutdownStrategyTerminate) && !wfc.throttler.Admit(key) {

Version(s)

v3.5.2

Paste a minimal workflow that reproduces the issue. We must be able to run the workflow; don't enter a workflow that uses private images.

.

Logs from the workflow controller

"@timestamp",log,"kubernetes.pod_id"
"Jan 20, 2025 @ 14:17:52.598","time=""2025-01-20T08:47:52.598Z"" level=info msg=""Workflow processing has been postponed due to max parallelism limit"" key=default/hive-1698426951-cron-1737275400","6aa8356f-0e1f-4842-b7f1-8035a97e8c95"
"Jan 20, 2025 @ 14:17:51.660","time=""2025-01-20T08:47:51.660Z"" level=info msg=""Workflow update successful"" namespace=default phase=Running resourceVersion=162586627 workflow=tableau-1717455078-cron-1737342000","6aa8356f-0e1f-4842-b7f1-8035a97e8c95"
"Jan 20, 2025 @ 14:17:51.646","time=""2025-01-20T08:47:51.646Z"" level=info msg=""Workflow update successful"" namespace=default phase=Running resourceVersion=162586626 workflow=snowflake-1700253343-cron-1737360000","6aa8356f-0e1f-4842-b7f1-8035a97e8c95"
"Jan 20, 2025 @ 14:17:51.445","time=""2025-01-20T08:47:51.445Z"" level=info msg=""Workflow to be dehydrated"" Workflow Size=1459378","6aa8356f-0e1f-4842-b7f1-8035a97e8c95"
"Jan 20, 2025 @ 14:17:51.401","time=""2025-01-20T08:47:51.401Z"" level=info msg=""TaskSet Reconciliation"" namespace=default workflow=tableau-1717455078-cron-1737342000","6aa8356f-0e1f-4842-b7f1-8035a97e8c95"
"Jan 20, 2025 @ 14:17:51.401","time=""2025-01-20T08:47:51.401Z"" level=info msg=reconcileAgentPod namespace=default workflow=tableau-1717455078-cron-1737342000","6aa8356f-0e1f-4842-b7f1-8035a97e8c95"
"Jan 20, 2025 @ 14:17:51.386","time=""2025-01-20T08:47:51.386Z"" level=info msg=""Could not acquire lock named: &{default atlas-config publish ConfigMap}"" namespace=default workflow=tableau-1717455078-cron-1737342000","6aa8356f-0e1f-4842-b7f1-8035a97e8c95"
"Jan 20, 2025 @ 14:17:51.386","time=""2025-01-20T08:47:51.386Z"" level=info msg=""Workflow step group node tableau-1717455078-cron-1737342000-2350978509 not yet completed"" namespace=default workflow=tableau-1717455078-cron-1737342000","6aa8356f-0e1f-4842-b7f1-8035a97e8c95"
"Jan 20, 2025 @ 14:17:51.386","time=""2025-01-20T08:47:51.386Z"" level=info msg=""Workflow step group node tableau-1717455078-cron-1737342000-2050333199 not yet completed"" namespace=default workflow=tableau-1717455078-cron-1737342000","6aa8356f-0e1f-4842-b7f1-8035a97e8c95"
"Jan 20, 2025 @ 14:17:51.386","time=""2025-01-20T08:47:51.386Z"" level=info msg=""Workflow step group node tableau-1717455078-cron-1737342000-3655143290 not yet completed"" namespace=default workflow=tableau-1717455078-cron-1737342000","6aa8356f-0e1f-4842-b7f1-8035a97e8c95"
"Jan 20, 2025 @ 14:17:51.378","time=""2025-01-20T08:47:51.378Z"" level=info msg=""Could not acquire lock named: &{default atlas-config publish ConfigMap}"" namespace=default workflow=tableau-1717455078-cron-1737342000","6aa8356f-0e1f-4842-b7f1-8035a97e8c95"
"Jan 20, 2025 @ 14:17:51.369","time=""2025-01-20T08:47:51.369Z"" level=info msg=""Could not acquire lock named: &{default atlas-config publish ConfigMap}"" namespace=default workflow=tableau-1717455078-cron-1737342000","6aa8356f-0e1f-4842-b7f1-8035a97e8c95"
"Jan 20, 2025 @ 14:17:51.366","time=""2025-01-20T08:47:51.366Z"" level=info msg=""Workflow to be dehydrated"" Workflow Size=430327","6aa8356f-0e1f-4842-b7f1-8035a97e8c95"
"Jan 20, 2025 @ 14:17:51.362","time=""2025-01-20T08:47:51.362Z"" level=info msg=""TaskSet Reconciliation"" namespace=default workflow=snowflake-1700253343-cron-1737360000","6aa8356f-0e1f-4842-b7f1-8035a97e8c95"
"Jan 20, 2025 @ 14:17:51.362","time=""2025-01-20T08:47:51.362Z"" level=info msg=reconcileAgentPod namespace=default workflow=snowflake-1700253343-cron-1737360000","6aa8356f-0e1f-4842-b7f1-8035a97e8c95"
"Jan 20, 2025 @ 14:17:51.360","time=""2025-01-20T08:47:51.360Z"" level=info msg=""Workflow step group node snowflake-1700253343-cron-1737360000-862133027 not yet completed"" namespace=default workflow=snowflake-1700253343-cron-1737360000","6aa8356f-0e1f-4842-b7f1-8035a97e8c95"
"Jan 20, 2025 @ 14:17:51.360","time=""2025-01-20T08:47:51.360Z"" level=info msg=""Could not acquire lock named: &{default atlas-config publish ConfigMap}"" namespace=default workflow=tableau-1717455078-cron-1737342000","6aa8356f-0e1f-4842-b7f1-8035a97e8c95"
"Jan 20, 2025 @ 14:17:51.360","time=""2025-01-20T08:47:51.360Z"" level=info msg=""Workflow step group node snowflake-1700253343-cron-1737360000-2003898834 not yet completed"" namespace=default workflow=snowflake-1700253343-cron-1737360000","6aa8356f-0e1f-4842-b7f1-8035a97e8c95"
"Jan 20, 2025 @ 14:17:51.360","time=""2025-01-20T08:47:51.360Z"" level=info msg=""Workflow step group node snowflake-1700253343-cron-1737360000-3387130128 not yet completed"" namespace=default workflow=snowflake-1700253343-cron-1737360000","6aa8356f-0e1f-4842-b7f1-8035a97e8c95"
"Jan 20, 2025 @ 14:17:51.360","time=""2025-01-20T08:47:51.360Z"" level=warning msg=""was unable to obtain the node for snowflake-1700253343-cron-1737360000-1818203418, taskName tag-attachment""","6aa8356f-0e1f-4842-b7f1-8035a97e8c95"
"Jan 20, 2025 @ 14:17:51.360","time=""2025-01-20T08:47:51.360Z"" level=warning msg=""was unable to obtain the node for snowflake-1700253343-cron-1737360000-1818203418, taskName tag-attachment""","6aa8356f-0e1f-4842-b7f1-8035a97e8c95"
"Jan 20, 2025 @ 14:17:51.357","time=""2025-01-20T08:47:51.357Z"" level=info msg=""Workflow step group node snowflake-1700253343-cron-1737360000-3332696593 not yet completed"" namespace=default workflow=snowflake-1700253343-cron-1737360000","6aa8356f-0e1f-4842-b7f1-8035a97e8c95"
"Jan 20, 2025 @ 14:17:51.356","time=""2025-01-20T08:47:51.356Z"" level=info msg=""Could not acquire lock named: &{default atlas-config publish ConfigMap}"" namespace=default workflow=snowflake-1700253343-cron-1737360000","6aa8356f-0e1f-4842-b7f1-8035a97e8c95"
"Jan 20, 2025 @ 14:17:51.351","time=""2025-01-20T08:47:51.351Z"" level=info msg=""Could not acquire lock named: &{default atlas-config publish ConfigMap}"" namespace=default workflow=tableau-1717455078-cron-1737342000","6aa8356f-0e1f-4842-b7f1-8035a97e8c95"
"Jan 20, 2025 @ 14:17:51.342","time=""2025-01-20T08:47:51.342Z"" level=info msg=""Could not acquire lock named: &{default atlas-config publish ConfigMap}"" namespace=default workflow=tableau-1717455078-cron-1737342000","6aa8356f-0e1f-4842-b7f1-8035a97e8c95"
"Jan 20, 2025 @ 14:17:51.341","time=""2025-01-20T08:47:51.341Z"" level=info msg=""SG Outbound nodes of snowflake-1700253343-cron-1737360000-3382269487 are [snowflake-1700253343-cron-1737360000-986958898]"" namespace=default workflow=snowflake-1700253343-cron-1737360000","6aa8356f-0e1f-4842-b7f1-8035a97e8c95"
"Jan 20, 2025 @ 14:17:51.341","time=""2025-01-20T08:47:51.341Z"" level=info msg=""SG Outbound nodes of snowflake-1700253343-cron-1737360000-4181863139 are [snowflake-1700253343-cron-1737360000-769128878]"" namespace=default workflow=snowflake-1700253343-cron-1737360000","6aa8356f-0e1f-4842-b7f1-8035a97e8c95"
"Jan 20, 2025 @ 14:17:51.341","time=""2025-01-20T08:47:51.341Z"" level=info msg=""SG Outbound nodes of snowflake-1700253343-cron-1737360000-2350211669 are [snowflake-1700253343-cron-1737360000-132543156]"" namespace=default workflow=snowflake-1700253343-cron-1737360000","6aa8356f-0e1f-4842-b7f1-8035a97e8c95"
"Jan 20, 2025 @ 14:17:51.341","time=""2025-01-20T08:47:51.341Z"" level=info msg=""SG Outbound nodes of snowflake-1700253343-cron-1737360000-2284221594 are [snowflake-1700253343-cron-1737360000-2203160129]"" namespace=default workflow=snowflake-1700253343-cron-1737360000","6aa8356f-0e1f-4842-b7f1-8035a97e8c95"
"Jan 20, 2025 @ 14:17:51.328","time=""2025-01-20T08:47:51.328Z"" level=info msg=""Could not acquire lock named: &{default atlas-config publish ConfigMap}"" namespace=default workflow=tableau-1717455078-cron-1737342000","6aa8356f-0e1f-4842-b7f1-8035a97e8c95"
"Jan 20, 2025 @ 14:17:51.318","time=""2025-01-20T08:47:51.318Z"" level=info msg=""SG Outbound nodes of tableau-1717455078-cron-1737342000-3921439732 are [tableau-1717455078-cron-1737342000-2725107359]"" namespace=default workflow=tableau-1717455078-cron-1737342000","6aa8356f-0e1f-4842-b7f1-8035a97e8c95"
"Jan 20, 2025 @ 14:17:51.317","time=""2025-01-20T08:47:51.317Z"" level=info msg=""SG Outbound nodes of tableau-1717455078-cron-1737342000-2231336966 are [tableau-1717455078-cron-1737342000-1671877509]"" namespace=default workflow=tableau-1717455078-cron-1737342000","6aa8356f-0e1f-4842-b7f1-8035a97e8c95"
"Jan 20, 2025 @ 14:17:51.317","time=""2025-01-20T08:47:51.317Z"" level=info msg=""SG Outbound nodes of tableau-1717455078-cron-1737342000-2791877580 are [tableau-1717455078-cron-1737342000-2791877580]"" namespace=default workflow=tableau-1717455078-cron-1737342000","6aa8356f-0e1f-4842-b7f1-8035a97e8c95"
"Jan 20, 2025 @ 14:17:51.317","time=""2025-01-20T08:47:51.317Z"" level=info msg=""SG Outbound nodes of tableau-1717455078-cron-1737342000-4294889011 are [tableau-1717455078-cron-1737342000-3883667262]"" namespace=default workflow=tableau-1717455078-cron-1737342000","6aa8356f-0e1f-4842-b7f1-8035a97e8c95"
"Jan 20, 2025 @ 14:17:51.295","time=""2025-01-20T08:47:51.295Z"" level=warning msg=""was unable to obtain the node for snowflake-1700253343-cron-1737360000-1818203418, taskName tag-attachment""","6aa8356f-0e1f-4842-b7f1-8035a97e8c95"
"Jan 20, 2025 @ 14:17:51.275","time=""2025-01-20T08:47:51.275Z"" level=info msg=""Task-result reconciliation"" namespace=default numObjs=60 workflow=tableau-1717455078-cron-1737342000","6aa8356f-0e1f-4842-b7f1-8035a97e8c95"
"Jan 20, 2025 @ 14:17:51.274","time=""2025-01-20T08:47:51.274Z"" level=info msg=""Processing workflow"" namespace=default workflow=tableau-1717455078-cron-1737342000","6aa8356f-0e1f-4842-b7f1-8035a97e8c95"
"Jan 20, 2025 @ 14:17:51.265","time=""2025-01-20T08:47:51.265Z"" level=info msg=""Task-result reconciliation"" namespace=default numObjs=18 workflow=snowflake-1700253343-cron-1737360000","6aa8356f-0e1f-4842-b7f1-8035a97e8c95"
"Jan 20, 2025 @ 14:17:51.264","time=""2025-01-20T08:47:51.264Z"" level=info msg=""Processing workflow"" namespace=default workflow=snowflake-1700253343-cron-1737360000","6aa8356f-0e1f-4842-b7f1-8035a97e8c95"
"Jan 20, 2025 @ 14:17:41.091","pod/argo-workflow-controller-6c748446bd-jhkfc not labeled","35d26b06-fbd9-4ec9-b0ae-82baee82a210"
"Jan 20, 2025 @ 14:17:40.954","Mon Jan 20 08:47:40 UTC 2025: Status: standby","35d26b06-fbd9-4ec9-b0ae-82baee82a210"
"Jan 20, 2025 @ 14:17:38.690","pod/argo-workflow-controller-6c748446bd-szdj7 not labeled","6aa8356f-0e1f-4842-b7f1-8035a97e8c95"
"Jan 20, 2025 @ 14:17:38.580","Mon Jan 20 08:47:38 UTC 2025: Status: standby","6aa8356f-0e1f-4842-b7f1-8035a97e8c95"
"Jan 20, 2025 @ 14:17:32.699","time=""2025-01-20T08:47:32.699Z"" level=info msg=""Workflow to be dehydrated"" Workflow Size=10639167","6aa8356f-0e1f-4842-b7f1-8035a97e8c95"
"Jan 20, 2025 @ 14:17:32.463","time=""2025-01-20T08:47:32.463Z"" level=info msg=""TaskSet Reconciliation"" namespace=default workflow=hive-1698426951-cron-1737275400","6aa8356f-0e1f-4842-b7f1-8035a97e8c95"
"Jan 20, 2025 @ 14:17:32.463","time=""2025-01-20T08:47:32.463Z"" level=info msg=reconcileAgentPod namespace=default workflow=hive-1698426951-cron-1737275400","6aa8356f-0e1f-4842-b7f1-8035a97e8c95"
"Jan 20, 2025 @ 14:17:32.462","time=""2025-01-20T08:47:32.462Z"" level=info msg=""Workflow step group node hive-1698426951-cron-1737275400-1091336000 not yet completed"" namespace=default workflow=hive-1698426951-cron-1737275400","6aa8356f-0e1f-4842-b7f1-8035a97e8c95"
"Jan 20, 2025 @ 14:17:32.462","time=""2025-01-20T08:47:32.462Z"" level=info msg=""Workflow step group node hive-1698426951-cron-1737275400-1768870649 not yet completed"" namespace=default workflow=hive-1698426951-cron-1737275400","6aa8356f-0e1f-4842-b7f1-8035a97e8c95"
"Jan 20, 2025 @ 14:17:32.462","time=""2025-01-20T08:47:32.462Z"" level=info msg=""Workflow step group node hive-1698426951-cron-1737275400-4028582987 not yet completed"" namespace=default workflow=hive-1698426951-cron-1737275400","6aa8356f-0e1f-4842-b7f1-8035a97e8c95"
"Jan 20, 2025 @ 14:17:32.459","time=""2025-01-20T08:47:32.459Z"" level=info msg=""Could not acquire lock named: &{default atlas-config publish ConfigMap}"" namespace=default workflow=hive-1698426951-cron-1737275400","6aa8356f-0e1f-4842-b7f1-8035a97e8c95"
"Jan 20, 2025 @ 14:17:32.459","time=""2025-01-20T08:47:32.459Z"" level=info msg=""Workflow step group node hive-1698426951-cron-1737275400-1517863364 not yet completed"" namespace=default workflow=hive-1698426951-cron-1737275400","6aa8356f-0e1f-4842-b7f1-8035a97e8c95"
"Jan 20, 2025 @ 14:17:32.450","time=""2025-01-20T08:47:32.450Z"" level=info msg=""Could not acquire lock named: &{default atlas-config publish ConfigMap}"" namespace=default workflow=hive-1698426951-cron-1737275400","6aa8356f-0e1f-4842-b7f1-8035a97e8c95"
"Jan 20, 2025 @ 14:17:32.442","time=""2025-01-20T08:47:32.442Z"" level=info msg=""Could not acquire lock named: &{default atlas-config publish ConfigMap}"" namespace=default workflow=hive-1698426951-cron-1737275400","6aa8356f-0e1f-4842-b7f1-8035a97e8c95"
"Jan 20, 2025 @ 14:17:32.433","time=""2025-01-20T08:47:32.433Z"" level=info msg=""Could not acquire lock named: &{default atlas-config publish ConfigMap}"" namespace=default workflow=hive-1698426951-cron-1737275400","6aa8356f-0e1f-4842-b7f1-8035a97e8c95"
"Jan 20, 2025 @ 14:17:32.425","time=""2025-01-20T08:47:32.425Z"" level=info msg=""Could not acquire lock named: &{default atlas-config publish ConfigMap}"" namespace=default workflow=hive-1698426951-cron-1737275400","6aa8356f-0e1f-4842-b7f1-8035a97e8c95"
"Jan 20, 2025 @ 14:17:32.418","time=""2025-01-20T08:47:32.418Z"" level=info msg=""Could not acquire lock named: &{default atlas-config publish ConfigMap}"" namespace=default workflow=hive-1698426951-cron-1737275400","6aa8356f-0e1f-4842-b7f1-8035a97e8c95"
"Jan 20, 2025 @ 14:17:32.410","time=""2025-01-20T08:47:32.410Z"" level=info msg=""Could not acquire lock named: &{default atlas-config publish ConfigMap}"" namespace=default workflow=hive-1698426951-cron-1737275400","6aa8356f-0e1f-4842-b7f1-8035a97e8c95"
"Jan 20, 2025 @ 14:17:32.400","time=""2025-01-20T08:47:32.400Z"" level=info msg=""Could not acquire lock named: &{default atlas-config publish ConfigMap}"" namespace=default workflow=hive-1698426951-cron-1737275400","6aa8356f-0e1f-4842-b7f1-8035a97e8c95"
"Jan 20, 2025 @ 14:17:32.386","time=""2025-01-20T08:47:32.386Z"" level=info msg=""Could not acquire lock named: &{default atlas-config publish ConfigMap}"" namespace=default workflow=hive-1698426951-cron-1737275400","6aa8356f-0e1f-4842-b7f1-8035a97e8c95"
"Jan 20, 2025 @ 14:17:32.375","time=""2025-01-20T08:47:32.375Z"" level=info msg=""Could not acquire lock named: &{default atlas-config publish ConfigMap}"" namespace=default workflow=hive-1698426951-cron-1737275400","6aa8356f-0e1f-4842-b7f1-8035a97e8c95"
"Jan 20, 2025 @ 14:17:32.365","time=""2025-01-20T08:47:32.365Z"" level=info msg=""Could not acquire lock named: &{default atlas-config publish ConfigMap}"" namespace=default workflow=hive-1698426951-cron-1737275400","6aa8356f-0e1f-4842-b7f1-8035a97e8c95"
"Jan 20, 2025 @ 14:17:32.354","time=""2025-01-20T08:47:32.354Z"" level=info msg=""Could not acquire lock named: &{default atlas-config publish ConfigMap}"" namespace=default workflow=hive-1698426951-cron-1737275400","6aa8356f-0e1f-4842-b7f1-8035a97e8c95"
"Jan 20, 2025 @ 14:17:32.345","time=""2025-01-20T08:47:32.345Z"" level=info msg=""Could not acquire lock named: &{default atlas-config publish ConfigMap}"" namespace=default workflow=hive-1698426951-cron-1737275400","6aa8356f-0e1f-4842-b7f1-8035a97e8c95"
"Jan 20, 2025 @ 14:17:32.335","time=""2025-01-20T08:47:32.335Z"" level=info msg=""Could not acquire lock named: &{default atlas-config publish ConfigMap}"" namespace=default workflow=hive-1698426951-cron-1737275400","6aa8356f-0e1f-4842-b7f1-8035a97e8c95"
"Jan 20, 2025 @ 14:17:32.327","time=""2025-01-20T08:47:32.327Z"" level=info msg=""Could not acquire lock named: &{default atlas-config publish ConfigMap}"" namespace=default workflow=hive-1698426951-cron-1737275400","6aa8356f-0e1f-4842-b7f1-8035a97e8c95"
"Jan 20, 2025 @ 14:17:32.320","time=""2025-01-20T08:47:32.320Z"" level=info msg=""Could not acquire lock named: &{default atlas-config publish ConfigMap}"" namespace=default workflow=hive-1698426951-cron-1737275400","6aa8356f-0e1f-4842-b7f1-8035a97e8c95"
"Jan 20, 2025 @ 14:17:32.312","time=""2025-01-20T08:47:32.312Z"" level=info msg=""Could not acquire lock named: &{default atlas-config publish ConfigMap}"" namespace=default workflow=hive-1698426951-cron-1737275400","6aa8356f-0e1f-4842-b7f1-8035a97e8c95"
"Jan 20, 2025 @ 14:17:32.305","time=""2025-01-20T08:47:32.305Z"" level=info msg=""Could not acquire lock named: &{default atlas-config publish ConfigMap}"" namespace=default workflow=hive-1698426951-cron-1737275400","6aa8356f-0e1f-4842-b7f1-8035a97e8c95"
"Jan 20, 2025 @ 14:17:32.299","time=""2025-01-20T08:47:32.299Z"" level=info msg=""Could not acquire lock named: &{default atlas-config publish ConfigMap}"" namespace=default workflow=hive-1698426951-cron-1737275400","6aa8356f-0e1f-4842-b7f1-8035a97e8c95"
"Jan 20, 2025 @ 14:17:32.290","time=""2025-01-20T08:47:32.290Z"" level=info msg=""Could not acquire lock named: &{default atlas-config publish ConfigMap}"" namespace=default workflow=hive-1698426951-cron-1737275400","6aa8356f-0e1f-4842-b7f1-8035a97e8c95"
"Jan 20, 2025 @ 14:17:32.281","time=""2025-01-20T08:47:32.281Z"" level=info msg=""Could not acquire lock named: &{default atlas-config publish ConfigMap}"" namespace=default workflow=hive-1698426951-cron-1737275400","6aa8356f-0e1f-4842-b7f1-8035a97e8c95"
"Jan 20, 2025 @ 14:17:32.274","time=""2025-01-20T08:47:32.274Z"" level=info msg=""Could not acquire lock named: &{default atlas-config publish ConfigMap}"" namespace=default workflow=hive-1698426951-cron-1737275400","6aa8356f-0e1f-4842-b7f1-8035a97e8c95"
"Jan 20, 2025 @ 14:17:32.266","time=""2025-01-20T08:47:32.266Z"" level=info msg=""Could not acquire lock named: &{default atlas-config publish ConfigMap}"" namespace=default workflow=hive-1698426951-cron-1737275400","6aa8356f-0e1f-4842-b7f1-8035a97e8c95"
"Jan 20, 2025 @ 14:17:32.257","time=""2025-01-20T08:47:32.257Z"" level=info msg=""Workflow step group node hive-1698426951-cron-1737275400-2410967227 not yet completed"" namespace=default workflow=hive-1698426951-cron-1737275400","6aa8356f-0e1f-4842-b7f1-8035a97e8c95"
"Jan 20, 2025 @ 14:17:32.241","time=""2025-01-20T08:47:32.241Z"" level=info msg=""Node hive-1698426951-cron-1737275400(0).run(0).publish(0)[0].publish(0)[0].publish-prod-v2(0)[2].publish(0)[2].publish-columns(19:19) acquired synchronization lock"" namespace=default workflow=hive-1698426951-cron-1737275400","6aa8356f-0e1f-4842-b7f1-8035a97e8c95"
"Jan 20, 2025 @ 14:17:32.233","time=""2025-01-20T08:47:32.233Z"" level=info msg=""Workflow step group node hive-1698426951-cron-1737275400-3406651981 not yet completed"" namespace=default workflow=hive-1698426951-cron-1737275400","6aa8356f-0e1f-4842-b7f1-8035a97e8c95"
"Jan 20, 2025 @ 14:17:32.222","time=""2025-01-20T08:47:32.222Z"" level=info msg=""Node hive-1698426951-cron-1737275400(0).run(0).publish(0)[0].publish(0)[0].publish-prod-v2(0)[2].publish(0)[2].publish-columns(18:18) acquired synchronization lock"" namespace=default workflow=hive-1698426951-cron-1737275400","6aa8356f-0e1f-4842-b7f1-8035a97e8c95"
"Jan 20, 2025 @ 14:17:32.214","time=""2025-01-20T08:47:32.214Z"" level=info msg=""Workflow step group node hive-1698426951-cron-1737275400-3249763399 not yet completed"" namespace=default workflow=hive-1698426951-cron-1737275400","6aa8356f-0e1f-4842-b7f1-8035a97e8c95"
"Jan 20, 2025 @ 14:17:32.202","time=""2025-01-20T08:47:32.202Z"" level=info msg=""Node hive-1698426951-cron-1737275400(0).run(0).publish(0)[0].publish(0)[0].publish-prod-v2(0)[2].publish(0)[2].publish-columns(17:17) acquired synchronization lock"" namespace=default workflow=hive-1698426951-cron-1737275400","6aa8356f-0e1f-4842-b7f1-8035a97e8c95"
"Jan 20, 2025 @ 14:17:32.194","time=""2025-01-20T08:47:32.194Z"" level=info msg=""Workflow step group node hive-1698426951-cron-1737275400-136933629 not yet completed"" namespace=default workflow=hive-1698426951-cron-1737275400","6aa8356f-0e1f-4842-b7f1-8035a97e8c95"
"Jan 20, 2025 @ 14:17:32.175","time=""2025-01-20T08:47:32.175Z"" level=info msg=""Node hive-1698426951-cron-1737275400(0).run(0).publish(0)[0].publish(0)[0].publish-prod-v2(0)[2].publish(0)[2].publish-columns(16:16) acquired synchronization lock"" namespace=default workflow=hive-1698426951-cron-1737275400","6aa8356f-0e1f-4842-b7f1-8035a97e8c95"
"Jan 20, 2025 @ 14:17:32.165","time=""2025-01-20T08:47:32.165Z"" level=info msg=""Workflow step group node hive-1698426951-cron-1737275400-4072834507 not yet completed"" namespace=default workflow=hive-1698426951-cron-1737275400","6aa8356f-0e1f-4842-b7f1-8035a97e8c95"
"Jan 20, 2025 @ 14:17:32.146","time=""2025-01-20T08:47:32.146Z"" level=info msg=""Node hive-1698426951-cron-1737275400(0).run(0).publish(0)[0].publish(0)[0].publish-prod-v2(0)[2].publish(0)[2].publish-columns(15:15) acquired synchronization lock"" namespace=default workflow=hive-1698426951-cron-1737275400","6aa8356f-0e1f-4842-b7f1-8035a97e8c95"
"Jan 20, 2025 @ 14:17:32.088","time=""2025-01-20T08:47:32.088Z"" level=info msg=""SG Outbound nodes of hive-1698426951-cron-1737275400-2880835072 are [hive-1698426951-cron-1737275400-4206186851]"" namespace=default workflow=hive-1698426951-cron-1737275400","6aa8356f-0e1f-4842-b7f1-8035a97e8c95"
"Jan 20, 2025 @ 14:17:32.088","time=""2025-01-20T08:47:32.088Z"" level=info msg=""SG Outbound nodes of hive-1698426951-cron-1737275400-932093800 are [hive-1698426951-cron-1737275400-3709443067]"" namespace=default workflow=hive-1698426951-cron-1737275400","6aa8356f-0e1f-4842-b7f1-8035a97e8c95"
"Jan 20, 2025 @ 14:17:32.088","time=""2025-01-20T08:47:32.088Z"" level=info msg=""Step 'nil' has no expanded child nodes"" namespace=default workflow=hive-1698426951-cron-1737275400","6aa8356f-0e1f-4842-b7f1-8035a97e8c95"
"Jan 20, 2025 @ 14:17:32.088","time=""2025-01-20T08:47:32.088Z"" level=info msg=""SG Outbound nodes of hive-1698426951-cron-1737275400-3949639979 are [hive-1698426951-cron-1737275400-3949639979]"" namespace=default workflow=hive-1698426951-cron-1737275400","6aa8356f-0e1f-4842-b7f1-8035a97e8c95"
"Jan 20, 2025 @ 14:17:32.088","time=""2025-01-20T08:47:32.088Z"" level=info msg=""SG Outbound nodes of hive-1698426951-cron-1737275400-1051173126 are [hive-1698426951-cron-1737275400-1051173126]"" namespace=default workflow=hive-1698426951-cron-1737275400","6aa8356f-0e1f-4842-b7f1-8035a97e8c95"
"Jan 20, 2025 @ 14:17:32.088","time=""2025-01-20T08:47:32.088Z"" level=info msg=""SG Outbound nodes of hive-1698426951-cron-1737275400-2749996612 are [hive-1698426951-cron-1737275400-2749996612]"" namespace=default workflow=hive-1698426951-cron-1737275400","6aa8356f-0e1f-4842-b7f1-8035a97e8c95"
"Jan 20, 2025 @ 14:17:32.086","time=""2025-01-20T08:47:32.086Z"" level=info msg=""Step 'nil' has no expanded child nodes"" namespace=default workflow=hive-1698426951-cron-1737275400","6aa8356f-0e1f-4842-b7f1-8035a97e8c95"
"Jan 20, 2025 @ 14:17:32.086","time=""2025-01-20T08:47:32.086Z"" level=info msg=""SG Outbound nodes of hive-1698426951-cron-1737275400-877246396 are [hive-1698426951-cron-1737275400-1819683959]"" namespace=default workflow=hive-1698426951-cron-1737275400","6aa8356f-0e1f-4842-b7f1-8035a97e8c95"
"Jan 20, 2025 @ 14:17:32.086","time=""2025-01-20T08:47:32.086Z"" level=info msg=""SG Outbound nodes of hive-1698426951-cron-1737275400-2811568410 are [hive-1698426951-cron-1737275400-2811568410]"" namespace=default workflow=hive-1698426951-cron-1737275400","6aa8356f-0e1f-4842-b7f1-8035a97e8c95"
"Jan 20, 2025 @ 14:17:32.086","time=""2025-01-20T08:47:32.086Z"" level=info msg=""SG Outbound nodes of hive-1698426951-cron-1737275400-2714757145 are [hive-1698426951-cron-1737275400-2714757145]"" namespace=default workflow=hive-1698426951-cron-1737275400","6aa8356f-0e1f-4842-b7f1-8035a97e8c95"
"Jan 20, 2025 @ 14:17:32.086","time=""2025-01-20T08:47:32.086Z"" level=info msg=""SG Outbound nodes of hive-1698426951-cron-1737275400-2400259535 are [hive-1698426951-cron-1737275400-2400259535]"" namespace=default workflow=hive-1698426951-cron-1737275400","6aa8356f-0e1f-4842-b7f1-8035a97e8c95"
"Jan 20, 2025 @ 14:17:32.086","time=""2025-01-20T08:47:32.086Z"" level=info msg=""SG Outbound nodes of hive-1698426951-cron-1737275400-560886924 are [hive-1698426951-cron-1737275400-1394307591]"" namespace=default workflow=hive-1698426951-cron-1737275400","6aa8356f-0e1f-4842-b7f1-8035a97e8c95"
"Jan 20, 2025 @ 14:17:32.086","time=""2025-01-20T08:47:32.086Z"" level=info msg=""SG Outbound nodes of hive-1698426951-cron-1737275400-1360375167 are [hive-1698426951-cron-1737275400-2923482626]"" namespace=default workflow=hive-1698426951-cron-1737275400","6aa8356f-0e1f-4842-b7f1-8035a97e8c95"
"Jan 20, 2025 @ 14:17:31.999","time=""2025-01-20T08:47:31.999Z"" level=info msg=""Task-result reconciliation"" namespace=default numObjs=1604 workflow=hive-1698426951-cron-1737275400","6aa8356f-0e1f-4842-b7f1-8035a97e8c95"
"Jan 20, 2025 @ 14:17:31.998","time=""2025-01-20T08:47:31.998Z"" level=info msg=""Processing workflow"" namespace=default workflow=hive-1698426951-cron-1737275400","6aa8356f-0e1f-4842-b7f1-8035a97e8c95"

Logs from in your workflow's wait container

kubectl logs -n argo -c wait -l workflows.argoproj.io/workflow=${workflow},workflow.argoproj.io/phase!=Succeeded
@tczhao tczhao changed the title workflow postponed due parallelism limit after running a running workflow turn postponed due parallelism limit Jan 24, 2025
@tczhao tczhao added the area/controller Controller issues, panics label Jan 24, 2025
@tooptoop4
Copy link
Contributor

maybe similar to #14100

@jswxstw
Copy link
Member

jswxstw commented Jan 26, 2025

maybe similar to #14100

@tooptoop4 same as #12103?

@tooptoop4
Copy link
Contributor

that too yes

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
area/controller Controller issues, panics type/bug
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@jswxstw @tczhao @tooptoop4