[Bug]: test_pfcwd_wb.py async storm always fails in KVM T0 testbed

[xwjiang-ms]

Issue Description

test_pfcwd_wb.py async storm always fails in KVM T0 testbed with config reload failed.

Results you see

config reload failed with error message:

2024 Dec 23 03:03:10.477773 vlab-01 ERR monit[12306]: Unix socket /var/run/monit.sock connection error -- No such file or directory

Results you expected to see

No error in config reload

Is it platform specific

generic

Relevant log output

No response

Output of show version

No response

Attach files (if any)

No response

[bingwang-ms]

The error below is caused by a known regression, which has been fixed by PR sonic-net/sonic-utilities#3698. Can you check if the issue has been addressed?

2024 Dec 23 03:03:10.477773 vlab-01 ERR monit[12306]: Unix socket /var/run/monit.sock connection error -- No such file or directory

[yxieca]

@xwjiang-ms can you check according to Bing's update?

[xwjiang-ms]

The socket connection error seems got resolved, but it still fails for:

08:55:34 __init__.pytest_runtest_call             L0040 ERROR  | Traceback (most recent call last):
  File "/usr/local/lib/python3.8/dist-packages/_pytest/python.py", line 1788, in runtest
    self.ihook.pytest_pyfunc_call(pyfuncitem=self)
  File "/usr/local/lib/python3.8/dist-packages/pluggy/_hooks.py", line 513, in __call__
    return self._hookexec(self.name, self._hookimpls.copy(), kwargs, firstresult)
  File "/usr/local/lib/python3.8/dist-packages/pluggy/_manager.py", line 120, in _hookexec
    return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
  File "/usr/local/lib/python3.8/dist-packages/pluggy/_callers.py", line 139, in _multicall
    raise exception.with_traceback(exception.__traceback__)
  File "/usr/local/lib/python3.8/dist-packages/pluggy/_callers.py", line 103, in _multicall
    res = hook_impl.function(*args)
  File "/usr/local/lib/python3.8/dist-packages/_pytest/python.py", line 194, in pytest_pyfunc_call
    result = testfunction(**testargs)
  File "/var/src/sonic-mgmt/tests/pfcwd/test_pfcwd_warm_reboot.py", line 635, in test_pfcwd_wb
    self.pfcwd_wb_helper(fake_storm, TESTCASE_INFO[testcase_action]['test_sequence'], setup_pfc_test,
  File "/var/src/sonic-mgmt/tests/pfcwd/test_pfcwd_warm_reboot.py", line 530, in pfcwd_wb_helper
    reboot(self.dut, localhost, reboot_type="warm", wait_warmboot_finalizer=True)
  File "/var/src/sonic-mgmt/tests/common/reboot.py", line 319, in reboot
    raise Exception('warmboot-finalizer service timeout on DUT {}'.format(hostname))
Exception: warmboot-finalizer service timeout on DUT vlab-01

I'm not sure if below error message is related to warmboot-finalizer inactive, @vaibhavhd @saiarcot895 could you help to check?

2025 Jan  9 08:55:29.809246 vlab-01 ERR swss#supervisor-proc-exit-listener: Process 'orchagent' is not running in namespace 'host' (4.0 minutes).