Have default memory be 3.0 GiB

[LinaresToine]

Fixes #12086

Status

Ready

Description

Express merge tasks now require 3.0 GiB

Is it backward compatible (if not, which system it affects?)

YES

Related PRs

#12085

12085 is closed now.

External dependencies / deployment changes

NO

[dmwm-bot]

Jenkins results:

• Python3 Unit tests: succeeded
  • 3 tests no longer failing
  • 2 changes in unstable tests
• Python3 Pylint check: failed
  • 7 warnings and errors that must be fixed
  • 2 warnings
  • 88 comments to review
• Pycodestyle check: succeeded

Details at https://cmssdt.cern.ch/dmwm-jenkins/view/All/job/WMCore-PR-Report/275/artifact/artifacts/PullRequestReport.html

[amaltaro]

@LinaresToine thank you for providing these changes.

I was probably not clear with my statement, as the current change would affect the default merge job memory requirements to 3GB for everything using WMCore - not only Express workflows. Actually, this does not even change the memory requirements, but only the memory watchdog.

Looking into details of the Express factory, I see it has a custom method for creating a merge task:

WMCore/src/python/WMCore/WMSpec/StdSpecs/Express.py

Line 249 in 4c4b5cf

def addExpressMergeTask(self, parentTask, parentStepName, parentOutputModuleName):

Perhaps a better change for the maxpss would be to make it configurable here in the addRuntimeMonitors() method, through an extra argument that would have a default of maxpss = 2.3 * 1024.

Then in the Express factory, you pass the new value for maxpss when calling addRuntimeMonitors(), in this line:
https://github.com/dmwm/WMCore/blob/master/src/python/WMCore/WMSpec/StdSpecs/Express.py#L268

What is left to do is, to check where Merge memory requirement can be defined and define it over there as well. Apparently done here: https://github.com/dmwm/WMCore/blob/master/src/python/WMCore/WMSpec/WMTask.py#L599, but we don't seem to ever set these requirements... Probably something that we need to change for central production as well!

[LinaresToine]

Thank you for your clarification @amaltaro. Indeed what I was proposing would have affected every type of merge task. Also thank you for your suggestions, with them I updated the PR and created the PR dmwm/T0#5030 in the T0 repository. Let me know what you think.

[dmwm-bot]

Jenkins results:

• Python3 Unit tests: failed
  • 5 new failures
  • 3 tests no longer failing
  • 2 changes in unstable tests
• Python3 Pylint check: failed
  • 14 warnings and errors that must be fixed
  • 2 warnings
  • 105 comments to review
• Pycodestyle check: succeeded
  • 2 comments to review

Details at https://cmssdt.cern.ch/dmwm-jenkins/view/All/job/WMCore-PR-Report/276/artifact/artifacts/PullRequestReport.html