Test and apply the BB8 Daemons

[cmoore24-24]

Looking into how BB8 rebalancing works, and how it can be applied system-wide. Will create transfer and rebalancing benchmarks, and identify proper use cases.

[dynamic-entropy]

Hi @cmoore24-24
Maybe you can evaluate against US T2 being highly locked scenarios, https://its.cern.ch/jira/browse/CMSTRANSF-709

[cmoore24-24]

As it stands, BB8 is a little limited. If we could add some way to rebalance RSEs by account (so as to avoid workflow related rules that shouldn't be touched) I think it would have much more potential.

[cmoore24-24]

Adding a task list to more finely track this issue; progress has been slow but I will be committing more time until it is complete.

Tasks

• BB8 Core Function Testing

  • Rebalance Rule Complete: Used the rebalance rule to successfully move a rule to a new RSE with the BB8 signature
  • List Rebalance Rule Candidates Complete: Returns a list of rules that match all of the parameters in the rule_clause list. This can be updated manually, but the majority of the additions are made automatically using options given directly to BB8.
  • Select Target RSE In Progress: Running into problems with RSESelector; unable to produce a list of target RSEs because of the error rucio.common.exception.InsufficientTargetRSEs: There are not enough target RSEs to fulfil the request at this time. Details: Target RSE set not sufficient for number of copies. (1 copies requested, RSE set size 0)
  • Rebalance RSE Waiting

• Implement a small scale test of BB8

[ericvaandering]

Can we update this? I opened a new issue, but that may not be quite right with the info that's here.

[cmoore24-24]

Over the course of Summer 2024, BB8 was brought into working order by a Fermilab summer student. The final product was passed to DM, who discovered that there was an issue with how BB8 was constructing its list of data that can be moved (prioritizing datasets over containers). This led to a test where much more data was rebalanced from T2_CH_CERN than was intended (which has since been cleaned up).

This was addressed by @amanrique1, who made changes to the SQL query that built this list. A pull request is currently open for CMSRucio #850 to add this change, as well as the summer student's work, as a patch. Simultaneously, there is an outstanding pull request to add this SQL change to core rucio.

@amanrique1 has worked on adding subscription functionality to BB8. We've tested this functionality, and I believe @amanrique1 has discussed the result with the DM team.