ReqMgr2 MicroService Pileup

MSPileup micro-service

Data management layer

The MSPileup data layer defines the following MSPileup data-structure:

{"pileupName": string,
  "pileupType": string,
  "insertTime": integer,
  "lastUpdateTime": integer,
  "expectedRSEs": list of strings,
  "currentRSEs": list of strings,
  "fullReplicas": integer,
  "campaigns": list of strings,
  "containerFraction": float,
  "replicationGrouping": string,
  "activatedOn": integer,
  "deactivatedOn": integer,
  "pileupSize": integer,
  "rulesList": list of strings,
  "active": boolean
}

where:

• pileupName: pileup dataset
• pileupType: either "premix" or "classic" value. Identifying which pileup data it corresponds to.
• insertTime: the timestamp (seconds since epoch in GMT) of when this pileup id was first defined in the microservice (by an user)
• `lastUpdateTime: the timestamp (seconds since epoch in GMT) of the last modification made to this pileup structure (by an user)
• expectedRSEs: to be provided by someone (P&R and/or DM), ["Disk1", "Disk2"]. This list of RSEs need to be properly validated against the known RSE names and ensured that it corresponds to a Disk RSE.
• currentRSEs: to be filled up by the micro-service itself. e.g. ["Disk3", "Disk4"], as a result of continuous data location. Once there is a lock for the pileup id in question, the RSE name should be written here.
• fullReplicas: will eventually supersede "expectedRSEs". To be used whenever the micro-service is mature enough and automatic data placement decisions can be performed.
• active: whether the pileup id is meant to be used by any campaign/workflow. False means it's no longer used and that replicas should be removed from Disk.
• campaigns: a list of campaigns
• containerFraction: a real number that corresponds to the fraction number of blocks to be replicated in a given RSE (default value is 1.0, full container at the same RSE - only supported case at this very moment)
• replicationGrouping: matching the grouping granularity field provided by Rucio. Allowed values would be DATASET (data placement by block) or ALL (whole container is placed under the same RSE)
• activatedOn: seconds since epoch in GMT, a date of when this pileup last became active
• deactivatedOn: seconds since epoch in GMT, a date of when this pileup last became deactive.
• pileupSize: to be updated by the microservice itself with the current size of the pileup in bytes
• rulesList: list of strings (rules) used to lock the pileup id.

These pileup objects are supposed to be stored in a central database (MongoDB?) in a single document, with a list data structure like:

[<pileup object 1>, <pileup object 2>, ..., <pileup object n>]