Skip to content

Latest commit

 

History

History
67 lines (60 loc) · 2.81 KB

20240503-minutes.md

File metadata and controls

67 lines (60 loc) · 2.81 KB

S3 Working Group Minutes

Friday, 2024/05/03

3:00 pm ET

Attendees:

Terrell Russell, Alan King, Illyoung Choi (CyVerse), Tony Edgin (CyVerse)

Minutes

DISCUSSION

  • iRODS 4.3.2 (and S3 plugin) released
    • CMake updates
    • Bugfixes and server response cleanup
  • iRODS S3 API
    • Multipart was merged
    • 0.2.0 was released
    • Tony's speed tests added to March minutes
      • v0.1.0 - 1404 seconds for 1G upload
      • v0.2.0 - 180 seconds for 1G upload
      • Still not the 13s from iCommands
      • Still doing store-and-forward, opportunity for good ideas here
  • Tony use case
    • S3 bucket with lots of files
    • Wants to run CyVerse analysis on it
    • Can we mount a bucket as s3 resource
    • This is related to BYOB (bring your own bucket)
    • Can we set the root / mount path for s3 plugin into the irods namespace
      • New context string option...
        • S3_TRUNCATE_PHYSICAL_PATH_INITIAL_ELEMENTS=2
        • S3_TRUNCATE_ROOT_PHYSICAL_PATH_ELEMENTS=3
        • S3_TRUNCATE_PHYSICAL_PATH_STRING='/zonename/home/user'
        • S3_TRUNCATE_VAULT_PATH_STRING='/zonename/home/user'
        • S3_TRUNCATE_VAULT_PATH_ELEMENTS=3
        • S3_CHOP_INITIAL_VAULT_PATH_ELEMENTS=3
        • S3_LTRIM_VAULT_PATH='/zonename/home/user'
        • S3_LTRIM_LOGICAL_PATH='/zonename/home/user'
        • S3_TRUNCATE_INITIAL_PATH_STRING='/zonename/home/user'
        • S3_CHROOT='/zonename/home/user'
        • S3_SPLITROOT='/zonename/home/user'
        • ROOT_COLL - CHROOT from the zone to something deeper
        • MOUNT_COLL - same as ROOT_COLL - synonym
        • And what's the default? Just get the fullpath, like always.
        • Doesn't want `/zonename/home/username/remotes3'
        • A third ARCHIVE_NAMING_POLICY
          • in addition to decoupled/consistent...
          • calling this one 'chroot', like consistent, but trimmed/shortened/reduced
          • and if in this mode, the plugin honors this new context string option
      • Concern: user can now touch 'out-of-band' from irods protocol
        • Going to have drift
        • Have to use iscan / ifsck, etc.
        • Registration tool
      • But only for a single user?
      • Personal resource (resource per person) -> s3 bucket
        • and cyverse stores the keypair and is trusted by the user(!)
    • Decoupled mode wouldn't help - would be gibberish to the human later
    • iadmin modrescdatapaths
      • modrescdatapaths Name oldpath newpath [user] (update data-object paths, sometimes needed after modresc path)
    • A rule would have to be written that protects that resource to only be written by a certain user
      • otherwise, other users could write into a different user's bucket
      • Resource level operations would be where the PEP would be most effective (open/read/write)
        • RESOURCE_OP_REGISTERED is called when something is created
  • Next meeting
    • June 2024