Skip to content

Latest commit

 

History

History
196 lines (154 loc) · 6.3 KB

20211105-minutes.md

File metadata and controls

196 lines (154 loc) · 6.3 KB

S3 Working Group Minutes

Friday, 2021/11/05

3:00 pm ET

Attendees:

Terrell Russell, Dave Fellinger, Alan King, Kory Draughn, Justin James, Deep Patel (NIEHS), Illyoung Choi (CyVerse), Tony Edgin (CyVerse), Nirav Merchant (CyVerse)

Minutes

DISCUSSION

  • RENCI Update
    • Designed and built a python flask app, but it would require admin account
    • Then learned anonymous could GenQuery for which logical collection (not a data object) is associated with an existing ticket
    • So...
    • Need to confirm "anonymous:" with empty password creates correct auth header - REST C++ needs to learn about 'anonymous' special user
    • Hope this means that Minio can just catch this ticket value and use as "ticket:kjsdfkjsdfkjdfkjaskjas" in the minio login form.
  • CyVerse Update
    • Tested parallel upload with 4.2.10 in container - success
    • Will start testing minio integration
    • Go-irodsclient on windows also worked well
      • Interested in packaging for windows/mac/linux
    • Gateway will use 'anonymous' to query for associated target collection
      • iquest "select TICKET_COLL_NAME where TICKET_ID = 'xxxxxx'"
    • Then use ticket access to drop user into that collection in minio GUI
  • Next Meeting - Friday, Dec 3

WORKING NOTES - DEVELOPMENT PRIOR TO MEETING

Mental Model - App wants to put data into S3

  • App gets s3 credentials from Amazon or someone else...
  • App talks to s3 via s3 protocol
  • Data is now in s3

Actual Model - App wants to put data in iRODS (but still thinks it's s3)

  • App goes to ticketbooth (gets s3 credentials, actually just "ticket"/idfhiuer398343984tiuh
  • App talks to minIO via s3 protocol (with "ticket"/idfhiuer398343984tiuh)
  • Data is now in iRODS

So maybe our code is only an overlay over the existing REST API?

Candidate REST endpoints...

  • Create (new ticket)
    • Authentication - iRODS credentials
    • Requires
      • Collection fullpath
    • Actions
      • Generates a ticket (GUID)
      • puts the GUID as AVU on the collection
    • Returns
      • GUID
  • Resolve (ticket to collection) (for MinIO / other GUIs)
    • Authentication - not sure… none? Admin?
      • If none… can none run a genquery?
      • Can a ticket provide genquery permission? Instead of read/write?
        • No
      • Solution… ticketbooth deploy requires an admin account
        • Would only be used here, to do this lookup/resolve
        • Because minIO isn't iRODS authenticated itself...
    • Requires
      • GUID
    • Actions
      • GenQuery to lookup collection via GUID
    • Returns
      • Collection fullpath
  • Revoke (my ticket)
    • Authentication - iRODS credentials
    • Requires
      • GUID
    • Actions
      • Deletes ticket
    • Returns
      • success
  • List (all (my) existing tickets)
    • Authentication - iRODS credentials
    • Requires
      • Nothing
    • Actions
      • GenQuery for my tickets
    • Returns
      • List of GUIDs?

Questions...

  • Ticketbooth is authenticated?
  • iRODS credentials?
  • Does it need admin credentials?
  • Seems we need to know that joe is asking for a ticket for joe's collection.

If this is a subset/clone of https://github.com/irods/irods_client_rest_cpp ?

  • It would not need more than host, zone, port in a config file
  • Perfect to be dockerized
  • Nginx reverse proxy... to four endpoints?
  • Docker-compose can do it all?
  • Deployment really could be just 'docker-compose up' with 3 environment variables... does that cover TLS/SSL to iRODS? a certificate?
  • Resolve endpoint apparently needs an admin account... so we're back to a config file with rodsadmin/password...

Use case, we want to encode the fullpath in the ticketvalue

  • So GUI doesn't have to hit 'resolve', which would have required the ticketbooth to have admin account information
  • Can we just use a JWT to hold the fullpath? And then gateway-irods.go can decode the JWT with a shared secret and know where to navigate?

JWT BRAINSTORMING

  • TicketBooth and MinIO will share a secret
    • Ticketbooth - can hold the secret in the config file
    • MinIO - secret can be an ENV or docker mount (from a config file)

Candidate REST endpoints...

  • Create (new ticket)
    • Authentication - iRODS credentials
    • Requires
      • Collection fullpath
    • Actions
      • Generates a ticket (JWT, which encodes fullpath, via shared secret)
    • Returns
      • JWT
  • Revoke (my ticket)
    • Authentication - iRODS credentials
    • Requires
      • JWT
    • Actions
      • Deletes ticket
    • Returns
      • Success
  • List (all (my) existing tickets)
    • Authentication - iRODS credentials
    • Requires
      • Nothing
    • Actions
      • GenQuery for my tickets
    • Returns
      • List of JWTs?

If this is true...

  • No rodsadmin required for ticketbooth app
    • Config file just has
      • Zone, port, host
      • Shared secret
      • SSL stuff? Other irods environment things?
  • MinIO just gets shared secret
    • IRODS_TICKETBOOTH_SHARED_SECRET ?
  • Nothing special happens in iRODS Server, ... doesn't have to be the latest server, this is vanilla stuff.

Why do we need fullpath?

  • Only for GUIs
  • CLIs supply a fullpath directly
    • mc cp mylocalfile.txt s3://bucketname/prefix1/2/3/filename.txt

minIO GUI

  • access/secret
  • Draws a page, drops me somewhere...? But where...
  • With a ticket... it shouldn't drop me at the root... or my home collection, i don't have one (not a iRODS user)...
    • so it has to 'resolve' or 'decode' my fullpath... and drop me there, and draw a directory (collection) tree.

Made a repository…

Can it just be a Python Flask app?

  • MVP is a single endpoint
    • /create
  • Other two endpoints are nice-to-haves
    • /revoke
    • /list
  • Easy to dockerize
    • Single config_file

Developed the s3 ticketbooth flask app...

  • determined it looks extremely similar/identical to the tickets section of the C++ REST API
  • C++ REST API just needs more flexibility/functionality for the ticket endpoint
    • more knobs for uses/bytes/expiry/etc.

So... added the functionality to the C++ REST API

Will probably abandon/archive the new repository for S3 Ticketbooth