Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Improve data directory functionality #1509

Merged
merged 23 commits into from
Jan 13, 2025
Merged

Improve data directory functionality #1509

merged 23 commits into from
Jan 13, 2025

Conversation

Tansito
Copy link
Member

@Tansito Tansito commented Oct 7, 2024 •
edited
Loading

Summary

This PR pretends to standardize the way that users will access to the file system to upload and interact with files in the Qiskit Functions. The new interface will generate the next UX:

  • User persona
from qiskit_ibm_catalog import QiskitFunctionsCatalog

catalog = QiskitFunctionsCatalog(token=...)

function = catalog.load(".../...")

# It will list files in the user's function folder
catalog.files(function)

# It will upload a file in the user's function folder
catalog.file_upload(filename, function)

# It will download a file in the user's function folder
catalog.file_download(filename, function)

# It will delete a file in the user's function folder
serverless.file_delete(filename, function)
  • Provider persona
from qiskit_ibm_catalog import QiskitFunctionsCatalog

catalog = QiskitFunctionsCatalog(token=...)

function = catalog.load(".../...")

# It will list files in the user's function folder
catalog.provider_files(function)

# It will upload a file in the user's function folder
catalog.provider_file_upload(filename, function)

# It will download a file in the user's function folder
catalog.provider_file_download(filename, function)

# It will delete a file in the user's function folder
catalog.provider_file_delete(filename, function)

The functionality inside the Qiskit Function will remain the same. The same two folders to manage the different kind of content:

  • /data: to store files for the user
  • /function_data: to store files for the partners

Details and comments

  • End-Points migrated
  • Qiskit Serverless client migrated
  • Scheduler migrated
  • Documentation updated
  • Qiskit IBM Catalog migrated

@Tansito Tansito mentioned this pull request Oct 14, 2024
Merged
@Tansito Tansito mentioned this pull request Oct 29, 2024
Merged
@Tansito
Merge branch 'main' into data-folder
a9ad7ba
@Tansito
Files list end point refactor (#1529)
71542fd 
* First refactorization for files

* Completed use cases for list end-point

* Renamed storage to file_storage

* Migrated files to file_storage

* Migrated get files to the new service

* Check if function exists first

* Separated end-points to the new provider

* Improved tests

* Restore original fixtures

* Solved the problem with the external if

* Added Literal type in the FileStorage service

* functions methods refactorization

* use self.username instead of the variable

* run path sanitization only one time

* function title is mandatory for file storage

* fix black

* Updated 403 by 404

* Updated swagger for list

* Makes use of enum with integers

* use is instead of equals

* unify path build

* get_function refactor
@Tansito
Copy link
Member Author

Tansito commented Nov 21, 2024

Failing tests at this point of the development are expected. API changes will require client migrations.

@Tansito Tansito changed the title 🏗️ Data folder refactorization 🏗️ Data folder refactor Nov 27, 2024
@Tansito Tansito changed the title 🏗️ Data folder refactor 🏗️ Refactor: data folder Dec 10, 2024
korgan00 and others added 8 commits December 10, 2024 14:32
@korgan00
Add support new files (#1546)
19d9602 
* new list files for the gateway refactor

* unify files url

* remove enum and linter

* remove provider parameter

* Check provider
@Tansito @korgan00
Download end point refactor (#1547)
1e9fae0 
* refactor of download end-points

* additional test to check non existing file

* renamed files test to v1_files

* added additional checks for the query

* included the checks in the list end-point

* include swagger documentation updated

* remove unneeded try except

* make use of regex instead of a manual parsing

* check not all instead of None

* rename file_extension_is_valid

* Update gateway/api/utils.py

Co-authored-by: Goyo <[email protected]>

---------

Co-authored-by: Goyo <[email protected]>
@Tansito
Merge branch 'main' into data-folder
1d0803e
@korgan00
add support to new download files endpoints (#1550)
ac33aab
@Tansito
Merge branch 'main' into data-folder
47c3064
@korgan00
Gateway/delete end point (#1554)
41be2f0 
* add support to new delete files endpoints

* fix get parameter

* fix tests

* delete programDelete fixture

* update file_storage/remove_file description
@Tansito
Upload end point (#1555)
5bdb776 
* Update upload end-point

* removed unused imports

* updated comment in upload end-point
@Tansito
Merge branch 'main' into data-folder
7ce99f8
@Tansito Tansito mentioned this pull request Dec 17, 2024
Merged
Tansito and others added 6 commits December 18, 2024 10:21
@Tansito
remove file extension limitation from files (#1559)
a0de41d
@korgan00
Client - files upload refactor (#1557)
97fe205 
* add support to new upload file endpoints

* replace data with params
@korgan00
Client - files delete refactor (#1556)
0368d2d 
* add support to new delete files endpoints

* fix get parameter

* fix tests

* adapt files delete to the refactor

* replace data with params

* remove not used fixture
@korgan00 @Tansito
Integration tests fix (#1561)
587ea86 
* add support to new delete files endpoints

* fix get parameter

* fix tests

* adapt files delete to the refactor

* replace data with params

* remove not used fixture

* tests fixed

* fix client integration

* fix context manager in files when download

* added a new test for provider end-points

* fix black

* migrated old tests

* Update gateway/api/services/file_storage.py

Co-authored-by: Goyo <[email protected]>

* remove additional line

---------

Co-authored-by: David <[email protected]>
@korgan00 @Tansito
Trace decorator (#1553)
f3b8de9 
* create a trace decorator

* fix decorator

* change documentation

---------

Co-authored-by: David <[email protected]>
@Tansito
Files refactor to include repositories logic (#1560)
f50a7f9 
* remove unneeded provider methods

* create access policies file

* refactor get_functions repository method

* refactor groups and repositories

* repository refactor from files

* fix some linter problems

* fixed a bug when the user retrieves a function

* fix lint

* refactor of get_function method

* remove artifact test file

* remove programs access policies

* refactor programs references to functions

* group repository refactor

* rename groups repository into user repository

* simplified get_function methods

* fix query

* adapt get_functions methods

* updated comments

* create path if doesn't exist

* remove some unused code

* fix files client

* fix typos

* fixed the creation of the directory

* added a test for the provider end-points

* fix some typos from the provider end-points

* fix black on tests
This was referenced Jan 2, 2025
Merged
Merged
Tansito added 2 commits January 7, 2025 11:26
@Tansito
files documentation updated (#1565)
62848f0
@Tansito
Refactor storage in scheduler (#1563)
be6c538 
* refactor storage in scheduler

* fix typo in ray template

* remove unused node_image

* use sub-paths for the cluster template

* storage absolute path refactor
@Tansito Tansito marked this pull request as ready for review January 10, 2025 15:25
@Tansito Tansito requested a review from pandasa123 January 10, 2025 15:26
@Tansito Tansito changed the title 🏗️ Refactor: data folder Improve data directory functionality Jan 10, 2025
@Tansito Tansito mentioned this pull request Jan 13, 2025
Open
@Tansito Tansito merged commit ccd159a into main Jan 13, 2025
9 checks passed
@Tansito Tansito deleted the data-folder branch January 13, 2025 18:33
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@pandasa123 pandasa123 pandasa123 approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@Tansito @pandasa123 @korgan00