Secrets #12
Comments
lmangani
added
enhancement
|
Ciao @lmangani , I have a related question: now is there no way to save default api_url, api_token and api_key? Thank you for this great tool |
|
CIao @aborruso at this time the settings are indeed required, but if there's interest I'll get started on stored SECRETS 👍 |
Grazie @lmangani, my use case is very trivial: I would like to open duckdb cli and use this fantastic work of yours, as available by default (with automatic load in .rc file), and without every time writing these parameters. And set these parameters only when I don't want to use the default ones. If you think this is helpful, and you can put time and head into it, thank you very much. I would like to help you, but I don't know how to do it. |
|
@aborruso here's an untested PR with SECRETS support and instructions. Would you able to build and validate it locally? InstructionsClone and build the PR branch locally: cd /usr/src
git clone -b secret-manager --recurse-submodules https://github.com/quackscience/duckdb-extension-openprompt
cd duckdb-extension-openprompt
OVERRIDE_GIT_DESCRIBE="v1.1.3" GEN=ninja makeRun the local DuckDB build: ./build/release/duckdbAdd and test SECRETS D CREATE SECRET IF NOT EXISTS open_prompt (
TYPE open_prompt,
PROVIDER config,
api_token 'your-api-token',
api_url 'http://localhost:11434/v1/chat/completions',
model_name 'qwen2.5:0.5b',
api_timeout '30'
);TestingAt this point the parameters should be derived from SECRETS instead of SET parameters. |
|
I also added ENV parameters to allow provisioning the process from the system at runtime |
|
Grazie @lmangani , I will try tomorrow. |
|
ciao @aborruso did you get a chance to try this? |
|
@aborruso i'm sorry about this! is the error at build time or run time? |
|
This one |
|
Looks like OOM to me. What's your OS/ARCH and how much memory does it have? |
It's my small Debian 12, with 8 GB ram :(
|
|
I increased the memory, but now I have this |
|
It's alright - don't worry, I'll find some time to test it and merge it. Thanks for trying I appreciate it no less! Grazie Mille 👋 |
|
issue should be patched 🤞 waiting for builders to confirm (or punish me) |
|
@aborruso the build should be fixed if you want to try again - it runs on my Linux/x86 and the action builder seems happy |
Do I must restart from here? |
|
If you have already clone, you can just do: and it should pull the patch and continue where it left off. Once again thanks! |
|
If I can do the build, do I have to uninstall the previous one first? Grazie |
|
No need to overwrite the system installation, this will simply build a local version you can run for testing: From within the test build shell you should be able to perform the validation without impacting anything else 🤞 if you have any questions ping me |
|
Ciao @lmangani , I am stupid :) When I run I have Do I must insert Thank you |
|
Interesting... I copy pasted the same in my local build and it seems to succeed on my Linux/x86 w/ Debian 12 D CREATE SECRET IF NOT EXISTS open_prompt (
TYPE open_prompt,
PROVIDER config,
api_token 'xxxxx',
api_url 'https://api.groq.com/openai/v1/chat/completions',
model_name 'llama-3.3-70b-versatile',
api_timeout '30'
);
┌─────────┐
│ Success │
│ boolean │
├─────────┤
│ true │
└─────────┘
D FROM duckdb_secrets();
┌─────────────┬─────────────┬──────────┬────────────┬─────────┬───────────┬──────────────────────────────────────────────────────────┐
│ name │ type │ provider │ persistent │ storage │ scope │ secret_string │
│ varchar │ varchar │ varchar │ boolean │ varchar │ varchar[] │ varchar │
├─────────────┼─────────────┼──────────┼────────────┼─────────┼───────────┼──────────────────────────────────────────────────────────┤
│ open_prompt │ open_prompt │ config │ false │ memory │ [] │ name=open_prompt;type=open_prompt;provider=config;seri… │
└─────────────┴─────────────┴──────────┴────────────┴─────────┴───────────┴──────────────────────────────────────────────────────────┘also persistent seems to work here: D CREATE PERSISTENT SECRET IF NOT EXISTS open_prompt (
TYPE open_prompt,
PROVIDER config,
api_token 'xxxxx',
api_url 'https://api.groq.com/openai/v1/chat/completions',
model_name 'llama-3.3-70b-versatile',
api_timeout '30'
);
┌─────────┐
│ Success │
│ boolean │
├─────────┤
│ true │
└─────────┘
D {exit}
D FROM duckdb_secrets();
┌─────────────┬─────────────┬──────────┬────────────┬────────────┬───────────┬───────────────────────────────────────────────────────┐
│ name │ type │ provider │ persistent │ storage │ scope │ secret_string │
│ varchar │ varchar │ varchar │ boolean │ varchar │ varchar[] │ varchar │
├─────────────┼─────────────┼──────────┼────────────┼────────────┼───────────┼───────────────────────────────────────────────────────┤
│ open_prompt │ open_prompt │ config │ true │ local_file │ [] │ name=open_prompt;type=open_prompt;provider=config;s… │
└─────────────┴─────────────┴──────────┴────────────┴────────────┴───────────┴───────────────────────────────────────────────────────┘This is using duckdb in |
I'm using the duckdb I have in my ./duckdb-extension-openprompt/build/release folder. The |
|
I updated the submodules to point at the latest version of duckdb and ci tools - that's the only reason I see for getting v1.1.1 |
|
It works!!! Thank you very much
|
|
@aborruso fantastico! does it work with the actual |
If I run I have used both |
|
In this way, it works |
|
Ok that's useful let me recheck as soon as possible what the secrets mistake is - I assume you either used persistent secrets and/or tested in the same session you saved secrets within correct? secrets are not persistent unless specified. Any chance you could try with ENVs on the process to see if those work? Grazie this really helps! |
|
Hi @lmangani , after I export the variables in the shell, I run duckdb and this query |
|
@aborruso Andrea thank you this helped me locate the logical issue (and why it worked locally) so now I should be able to release a working version! I really appreciate your time 👍 |
|
@aborruso If you feel like trying, the latest patches to the branch correctly initializes ENV, SECRET, SET or DEFAULT (in this order) once again thanks for your kind assistance testing this! If you have any suggestions as of the override order, feel free to share it and I'll include it |
how to update my cloned folder? |
|
Yes, the branch is still there so you can |
|
I'm probably tired. I launched git pull, and then after |
|
Testing can indeed tedious. Sorry for the trouble and once again thanks! I've tested it locally usiong ENV variables and SECRET settings and it all seems to work. Did not test with remote services. Not the ENV settings takes precedence over SECRETS and SET variables. I'll try again tomorrow as well to be sure. |
|
Ciao @lmangani the "secret-manager" branch is no longer present. Do I start over by cloning the main branch? Thank you |
|
Ciao @aborruso that's right, you can test using the current main branch 👋 |
|
It works great, thank you very much @lmangani |
|
Ciao @lmangani to use it with my standard duckdb, do I now have to wait for the update to be installable? |
|
Grazie @aborruso glad we made this improvement and learned a few things along the way, too! There's a PR open on the community repo: duckdb/community-extensions#255 |
The extension should leverage Secrets as an option to safely store parameters across sessions.
The text was updated successfully, but these errors were encountered: