Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merge feature into main #134

Merged
merged 59 commits into from
Jan 16, 2025
Merged
Changes from 1 commit
Commits
Show all changes
59 commits
Select commit Hold shift + click to select a range
1261ea9
Support attaching delta tables as catalogs
samansmink Oct 24, 2024
cc4eea6
minor fixes
samansmink Oct 25, 2024
3c19a20
bump duckdb to main
samansmink Nov 6, 2024
2374e61
bump to recent nightly
samansmink Nov 7, 2024
56200f8
disable mingw build
samansmink Nov 8, 2024
ecfca60
small ci fixes
samansmink Nov 8, 2024
5cf23c8
make python3 available
samansmink Nov 8, 2024
77373bc
bump vcpkg
samansmink Nov 8, 2024
ca12e53
fix mismatching duckdb version
samansmink Nov 8, 2024
49c902b
Merge pull request #110 from samansmink/attach-delta-squashed
samansmink Nov 8, 2024
e92266b
Merge branch 'main' into main-into-feature
samansmink Nov 12, 2024
296f514
format
samansmink Nov 12, 2024
f240e45
add ci tools version
samansmink Nov 12, 2024
febbb79
also skip old arch label
samansmink Nov 12, 2024
756526b
Merge pull request #118 from samansmink/main-into-feature
samansmink Nov 12, 2024
a2364a1
enable deploy for feature
samansmink Nov 14, 2024
ad12b36
add table function tostring
samansmink Nov 14, 2024
1fe8e0d
bump duckdb to 1.1.4.dev2005
samansmink Nov 20, 2024
fdb0b4e
Merge branch 'bump-duckdb-3' into improve-table-function-to-string
samansmink Nov 20, 2024
35114ae
fix upstream MultiFileReader API changes
samansmink Nov 20, 2024
6f25451
add explain table name test
samansmink Nov 20, 2024
35e9d52
format
samansmink Nov 20, 2024
b77a9da
Merge pull request #122 from samansmink/improve-table-function-to-string
samansmink Nov 20, 2024
bc3434f
add filtered files to explain output
samansmink Nov 20, 2024
c08e66b
hide behind option
samansmink Nov 20, 2024
0c815b9
Merge pull request #123 from samansmink/add-total-files-filtered
samansmink Nov 20, 2024
ac80e3a
bump-kernel
samansmink Dec 5, 2024
582202e
fix enum util
samansmink Dec 5, 2024
14e2e45
checkout v4
samansmink Dec 5, 2024
307049a
fix old manylinux based ci job
samansmink Dec 5, 2024
526f5f3
remove old line of azurite initialization
samansmink Dec 5, 2024
8a7721e
Merge pull request #127 from samansmink/bump-kernel-to-0.5.0
samansmink Dec 5, 2024
0c87fd6
minor cleanup
samansmink Dec 19, 2024
4a32608
add locking and parallel test
samansmink Dec 19, 2024
f945b66
disable micro benchmarks for now
samansmink Dec 19, 2024
a9f1977
correctly set endpoint for gcs
samansmink Dec 19, 2024
3da4e72
correctly set endpoint for gcs
samansmink Dec 19, 2024
041a838
bump to v0.6.0, build kernel either debug or release not both
samansmink Dec 19, 2024
6c3c795
add missing windows lib
samansmink Dec 19, 2024
7140762
Merge pull request #128 from samansmink/thread-safety
samansmink Dec 19, 2024
9478771
Merge branch 'fix-gcs-endpoint' into bump-kernel-to-0.6.0
samansmink Dec 19, 2024
2a001d5
Merge branch 'delta-code-cleanup' into bump-kernel-to-0.6.0
samansmink Dec 19, 2024
829103e
add missing test
samansmink Dec 19, 2024
9876698
add kernel expression test function based test
samansmink Dec 19, 2024
fd9dc3e
Merge pull request #129 from samansmink/bump-kernel-to-0.6.0
samansmink Dec 19, 2024
da26428
Merge branch 'feature' into add-expression-test
samansmink Dec 19, 2024
252bad8
format
samansmink Dec 19, 2024
1ed221c
fix booleans
samansmink Dec 19, 2024
23f4e9c
fix optional_idx issue
samansmink Dec 20, 2024
3b768d9
Merge pull request #130 from samansmink/add-expression-test
samansmink Dec 20, 2024
6810aa3
bump duckdb to main
samansmink Jan 13, 2025
e39f308
poc hooking up logging
samansmink Dec 12, 2024
020eb46
finish delta kernel logging forwarding
samansmink Jan 13, 2025
e5ee3e1
format
samansmink Jan 13, 2025
07955a1
remove forgotten print statement
samansmink Jan 13, 2025
89ebdb7
re-enable building test extensions
samansmink Jan 13, 2025
0ece666
fix thread-safety issue for logging singleton
samansmink Jan 13, 2025
7e99954
skip musl
samansmink Jan 13, 2025
4691801
Merge pull request #132 from samansmink/logging-wip
samansmink Jan 14, 2025
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
correctly set endpoint for gcs
samansmink committed Dec 19, 2024
commit 3da4e72fa22d990f4b3fe4da57d6194d5bbc9a6e
69 changes: 34 additions & 35 deletions src/functions/delta_scan.cpp
Original file line number Diff line number Diff line change
@@ -241,42 +241,41 @@ static ffi::EngineBuilder *CreateBuilder(ClientContext &context, const string &p
// Here you would need to add the logic for setting the builder options for Azure
// This is just a placeholder and will need to be replaced with the actual logic
if (secret_type == "s3" || secret_type == "gcs" || secret_type == "r2") {
string key_id, secret, session_token, region, endpoint, url_style;
bool use_ssl = true;
secret_reader.TryGetSecretKey("key_id", key_id);
secret_reader.TryGetSecretKey("secret", secret);
secret_reader.TryGetSecretKey("session_token", session_token);
secret_reader.TryGetSecretKey("region", region);
secret_reader.TryGetSecretKey("endpoint", endpoint);
secret_reader.TryGetSecretKey("url_style", url_style);
secret_reader.TryGetSecretKey("use_ssl", use_ssl);

if (key_id.empty() && secret.empty()) {
ffi::set_builder_option(builder, KernelUtils::ToDeltaString("skip_signature"),
KernelUtils::ToDeltaString("true"));
}

string key_id, secret, session_token, region, endpoint, url_style;
bool use_ssl = true;
secret_reader.TryGetSecretKey("key_id", key_id);
secret_reader.TryGetSecretKey("secret", secret);
secret_reader.TryGetSecretKey("session_token", session_token);
secret_reader.TryGetSecretKey("region", region);
secret_reader.TryGetSecretKey("endpoint", endpoint);
secret_reader.TryGetSecretKey("url_style", url_style);
secret_reader.TryGetSecretKey("use_ssl", use_ssl);

if (key_id.empty() && secret.empty()) {
ffi::set_builder_option(builder, KernelUtils::ToDeltaString("skip_signature"),
KernelUtils::ToDeltaString("true"));
}

if (!key_id.empty()) {
ffi::set_builder_option(builder, KernelUtils::ToDeltaString("aws_access_key_id"),
KernelUtils::ToDeltaString(key_id));
}
if (!secret.empty()) {
ffi::set_builder_option(builder, KernelUtils::ToDeltaString("aws_secret_access_key"),
KernelUtils::ToDeltaString(secret));
}
if (!session_token.empty()) {
ffi::set_builder_option(builder, KernelUtils::ToDeltaString("aws_session_token"),
KernelUtils::ToDeltaString(session_token));
}
if (!endpoint.empty() && endpoint != "s3.amazonaws.com") {
if (!StringUtil::StartsWith(endpoint, "https://") && !StringUtil::StartsWith(endpoint, "http://")) {
if (use_ssl) {
endpoint = "https://" + endpoint;
} else {
endpoint = "http://" + endpoint;
}
}
if (!key_id.empty()) {
ffi::set_builder_option(builder, KernelUtils::ToDeltaString("aws_access_key_id"),
KernelUtils::ToDeltaString(key_id));
}
if (!secret.empty()) {
ffi::set_builder_option(builder, KernelUtils::ToDeltaString("aws_secret_access_key"),
KernelUtils::ToDeltaString(secret));
}
if (!session_token.empty()) {
ffi::set_builder_option(builder, KernelUtils::ToDeltaString("aws_session_token"),
KernelUtils::ToDeltaString(session_token));
}
if (!endpoint.empty() && endpoint != "s3.amazonaws.com") {
if (!StringUtil::StartsWith(endpoint, "https://") && !StringUtil::StartsWith(endpoint, "http://")) {
if (use_ssl) {
endpoint = "https://" + endpoint;
} else {
endpoint = "http://" + endpoint;
}
}

if (StringUtil::StartsWith(endpoint, "http://")) {
ffi::set_builder_option(builder, KernelUtils::ToDeltaString("allow_http"),