Add iceberg catalog read support #98

Tmonster · 2025-01-29T14:59:08Z

Piggy backing off of #95

This PR adds support for attaching an Iceberg catalog and be able to read iceberg tables from a datalake. It specifically supports the following sql commands:

Create an ICEBERG secret to access your catalog
CREATE SECRET (
     TYPE ICEBERG,
     CLIENT_ID <your-catalog-client-id>,
     CLIENT_SECRET <your-catalog-client-secret>,
     ENDPOINT 'https://<your-catalog-host>/api/catalog',
     AWS_REGION 'us-east-1'
);

Attach your catalog

ATTACH 'my_catalog' AS my_datalake (TYPE ICEBERG);
-- Select your iceberg tables in the datalake via the catalog
SHOW ALL TABLES;
SELECT * FROM my_datalake.schema.table;

Some tests as well to make sure everything works. There is also a double check on the config since it's possible some catalogs won't return configs. In this case the user is required to have a second key.

samansmink

Looks great! I've added a few comments

samansmink · 2025-01-29T15:03:15Z

CMakeLists.txt

@@ -4,7 +4,7 @@ cmake_minimum_required(VERSION 2.8.12)
 set(TARGET_NAME iceberg)
 project(${TARGET_NAME})

-set(CMAKE_CXX_STANDARD 14)
+set(CMAKE_CXX_STANDARD 17)


do we really need to bump this? I would prefer not to, to avoid any CI issues

samansmink · 2025-01-29T15:05:16Z

src/catalog_api.cpp

+	}
+  return result;
+
+// throw std::runtime_error("No AWS credentials found for table");


nit: can be removed

samansmink · 2025-01-29T15:14:09Z

src/storage/ic_transaction.cpp

+	}
+}
+
+// ICConnection &ICTransaction::GetConnection() {


nit: let's just remove all commented out code. This has been copied from the uc_catalog extension which has a prototype-quality level codebase

samansmink · 2025-01-29T15:21:17Z

test/sql/iceberg_catalog_read.test

+require httpfs
+
+statement ok
+CREATE SECRET (


I think this is fine for a first PR, but we should think about the UX a little here: right now there is only 1 iceberg secret which is then fetched when calling ATTACH '' AS my_datalake (TYPE ICEBERG);

However it's probably desirable to be able to do something like:

CREATE SECRET irc_secret_1 (TYPE ICEBERG, ENDPOINT 'http://127.0.0.1:8181', BEARER_TOKEN 'bla'); CREATE SECRET irc_secret_2 (TYPE ICEBERG, ENDPOINT 'http://some.other.thing.com', BEARER_TOKEN 'bla'); ATTACH 'irc_secret_1' AS irc1 (TYPE ICEBERG); ATTACH 'irc_secret_2' AS irc2 (TYPE ICEBERG);

or perhaps

CREATE SECRET irc_secret_1 (TYPE ICEBERG, SCOPE 'http://127.0.0.1:8181', BEARER_TOKEN 'bla'); CREATE SECRET irc_secret_2 (TYPE ICEBERG, SCOPE 'http://some.other.thing.com', BEARER_TOKEN 'bla'); ATTACH 'http://127.0.0.1:8181' AS irc1 (TYPE ICEBERG); ATTACH 'http://some.other.thing.com' AS irc2 (TYPE ICEBERG);

We can have some discussion on what is nicest, these are currently also open questions for other catalog extensions such as postgres and mysql i think

samansmink · 2025-01-29T15:25:25Z

src/storage/ic_table_set.cpp

+	}
+
+	auto &ic_catalog = catalog.Cast<ICCatalog>();
+	// TODO: handle out-of-order columns using position property


This TODO is copy pasted from uc_catalog: we should figure out if this is also relevant to iceberg, if so it can stay otherwise we should remove it to avoid confusion

samansmink · 2025-01-29T15:31:43Z

src/storage/ic_table_entry.cpp

+    auto &get = (LogicalGet &)*op;
+	bind_data = std::move(get.bind_data);
+
+	return parquet_scan_function;


I think this doesn't work with tables with deletes and potentially return incorrect results. Can we add a test for this?

If it does indeed fail, we should take a look into how to solve this, or at least throw an error

samansmink · 2025-01-29T15:32:11Z

src/storage/ic_table_entry.cpp

+	auto table_ref = iceberg_scan_function.bind_replace(context, bind_input);
+
+	// 1) Create a Binder and bind the parser-level TableRef -> BoundTableRef
+    auto binder = Binder::CreateBinder(context);


nit: formatting

samansmink · 2025-01-29T15:33:41Z

src/storage/ic_table_entry.cpp

+	throw NotImplementedException("BindUpdateConstraints");
+}
+
+struct MyIcebergFunctionData : public FunctionData {


this is dead code i think?

samansmink · 2025-01-29T16:04:40Z

CMakeLists.txt

    src/common/utils.cpp
    src/common/schema.cpp
    src/common/iceberg.cpp
    src/iceberg_functions/iceberg_snapshots.cpp
    src/iceberg_functions/iceberg_scan.cpp
-    src/iceberg_functions/iceberg_metadata.cpp)
+    src/iceberg_functions/iceberg_metadata.cpp
+    src/storage/ic_catalog.cpp


I feel like this structure could be a bit clearer

What do you think about renaming the IC to IRC (this code is meant for the Iceberg Rest Catalog after all) and moving everything:
src/storage/irc/irc_catalog.cpp?

This would allow us to later on create another catalog which can handle static iceberg tables, similar to duckdb/duckdb-delta#110 which could then live in src/storage/static/static_catalog.cpp

Do you want all of these files to be icr_*or just ic_catalog.cpp to be icr_catalog.cpp

I think irc_* (not icr_ 😄) everywhere that is related to the Iceberg Rest Catalog makes the most sense. We can also look to using C++ namespacing for the iceberg extension at some point, but I think just prefixing everything with IRC is fine for now.

sean-lynch · 2025-02-04T07:17:55Z

Very excited future user here! Chiming in on a review like this may be frowned upon. Please ignore if it's a distraction.

One minor request to make the ICEBERG secret agnostic of object storage provider. Rather than accepting AWS_REGION here, can that be left entirely to the existing httpfs secret? Taking some inspiration from @samansmink's feedback, maybe a future addition lets you specify which object storage provider secret to use, but at least leaving out AWS_REGION here avoids mixing up that configuration.

Tmonster · 2025-02-04T08:19:07Z

src/storage/irc_table_entry.cpp

+		                              table_data->data_source_format);
+	}
+
+	if (table_data->storage_location.find("file://") != 0) {


the hardcoded file:// maybe is unnecessary? I feel like we can omit the condition and just try to pull the credentials

Tmonster · 2025-02-04T08:20:39Z

src/storage/irc_table_entry.cpp

+			// Inject secret into secret manager scoped to this path
+			CreateSecretInfo info(OnCreateConflict::ERROR_ON_CONFLICT, SecretPersistType::TEMPORARY);
+			info.name = "__internal_ic_" + table_data->table_id;
+			info.type = "s3";


Also, based on the comment here, we should see if the spec returns information on the type of the secret so we don't have to hard code this kind of information

ediril and others added 15 commits January 14, 2025 02:49

Merge is done

1710ff9

Rename files uc -> ic

f3d5d87

Started renaming classes

1be2fa3

FInished renaming

63ef6e0

Delete duplicate

e6f671a

Lazy load metadata for tables

46a9946

Extract precision and scale

26d9f1b

Get rid of is_loaded

645e83c

Finish implementing create/drop schema

3a63af6

Reorg classes for clarity

f24fc3a

Minor fixes

56e4c76

Rename classes for clarity

aee6c1d

Complete table create & drop

dc0ebb3

fixes so it can work with local testing information

f3007ae

add tests

22bcffe

samansmink reviewed Jan 29, 2025

View reviewed changes

Tmonster added 8 commits January 30, 2025 09:53

PR comments duckdb#1

dfb95f9

remove dead code

bffe013

renaming files and classes

92046e3

fix include errors

49c8fba

also check delete and insert statements

bb4cb29

fix formatting

40dd36c

also rename ic_catalog.cpp to icr_catalog.cpp

a8c7fe6

change ic to irc

43daceb

Tmonster commented Feb 4, 2025

View reviewed changes

Tmonster added 3 commits February 4, 2025 15:17

rename ICRCatalog to IRCatalog

c288e88

remove check for 'file://'.

bc57d82

remove check for is internal table

b2259f0

Tmonster added 2 commits February 5, 2025 14:42

more renaming ICR to IRC and remove UC references

1fa5bd8

modify test so it reads from correct files

07ef8dc

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add iceberg catalog read support #98

Add iceberg catalog read support #98

Tmonster commented Jan 29, 2025

samansmink left a comment

samansmink Jan 29, 2025

samansmink Jan 29, 2025

samansmink Jan 29, 2025

samansmink Jan 29, 2025

samansmink Jan 29, 2025

samansmink Jan 29, 2025

samansmink Jan 29, 2025

samansmink Jan 29, 2025

samansmink Jan 29, 2025

Tmonster Feb 3, 2025

samansmink Feb 3, 2025

sean-lynch commented Feb 4, 2025 •

edited

Loading

Tmonster Feb 4, 2025

Tmonster Feb 4, 2025

Add iceberg catalog read support #98

Are you sure you want to change the base?

Add iceberg catalog read support #98

Conversation

Tmonster commented Jan 29, 2025

samansmink left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sean-lynch commented Feb 4, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sean-lynch commented Feb 4, 2025 •

edited

Loading