feat: `SnapshotSummary` outline #939

jonathanc-n · 2025-02-02T22:30:36Z

Part of #724.

Built the outline of the SnapshotSummary. Functionality into the code can be added in a separate pr.

…berg-rust into snapshot-summary

jonathanc-n · 2025-02-04T01:11:16Z

Is someone able to take a look at why the integration tests are failing?

kevinjqliu · 2025-02-04T01:33:23Z

From CI

test test_scan_all_type ... FAILED

failures:

---- test_scan_all_type stdout ----
thread 'test_scan_all_type' panicked at crates/integration_tests/tests/scan_all_type.rs:324:56:
called `Result::unwrap()` on an `Err` value: Unexpected => Failed to parse response from rest catalog server!

Context:
   code: 500 Internal Server Error
   method: POST
   url: http://172.29.0.3:8181/v1/namespaces/apple%1Fios/tables/t1
   json: <html>
<head>
<meta http-equiv="Content-Type" content="text/html;charset=ISO-8859-1"/>
<title>Error 500 java.lang.IllegalArgumentException: Cannot parse to a string value: added_data_files: 0</title>
</head>
<body><h2>HTTP ERROR 500 java.lang.IllegalArgumentException: Cannot parse to a string value: added_data_files: 0</h2>
<table>
<tr><th>URI:</th><td>/v1/namespaces/apple%1Fios/tables/t1</td></tr>
<tr><th>STATUS:</th><td>500</td></tr>
<tr><th>MESSAGE:</th><td>java.lang.IllegalArgumentException: Cannot parse to a string value: added_data_files: 0</td></tr>
<tr><th>SERVLET:</th><td>org.apache.iceberg.rest.RESTCatalogServlet-1ddf84b8</td></tr>
<tr><th>CAUSED BY:</th><td>java.lang.IllegalArgumentException: Cannot parse to a string value: added_data_files: 0</td></tr>
</table>
<h3>Caused by:</h3><pre>java.lang.IllegalArgumentException: Cannot parse to a string value: added_data_files: 0
	at org.apache.iceberg.relocated.com.google.common.base.Preconditions.checkArgument(Preconditions.java:446)
	at org.apache.iceberg.util.JsonUtil.getString(JsonUtil.java:168)
	at org.apache.iceberg.SnapshotParser.fromJson(SnapshotParser.java:153)
	at org.apache.iceberg.MetadataUpdateParser.readAddSnapshot(MetadataUpdateParser.java:544)
	at org.apache.iceberg.MetadataUpdateParser.fromJson(MetadataUpdateParser.java:310)
	at org.apache.iceberg.rest.requests.UpdateTableRequestParser.lambda$fromJson$2(UpdateTableRequestParser.java:105)
	at java.base/java.lang.Iterable.forEach(Iterable.java:75)
	at org.apache.iceberg.rest.requests.UpdateTableRequestParser.fromJson(UpdateTableRequestParser.java:105)
	at org.apache.iceberg.rest.RESTSerializers$UpdateTableRequestDeserializer.deserialize(RESTSerializers.java:367)
	at org.apache.iceberg.rest.RESTSerializers$UpdateTableRequestDeserializer.deserialize(RESTSerializers.java:362)
	at com.fasterxml.jackson.databind.deser.DefaultDeserializationContext.readRootValue(DefaultDeserializationContext.java:342)
	at com.fasterxml.jackson.databind.ObjectMapper._readMapAndClose(ObjectMapper.java:4917)
	at com.fasterxml.jackson.databind.ObjectMapper.readValue(ObjectMapper.java:3873)
	at org.apache.iceberg.rest.RESTCatalogServlet$ServletRequestContext.from(RESTCatalogServlet.java:179)
	at org.apache.iceberg.rest.RESTCatalogServlet.doPost(RESTCatalogServlet.java:78)
	at jakarta.servlet.http.HttpServlet.service(HttpServlet.java:520)
	at jakarta.servlet.http.HttpServlet.service(HttpServlet.java:587)
	at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:764)
	at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:529)
	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:131)
	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:122)
	at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:822)
	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:122)
	at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:223)
	at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1381)
	at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:176)
	at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:484)
	at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:174)
	at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1303)
	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:129)
	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:122)
	at org.eclipse.jetty.server.Server.handle(Server.java:563)
	at org.eclipse.jetty.server.HttpChannel$RequestDispatchable.dispatch(HttpChannel.java:1598)
	at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:753)
	at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:501)
	at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:287)
	at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:314)
	at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:100)
	at org.eclipse.jetty.io.SelectableChannelEndPoint$1.run(SelectableChannelEndPoint.java:53)
	at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.runTask(AdaptiveExecutionStrategy.java:421)
	at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.consumeTask(AdaptiveExecutionStrategy.java:390)
	at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.tryProduce(AdaptiveExecutionStrategy.java:277)
	at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.run(AdaptiveExecutionStrategy.java:199)
	at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:411)
	at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:969)
	at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.doRunJob(QueuedThreadPool.java:1194)
	at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1149)
	at java.base/java.lang.Thread.run(Thread.java:840)
</pre>
<hr/><a href="https://jetty.org/">Powered by Jetty:// 11.0.24</a><hr/>

</body>
</html>


Source: expected value at line 1 column 1

note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace


failures:
    test_scan_all_type

test result: FAILED. 0 passed; 1 failed; 0 ignored; 0 measured; 0 filtered out; finished in 36.11s

error: test failed, to rerun pass `-p iceberg-integration-tests --test scan_all_type`
     Running unittests src/lib.rs (target/debug/deps/iceberg_test_utils-5a271539e841efee)

running 0 tests

test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s

     Running unittests src/lib.rs (target/debug/deps/sqllogictest-938b48cff3f7d53f)

running 0 tests

test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s

error: 4 targets failed:
    `-p iceberg-integration-tests --test append_data_file_test`
    `-p iceberg-integration-tests --test append_partition_data_file_test`
    `-p iceberg-integration-tests --test conflict_commit_test`
    `-p iceberg-integration-tests --test scan_all_type`
Error: Process completed with exit code 101.

…berg-rust into snapshot-summary

jonathanc-n · 2025-02-05T05:27:37Z

cc @Fokko or others. I'm trying to debug a deserialization issue, and I suspect the server might be encountering a schema mismatch when handling the SnapshotSummary. However, since serde(default) is applied, missing fields should default to their expected values. Could the server be failing to serialize the response correctly, or is the client running into an unexpected parsing error despite the defaults?

jonathanc-n added 10 commits February 1, 2025 23:21

feat: Add SnapshotSummary outline

6ea60fb

fix structure

d1c4fcf

fix

1af4bb7

Merge branch 'main' into snapshot-summary

adf1aa3

small fix

c7f25e4

Merge branch 'snapshot-summary' of https://github.com/jonathanc-n/ice…

e4f6a88

…berg-rust into snapshot-summary

clippy fix

55fe0a9

test fix

5905e7d

fix

3e8e45e

fmt

ce63c68

jonathanc-n marked this pull request as draft February 5, 2025 01:50

jonathanc-n added 6 commits February 4, 2025 21:10

fix

3886a0c

fmt

c0e348d

Merge branch 'main' into snapshot-summary

7f7b540

log

27d4057

Merge branch 'snapshot-summary' of https://github.com/jonathanc-n/ice…

c2a93ad

…berg-rust into snapshot-summary

fix

5e5bbb4

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: `SnapshotSummary` outline #939

feat: `SnapshotSummary` outline #939

jonathanc-n commented Feb 2, 2025

jonathanc-n commented Feb 4, 2025

kevinjqliu commented Feb 4, 2025

jonathanc-n commented Feb 5, 2025

feat: SnapshotSummary outline #939

Are you sure you want to change the base?

feat: SnapshotSummary outline #939

Conversation

jonathanc-n commented Feb 2, 2025

jonathanc-n commented Feb 4, 2025

kevinjqliu commented Feb 4, 2025

jonathanc-n commented Feb 5, 2025

feat: `SnapshotSummary` outline #939

feat: `SnapshotSummary` outline #939