support source watermark for flink sql windows #12191
Open
+112
−2
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
pvary
reviewed
Feb 7, 2025
| @@ -86,7 +86,7 @@ protected void dropDatabase(String database, boolean ifExists) { | |||
| sql("DROP DATABASE %s %s", ifExists ? "IF EXISTS" : "", database); | |||
| } | |||
|
|
|||
| protected static String toWithClause(Map<String, String> props) { | |||
| public static String toWithClause(Map<String, String> props) { | |||
There was a problem hiding this comment.
Why is this change? Did this become a utility method which is shared between tests? Can we find a better place for it then?
pvary
reviewed
Feb 7, 2025
| .isInstanceOf(NullPointerException.class) | ||
| .hasMessage("watermark-column needs to be configured to use source watermark."); | ||
| } finally { | ||
| SqlHelpers.sql(getStreamingTableEnv(), "DROP TABLE IF EXISTS %s", flinkTable); |
There was a problem hiding this comment.
Why is this try finally? Can we just do a cleanup in an after method? That would make the tests easier to read
pvary
reviewed
Feb 7, 2025
| } | ||
|
|
||
| @NotNull | ||
| private static Map<String, String> getConnectorOptions() { |
There was a problem hiding this comment.
If we do the PRs in a different order (first merging the optionless part), could we get rid of this method?
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Iceberg Source to support Source Watermark, so it can be used in Flink WINDOW functions. https://github.com/apache/flink/blob/release-1.18/flink-table/flink-table-common/src/main/java/org/apache/flink/table/connector/source/abilities/SupportsSourceWatermark.java enables Flink to rely on the watermark strategy provided by the ScanTableSource itself.
Reference:
#10219
#9346
Previous discussion in PR, #12116 . Split into separate PR for easy review.