You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
I searched in the issues and found nothing similar.
Motivation
We use ticdc to collect data from tidb and send to kafka in canal-json format, then sink the data with paimon-flink-action.
But actually only type name are collected and sent to kafka for the metadata part. For example, the decimal fields, only the name 'decimal' provided, without the precision and scale.
In paimon-flink-action, the missing precision and scale will be set to maximum, which is decimal(38,16). That is too large and not expected.
In this case, the original format is prefer, which means not changing the deciaml type during cdc.
Solution
Add a type-mappint option for kafka cdc, like
# to ignore only decimal change
--type_mapping decimal-no-change
# to ingore any type change
--type_mapping no_change
And ignore the type change in $$RichEventParser::parseSchemaChange$$
Anything else?
Enlarging type also occurs to char/varchar, which is acceptable.
Are you willing to submit a PR?
I'm willing to submit a PR!
The text was updated successfully, but these errors were encountered:
JackeyLee007
changed the title
[Feature] Ignore decimal type evolution if no precision and scale provided during CDC.
[Feature] Ignore decimal type evolution in special case
Dec 29, 2024
Search before asking
Motivation
We use ticdc to collect data from tidb and send to kafka in canal-json format, then sink the data with paimon-flink-action.
But actually only type name are collected and sent to kafka for the metadata part. For example, the decimal fields, only the name 'decimal' provided, without the precision and scale.
In paimon-flink-action, the missing precision and scale will be set to maximum, which is decimal(38,16). That is too large and not expected.
In this case, the original format is prefer, which means not changing the deciaml type during cdc.
Solution
Add a type-mappint option for kafka cdc, like
And ignore the type change in$$RichEventParser::parseSchemaChange$$
Anything else?
Enlarging type also occurs to char/varchar, which is acceptable.
Are you willing to submit a PR?
The text was updated successfully, but these errors were encountered: