Possible limitation of ReplicationStream schema

[ShahBinoy]

Hi,
When building a replication config, I wanted to mix/match default Sync and custom SQL Sync
along with usage of Default Patterns, so I construct this replication json file

{
  "source": "ACADIA",
  "target": "S3_LAKE_ZONE_A",
  "defaults": {
    "mode": "incremental",
    "object": "dev-binoy/elt/data_landing/datalake/{source_name}/{stream_table}/date={YYYY}-{MM}-{DD}/*",
    "update_key": "updated_on",
    "target_options": {
      "format": "parquet"
    }
  },
  "streams": {
    "acadia.form_mode": {
      "sql": "select * from form_mode where id = 433 and {incremental_where_cond}",
      "disabled": false
    },
    "acadia.user_token_mapping": {
      "sql": "select * from user_token_mapping where tm_id = 433022 and {incremental_where_cond}",
      "object": "user_token_mapping",
      "disabled": false
    }
  },
  "env": {
    "SLING_LOADED_AT_COLUMN": true,
    "SLING_STREAM_URL_COLUMN": true,
    "path_prefix": "${DATALAKE_ELT_LANDING_PREFIX}",
    "SLING_SHOW_PROGRESS": "false",
    "SLING_DISABLE_TELEMETRY": "false",
    "SLING_HOME_DIR": "/Users/bshah/workspace/data-engg/data-lake-workflows/src/vibrent_common/sling/../../sling",
    "SLING_LOGGING": "NO_COLOR",
    "SLING_ALLOW_EMPTY_TABLES": "true"
  }
}

I have read the documentation of it and it mentions the following 👇🏼

And so I add the `object` for custom sql,

• And now if I insert object name for custom sql, it does not respect the default objects for where I need to save the objects via a pattern and it saves parquet file without extension to root of the S3 bucket with the object name
• If I dont add the object keyword, the {stream_table} token as applied in the default object, stays unresolved and creates a path with verbatim {stream_table} as value

Could it be possible to not have to use the object in the custom sql based outputs and rather rely on different parameter or use the name of the dictionary, like eg in above case "acadia.form_mode" such that defaults are still respected and utilized ?

[flarco]

Hi, interesting take. Yea I think that's a good suggestion, sling could attempt to parse the stream name and assume that it's a table, as normal. Should work 👍 . Will create issue for that.

You could also specify the object name both them, and fill in the {stream_table} yourself manually, like

• "object": "dev-binoy/elt/data_landing/datalake/{source_name}/form_mode/date={YYYY}-{MM}-{DD}/*"
• "object": "dev-binoy/elt/data_landing/datalake/{source_name}/user_token_mapping/date={YYYY}-{MM}-{DD}/*"

[ShahBinoy]

I thought about constructing the full name, but doing in pythonic way dynamically to update the object with a full constructed way and also aligning with default object, and utilizing it with current sling-python module, it appeared a bit clunky and too wordy

Thanks for taking up the suggestion.

[flarco]

Agreed.
Oh, there is also the {stream_name} placeholder already implemented, that could work for you. But that would conflict with the other non-custom SQL... So, maybe not ha.
Opened issue #125.