Feature Request: Support for Dimension Table Join

[boer0924]

I would like to request the implementation of dimension table join functionality in Arroyo, similar to the dimension table support in Flink.

Use Case:
Dimension table joins are essential for enriching streaming data by querying external static or slowly changing datasets (e.g., MySQL, PostgreSQL). This feature is particularly useful for scenarios such as:

• Joining streaming order data with user profiles stored in a database.
• Enriching event streams with metadata from external sources.

Key Features:

1. Caching Strategy:
   Support caching of dimension table data in memory to enhance join performance.
   Configurable cache expiration and refresh intervals.

2. Lookup and Join:
   Allow efficient lookups against dimension tables during streaming data processing.
   Support for handling missing or unmatched keys with default values.

3. Data Consistency:
   Enable periodic or incremental updates to keep dimension table data consistent with the source.
   Optional support for transactional updates via mechanisms like CDC (Change Data Capture).

It would be great to see a similar feature in Arroyo to enrich its functionality for real-time data processing.

Thank you for considering this request!

[boer0924]

I would like to request the implementation of Lookup Join functionality in Arroyo, similar to Flink's, which enables efficient joins between streaming data and dimension tables, allowing real-time enrichment with static or slowly changing data from external sources.
https://nightlies.apache.org/flink/flink-docs-release-1.20/docs/dev/table/sql/queries/joins/#lookup-join