diff --git a/README.md b/README.md index 84cab508..30607d4e 100644 --- a/README.md +++ b/README.md @@ -468,6 +468,35 @@ handy paired with `union_relations`. * `database` (optional, default = `target.database`): The database to inspect for relations. +#### get_relations_by_pattern +> This was built from the get_relations_by_prefix macro. + +Returns a list of [Relations](https://docs.getdbt.com/docs/writing-code-in-dbt/class-reference/#relation) +that match a given schema or table pattern and table/view name with an optional exclusion pattern. Like its cousin +get_relations_by_prefix, it's particularly handy paired with `union_relations`. +**Usage:** +``` +-- Returns a list of relations that match schema%.table +{% set relations = dbt_utils.get_relations_by_pattern('schema_pattern%', 'table_pattern') %} + +-- Returns a list of relations that match schema.table% +{% set relations = dbt_utils.get_relations_by_pattern('schema_pattern', 'table_pattern%') %} + +-- Returns a list of relations as above, excluding any that end in `deprecated` +{% set relations = dbt_utils.get_relations_by_pattern('schema_pattern', 'table_pattern%', '%deprecated') %} + +-- Example using the union_relations macro +{% set event_relations = dbt_utils.get_relations_by_pattern('venue%', 'clicks') %} +{{ dbt_utils.union_relations(relations = event_relations) }} +``` + +**Args:** +* `schema_pattern` (required): The schema pattern to inspect for relations. +* `table_pattern` (required): The name of the table/view (case insensitive). +* `exclude` (optional): Exclude any relations that match this table pattern. +* `database` (optional, default = `target.database`): The database to inspect +for relations. + #### group_by ([source](macros/sql/groupby.sql)) This macro build a group by statement for fields 1...N diff --git a/integration_tests/models/sql/test_get_relations_by_pattern.sql b/integration_tests/models/sql/test_get_relations_by_pattern.sql new file mode 100644 index 00000000..5a51dd05 --- /dev/null +++ b/integration_tests/models/sql/test_get_relations_by_pattern.sql @@ -0,0 +1,4 @@ +{{ config(materialized = 'table') }} + +{% set relations = dbt_utils.get_relations_by_pattern(target.schema, 'data_events_') %} +{{ dbt_utils.union_relations(relations) }} \ No newline at end of file diff --git a/macros/sql/get_relations_by_pattern.sql b/macros/sql/get_relations_by_pattern.sql new file mode 100644 index 00000000..78a766ad --- /dev/null +++ b/macros/sql/get_relations_by_pattern.sql @@ -0,0 +1,23 @@ +{% macro get_relations_by_pattern(schema_pattern, table_pattern, exclude='', database=target.database) %} + + {%- call statement('get_tables', fetch_result=True) %} + + {{ dbt_utils.get_tables_by_pattern(schema_pattern, table_pattern, exclude, database) }} + + {%- endcall -%} + + {%- set table_list = load_result('get_tables') -%} + + {%- if table_list and table_list['table'] -%} + {%- set tbl_relations = [] -%} + {%- for row in table_list['table'] -%} + {%- set tbl_relation = api.Relation.create(database, row.table_schema, row.table_name) -%} + {%- do tbl_relations.append(tbl_relation) -%} + {%- endfor -%} + + {{ return(tbl_relations) }} + {%- else -%} + {{ return([]) }} + {%- endif -%} + +{% endmacro %} \ No newline at end of file diff --git a/macros/sql/get_tables_by_pattern.sql b/macros/sql/get_tables_by_pattern.sql new file mode 100644 index 00000000..ad388ca4 --- /dev/null +++ b/macros/sql/get_tables_by_pattern.sql @@ -0,0 +1,12 @@ +{% macro get_tables_by_pattern(schema_pattern, table_pattern, exclude='', database=target.database) %} + + select distinct + table_schema as "table_schema", table_name as "table_name" + from {{database}}.information_schema.tables + where table_schema ilike '{{ schema_pattern }}' + and table_name ilike '{{ table_pattern }}' + and table_name not ilike '{{ exclude }}' + +{% endmacro %} + +