Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support overriding the profile_config per dbt node or folder using config #1492

Merged
merged 17 commits into from
Jan 29, 2025

Conversation

tatiana
Copy link
Collaborator

@tatiana tatiana commented Jan 27, 2025

Allow users to override profile_config per operator or folder using config. Extends the work introduced by @wornjs in #1339, when we started supporting operator-level arguments defined in the dbt project/model YAML.

This PR includes a generic implementation that introduces the [meta][cosmos][profile_config] section, at the same level as operator_kwargs, and ensures these configs override the original profile config.

Example overriding a profile_config that uses profiles_yml_filepath:

      +meta:
        cosmos:
          operator_kwargs:
              (...)
          profile_config:
             profile_name: non_default_profile
             target_name: stage
             profiles_yml_filepath: /some/dir/profiles.yml

Example of an interface when using profile_mapping:

      +meta:
        cosmos:
          operator_kwargs:
            (...)    
          profile_config:
             profile_name: non_default_profile
             target_name: stage
             profile_mapping:
               conn_id: non_default_connection
               profile_args:
                 schema: non_default_schema   

Copy link

netlify bot commented Jan 27, 2025

Deploy Preview for sunny-pastelito-5ecb04 canceled.

Name Link
🔨 Latest commit 1a53762
🔍 Latest deploy log https://app.netlify.com/sites/sunny-pastelito-5ecb04/deploys/679a4521bf2d96000894a15b

Copy link

cloudflare-workers-and-pages bot commented Jan 27, 2025

Deploying astronomer-cosmos with  Cloudflare Pages  Cloudflare Pages

Latest commit: 1a53762
Status:⚡️  Build in progress...

View logs

@tatiana tatiana added the customer request An Astronomer customer made requested this label Jan 27, 2025
@tatiana
Copy link
Collaborator Author

tatiana commented Jan 28, 2025

@wornjs @dwreeves @pankajastro @pankajkoti would love your thoughts. Please look at the description and the current/ possible alternative implementation. I'm happy to hear other ideas as well! I'd rather take a bit longer and have an interface that makes sense.

@tatiana tatiana changed the title WIP: Support overriding the profile_name per node or folder using config WIP: Support overriding the profile_name per dbt node or folder using config Jan 28, 2025
@tatiana tatiana changed the title WIP: Support overriding the profile_name per dbt node or folder using config WIP: Support overriding the profile_config per dbt node or folder using config Jan 28, 2025
@tatiana tatiana changed the title WIP: Support overriding the profile_config per dbt node or folder using config Support overriding the profile_config per dbt node or folder using config Jan 28, 2025
Copy link

codecov bot commented Jan 28, 2025

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 97.11%. Comparing base (c4e4abc) to head (cbf97d1).

Additional details and impacted files
@@            Coverage Diff             @@
##             main    #1492      +/-   ##
==========================================
+ Coverage   97.05%   97.11%   +0.05%     
==========================================
  Files          77       77              
  Lines        4484     4501      +17     
==========================================
+ Hits         4352     4371      +19     
+ Misses        132      130       -2     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@tatiana tatiana marked this pull request as ready for review January 28, 2025 22:45
@dosubot dosubot bot added size:L This PR changes 100-499 lines, ignoring generated files. area:config Related to configuration, like YAML files, environment variables, or executer configuration area:profile Related to ProfileConfig, like Athena, BigQuery, Clickhouse, Spark, Trino, etc labels Jan 28, 2025
@tatiana tatiana added this to the Cosmos 1.9.0 milestone Jan 28, 2025
@tatiana tatiana self-assigned this Jan 28, 2025
cosmos/airflow/graph.py Outdated Show resolved Hide resolved
@tatiana tatiana force-pushed the override_profile_name_by_task branch from 4db7628 to cd87b6f Compare January 29, 2025 13:26
@dosubot dosubot bot added size:XL This PR changes 500-999 lines, ignoring generated files. and removed size:L This PR changes 100-499 lines, ignoring generated files. labels Jan 29, 2025
@tatiana tatiana merged commit b7e8ceb into main Jan 29, 2025
5 of 34 checks passed
@tatiana tatiana deleted the override_profile_name_by_task branch January 29, 2025 15:12
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area:config Related to configuration, like YAML files, environment variables, or executer configuration area:profile Related to ProfileConfig, like Athena, BigQuery, Clickhouse, Spark, Trino, etc customer request An Astronomer customer made requested this size:XL This PR changes 500-999 lines, ignoring generated files.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants