Real-time log ingestion to Datadog from dbt Cloud

[dpguthrie]

Hey All!

I have been playing around with a solution that sends data from the metadata API to Datadog following a successful job run within dbt Cloud. A few things to note about the solution so you can try it on your own:

• I'm leveraging github actions to trigger the job on a cron schedule. Make sure to deselect "Run on Schedule" within your job settings so it doesn't run more than once.
• The job_id is hardcoded within the yml file right now (along with the Datadog site). The reasoning behind that was so you can use this for multiple jobs!
• Set the following secrets within your repo:
  • DBT_CLOUD_ACCOUNT_ID (look in the url -> /accounts/<account_id>)
  • DBT_CLOUD_SERVICE_TOKEN (Account Settings -> Service Tokens)
  • DBT_CLOUD_API_KEY (Profile -> API Access)
  • DD_API_KEY (Datadog - Organization Settings -> API Keys)

You can find all the code within this repo. The pertinent files are:

• run_and_log.py
• run_and_log.yml
• requirements.txt

I'm sure there's a better way of doing some of this, so please, feel free to provide any feedback.

Edit

The service token I used has the "Metadata Only" permission, so it's unable to be used to trigger a job and is the reason why I am also including an API key. If your service token contains the "Job Admin" permission, you can use that for both arguments.