Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Display rendered SQL from async operators in the Airflow UI using the monkeypatched approach #1490

Open
2 tasks
pankajkoti opened this issue Jan 24, 2025 · 0 comments
Open
2 tasks

Display rendered SQL from async operators in the Airflow UI using the monkeypatched approach #1490

pankajkoti opened this issue Jan 24, 2025 · 0 comments
Assignees
@pankajkoti
Labels
area:rendering Related to rendering, like Jinja, Airflow tasks, etc execution:async Related to the Async execution mode priority:high High priority issues are blocking or critical issues without a workaround and large impact
Milestone
Cosmos 1.9.0

Comments

@pankajkoti
Copy link
Contributor

Issue Description

In PR #1474, the monkeypatched approach generates SQL that is submitted to asynchronous Airflow operators. It would be helpful for users to see this rendered SQL directly in the Airflow UI to enhance visibility and debugging.

In my initial attempt within PR #1474 itself, I noticed that the SQL is stored in the rendered template field while the operator is running but gets cleared after transitioning into and out of deferred mode. Investigate ways to persist the SQL during deferrable execution.

Acceptance Criteria

  • The Airflow UI displays the rendered SQL submitted as part of async operators.
  • Validate that the displayed SQL matches what is being executed by the operator.

This feature will improve the user experience by making the generated SQL more transparent and easier to inspect.
@pankajkoti pankajkoti added this to the Cosmos 1.9.0 milestone Jan 24, 2025
@pankajkoti pankajkoti self-assigned this Jan 24, 2025
@dosubot dosubot bot added the area:rendering Related to rendering, like Jinja, Airflow tasks, etc label Jan 24, 2025
@tatiana tatiana added the priority:high High priority issues are blocking or critical issues without a workaround and large impact label Jan 27, 2025
@pankajkoti pankajkoti mentioned this issue Feb 3, 2025
Merged
@pankajkoti pankajkoti added the execution:async Related to the Async execution mode label Feb 5, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@pankajkoti pankajkoti

Labels
area:rendering Related to rendering, like Jinja, Airflow tasks, etc execution:async Related to the Async execution mode priority:high High priority issues are blocking or critical issues without a workaround and large impact
Projects
None yet
Milestone
Cosmos 1.9.0
Development

No branches or pull requests
3 participants
@tatiana @pankajkoti @pankajastro