Feature/spark expectation enhancements#123

[sudeep7978]

Add Column for Column-Level Visibility in Data Quality Framework Result Table

Description

Schema Evolution with AutoMerge:
Enabled Delta Lake's spark.databricks.delta.schema.autoMerge.enabled configuration to allow schema evolution during write operations.
Modified the data quality framework to include the affected_column_name field dynamically if not already present.

ENHANCEMENT

• Enhanced granularity in data quality reporting.
• Improved ease of debugging and resolving data quality issues.
• Better alignment with industry practices for data governance and observability.

Motivation and Context

Increased Transparency: Builds trust by providing clear visibility into how data quality rules are applied and which columns are impacted.
Operational Efficiency: Reduces manual intervention and effort required to diagnose data issues, optimizing resource utilization.

How Has This Been Tested?

Ensure backward compatibility is maintained for legacy workflows making sure that exusting pipelines donot break

Screenshots (if appropriate):

Types of changes

Checklist:

• My code follows the code style of this project.
• My change requires a change to the documentation.
• I have updated the documentation accordingly.
• I have read the CONTRIBUTING document.
• I have added tests to cover my changes.
• All new and existing tests passed.

[sudeep7978]

@asingamaneni Can you please look into it.
Let me know any changes required.
THANK YOU.

[sudeep7978]

@asingamaneni
closing this PR will raise a new PR taking the SMTP authentication changes after the other PR is merged
#54