Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Load job failure related has succeed #420

Open
moriyasror opened this issue Jan 22, 2025 · 0 comments
Open

Load job failure related has succeed #420

moriyasror opened this issue Jan 22, 2025 · 0 comments

Comments

@moriyasror
Copy link

moriyasror commented Jan 22, 2025

I use the biqquery connector and I have schema mismatch, which cause the load job to fail so data isn't uploaded to biq query,
The file in the bucket contains field that isn't exist in the table, but I see no errors or warnings in the logs.
Actually the code relate this as success:

I see this log message: "GCS To BQ job tally: 1 successful jobs, 0 failed jobs"
the file is deleted from the google storage bucket.

From my understanding and testing the issue is that job.isDone return true, since the job object is not refresh with the current status.
Since we look on the job without reload we don't have the right status, adding this job.reload raise exception as I expected and solve this issue, we relate this as error and do retry as expected.
As the code today, there is no flow which the BigQueryException can be raised, so we don't identify ant error in bq load

logger.debug("Checking next job: {}", job.getJobId());
try {
// Reload the job to get the latest status - I have added this line
job = job.reload();
if (job.isDone()) {
......
}

moriyasror pushed a commit to moriyasror/kafka-connect-bigquery that referenced this issue Jan 22, 2025
(The code which handle errors in bq load job, will never be called as we don't identify error correctly)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant