You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
I use the biqquery connector and I have schema mismatch, which cause the load job to fail so data isn't uploaded to biq query,
The file in the bucket contains field that isn't exist in the table, but I see no errors or warnings in the logs.
Actually the code relate this as success:
I see this log message: "GCS To BQ job tally: 1 successful jobs, 0 failed jobs"
the file is deleted from the google storage bucket.
From my understanding and testing the issue is that job.isDone return true, since the job object is not refresh with the current status.
Since we look on the job without reload we don't have the right status, adding this job.reload raise exception as I expected and solve this issue, we relate this as error and do retry as expected.
As the code today, there is no flow which the BigQueryException can be raised, so we don't identify ant error in bq load
logger.debug("Checking next job: {}", job.getJobId());
try {
// Reload the job to get the latest status - I have added this line
job = job.reload();
if (job.isDone()) {
......
}
The text was updated successfully, but these errors were encountered:
moriyasror
pushed a commit
to moriyasror/kafka-connect-bigquery
that referenced
this issue
Jan 22, 2025
I use the biqquery connector and I have schema mismatch, which cause the load job to fail so data isn't uploaded to biq query,
The file in the bucket contains field that isn't exist in the table, but I see no errors or warnings in the logs.
Actually the code relate this as success:
I see this log message: "GCS To BQ job tally: 1 successful jobs, 0 failed jobs"
the file is deleted from the google storage bucket.
From my understanding and testing the issue is that job.isDone return true, since the job object is not refresh with the current status.
Since we look on the job without reload we don't have the right status, adding this job.reload raise exception as I expected and solve this issue, we relate this as error and do retry as expected.
As the code today, there is no flow which the BigQueryException can be raised, so we don't identify ant error in bq load
logger.debug("Checking next job: {}", job.getJobId());
try {
// Reload the job to get the latest status - I have added this line
job = job.reload();
if (job.isDone()) {
......
}
The text was updated successfully, but these errors were encountered: