Skip to content

Commit

Permalink
Early return for num_shards==0 in the Beam pipeline.
Browse files Browse the repository at this point in the history
  • Loading branch information
marcenacp committed Dec 2, 2024
1 parent 40cfbd9 commit b3040e8
Showing 1 changed file with 3 additions and 0 deletions.
Original file line number Diff line number Diff line change
Expand Up @@ -195,6 +195,9 @@ def execute_operations_in_beam(
enumerate(files)
)
num_shards = len(files)
if not num_shards:
logging.warning(f"Empty {record_set=} for filters={json.dumps(filters)}")
return pipeline

# We don't know in advance the number of records per shards. So we just allocate the
# maximum number which is `sys.maxsize // num_shards`. Taking the practical case of
Expand Down

0 comments on commit b3040e8

Please sign in to comment.