Skip to content

Commit

Permalink
PB-1091 Flag the inexistent files in the file size updater
Browse files Browse the repository at this point in the history
For some assets the file on the bucket is missing. This makes the
file_size_upgrade cronjob be stuck in a infinite loop, as it'll always
try to update the next N assets.
Therefore we flag these assets with a `file_size` of `None`, to avoid
repeating the update and to be able to produce a list of missing files.
  • Loading branch information
schtibe committed Jan 21, 2025
1 parent 3911fb8 commit 7c1c923
Showing 1 changed file with 15 additions and 1 deletion.
16 changes: 15 additions & 1 deletion app/stac_api/management/commands/update_asset_file_size.py
Original file line number Diff line number Diff line change
Expand Up @@ -45,6 +45,13 @@ def update(self):
asset.save()
print(".", end="", flush=True)
except ClientError:
# We set file_size to None to indicate that this asset couldn't be
# found on the bucket. That way the script won't get stuck with the
# same 100 inexistent assets on one hand and we'll be able to
# produce a list of missing files on the other hand
asset.file_size = None
asset.save()
print("_", end="", flush=True)
logger.error(
'file size could not be read from s3 bucket [%s] for asset %s', bucket, key
)
Expand All @@ -56,7 +63,7 @@ def update(self):

self.print_success(
f"Update file size for {len(collection_assets)} collection assets out of "
"{total_asset_count}"
f"{total_asset_count}"
)

for collection_asset in collection_assets:
Expand All @@ -71,6 +78,13 @@ def update(self):
collection_asset.save()
print(".", end="", flush=True)
except ClientError:
# We set file_size to None to indicate that this asset couldn't be
# found on the bucket. That way the script won't get stuck with the
# same 100 inexistent assets on one hand and we'll be able to
# produce a list of missing files on the other hand
collection_asset.file_size = None
collection_asset.save()
print("_", end="", flush=True)
logger.error(
'file size could not be read from s3 bucket [%s] for collection asset %s'
)
Expand Down

0 comments on commit 7c1c923

Please sign in to comment.