Skip to content

Commit

Permalink
issue apache#3247 : Snowflake Bulk Uploader always deletes staging files
Browse files Browse the repository at this point in the history
  • Loading branch information
mattcasters committed Nov 14, 2023
1 parent 7efa1c5 commit 7dc8292
Show file tree
Hide file tree
Showing 2 changed files with 2 additions and 2 deletions.
Original file line number Diff line number Diff line change
Expand Up @@ -58,7 +58,7 @@ The Snowflake Bulk Loader transform utilizes the Snowflake Copy command to load
- **On Error**: (Abort, Skip File, Skip File Percent, Continue) The behavior when errors are encountered on a load.
- **Error limit**: (When On Error = Skip File or Skip File Percent) The error limit before the file should be skipped. If empty or 0 the file will be skipped on the first error.
- **Split load every ... rows**: Breaking the temp files into multiple smaller files will allow Snowflake to perform the bulk load in parallel, thus improving performance. This is the number of rows each file should contain.
- **Remove files after load**: (Y/N) Should the files be removed from the Snowflake stage after the load. (Local temp files are always removed.)
- **Remove files after load**: (Y/N) Should the files be removed from the Snowflake stage after the load? You also need to set variable `SNOWFLAKE_DEBUG_MODE` to `true` to keep the files.

=== Data type tab

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -775,7 +775,7 @@ public void dispose() {
setErrors(1);
}

if (!Boolean.parseBoolean(resolve(SnowflakeBulkLoaderMeta.DEBUG_MODE_VAR))) {
if (meta.isRemoveFiles() || !Boolean.parseBoolean(resolve(SnowflakeBulkLoaderMeta.DEBUG_MODE_VAR))) {
for (String filename : data.previouslyOpenedFiles) {
try {
HopVfs.getFileObject(filename).delete();
Expand Down

0 comments on commit 7dc8292

Please sign in to comment.