Skip to content

Commit

Permalink
Use --rows-per-insert instead of merge_inserts.awk
Browse files Browse the repository at this point in the history
Older versions pg_dump didn't have this option so we had to use
our custom AWK script to get a significant speed-up of loading
dumps from feeders on a superhub. All versions of PostgreSQL
CFEngine Enterprise is shipped with now have pg_dump supporting
the new --rows-per-insert option.

Changelog: Data dumping on Federated Reporting feeders no longer
           uses an AWK filter to merge INSERT lines in the dumps
Ticket: None
  • Loading branch information
vpodzime committed Jul 30, 2024
1 parent efb9b49 commit a785de9
Show file tree
Hide file tree
Showing 3 changed files with 1 addition and 123 deletions.
4 changes: 0 additions & 4 deletions cfe_internal/enterprise/federation/federation.cf
Original file line number Diff line number Diff line change
Expand Up @@ -591,10 +591,6 @@ bundle agent federation_manage_files
copy_from => default:local_dcp( "$(this.promise_dirname)/../../../templates/federated_reporting/dump.sh" ),
perms => default:mog( "700", "root", "root" );

"$(cfengine_enterprise_federation:config.federation_dir)/fedhub/dump/filters/50-merge_inserts.awk"
copy_from => default:local_dcp( "$(this.promise_dirname)/../../../templates/federated_reporting/50-merge_inserts.awk" ),
perms => default:mog( "600", "root", "root" );

am_transporter::
"$(cfengine_enterprise_federation:config.bin_dir)/transport.sh" -> { "CFE-951" }
copy_from => default:local_dcp( "$(this.promise_dirname)/../../../templates/federated_reporting/transport.sh" ),
Expand Down
118 changes: 0 additions & 118 deletions templates/federated_reporting/50-merge_inserts.awk

This file was deleted.

2 changes: 1 addition & 1 deletion templates/federated_reporting/dump.sh
Original file line number Diff line number Diff line change
Expand Up @@ -55,7 +55,7 @@ in_progress_file="$CFE_FR_DUMP_DIR/$CFE_FR_FEEDER_$ts.sql.$CFE_FR_COMPRESSOR_EXT

log "Dumping tables: $CFE_FR_TABLES"
{
"$CFE_BIN_DIR"/pg_dump --serializable-deferrable --column-inserts --data-only $(printf ' -t %s' $CFE_FR_TABLES) cfdb
"$CFE_BIN_DIR"/pg_dump --serializable-deferrable --column-inserts --rows-per-insert=10000 --data-only $(printf ' -t %s' $CFE_FR_TABLES) cfdb

# in case of 3.12 must copy m_inventory as if it was __inventory
if [[ "$CFE_VERSION" =~ "3.12." ]]; then
Expand Down

0 comments on commit a785de9

Please sign in to comment.