Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

SpeedUp: polishing #893

Open
matyaskopp opened this issue Jan 14, 2025 · 0 comments
Open

SpeedUp: polishing #893

matyaskopp opened this issue Jan 14, 2025 · 0 comments
Assignees

Comments

@matyaskopp
Copy link
Collaborator

Polishing of annotated version of the ParlaMint-IL corpus takes 17 hours on AMD EPYC 7313, when running on a single thread:

#Format XML file to be a bit nicer & smaller
sub polish {
my $dir = shift;
foreach my $file (glob("$dir/*.xml $dir/*/*.xml")) {
`$scriptPolish < $file > $file.tmp`;
rename("$file.tmp", $file);
}
}

It can be easily parallelized:

  • use parallel command
  • add --procThreads to set a degree of parallelization
@matyaskopp matyaskopp self-assigned this Jan 14, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant