Create separate files for each deduplication class

[sarahyurick]

Re-opening #389 because of merge issues.

[ayushdg]

Generally in favor of this.

Would also check and update the api docs if needed: https://github.com/NVIDIA/NeMo-Curator/blob/main/docs/user-guide/api/deduplication.rst (not sure if it is or not).

Would also update any nemo-ci scripts if needed to accomodate the changes here.

[ayushdg]

cc: @VibhuJawa Could you take a look for semdedup as well?

[VibhuJawa]

Reviwed the semdedup functions, my only nit pick is to keep the module level import for stuff like ClusteringModel , EmbeddingCreator etc because they can also be generic in nature so it is a easier mental model.

[VibhuJawa]

We should keep this as semnatic dedup specific import (because clustering can be generic). So something like:

[sarahyurick]

It looks like EmbeddingCreator, ClusteringModel, and SemanticClusterLevelDedup are currently available as generic imports in NeMo Curator: https://github.com/NVIDIA/NeMo-Curator/blob/main/docs/user-guide/api/deduplication.rst

Should I change it to only be importable via nemo_curator.semantic_dedup instead?