CUDA out of memory

[FelixKStudent]

I have been trying to train my own model on custom data, however in the pre-hubert step no matter what I do I get the error bellow. At first thinking this was caused due to the fact that I was training on 30 min of data, I cut out a section only 1.5 min long but nothing seemed to change. Does someone know what might be casuing this? (I have a rtx 4090 on a linux system)
Thanks in advance!

Traceback (most recent call last):
File "/home/felix/svc/venv/bin/svc", line 8, in
sys.exit(cli())
File "/home/felix/svc/venv/lib/python3.10/site-packages/click/core.py", line 1130, in call
return self.main(*args, **kwargs)
File "/home/felix/svc/venv/lib/python3.10/site-packages/click/core.py", line 1055, in main
rv = self.invoke(ctx)
File "/home/felix/svc/venv/lib/python3.10/site-packages/click/core.py", line 1657, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/home/felix/svc/venv/lib/python3.10/site-packages/click/core.py", line 1404, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "/home/felix/svc/venv/lib/python3.10/site-packages/click/core.py", line 760, in invoke
return __callback(*args, **kwargs)
File "/home/felix/svc/venv/lib/python3.10/site-packages/so_vits_svc_fork/main.py", line 613, in pre_hubert
preprocess_hubert_f0(
File "/home/felix/svc/venv/lib/python3.10/site-packages/so_vits_svc_fork/preprocessing/preprocess_hubert_f0.py", line 148, in preprocess_hubert_f0
Parallel(n_jobs=n_jobs)(
File "/home/felix/svc/venv/lib/python3.10/site-packages/joblib/parallel.py", line 1085, in call
if self.dispatch_one_batch(iterator):
File "/home/felix/svc/venv/lib/python3.10/site-packages/joblib/parallel.py", line 901, in dispatch_one_batch
self._dispatch(tasks)
File "/home/felix/svc/venv/lib/python3.10/site-packages/joblib/parallel.py", line 819, in _dispatch
job = self._backend.apply_async(batch, callback=cb)
File "/home/felix/svc/venv/lib/python3.10/site-packages/joblib/_parallel_backends.py", line 208, in apply_async
result = ImmediateResult(func)
File "/home/felix/svc/venv/lib/python3.10/site-packages/joblib/_parallel_backends.py", line 597, in init
self.results = batch()
File "/home/felix/svc/venv/lib/python3.10/site-packages/joblib/parallel.py", line 288, in call
return [func(*args, **kwargs)
File "/home/felix/svc/venv/lib/python3.10/site-packages/joblib/parallel.py", line 288, in
return [func(*args, **kwargs)
File "/home/felix/svc/venv/lib/python3.10/site-packages/so_vits_svc_fork/preprocessing/preprocess_hubert_f0.py", line 112, in _process_batch
_process_one(
File "/home/felix/svc/venv/lib/python3.10/site-packages/so_vits_svc_fork/preprocessing/preprocess_hubert_f0.py", line 58, in _process_one
c = utils.get_content(
File "/home/felix/svc/venv/lib/python3.10/site-packages/so_vits_svc_fork/utils.py", line 182, in get_content
c = cmodel(audio, output_hidden_states=True)["hidden_states"][9]
File "/home/felix/svc/venv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
return forward_call(*args, **kwargs)
File "/home/felix/svc/venv/lib/python3.10/site-packages/transformers/models/hubert/modeling_hubert.py", line 1073, in forward
encoder_outputs = self.encoder(
File "/home/felix/svc/venv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
return forward_call(*args, **kwargs)
File "/home/felix/svc/venv/lib/python3.10/site-packages/transformers/models/hubert/modeling_hubert.py", line 709, in forward
layer_outputs = layer(
File "/home/felix/svc/venv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
return forward_call(*args, **kwargs)
File "/home/felix/svc/venv/lib/python3.10/site-packages/transformers/models/hubert/modeling_hubert.py", line 588, in forward
hidden_states, attn_weights, _ = self.attention(
File "/home/felix/svc/venv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
return forward_call(*args, **kwargs)
File "/home/felix/svc/venv/lib/python3.10/site-packages/transformers/models/hubert/modeling_hubert.py", line 503, in forward
attn_weights = nn.functional.softmax(attn_weights, dim=-1)
File "/home/felix/svc/venv/lib/python3.10/site-packages/torch/nn/functional.py", line 1843, in softmax
ret = input.softmax(dim)
torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 8.62 GiB (GPU 0; 23.65 GiB total capacity; 9.69 GiB already allocated; 3.65 GiB free; 18.40 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation. See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF

[Z3Coder]

1.5 min pieces are too long. you need pieces of around 10 secs.

[SamueleLorefice]

Split into max 10 seconds a piece, you want to have multiple pieces of 10 seconds or less, a quick and dirty way is to use ffmpeg to split in 10 seconds chunks.