Testing and evaluating custom dataset and retrieving model analytics.

[denmonz]

I've just trained a model on ~13 hours of data, and saved the model. How do I test the model and collect model analytics?

After I train the model, I serialize it and save it, then load it and run testing using the following code:

model = DPRNNTasNet.from_pretrained(pretrained_model_conf_or_path=model_file)

model.to(device, non_blocking=True)

loss = PITLossWrapper(pairwise_neg_sisdr, pit_from="pw_mtx")

optimizer = optim.Adam(model.parameters(), lr=1e-3)

trainer = Trainer(max_epochs=100, gpus=1, log_every_n_steps=10, logger=wandb_logger, precision=16)

system = System(model=model, optimizer=optimizer, loss_func=loss, train_loader=train_dl, val_loader=val_dl,
                test_loader=test_dl)

results = trainer.test(model=system, dataloaders=test_dl)

I only receive back a loss metric. Does Asteroid provide the means of evaluating a model, or does Asteroid only provide training/validating that model?

[mpariente]

Thanks for a new on-point question! Have a look at the eval.py scripts in the recipes (they all look similar) and you might find the solution. However, this is not optimal and we've been wanting to migrate to lightning-based test loops for a while but didn't get to it. Le sam. 7 août 2021 à 01:29, Zane Denmon ***@***.***> a écrit :

…

I've just trained a model on ~13 hours of data, and saved the model. How do I test the model and collect model analytics? After I train the model, I serialize it and save it, then load it and run testing using the following code: model = DPRNNTasNet.from_pretrained(pretrained_model_conf_or_path=model_file) model.to(device, non_blocking=True) loss = PITLossWrapper(pairwise_neg_sisdr, pit_from="pw_mtx") optimizer = optim.Adam(model.parameters(), lr=1e-3) trainer = Trainer(max_epochs=100, gpus=1, log_every_n_steps=10, logger=wandb_logger, precision=16) system = System(model=model, optimizer=optimizer, loss_func=loss, train_loader=train_dl, val_loader=val_dl, test_loader=test_dl) results = trainer.test(model=system, dataloaders=test_dl) I only receive back a loss metric. Does Asteroid provide the means of evaluating a model, or does Asteroid only provide training/validating that model? — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#538>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AEND2HFUDM2IBOMTHRHD66LT3RV6VANCNFSM5BWYM4RQ> .

[denmonz]

I was able to add a def test_step(self, batch, batch_nb) to asteroid\engine\system.py that seems to do the trick, though, I get errors when attempting to retrieve a PESQ score:

    def test_step(self, batch, batch_nb):
            """Pass data through the model and compute the loss, and other metrics such as:
            SI-SDR: Scale-Invariant Source-to-Distortion Ratio
            SDR: Source-to-Distortion Ratio
                SDR is usually considered to be an overall measure of how good a source sounds. If a paper only reports
                 one number for estimated quality, it is usually SDR.
            SIR: Source-to-Interference Ratio
                This is usually interpreted as the amount of other sources that can be heard in a source estimate. This
                 is most close to the concept of “bleed”, or “leakage”.
            SAR: Source-to-Artifact Ratio
                This is usually interpreted as the amount of unwanted artifacts a source estimate has with relation to
                 the true source.
            STOI: Short Term Objective Intelligibility
            PESQ

            Args:
                batch: the object returned by the loader (a list of torch.Tensor
                    in most cases) but can be something else.
                batch_nb (int): The number of the batch in the epoch.
            """
            loss = self.common_step(batch, batch_nb, train=False)
            self.log("test_loss", loss, on_epoch=True, prog_bar=True)

            inputs, targets = batch
            est_targets = self(inputs)
    
            # Move tensors from GPU to CPU
            inputs = inputs.cpu().numpy()
            targets = targets.cpu().numpy()
            est_targets = est_targets.cpu().numpy()
            sample_rate = 2521
            
            for x in range(inputs.shape[0]): # Iterate per batch
                # Iterate per source?
                metrics_list = ["si_sdr", "sdr", "sir", "sar", "stoi"]
                metrics = get_metrics(inputs[x], targets[x], est_targets[x], sample_rate=sample_rate,
                                      metrics_list=metrics_list, ignore_metrics_errors=True)
                if 'si_sdr' in metrics.keys() and metrics['si_sdr'] is not None:
                    self.log("SI-SDR", metrics['si_sdr'], on_epoch=True, prog_bar=True)
                if 'sdr' in metrics.keys() and metrics['sdr'] is not None:
                    self.log("SDR", metrics['sdr'], on_epoch=True, prog_bar=True)
                if 'sir' in metrics.keys() and metrics['sir'] is not None:
                    self.log("SIR", metrics['sir'], on_epoch=True, prog_bar=True)
                if 'sar' in metrics.keys() and metrics['sar'] is not None:
                    self.log("SAR", metrics['sar'], on_epoch=True, prog_bar=True)
                if 'stoi' in metrics.keys() and metrics['stoi'] is not None:
                    self.log("STOI", metrics['stoi'], on_epoch=True, prog_bar=True)
                if 'pesq' in metrics.keys() and metrics['pesq'] is not None:
                    self.log("PESQ", metrics['pesq'], on_epoch=True, prog_bar=True)

which is called using this code in my script:

trainer.test(model=system, dataloaders=test_dl)