finetune model

[github163sl]

Hello, I have fine-tuned the training model according to the meaning of the article, but the results are different from yours. I don't know why the result is so bad. Can you help me solve it? Thank you！

Run command “python train.py --id finetune --freeze_earlier_blocks 4"

This is the code I modified.

Create dataloader

####################### #modify#########################
dataset_train_finetune = PanoCorBonDataset(
root_dir=args.train_finetune_dir,
flip=not args.no_flip, rotate=not args.no_rotate, gamma=not args.no_gamma,
stretch=not args.no_pano_stretch)

loader_train_finetune = DataLoader(dataset_train_finetune, args.batch_size_train,
                          shuffle=True, drop_last=True,
                          num_workers=args.num_workers,
                          pin_memory=not args.no_cuda,
                          worker_init_fn=lambda x: np.random.seed())
 ####################### #modify#########################
dataset_train = PanoCorBonDataset(
    root_dir=args.train_root_dir,
    flip=not args.no_flip, rotate=not args.no_rotate, gamma=not args.no_gamma,
    stretch=not args.no_pano_stretch)
loader_train = DataLoader(dataset_train, args.batch_size_train,
                          shuffle=True, drop_last=True,
                          num_workers=args.num_workers,
                          pin_memory=not args.no_cuda,
                          worker_init_fn=lambda x: np.random.seed())
if args.valid_root_dir:
    dataset_valid = PanoCorBonDataset(
        root_dir=args.valid_root_dir,
        flip=False, rotate=False, gamma=False,
        stretch=False)
    loader_valid = DataLoader(dataset_valid, args.batch_size_valid,
                              shuffle=False, drop_last=False,
                              num_workers=args.num_workers,
                              pin_memory=not args.no_cuda)

Start training

for ith_epoch in trange(1, args.epochs + 1, desc='Epoch', unit='ep'):

    # Train phase
    net.train()
    if args.freeze_earlier_blocks != -1:
        b0, b1, b2, b3, b4 = net.feature_extractor.list_blocks()
        blocks = [b0, b1, b2, b3, b4]
        for i in range(args.freeze_earlier_blocks + 1):
            for m in blocks[i]:
                m.eval()
    iterator_train = iter(loader_train)
    iterator_train_finetune = iter(loader_train_finetune)

    ith_batch = 0

#######################################modify##################################
for _ in trange(len(loader_train_finetune),
desc='Train ep%s' % ith_epoch, position=1):
# Set learning rate
adjust_learning_rate(optimizer, args)

        args.cur_iter += 1
        x1, y_bon1, y_cor1 = next(iterator_train_finetune)
        x2, y_bon2, y_cor2 = next(iterator_train)

        x = torch.cat([x1,x2],0)
        y_bon = torch.cat([y_bon1, y_bon2], 0)
        y_cor = torch.cat([y_cor1, y_cor2], 0)

        losses = feed_forward(net, x, y_bon, y_cor)
        for k, v in losses.items():
            k = 'train/%s' % k
            tb_writer.add_scalar(k, v.item(), args.cur_iter)
        tb_writer.add_scalar('train/lr', args.running_lr, args.cur_iter)
        loss = losses['total']

#######################################modify##################################
# backprop
optimizer.zero_grad()
loss.backward()
nn.utils.clip_grad_norm_(net.parameters(), 3.0, norm_type='inf')
optimizer.step()
ith_batch += 1
179,24 82%

[sunset1995]

Sorry for late reply.
You didn't load any panorama/layout pertained weight for encoder and freeze them on ImageNet pertained only weight.
This could be the reason for your bad result.
If you really want to freeze the entire encoder, please consider load my pertained weight mentioned in README (here).

Some other questions:

• Did you use the latest version?
• Did you check the correctness of your dataset by visualization? (python dataset.py -h can help you)

[github163sl]

Thank you for your reply!

I fine-tuned the model according to the article, freezing all layers, the effect is bad.
Run command “python train.py --id finetune --freeze_earlier_blocks 4 --pth resnet50_rnn__panos2d3d.pth"

When I froze some layers, the results were relatively good. The fewer layers are frozen, the better the results will be?
Run command “python train.py --id finetune --freeze_earlier_blocks 2 --pth resnet50_rnn__panos2d3d.pth"