Adapter

[wangxiang-wx]

Do you have any optimization suggestions about your adapter? thank you！
class Adapter(nn.Module):
def init(self, blk) -> None:
super(Adapter, self).init()
self.block = blk
dim = blk.attn.qkv.in_features # dim = 144,288,576
self.prompt_learn = nn.Sequential(
nn.Linear(dim, 32),
nn.GELU(),
nn.Linear(32, dim),
nn.GELU()
)

def forward(self, x):
    prompt = self.prompt_learn(x)   #  x.torch.Size = [24, 88, 88, 144] (batch, w, h, channel)
    promped = x + prompt
    net = self.block(promped)
    return net

[xiongxyowo]

Hi, below are some common design insights for Adapter:

1. Low-level structural information, such as EVP [1], SAM2-Adapter [2].
2. Multi-scale context, such as Mammo-SAM [3], MDSAM [4].
3. Additional parallel encoder branch, such as ViT-Adapter [5], SAMUS [6].
4. Insertion position, such as AdaptFormer [7].

[1] Explicit visual prompting for low-level structure segmentations
[2] SAM2-Adapter: Evaluating & Adapting Segment Anything 2 in Downstream Tasks
[3] Mammo-sam: Adapting foundation segment anything model for automatic breast mass segmentation in whole mammograms
[4] Multi-Scale and Detail-Enhanced Segment Anything Model for Salient Object Detection
[5] Vision Transformer Adapter for Dense Predictions
[6] SAMUS: Adapting segment anything model for clinically-friendly and generalizable ultrasound image segmentation
[7] Adaptformer: Adapting vision transformers for scalable visual recognition

---

同学你好，Adapter的改进思路大致分为如下几种：

1. 引入低层结构信息，如EVP [1], SAM2-Adapter [2]
2. 引入多尺度上下文，如Mammo-SAM [3], MDSAM [4]
3. 独立的Adapter分支，如ViT-Adapter [5], SAMUS [6]
4. 改进插入位置，如AdaptFormer [7]

[wangxiang-wx]

Thank you ！！！