Add DAB-DETR Object detection/segmentation model #30803

conditionedstimulus · 2024-05-14T14:19:50Z

What does this PR do?

Add DAB-DETR Object detection model. Paper: https://arxiv.org/abs/2201.12329
Original code repo: https://github.com/IDEA-Research/DAB-DETR

Fixes # (issue)
[WIP] This model is part of how DETR models have evolved, alongside DN DETR (not part of this PR), to pave the way for newer and better models like Dino and Stable Dino in object detection

Who can review?

@amyeroberts

amyeroberts · 2024-05-15T14:36:33Z

Hi @conditionedstimulus, thanks for opening a PR!

Just skimming over the modeling files, it looks like all of the modules are copied from, or can be copied from conditional DETR. Are there any architectural changes this model brings? If not, then all we need to do is convert the checkpoints and upload those to the hub such that they can be loaded in ConditionalDETR directly

conditionedstimulus · 2024-05-15T20:38:01Z

Hi @conditionedstimulus, thanks for opening a PR!

Just skimming over the modeling files, it looks like all of the modules are copied from, or can be copied from conditional DETR. Are there any architectural changes this model brings? If not, then all we need to do is convert the checkpoints and upload those to the hub such that they can be loaded in ConditionalDETR directly

Hi Amy,

I attached a photo comparing the cross-attention of the decoder in DETR, Conditional DETR, and DAB DETR, as this is the main architectural difference. I copied the code from Conditional DETR because this model is an extension/evolved version of Conditional DETR. I believe it would be cool and useful to include this model in the HF object detection collection.

amyeroberts · 2024-05-17T12:00:10Z

@conditionedstimulus Thanks for sharing! OK, seems useful to have this available as an option as part of the DETR family in the library. Feel free to ping me when the PR is ready for review.

cc @qubvel for reference

qubvel · 2025-01-21T14:42:05Z

btw, I see there is a "DAB-Deformable-DETR-R50-v2" checkpoint on github which has the best metrics, is it possible to convert it as well?

qubvel

Tested inference and training as well, all goes smoothly:

Final steps to merge the model:

Merge/rebase on main
Run slow tests (I can trigger it, as soon as main merged)
~~Convert the last checkpoint, mentioned above~~
Add model cards for all checkpoints on the Hub
Transfer checkpoints to IDEA-Research org (also on my side, only if it's ok with you) + correct checkpoint's path in code
Merge!

Thank you, @conditionedstimulus, for the iterations, and apologize for the delays on our end. 🤗

conditionedstimulus · 2025-01-22T21:53:08Z

Hi @qubvel,

Thank you for your feedback and review! Here's an update on my progress:

I merged the main branch into the feature branch.
I ran the slow tests.
I updated the model cards.
Yes, please copy the model cards to IDEA-Research as it is their work, I already updated the 'how to use' part with their source in the model cards.
Unfortunately, I’m unable to convert the DAB-Deformable-DETR checkpoint, as it uses a different architecture. You can see the differences here. (c, e ,f)

Model cards:
davidhajdu/dab-detr-resnet-50
davidhajdu/dab-detr-resnet-50-dc5
davidhajdu/dab-detr-resnet-50-dc5-fixxy
davidhajdu/dab-detr-resnet-50-dc5-pat3
davidhajdu/dab-detr-resnet-50-pat3

If you would like any modifications to the model cards, please let me know!

qubvel · 2025-01-22T23:22:10Z

run-slow: dab_detr

github-actions · 2025-01-22T23:23:30Z

This comment contains run-slow, running the specified jobs: ['models/dab_detr'] ...

qubvel · 2025-01-22T23:28:10Z

I see these 3 tests are failing on CI:

FAILED tests/models/dab_detr/test_modeling_dab_detr.py::DabDetrModelTest::test_cpu_offload - RuntimeError: Attempted to call `variable.set_data(tensor)`, but `variable` and `tensor` have incompatible tensor type.
FAILED tests/models/dab_detr/test_modeling_dab_detr.py::DabDetrModelTest::test_disk_offload_safetensors - RuntimeError: Attempted to call `variable.set_data(tensor)`, but `variable` and `tensor` have incompatible tensor type.
FAILED tests/models/dab_detr/test_modeling_dab_detr.py::DabDetrModelTest::test_save_load_low_cpu_mem_usage - RuntimeError: Attempted to call `variable.set_data(tensor)`, but `variable` and `tensor` have incompatible tensor type.

Did they pass locally?

qubvel · 2025-01-22T23:30:25Z

Also it seems like images are not loading in model cards (maybe my issue only). Feel free to open a upload them into dataset with "Add file" button and ping me to merge.

conditionedstimulus · 2025-01-23T14:11:44Z

test_cpu_offload

Hey, yes I don't have these kind of errors locally. I have one issue but that related to the model weight sources. I already left the new 'IDEA-Research' one inside, but that throws a different kind of error.

But if I fix the source then it runs without errors:

conditionedstimulus · 2025-01-23T14:16:56Z

Hi @qubvel,

I opened the PR on huggingface datasets.
When it will be ready I'll modify the image sources!

Thank you!

qubvel · 2025-01-23T14:28:00Z

Thanks, merged!

Yeah, integration tests will be fixed as soon as we transfer checkpoints. test_cpu_offload is more interesting, are you running with GPU available?

conditionedstimulus · 2025-01-23T14:36:23Z

Thanks, merged!

Yeah, integration tests will be fixed as soon as we transfer checkpoints. test_cpu_offload is more interesting, are you running with GPU available?

Thanks! No i run actually on a really weak cpu only with 8gb ram (i5 dual core)

qubvel · 2025-01-23T14:43:55Z

I think that's the reason, the test is just skipped for cpu-only env. Try to run with pytest -v to see.

conditionedstimulus · 2025-01-23T18:15:18Z

Hi @qubvel ,

This test fails: FAILED tests/models/qwen2_5_vl/test_modeling_qwen2_5_vl.py::Qwen2_5_VLModelTest::test_generate_from_inputs_embeds_1_beam_search
but it's not related to DAB-DETR.
I found the error and fixed it and I also pulled the main branch into the feature branch.
I also updated the model cards.

Thanks!

qubvel · 2025-01-23T19:41:02Z

src/transformers/models/dab_detr/modeling_dab_detr.py

+            module.class_embed.bias.data = (
+                torch.ones(
+                    self.config.num_labels,
+                    device=module.class_embed.bias.data.device,
+                    dtype=module.class_embed.bias.data.dtype,
+                )
+                * bias_value
+            )


I suppose we can use module.class_embed.bias.data.fill_(bias_value) or smth similar

Yes, better solution. I've just updated the code.

qubvel · 2025-01-23T19:42:26Z

run-slow: dab_detr

github-actions · 2025-01-23T19:43:43Z

This comment contains run-slow, running the specified jobs: ['models/dab_detr'] ...

conditionedstimulus · 2025-01-23T20:44:03Z

run-slow: dab_detr

conditionedstimulus · 2025-01-23T20:47:59Z

@qubvel I can see my comment doesn't have the superpower to trigger the bot to run the tests from a comment. :)

qubvel · 2025-01-23T23:42:30Z

Yeah, it's available for maintainers only 😄

qubvel · 2025-01-23T23:42:41Z

run-slow: dab_detr

github-actions · 2025-01-23T23:44:02Z

This comment contains run-slow, running the specified jobs: ['models/dab_detr'] ...

qubvel

Thanks for fixing tests! While I asking the team to move checkpoints to the org, can you please update the last thing (I hope 😄)

qubvel · 2025-01-28T10:31:27Z

tests/models/dab_detr/test_modeling_dab_detr.py

+        self.assertEqual(len(results["scores"]), 5)
+        self.assertTrue(torch.allclose(results["scores"], expected_scores, atol=1e-4))
+        self.assertSequenceEqual(results["labels"].tolist(), expected_labels)
+        self.assertTrue(torch.allclose(results["boxes"][0, :], expected_boxes, atol=1e-4))


I hope the last thing! Can you please update to use torch.testing.assert_close instead of self.assertTrue(torch.allclose(...

Here and in other places in tests, for example:

https://github.com/huggingface/transformers/pull/35903/files

Hi, no problem I changed and ran the test w my model source. Is it enough to change only in that part of the tests or it should be in the whole file? Also apprx. how much time it's gonna take to move the model cards?:)
thanks

…t assert_close

initial commit

8adf1bb

encoder+decoder layer changes WIP

8291122

conditionedstimulus added 25 commits May 21, 2024 21:48

architecture checks

09e2516

working version of detection + segmentation

8a004cf

fix modeling outputs

defbc43

fix return dict + output att/hs

5cfbcfc

found the position embedding masking bug

6c7564a

pre-training version

35e056f

added iamge processors

24a9d7a

typo in init.py

d9b7af4

iterupdate set to false

a171339

fixed num_labels in class_output linear layer bias init

b8b2201

multihead attention shape fixes

abe0698

test improvements

e60b555

test update

6dafb79

dab-detr model_doc update

5bbdca1

dab-detr model_doc update2

4a5ac4f

test fix:test_retain_grad_hidden_states_attentions

592796b

config file clean and renaming variables

d76fda2

config file clean and renaming variables fix

ade9720

updated convert_to_hf file

6b58e5f

small fixes

eac19f5

style and qulity checks

460e9d6

Merge branch 'main' into add_dab_detr

0151f65

return_dict fix

97194c7

Merge branch main into add_dab_detr

3fc56b4

Merge branch main into add_dab_detr

ffbb1dc

qubvel approved these changes Jan 21, 2025

View reviewed changes

conditionedstimulus added 2 commits January 22, 2025 21:03

Merge branch 'main' into add_dab_detr

0eea3b5

[run_slow] dab_detr

194f62d

conditionedstimulus added 2 commits January 23, 2025 16:24

added device and type for head bias data part

7b58b25

Merge branch 'main' into add_dab_detr

675f3fd

[run_slow] dab_detr

7c1161b

qubvel reviewed Jan 23, 2025

View reviewed changes

fixed model head bias data fill

0967981

qubvel approved these changes Jan 28, 2025

View reviewed changes

conditionedstimulus added 2 commits January 28, 2025 12:36

changed test_inference_object_detection_head assertTrues to torch tes…

ac8f4cb

…t assert_close

Merge branch 'main' into add_dab_detr

3cf9b99

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add DAB-DETR Object detection/segmentation model #30803

Add DAB-DETR Object detection/segmentation model #30803

conditionedstimulus commented May 14, 2024

amyeroberts commented May 15, 2024

conditionedstimulus commented May 15, 2024

amyeroberts commented May 17, 2024

qubvel commented Jan 21, 2025

qubvel left a comment •

edited

Loading

conditionedstimulus commented Jan 22, 2025

qubvel commented Jan 22, 2025

github-actions bot commented Jan 22, 2025

qubvel commented Jan 22, 2025

qubvel commented Jan 22, 2025

conditionedstimulus commented Jan 23, 2025

conditionedstimulus commented Jan 23, 2025

qubvel commented Jan 23, 2025

conditionedstimulus commented Jan 23, 2025

qubvel commented Jan 23, 2025

conditionedstimulus commented Jan 23, 2025

qubvel Jan 23, 2025

conditionedstimulus Jan 23, 2025

qubvel commented Jan 23, 2025

github-actions bot commented Jan 23, 2025

conditionedstimulus commented Jan 23, 2025

conditionedstimulus commented Jan 23, 2025 •

edited

Loading

qubvel commented Jan 23, 2025

qubvel commented Jan 23, 2025

github-actions bot commented Jan 23, 2025

qubvel left a comment •

edited

Loading

qubvel Jan 28, 2025

qubvel Jan 28, 2025

conditionedstimulus Jan 28, 2025

Add DAB-DETR Object detection/segmentation model #30803

Are you sure you want to change the base?

Add DAB-DETR Object detection/segmentation model #30803

Conversation

conditionedstimulus commented May 14, 2024

What does this PR do?

Who can review?

amyeroberts commented May 15, 2024

conditionedstimulus commented May 15, 2024

amyeroberts commented May 17, 2024

qubvel commented Jan 21, 2025

qubvel left a comment • edited Loading

Choose a reason for hiding this comment

conditionedstimulus commented Jan 22, 2025

qubvel commented Jan 22, 2025

github-actions bot commented Jan 22, 2025

qubvel commented Jan 22, 2025

qubvel commented Jan 22, 2025

conditionedstimulus commented Jan 23, 2025

conditionedstimulus commented Jan 23, 2025

qubvel commented Jan 23, 2025

conditionedstimulus commented Jan 23, 2025

qubvel commented Jan 23, 2025

conditionedstimulus commented Jan 23, 2025

qubvel Jan 23, 2025

Choose a reason for hiding this comment

conditionedstimulus Jan 23, 2025

Choose a reason for hiding this comment

qubvel commented Jan 23, 2025

github-actions bot commented Jan 23, 2025

conditionedstimulus commented Jan 23, 2025

conditionedstimulus commented Jan 23, 2025 • edited Loading

qubvel commented Jan 23, 2025

qubvel commented Jan 23, 2025

github-actions bot commented Jan 23, 2025

qubvel left a comment • edited Loading

Choose a reason for hiding this comment

qubvel Jan 28, 2025

Choose a reason for hiding this comment

qubvel Jan 28, 2025

Choose a reason for hiding this comment

conditionedstimulus Jan 28, 2025

Choose a reason for hiding this comment

qubvel left a comment •

edited

Loading

conditionedstimulus commented Jan 23, 2025 •

edited

Loading

qubvel left a comment •

edited

Loading