[Feat] Add COCOPoseMetric #35

liqikai9 · 2022-10-28T07:10:57Z

Motivation

Add COCOPoseMetric for pose estimation task.

PR in mmpose: open-mmlab/mmpose#1777, about the results verification.

Modification

Add COCOPoseMetric under mmeval/metrics/coco_pose.py
Add some test data: tests/test_metrics/data/coco_pose_sample.json, tests/test_metrics/data/crowdpose_sample.json, tests/test_metrics/data/ap10k_sample.json and unittests.
Add nms function under mmeval/metrics/utils/nms.py

BC-breaking (Optional)

Use cases (Optional)

Checklist

Pre-commit or other linting tools are used to fix the potential lint issues.
The modification is covered by complete unit tests. If not, please add more unit test to ensure the correctness.
If the modification has potential influence on downstream projects, this PR should be tested with downstream projects, like MMDet or MMCls.
The documentation has been modified accordingly, like docstring or example tutorials.

CLAassistant · 2022-12-13T08:32:47Z

All committers have signed the CLA.

C1rN09

Please update api docs && suport_matrix documentation.

BTW I was told this metric should be renamed as COCOKeyPointDetection

C1rN09 · 2023-02-13T07:38:00Z

mmeval/metrics/coco_pose.py

+        >>> try:
+        ...     from xtcocotools.coco import COCO
+        ...     from xtcocotools.cocoeval import COCOeval
+        ...     HAS_XTCOCOTOOLS = True
+        ... except ImportError:
+        ...     HAS_XTCOCOTOOLS = False
+        ...
+        >>> ann_file = 'tests/test_metrics/data/coco_pose_sample.json'
+        >>> coco = COCO(ann_file)
+        loading annotations into memory...
+        Done (t=0.00s)
+        creating index...
+        index created!
+        >>> classes = coco.loadCats(coco.getCatIds())
+        >>> sigmas = np.array([
+        ...     0.026, 0.025, 0.025, 0.035, 0.035, 0.079, 0.079, 0.072, 0.072,
+        ...     0.062, 0.062, 0.107, 0.107, 0.087, 0.087, 0.089, 0.089
+        ... ]).astype(np.float32)
+        >>> coco_dataset_meta = {
+        ...     'CLASSES': classes,
+        ...     'num_keypoints': 17,
+        ...     'sigmas': sigmas,
+        ... }
+        >>> def _convert_ann_to_pred_and_gt(ann_file):
+        ...     predictions = []
+        ...     groundtruths = []
+        ...     db = load(ann_file)
+        ...     imgid2info = dict()
+        ...     for img in db['images']:
+        ...         imgid2info[img['id']] = img
+        ...     for ann in db['annotations']:
+        ...         bboxes = np.array(ann['bbox'], dtype=np.float32).reshape(-1, 4)
+        ...         keypoints = np.array(ann['keypoints']).reshape((1, -1, 3))
+        ...         prediction = {
+        ...             'id': ann['id'],
+        ...             'img_id': ann['image_id'],
+        ...             'bboxes': bboxes,
+        ...             'keypoints': keypoints[..., :2],
+        ...             'keypoint_scores': keypoints[..., -1],
+        ...             'bbox_scores': np.ones((1, ), dtype=np.float32),
+        ...         }
+        ...         groundtruth = {
+        ...             'img_id': ann['image_id'],
+        ...             'width': 640,
+        ...             'height': 480,
+        ...             'num_keypoints': ann['num_keypoints'],
+        ...             'raw_ann_info': [copy.deepcopy(ann)],
+        ...         }
+        ...         if 'area' in ann:
+        ...             groundtruth['area'] = ann['area']
+        ...         if 'crowdIndex' in imgid2info[ann['image_id']]:
+        ...             groundtruth['crowd_index'] = imgid2info[
+        ...                 ann['image_id']]['crowdIndex']
+        ...         predictions.append(prediction)
+        ...         groundtruths.append(groundtruth)
+        ...     return predictions, groundtruths
+        >>> predictions, groundtruths = _convert_ann_to_pred_and_gt(ann_file)


This example is too long and it doesn't give a clear input format. Should write a plain dict as input.

C1rN09 · 2023-02-13T07:38:39Z

mmeval/metrics/coco_pose.py

+        >>> coco_pose_metric(predictions, groundtruths)
+        loading annotations into memory...
+        Done (t=0.00s)
+        creating index...
+        index created!
+        >>> coco_pose_metric(predictions, groundtruths)
+        Loading and preparing results...
+        DONE (t=0.00s)
+        creating index...
+        index created!


Why are there 2 function calls?

C1rN09 · 2023-02-13T07:40:24Z

mmeval/metrics/coco_pose.py

+        >>> coco_pose_metric = CocoPoseMetric(
+        ...     ann_file=ann_file,
+        ...     dataset_meta=coco_dataset_meta)


There should be 2 example codes to cover the use cases:

ann_file=None, a simple __call__ method

ann_file is not None, call add and compute. Should illustrate what groundtruths should be in this case.

C1rN09 · 2023-02-13T07:47:50Z

mmeval/metrics/coco_pose.py

+        self.format_only = format_only
+        self.outfile_prefix = outfile_prefix
+
+    def add(self, predictions: Sequence[Dict], groundtruths: Sequence[Dict]) -> None:  # type: ignore # yapf: disable # noqa: E501


What about making groundtruths defaults to None so that it's easier to use this metric with ann_file? Creating an 'empty' groundtruths is ambiguous and inconvenient, since we require it has the same length as predictions

C1rN09 · 2023-02-13T07:51:26Z

mmeval/metrics/coco_pose.py

+            predictions (Sequence[dict]): A sequence of dict. Each dict
+                representing a pose estimation result for an instance, with
+                the following keys:
+                    - 'id'(int): The id of the instance


Suggested change

- 'id'(int): The id of the instance

- 'id' (int): The id of the instance

C1rN09 · 2023-02-13T08:00:17Z

mmeval/metrics/coco_pose.py

+
+        info = dict(
+            date_created=str(datetime.datetime.now()),
+            description='Coco json file converted by mmpose CocoMetric.')


Suggested change

description='Coco json file converted by mmpose CocoMetric.')

description='Coco json file converted by mmeval CocoMetric.')

C1rN09 · 2023-02-13T08:09:07Z

mmeval/metrics/coco_pose.py

+                instance['keypoints'] = np.concatenate([
+                    instance['keypoints'], instance['keypoint_scores'][:, None]
+                ],
+                                                       axis=-1)


The indent is strange

C1rN09 · 2023-02-13T08:30:54Z

mmeval/metrics/coco_pose.py

+        >>> coco_pose_metric = CocoPoseMetric(
+        ...     ann_file=ann_file,
+        ...     dataset_meta=coco_dataset_meta)


If dataset_meta is required in this metric, I think it should be explained in docstring

C1rN09 · 2023-02-13T08:37:17Z

mmeval/metrics/coco_pose.py

+
+        Args:
+            kpts (Dict[int, list]): keypoint prediction results. The keys are
+                '`img_id`' and the values are list that may contain


Suggested change

'`img_id`' and the values are list that may contain

``'img_id'`` and the values are list that may contain

C1rN09 · 2023-02-13T08:42:10Z

mmeval/metrics/utils/nms.py

+# ------------------------------------------------------------------------------
+# Adapted from https://github.com/leoxiaobin/deep-high-resolution-net.pytorch
+# Original licence: Copyright (c) Microsoft, under the MIT License.
+# ------------------------------------------------------------------------------
+


Maybe this file should be moved to metrics/_vendor? What do you think @zhouzaida

C1rN09 · 2023-02-24T08:12:23Z

Close this because duplicated with #97

[Feat] Add CocoPoseMetric

215745a

liqikai9 requested review from ly015 and ice-tong October 28, 2022 07:25

C1rN09 reviewed Feb 13, 2023

View reviewed changes

C1rN09 closed this Feb 24, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Feat] Add COCOPoseMetric #35

[Feat] Add COCOPoseMetric #35

liqikai9 commented Oct 28, 2022 •

edited

Loading

CLAassistant commented Dec 13, 2022 •

edited

Loading

C1rN09 left a comment •

edited

Loading

C1rN09 Feb 13, 2023

C1rN09 Feb 13, 2023

C1rN09 Feb 13, 2023 •

edited

Loading

C1rN09 Feb 13, 2023

C1rN09 Feb 13, 2023

C1rN09 Feb 13, 2023

C1rN09 Feb 13, 2023

C1rN09 Feb 13, 2023

C1rN09 Feb 13, 2023

C1rN09 Feb 13, 2023

C1rN09 commented Feb 24, 2023

	- 'id'(int): The id of the instance
	- 'id' (int): The id of the instance

	description='Coco json file converted by mmpose CocoMetric.')
	description='Coco json file converted by mmeval CocoMetric.')

	'`img_id`' and the values are list that may contain
	``'img_id'`` and the values are list that may contain

[Feat] Add COCOPoseMetric #35

[Feat] Add COCOPoseMetric #35

Conversation

liqikai9 commented Oct 28, 2022 • edited Loading

Motivation

Modification

BC-breaking (Optional)

Use cases (Optional)

Checklist

CLAassistant commented Dec 13, 2022 • edited Loading

C1rN09 left a comment • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

C1rN09 Feb 13, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

C1rN09 commented Feb 24, 2023

liqikai9 commented Oct 28, 2022 •

edited

Loading

CLAassistant commented Dec 13, 2022 •

edited

Loading

C1rN09 left a comment •

edited

Loading

C1rN09 Feb 13, 2023 •

edited

Loading