Enable RANSAC in triangulation #292

travisdriver · 2021-09-13T14:55:31Z

Enabled RANSAC in the triangulation module.

…values

travisdriver · 2021-09-13T14:58:03Z

The results on Skydio-32 look promising (will post other results once the benchmarking finishes):

Before

Metric name	Value
number_cameras	32
number_tracks_unfiltered	5724
mean_3d_track_lengths_unfiltered	2.67313
mean_reprojection_errors_unfiltered_px	10.9595
mean_rotation_error_angle_deg	7.326
mean_translation_error_distance	0.651919
mean_translation_angle_error_deg	11.4042
number_tracks_filtered	1256
mean_3d_track_lengths_filtered	2.08758
mean_reprojection_errors_filtered_px	1.39266

After

Metric name	Value
number_cameras	32
number_tracks_unfiltered	3482
mean_3d_track_lengths_unfiltered	2.04451
mean_reprojection_errors_unfiltered_px	0.669101
mean_rotation_error_angle_deg	6.7561
mean_translation_error_distance	0.920434
mean_translation_angle_error_deg	12.0292
number_tracks_filtered	3445
mean_3d_track_lengths_filtered	2.04325
mean_reprojection_errors_filtered_px	0.641061

johnwlambert · 2021-09-13T15:11:33Z

Thanks for the PR, Travis. Cool to see.

I'm curious about why the average camera distance from GT goes from 0.65 to 0.9. Could you try sharing what the visualizations look like before and after the change?

johnwlambert · 2021-09-13T15:24:20Z

Travis, is there any way you could share some examples of how RANSAC is doing? Like for some visualizations of tracks (using our DA track viz script that dumps vertically stacked patches), could we see which ones RANSAC is labeling as outliers? Do the outliers look correct in practice? I think having GT metrics here on 3d GT will help too : - )

travisdriver · 2021-09-13T15:30:19Z

Travis, is there any way you could share some examples of how RANSAC is doing? Like for some visualizations of tracks (using our DA track viz script that dumps vertically stacked patches), could we see which ones RANSAC is labeling as outliers? Do the outliers look correct in practice? I think having GT metrics here on 3d GT will help too : - )

Which GT metrics are you referring to?

johnwlambert · 2021-09-13T15:33:04Z

Travis, is there any way you could share some examples of how RANSAC is doing? Like for some visualizations of tracks (using our DA track viz script that dumps vertically stacked patches), could we see which ones RANSAC is labeling as outliers? Do the outliers look correct in practice? I think having GT metrics here on 3d GT will help too : - )

Which GT metrics are you referring to?

Oh I meant the track precision/recall from the 3d model -- I think we might need a bit more DA metrics before merging this DA PR

travisdriver · 2021-10-08T20:57:20Z

The performance with RANSAC is relatively positive so far. Will continue to update as I run tests.

Skydio-32 (deep_front_end, skydio-32, 32, 760, true)

Metric name	w/o RANSAC	w/ RANSAC
number_cameras	30	30
number_tracks_unfiltered	8532	8577
mean_3d_track_lengths_unfiltered	2.97703	2.69255
mean_reprojection_errors_unfiltered_px	3.37457	1.18834
mean_rotation_error_angle_deg	0.175664	0.227616
mean_translation_error_distance	0.0192357	0.0256501
mean_translation_angle_error_deg	0.396858	0.477618
Filtered tracks triangulated with GT cams: SUCCESS	6403	7201
Filtered tracks triangulated with GT cams: EXCEEDS_REPROJ_THRESH	106	109
Filtered tracks triangulated with GT cams: CHEIRALITY_FAILURE	12	14
number_tracks_filtered	6521	7324
mean_3d_track_lengths_filtered	2.50437	2.45508
mean_reprojection_errors_filtered_px	0.809112	0.797346

RC3 (deep_front_end, 2011205_rc3, 65, 1024, true)

Metric name	w/o RANSAC	w/ RANSAC
number_cameras	22	13
number_tracks_unfiltered	2664	2972
mean_3d_track_lengths_unfiltered	2.00188	2
mean_reprojection_errors_unfiltered_px	0.422914	0.37011
mean_rotation_error_angle_deg	8.10497	4.86258
mean_translation_error_distance	2553.99	1217.21
mean_translation_angle_error_deg	36.4078	27.7624
Filtered tracks triangulated with GT cams: SUCCESS	2663	2972
number_tracks_filtered	2664	2972
mean_3d_track_lengths_filtered	2.00188	2
mean_reprojection_errors_filtered_px	0.422914	0.37011

johnwlambert · 2021-11-28T00:22:11Z

How does the dashboard look for this PR Travis? Could you share the file if you have created it?

travisdriver · 2021-11-28T19:51:44Z

Here are the results for 100 RANSAC iterations with a 5 pixel threshold.

I think door-12 may be so "converged" that small perturbations in the results make it look like it's doing really bad in the dashboard. For example, the ~1000% increase in translation error is a result of it going from 0.002 to 0.02 (keep in mind the ground truth is the COLMAP solution and not the actual). Nevertheless, the results are still not great.

I still think there's something peculiar going on. I would have at least expected the number of unfiltered tracks to increase across the board (since were no longer rejecting entire tracks), but this is not the case. Also, the mean track length seems to have decreased substantially in most cases, suggesting that RANSAC is rejecting a significant amount of track correspondences (maybe we need to increase the pixel threshold).

I will investigate and debug.

…nsac-in-triangulation

travisdriver · 2021-11-28T20:05:01Z

Actually I guess I would expect the number of unfiltered tracks to decrease since the default reprojection error threshold is 100 pixels on master...

travisdriver · 2021-11-30T21:17:49Z

Here are the results with a RANSAC threshold of 100 pixels.

My hypothesis is that the poses from the averaging module are not accurate enough to perform RANSAC at more reasonable thresholds (this is supported by the huge performance increase afforded by #225).

…nsac-in-triangulation

johnwlambert · 2022-01-04T20:31:18Z

Thanks for the PR, Travis. I'm taking a look at the report now:

visual_comparison_dashboard 3.html.zip

Just trying to figure out which changes are sort of random changes vs. deterministic ones. I think you have a better sense from me from having run the CI a few times on this branch.

Maybe if we fix the determinism bug first it will be a bit clearer. What are your thoughts?

johnwlambert · 2022-01-17T22:48:17Z

Hi @travisdriver the PR looks good. Do you have a sense for the new runtime? I'm curious if it's 1% slower, 10% slower, or 100%+ slower with the new RANSAC loop.

travisdriver · 2022-01-17T23:00:14Z

Hi @travisdriver the PR looks good. Do you have a sense for the new runtime? I'm curious if it's 1% slower, 10% slower, or 100%+ slower with the new RANSAC loop.

I'm not sure what you're referring to; there's no increase in runtime according to the CI.

johnwlambert · 2022-01-17T23:03:07Z

Hi @travisdriver the PR looks good. Do you have a sense for the new runtime? I'm curious if it's 1% slower, 10% slower, or 100%+ slower with the new RANSAC loop.

I'm not sure what you're referring to; there's no increase in runtime according to the CI.

I was thinking since we were adding a new inner Python for-loop for our 2-view geometry estimation, so that could slow things down considerably. But I guess since we limit to just 100 hypotheses the effect on runtime is neglible?

travisdriver · 2022-01-17T23:04:52Z

Hi @travisdriver the PR looks good. Do you have a sense for the new runtime? I'm curious if it's 1% slower, 10% slower, or 100%+ slower with the new RANSAC loop.

I'm not sure what you're referring to; there's no increase in runtime according to the CI.

I was thinking since we were adding a new inner Python for-loop for our 2-view geometry estimation, so that could slow things down considerably. But I guess since we limit to just 100 hypotheses the effect on runtime is neglible?

Yes, triangulation is a relatively inexpensive operation.

travisdriver · 2022-01-18T19:39:08Z

@johnwlambert What else needs to be done in order to merge this PR?

johnwlambert · 2022-01-18T23:47:45Z

Hi Travis, thanks for your patience here. I'm a bit confused about the performance on skydio-8 and skydio-32. I triggered the CI one more time to see if those regressions were random. @akshay-krishnan @ayushbaid could you all take a look? I'm fine with landing the new config just for astronet right away, though.
visual_comparison_dashboard.html.zip

I saw that the runtime went from 58 min to 2 hours 20 min just for one benchmark (deep front end RC3). I'm running again to see if that's repeatable.

johnwlambert · 2022-01-19T14:35:36Z

I checked the dashboard for the second run, and still seeing some confusing things. Hopefully if we land the repeatability stuff first, it will be more clear for us

travisdriver · 2022-01-19T14:36:51Z

I saw that the runtime went from 58 min to 2 hours 20 min just for one benchmark (deep front end RC3). I'm running again to see if that's repeatable.

This is from the most recent commit on master; the benchmark runtime fluctuates a lot for some reason, but there are plenty of instances other than this branch where is takes >2 hours.

johnwlambert · 2022-01-19T14:54:54Z

I saw that the runtime went from 58 min to 2 hours 20 min just for one benchmark (deep front end RC3). I'm running again to see if that's repeatable.

This is from the most recent commit on master; the benchmark runtime fluctuates a lot for some reason, but there are plenty of instances other than this branch where is takes >2 hours.

I see, cool. Didn't realize runtime was fluctuating that much too. I wonder if the hardware specs are different between runs on the CI

travisdriver · 2022-01-22T20:04:49Z

I checked the dashboard for the second run, and still seeing some confusing things. Hopefully if we land the repeatability stuff first, it will be more clear for us

I don't think an increase in angular error from .3 to .8 should completely hold the PR, especially when there are significant improvements across the board. The repeatability issues effect all PRs, but it seems to only be holding up this one...

I checked the dashboard for the second run, and still seeing some confusing things. Hopefully if we land the repeatability stuff first, it will be more clear for us

Thanks for looking at the benchmarks. What's confusing from the output? A .5 degree increase in the mean rotation angle error seems insignificant compared to the other improvements.

johnwlambert · 2022-01-23T07:42:25Z

gtsfm/data_association/point3d_initializer.py

+        dyn_num_hypotheses = int(
+            np.log(1 - self.confidence)
+            / np.log(1 - self.min_inlier_ratio ** NUM_SAMPLES_PER_RANSAC_HYPOTHESIS)
+            * self.dyn_num_hypotheses_multiplier


Hi Travis, could you add a reference for this dynamic multiplier? If COLMAP is using it, do you mind adding an inline comment w/ URL to the COLMAP source where they do it?

COLMAP uses it, but all it does it multiply the number of dynamic hypotheses by a constant so I'm indifferent about whether it warrants a reference

gtsfm/data_association/point3d_initializer.py

akshay-krishnan · 2022-01-24T02:04:28Z

gtsfm/data_association/point3d_initializer.py

+    mode: TriangulationParam
+
+    # RANSAC parameters
+    min_inlier_ratio: float = 0.1


isnt that a really small ratio? do we have a min_num_inliers too? just to be sure this does not select tracks with one or two inliers?

Yes, a smaller ratio will result in a larger number of dynamic hypotheses.

Keep in mind that the measurements are the cameras (with the corresponding keypoint), so a track with one inlier would result in a underdetermined problem (which we check for and triangulation would fail anyways). Two inliers is allowed, but tracks are discarded according to min_track_length.

Okay, I wonder if a higher min inlier ratio produces better hypothesis? It would definitely save some compute.

gtsfm/data_association/point3d_initializer.py

Rename TriangulationParam to TriangulationSamplingMode

…nsac-in-triangulation

ayushbaid

LGTM except few minor comments.

Do you want to wait for the repeatability issues in 1dsfm to land before using it in astronet, or do you want to start using it regardless?

ayushbaid · 2022-01-25T05:13:09Z

gtsfm/configs/deep_front_end_astronet.yaml

@@ -0,0 +1,68 @@
+# Relaxes triangulation RANSAC threshold to 100 (10x threshold of deep_front_end.yaml)


Can we name it something which mentions this attribute value being high instead of tagging it with a database?

I agree, I think this would improve clarity

AstroNet is the only dataset that uses the config by default, so it seems fitting to name it accordingly.

ayushbaid · 2022-01-25T05:23:12Z

gtsfm/configs/scene_optimizer_unit_test_config.yaml

+      triangulation_options:
+        _target_: gtsfm.data_association.point3d_initializer.TriangulationOptions
+        reproj_error_threshold: 10
+        mode:
+          _target_: gtsfm.data_association.point3d_initializer.TriangulationSamplingMode
+          value: 0 # 0 corresponds to NO_RANSAC
+        max_num_hypotheses: 20


Why change to ransac in the unit test?

I didn't quite follow here Ayush -- I believe we were using NO_RANSAC before, and the test is still using NO_RANSAC now.

ayushbaid · 2022-01-25T05:29:37Z

gtsfm/data_association/point3d_initializer.py

+        mode: triangulation mode, which dictates whether or not to use robust estimation.
+        min_inlier_ratio: a priori assumed minimum probability that a point is an inlier.
+        confidence: desired confidence that at least one hypothesis is outlier free.
+        dyn_num_hypotheses_multiplier: multiplication factor for dynamically computed hyptheses based on confidence.


Can you elaborate what this is in the PR comments?

Also good to have an example computation in the docstring for the number of hypothesis.

@travisdriver we could add the following math:

The RANSAC module defaults to 2749 iterations, computed as np.log(1-0.9999) / np.log( 1 - 0.1 **2) * 3 = 2749.3

I added this to the docstring

ayushbaid · 2022-01-25T05:40:41Z

gtsfm/data_association/point3d_initializer.py

-    mode: TriangulationParam
-    reproj_error_thresh: float
-    num_ransac_hypotheses: Optional[int] = None
+    options: TriangulationOptions


can we rename to triangulation_options to prevent ambiguity?

I feel like maybe that would be too wordy/long. Just my two cents.

I think just options is sufficient since there aren't any similar attributes.

johnwlambert

LGTM

enabled RANSAC in triangulation, tested reprojection error threshold …

6089133

…values

travisdriver requested review from ayushbaid and akshay-krishnan September 13, 2021 14:58

travisdriver added 3 commits October 8, 2021 13:42

fix merge conflicts

acd4fdd

remove changes in config

e04bbca

remove changes in config

beaa85d

travisdriver added 3 commits November 27, 2021 11:52

Fixed merge conflicts

7febac4

Changed RANSAC threshold to 3 pixels

742d250

Removed dashboard file

998c446

Merge branch 'master' of https://github.com/borglab/gtsfm into use-ra…

c74954d

…nsac-in-triangulation

travisdriver added 2 commits November 28, 2021 15:05

Reject RANSAC support if <3 measurements

613fecb

Increase RANSAC threshold to 10 pixels

f194248

travisdriver marked this pull request as draft November 29, 2021 15:35

travisdriver changed the title ~~[WIP] Enabled RANSAC in triangulation~~ Enable RANSAC in triangulation Nov 29, 2021

Increase pixel threshold to 100

1a94590

travisdriver and others added 5 commits December 7, 2021 17:47

Merge branch 'master' of https://github.com/borglab/gtsfm into use-ra…

d4fe8c7

…nsac-in-triangulation

Decrease RANSAC threshold to 10 pixels

300e1b4

Decrease RANSAC threshold to 10 pixels

30956c1

Merge branch 'master' of https://github.com/borglab/gtsfm into use-ra…

1e8c9fd

…nsac-in-triangulation

Merge branch 'master' of https://github.com/borglab/gtsfm into use-ra…

9826848

…nsac-in-triangulation

travisdriver added 2 commits January 3, 2022 16:43

Fixed bugs

7f61b42

Fix astronet config

3fe2a08

travisdriver requested a review from johnwlambert January 4, 2022 01:53

johnwlambert reviewed Jan 23, 2022

View reviewed changes

akshay-krishnan reviewed Jan 24, 2022

View reviewed changes

gtsfm/data_association/point3d_initializer.py Outdated Show resolved Hide resolved

johnwlambert reviewed Jan 24, 2022

View reviewed changes

gtsfm/data_association/point3d_initializer.py Outdated Show resolved Hide resolved

John Lambert and others added 4 commits January 24, 2022 23:18

address Akshay comments

991e693

rename param

7de6481

Merge pull request #433 from borglab/rename-triangulation-param-class

6b15560

Rename TriangulationParam to TriangulationSamplingMode

Merge branch 'master' of https://github.com/borglab/gtsfm into use-ra…

c0342a0

…nsac-in-triangulation

akshay-krishnan approved these changes Jan 25, 2022

View reviewed changes

ayushbaid approved these changes Jan 25, 2022

View reviewed changes

johnwlambert added 3 commits January 25, 2022 11:03

add ViewGraphEstimator to new config

f4e3dd7

address Ayush comments

b3e3e78

use str Enum inheritance for config readability

ba1af74

johnwlambert approved these changes Jan 25, 2022

View reviewed changes

johnwlambert merged commit 185cb97 into master Jan 25, 2022

johnwlambert deleted the use-ransac-in-triangulation branch January 25, 2022 18:18

		@@ -0,0 +1,68 @@
		# Relaxes triangulation RANSAC threshold to 100 (10x threshold of deep_front_end.yaml)

Enable RANSAC in triangulation #292

Enable RANSAC in triangulation #292

Conversation

travisdriver commented Sep 13, 2021 • edited Loading

travisdriver commented Sep 13, 2021

Before

After

johnwlambert commented Sep 13, 2021

johnwlambert commented Sep 13, 2021

travisdriver commented Sep 13, 2021

johnwlambert commented Sep 13, 2021

travisdriver commented Oct 8, 2021 • edited Loading

Skydio-32 (deep_front_end, skydio-32, 32, 760, true)

RC3 (deep_front_end, 2011205_rc3, 65, 1024, true)

johnwlambert commented Nov 28, 2021

travisdriver commented Nov 28, 2021 • edited Loading

travisdriver commented Nov 28, 2021

travisdriver commented Nov 30, 2021

johnwlambert commented Jan 4, 2022

johnwlambert commented Jan 17, 2022

travisdriver commented Jan 17, 2022

johnwlambert commented Jan 17, 2022

travisdriver commented Jan 17, 2022

travisdriver commented Jan 18, 2022

johnwlambert commented Jan 18, 2022 • edited Loading

johnwlambert commented Jan 19, 2022

travisdriver commented Jan 19, 2022

johnwlambert commented Jan 19, 2022

travisdriver commented Jan 22, 2022

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ayushbaid left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

johnwlambert left a comment

Choose a reason for hiding this comment

travisdriver commented Sep 13, 2021 •

edited

Loading

travisdriver commented Oct 8, 2021 •

edited

Loading

travisdriver commented Nov 28, 2021 •

edited

Loading

johnwlambert commented Jan 18, 2022 •

edited

Loading