Failed on warp()

[cyhsu]

Hi again.

I was trying to following the tutorials but using my own images. However, when I reach to the following steps

from stitching.images import Images
img_handler = Images.of(images[10:13])

final_sizes = img_handler.get_scaled_img_sizes(Images.Resolution.FINAL)
camera_aspect = img_handler.get_ratio(Images.Resolution.MEDIUM, Images.Resolution.FINAL)

warped_final_imgs = list(warper.warp_images(final_imgs, cameras, camera_aspect))
warped_final_masks = list(warper.create_and_warp_masks(final_sizes, cameras, camera_aspect))
final_corners, final_sizes = warper.warp_rois(final_sizes, cameras, camera_aspect)

I see an error message saying

---------------------------------------------------------------------------
error                                     Traceback (most recent call last)
Cell In[15], line 4
      1 final_sizes = img_handler.get_scaled_img_sizes(Images.Resolution.FINAL)
      2 camera_aspect = img_handler.get_ratio(Images.Resolution.MEDIUM, Images.Resolution.FINAL)
----> 4 warped_final_imgs = list(warper.warp_images(final_imgs, cameras, camera_aspect))
      5 warped_final_masks = list(warper.create_and_warp_masks(final_sizes, cameras, camera_aspect))
      6 final_corners, final_sizes = warper.warp_rois(final_sizes, cameras, camera_aspect)

File [/opt/venv/lib/python3.9/site-packages/stitching/warper.py:41](https://vscode-remote+ssh-002dremote-002bcyhsu-002dwork-002ddev-002eokteto.vscode-resource.vscode-cdn.net/opt/venv/lib/python3.9/site-packages/stitching/warper.py:41), in Warper.warp_images(self, imgs, cameras, aspect)
     39 def warp_images(self, imgs, cameras, aspect=1):
     40     for img, camera in zip(imgs, cameras):
---> 41         yield self.warp_image(img, camera, aspect)

File [/opt/venv/lib/python3.9/site-packages/stitching/warper.py:45](https://vscode-remote+ssh-002dremote-002bcyhsu-002dwork-002ddev-002eokteto.vscode-resource.vscode-cdn.net/opt/venv/lib/python3.9/site-packages/stitching/warper.py:45), in Warper.warp_image(self, img, camera, aspect)
     43 def warp_image(self, img, camera, aspect=1):
     44     warper = cv.PyRotationWarper(self.warper_type, self.scale * aspect)
---> 45     _, warped_image = warper.warp(
     46         img,
     47         Warper.get_K(camera, aspect),
     48         camera.R,
     49         cv.INTER_LINEAR,
     50         cv.BORDER_REFLECT,
     51     )
     52     return warped_image

error: OpenCV(4.8.0) [/io/opencv/modules/imgproc/src/imgwarp.cpp:1730](https://vscode-remote+ssh-002dremote-002bcyhsu-002dwork-002ddev-002eokteto.vscode-resource.vscode-cdn.net/io/opencv/modules/imgproc/src/imgwarp.cpp:1730): error: (-215:Assertion failed) dst.cols < SHRT_MAX && dst.rows < SHRT_MAX && src.cols < SHRT_MAX && src.rows < SHRT_MAX in function 'remap'

I am not really familiar with OpenCV and also the stitching package. Could someone here help me to figure out what the issue is, please?

[lukasalexanderweber]

You should either share your input images or intermediate results so one could help you

[cyhsu]

Thanks for the responding.
Here are two figures I do the test.
For the code, I only do copy & paste from tutorial and modified the image path.

[lukasalexanderweber]

Have you tried the AffineStitcher? Your Images have no homographic relation

[cyhsu]

AffineStitcher works well. Thank you @lukasalexanderweber !!!!
Could you please explain it little in detail why the regular stitch() is not working?

[lukasalexanderweber]

On homography, all images are taken from the same spot:

With affine stitching the images are just "merged" into a joint image plane. The best use case are scanners or perfectly aligned drone images:

You could also try homography settings and use the plane warper

[cyhsu]

I will definitely to give this a try today!! Thank you!!

[cyhsu]

Hi Luka,

I have use the one you suggested for AffineStitch(), here is the code

settings = {
    "detector": "sift",
    "confidence_threshold": 1.0, 
    "crop": False,    
    'estimator': 'affine', 
    'wave_correct_kind': 'no', 
    'matcher_type': 'affine', 
    'adjuster': 'affine', 
    'warper_type': 'affine', 
    'compensator': 'no'
}

stitcher = AffineStitcher(**settings)
panorama = stitcher.stitch(images[30:41])

and then I got the error message.

---------------------------------------------------------------------------
StitchingError                            Traceback (most recent call last)
Cell In[95], line 29
     26 stitcher = AffineStitcher(**settings)
     27 # panorama01 = stitcher.stitch(images[10:21])
     28 # panorama02 = stitcher.stitch(images[20:31])
---> 29 panorama03 = stitcher.stitch(images[30:34])
     30 # panorama04 = stitcher.stitch(images[40:51])
     31 # panorama05 = stitcher.stitch(images[50:61])
     32 # panorama06 = stitcher.stitch(images[60:71])

File [~/miniconda3/lib/python3.10/site-packages/stitching/stitcher.py:100](https://file+.vscode-resource.vscode-cdn.net/Users/cyhsu/dev/gitlab/cyhsu-work/notebooks/projects/on-farm-uav/~/miniconda3/lib/python3.10/site-packages/stitching/stitcher.py:100), in Stitcher.stitch(self, images, feature_masks)
     98 features = self.find_features(imgs, feature_masks)
     99 matches = self.match_features(features)
--> 100 imgs, features, matches = self.subset(imgs, features, matches)
    101 cameras = self.estimate_camera_parameters(features, matches)
    102 cameras = self.refine_camera_parameters(features, matches, cameras)

File [~/miniconda3/lib/python3.10/site-packages/stitching/stitcher.py:146](https://file+.vscode-resource.vscode-cdn.net/Users/cyhsu/dev/gitlab/cyhsu-work/notebooks/projects/on-farm-uav/~/miniconda3/lib/python3.10/site-packages/stitching/stitcher.py:146), in Stitcher.subset(self, imgs, features, matches)
    145 def subset(self, imgs, features, matches):
--> 146     indices = self.subsetter.subset(self.images.names, features, matches)
    147     imgs = Subsetter.subset_list(imgs, indices)
    148     features = Subsetter.subset_list(features, indices)

File [~/miniconda3/lib/python3.10/site-packages/stitching/subsetter.py:29](https://file+.vscode-resource.vscode-cdn.net/Users/cyhsu/dev/gitlab/cyhsu-work/notebooks/projects/on-farm-uav/~/miniconda3/lib/python3.10/site-packages/stitching/subsetter.py:29), in Subsetter.subset(self, img_names, features, matches)
     27 def subset(self, img_names, features, matches):
     28     self.save_matches_graph_dot_file(img_names, matches)
---> 29     indices = self.get_indices_to_keep(features, matches)
     31     if len(indices) < len(img_names):
     32         warnings.warn(
     33             """Not all images are included in the final panorama.
     34                       If this is not intended, use the 'matches_graph_dot_file'
   (...)
     38             StitchingWarning,
     39         )

File [~/miniconda3/lib/python3.10/site-packages/stitching/subsetter.py:66](https://file+.vscode-resource.vscode-cdn.net/Users/cyhsu/dev/gitlab/cyhsu-work/notebooks/projects/on-farm-uav/~/miniconda3/lib/python3.10/site-packages/stitching/subsetter.py:66), in Subsetter.get_indices_to_keep(self, features, pairwise_matches)
     63 indices = indices.flatten()
     65 if len(indices) < 2:
---> 66     raise StitchingError(
     67         """No match exceeds the given confidence threshold.
     68                          Do your images have enough overlap and common
     69                          features? If yes, you might want to lower the
     70                          'confidence_threshold' or try another
     71                          'detector'."""
     72     )
     74 return indices

StitchingError: No match exceeds the given confidence threshold.
                                 Do your images have enough overlap and common
                                 features? If yes, you might want to lower the
                                 'confidence_threshold' or try another
                                 'detector'.

Now it looks like the confidence_threshold is too high, or I use the wrong detector. To investigate the error, I then use the tutorial and modify the functions to get the matches confidence level, which is like...

#-- image loads
from stitching.images import Images
img_handler = Images.of(images[30:33])
medium_imgs = list(img_handler.resize(Images.Resolution.MEDIUM))
low_imgs = list(img_handler.resize(Images.Resolution.LOW))
final_imgs = list(img_handler.resize(Images.Resolution.FINAL))

#-- feature detector                                 
from stitching.feature_detector import FeatureDetector
finder = FeatureDetector(detector='sift',)
features = [finder.detect_features(img) for img in medium_imgs]

#-- feature matching
from stitching.feature_matcher import FeatureMatcher
matcher = FeatureMatcher(matcher_type='affine')
matches = matcher.match_features(features)

#-- print out the confidence 
abc = matcher.get_confidence_matrix(matches)
print(np.max(abc, axis=1))

It then shows me the value of [1.45683177 1.45683177 1.21316077]

Is it not meaning that when I use "SIFT" detector, I can get confidence level of at least 1.2?

[cyhsu]

This is also showing when I use the Subsetter()

from stitching.subsetter import Subsetter
subsetter = Subsetter()
dot_notation = subsetter.get_matches_graph(img_handler.names, matches)
print(dot_notation)

graph matches_graph{
"DJI_0068.jpg" -- "DJI_0070.jpg"[label="Nm=2147, Ni=950, C=1.45683"];
"DJI_0070.jpg" -- "DJI_0072.jpg"[label="Nm=2111, Ni=778, C=1.21316"];
}

[lukasalexanderweber]

I can't quite follow it should work if the confidences are bigger than 1

[cyhsu]

Did you think I did anything wrong here?