Implementing custom Feature Detector in the Image Stitching pipeline

[Faris-Faiz]

Introduction to the issue

Hi! I'm writing this because I want to use a more in-depth feature detector since this library is using built-in OpenCV detectors like ORB, AKAZE, BRISK and SIFT. I've tried these detectors and have found that they perform poorly in detecting features of close-up images of a wall (as expected).

However, I find that using LoFTR (LoFTR: Detector-Free Local Feature Matching with Transformers) achieves what I want. However, it's under the Kornia library. I find that Kornia's built-in image stitching pipeline that uses LoFTR is quite poor in comparison to this library's image stitching output. The seams in between images are very much visible in comparison to the panorama images formed from this library. You can take a look here at what the panorama image would look like if using the Kornia library.

I've researched for a few days now, and have come with no output unfortunately. I believe I've hit a dead-end in this and finally resort to reaching out to this community in hopes of you bright-minded individuals might being able to give some ideas on my issue.

What I've tried

Religiously going back and forth on the Jupyter Notebook this library has provided, I've identified that I just need to make sure before passing into the FeatureMatcher() method, where you pass the 'features' variable, which is a list of cv2.detail.ImageFeatures in the list, corresponding to the number of images that had been passed earlier into the FeatureDetector() method.

Hence, I intuitively thought, maybe I should match the data type to pass into the FeatureMatcher() method, which is basically converting the output from LoFTR (a dictionary of Tensors) into the required data type before passing into the FeatureMatcher() variable.

Here's the progress so far in a Kaggle Notebook: https://www.kaggle.com/code/u2000421student/loftr-to-openstitching-progress

I've figured out that the cv2.detail.ImageFeatures data type requires information such as the keypoints, descriptors, img_idx (image index) as well as img_size (image size).

I was successful in adding the keypoints from the output of LoFTR (in the correspondences variable, referring to the Kaggle Notebook earlier).

However, whenever trying to pass in the descriptors, it always will have a conversion error. Further research and a couple of ChatGPT prompts later, I found out that apparently LoFTR apparently doesn't output the descriptors because "LoFTR (Local Feature TRansformer) is designed to perform feature matching without explicitly computing traditional descriptors. Instead, it leverages a transformer-based architecture to establish correspondences between images directly. This approach contrasts with traditional methods that follow a sequential pipeline of detection, description, and matching."

I'd like your input on where to go in this project. Any input is extremely valuable! Thanks for taking the time to read this.

[lukasalexanderweber]

Why don't you use LoFTR for the whole Image registration process. You should get some transformation parameters, can create camera objects and perform the Image composition with OpenStitching.

[Faris-Faiz]

This is something I never thought of, thanks so much for your valuable input!

So basically I have to calculate the homographies and pass it into the CameraEstimator object, correct?

I've tried a bit but with a bit of issues but I think I can try to get it to work and report back in a bit

[lukasalexanderweber]

You can start here (the cameras are basically the homographies)

[Faris-Faiz]

Hey, I've spent a few days trying to research based on what you've mentioned, but still came to a standstill. Just want to clarify, do you think it is possible to pass in the LoFTR outputs into the Image Stitching Pipeline?

As referenced in the notebook tutorial for this library, I was under the assumption it's possible to pass it into the Camera Estimator class, and since we're replacing the entire Image Registration process, I can still use this library from the Camera Estimator class onwards

At the moment, I am trying to pass it into the Camera Estimator class to no avail, but I'd just like to double confirm with you in this matter. I'll attach the code so far once I confirm this, cause I wouldn't want to show code that's not achieving what you meant 😅

Also as a side note, I've tried looking into OpenCV's own implementation of image stitching, as well as the detailed_stitching.py file by OpenCV you referenced, which is a great resource I just found today that would hopefully help me in my endeavor

[lukasalexanderweber]

pass in the LoFTR outputs

what are the outputs?