This project attempts to use projective warping and image stitching to seamlessly create 360 degree panoramas.
I was originally inspired to pursue 360 degree panoramic stitching by
my experience with Google PhotoSpheres. These are, as the name implies,
full spheres of photos that one could explore from the inside, after
taking many photos and letting Google stitch them together
(similar to the Google Street View experience).
More Info on Street View/PhotoSpheres: https://www.google.com/maps/streetview/
I set out to implement a similar experience, but as I encountered difficulties, the project changed focus.
In this part of the project, a series of images for the panorama (taken so that the images overlap, and the only change in camera orientation is horizontal panning) are warped into cylindrical/spherical coordinates, and then aligned using RANSAC over SIFT features.
In order to warp the image into cylindrical/spherical coordinates,
we can compute the warped pixel position corresponding to the planar image.
For cylindrical warping we assume that each row of the image is warped into a
circle segment. Thus, the warped coordinates (x',y') from the
normal coordinates (x,y,f) become
x' = s* arctan(x/f)
y' = s* y/( sqrt(x^2+f^2) )
where f is the focal length of the camera in pixels, and s is a scaling factor which I have set equal to f.
Spherical warping is quite similar; the warped coordinates (x',y') from the normal coordinates (x,y,f) become
x' = s* arctan(x/f)
y' = s* arctan( y/( sqrt(x^2+f^2) ) )
NOTE: in order to achieve the focal length in pixels, you have to take a few additional steps. Firstly, we need to know the focal length of the photo, which can be found in the photo's EXIF data. Secondly, we need to know the size of the camera sensor, or CCD width/height. Then f in pixels simply follows the forthcoming formula: f = photo_width*(focal_length/CCD_width).
After all images in the panorama series are warped, alignment becomes
the simple task of translating. To find the correct alignment I used
the RANSAC/SIFT algorithm from Project 5.
The algorithm first finds corresponding points between images, then computes a least-squares solution to the two tx and ty translation parameters.
The alignments I achieved this way were excellent.
Now with all the photos warped and aligned, the next step is to stitch
them into one composite panorama, using a blending technique.
This is the part of the project that I had the most trouble with. I tried many different ways to achieve a smooth panorama and minimize the artifacts in the image, each with varying degrees of success.
Simply combining the images with no blending. The result shows obvious lines separating the individual images of differing exposures.
This blending method averaged the overlapping areas between two photos.
The result shows obvious local errors in the alignment of the images,
drawing attention to the overlapping area. This also messes up the
high frequency features of the overlap. In addition, the exposure difference
between the photos is still highly obvious.
To try to fix this, the next technique I tried attempting to gradually transition between overlapping photos.
This blending method took a weighted average of the overlapping areas between two photos.
This was done by computing a mask for each new photo added, smoothing the mask with a gaussian filter,
and then also taking the inverse mask. These two masks are the weights
for each side of the blending.
The result is quite similar to a normal average. Although the blending differed slightly by the choice of the size of the gaussian smoothing filter, the results were still never very much better.
TThe next technique I tried was a 2-band blending.
This blending method is essentially an extension of the last. The idea
is to feather blend at two separate frequencies in a two-level laplacian
pyramid. Thus, the high-frequencies will be blended quickly while the
low-frequencies will be blended more slowly.
Despite the long hours I put into this, I could not get this form of blending to work with my panorama. Although I've tested my laplacian pyramid blending on outside test images, in the context of image masks, and multi-layer images I could not get it to cooperate. The top image is my result.
In my frustration, I sought to compare the performance of my implementation with a successful one, so I searched online to find a solution by Yan Ke @ THUEE, email@example.com. The bottom image is the result of using their laplacian code. As you can see, even with a correct laplacian pyramid implementation, some 'shadows' are very visible at the photo boundries. This led me to believe that something (in addition to my laplacian pyramid code) was wrong; my best guess was some sort of error in alignment/choice of image mask.
Regardless, by this time I was frustrated with the image blending (which wasn't supposed to take long), and decided to switch blending tactics entirely.
In this blending technique I found the lowest-cost vertical cut through
the overlapping area between photos. This is the same seam finding
technique described in project 2.
After finding the lowest-cost seam, the blend was as simple as sampling from one image if on one side of the seam, and from the other image if on the other side of the seam.
I see this as an improvement over all other blending techniques so far; however, I also saw room for improvement.
This blending technique is similar to the previous one, except instead of
simply sampling against the found seam, I combined it with feathering
to acheive a smoother transition across it.
This is the best blending that I have done.
This is as far as I have come to my original goal of stitching PhotoSpheres. However, below I highlight other difficulties I had, and the next steps to take.
I experienced other complications with a more general smart-cut solution,
and aligning/warping multi-layer spherically-warped.
Smart Cut: at first, I wanted to try a two-dimensional smart-cut: one that both cut across and down the overlapping section between photos. However, the resulting feathered mask wroght similar shadows I encountered in earlier blending techniques (solidifying my thoughts that it was a masking issue). After some time, I came to realize that graph-cut was often cutting along the borders of photos, which may have lead to the dark shadows (as part of the weighted average now pulled from the black pixels next to photo borders). Attempting to solve this, I increased the cost of cutting through a pixel in a thick border around the overlap mask. This however, did not work.
Additionally, my smart-cut system is uni-directional: it only cuts on the left of the image being added, assuming we construct the panorama from left to right. This is a problem, because full spherical panoramas can be constructed in an order, adding photos above, below, to the left, and right of the rest of the panorama. To implement directionality, I would need to first fix the above problem of two-way graph cuts, and then compare the center of the overlap region with the center of the photo-to-be-added, thus giving a direction the the cut and blend.
Spherical Warp/Alignment: another thing that works remarkably well when constructin a panorama from left to right is the image alignment. However, my current implementation of warp does not account for images above or below the first layer of images. To do a full photosphere, that must be fixed.