Scale invariant feature
detectors often find stable scales in only a few image pixels. Consequently, methods
for feature matching typically choose one of two extreme options: matching a sparse
set of scale invariant features, or dense matching using arbitrary scales. In this
paper, we turn our attention to the overwhelming majority of pixels, those where
stable scales are not found by standard techniques. We ask, is scale-selection necessary
for these pixels, when dense, scale-invariant matching is required and if so, how
can it be achieved? We make the following contributions: (i) We show that features
computed over different scales, even in low-contrast areas, can be different and
selecting a single scale, arbitrarily or otherwise, may lead to poor matches when
the images have different scales. (ii) We show that representing each pixel as a
set of SIFTs, extracted at multiple scales, allows for far better matches than single-scale
descriptors, but at a computational price. Finally, (iii) we demonstrate that each
such set may be accurately represented by a low-dimensional, linear subspace. A
subspace-to-point mapping may further be used to produce a novel descriptor representation,
the Scale-Less SIFT (SLS), as an alternative to single-scale descriptors. These
claims are verified by quantitative and qualitative tests, demonstrating significant
improvements over existing methods.
Reference (Extended journal version):
Tal Hassner, Shay Filosof,
Viki Mayzels, and Lihi Zelnik-Manor, SIFTing through Scales,
IEEE Trans. on Pattern Analysis and Machine Intelligence (TPAMI), 39(7): 1431-1443 (2017) (PDF
Reference (conference paper):
Tal Hassner, Viki Mayzels, and Lihi
Zelnik-Manor, On SIFTs and their Scales,
IEEE Conf. on Computer Vision
and Pattern Recognition (CVPR), Rhode Island, June 2012 (PDF
New! (13 June 2016)
Extended journal version accepted to IEEE-TPAMI. See above for reference
Followup project on
dense correspondences between scenes and scales
has been posted, including
code and related documents.
Slides from the ICCV'13 tutorial on Dense Image Correspondences for Computer Vision,
Sydney Australia, Dec. 2013. (PDF
Left input image
Right input image
Warp with DSIFT
Warp+crop with our SLS
Dense matches of different objects in different scales.
Left and Right input images. Bottom: Left image warped onto Right using the recovered flows: Using DSIFT
(bottom left) and our Scale-Less SIFT (SLS) descriptor (bottom right), cropped to
the region of confident matches and overlaid on the Right image. DSIFT fails to
capture the scale differences and produces an output in the same scale as the input.
Our SLS descriptor captures scale changes at each pixel: the output produced by
using SLS has the appearance of the Left image in the scale and position of the
Right. Please see text for further details.
MATLAB code for computing dense Scale-Less SIFT descriptors (SLS) is now available
for download here. The distribution contains also a
demo script for establishing dense correspondences between two images of different
scenes in different scales using the SLS descriptors.
Installing and running
The code uses the vlfeat distribution to compute
SIFT descriptors and the SIFT-Flow
release for computing dense correspondences. Please follow these instructions to
set these up before running our code.
Download the following software to your computer and modify the following lines
in demo.m and compute_dense_flow.m
The vl_feat code from http://www.vlfeat.org/
In the line "addpath vlfeat-0.9.14/toolbox" - replace "vlfeat-0.9.14"
with the name of the folder containing the vl_feat code.
The Sift-flow code from http://people.csail.mit.edu/celiu/ECCV2008/
In the lines "addpath release" and "addpath release/mex" - replace
"release" with the name of the folder containing the SIFT-flow code.
To run the algorithm on an example image pair, included in the distribution, simply
run the script demo.m. To use your own images or to modify parameters, see their
Alternatively, to run the demo as a function, you can call:
[flo,wrp,clrflo] = compute_dense_flow(Im_Source, Im_Target)
wrp provides warped output images such as those shown in the paper.
For more information, see
Please note that this code was not designed for optimal running times or memory
usage. It may therefore require quite a lot of time to produce multiple SIFTs for
each pixel and compute their subspaces. The same is true for the memory required
for the descriptors at each pixel.
our paper if you use this code for your research.
11-June-2012, Version 1.1: sls_distribution.1.1.zip
The code now includes an option to compute SLS descriptors with PCA reduced SIFTs.
This produces SLS descriptors which are an order of magnitude smaller than those
used in the paper, while still providing high quality dense flows. A more detailed
description of this addition will be described in an upcoming paper.
See the help for extract_scalelessdescs for
more information on the new usage options.
31-May-2012, Version 1.0: sls_distribution.1.0.zip
Contact and questions
For questions or concerns, please write email@example.com
Copyright and disclaimer
Copyright 2012, Tal Hassner, Viki Mayzels, Shay Filosof, and Lihi Zelnik-Manor
The SOFTWARE ("extract_scalelessdescs", "compute_dense_flow"
and "demo.m") is provided "as is", without any guarantee made
as to its suitability or fitness for any particular use. It may contain bugs, so
use of this tool is at your own risk. We take no responsibility for any damage that
may unintentionally be caused through its use.
ver 1.1, 11-June-2012