Evaluation toolkit for 2D image- and 3D structure-based localization

We provide the evaluation toolkit for benchmarking on the San Francisco Landmarks dataset [1,2] and their reference poses presented in our CVPR17 paper [3]. Please send bug reports and suggestions to htaira@ctrl.titech.ac.jp, torii@sc.e.titech.ac.jp .

Quick start

  • Run the following script in MATLAB
  • >> demo_benchmark

It shows all the localization results included in [3] and the result of [1] (Chen NoGPS). The simplest way to compare with [3] is to replace the line 5 in demo_benchmark.m to point your result.

Input format

This toolkit can evaluate both 2D image- and 3D structure-based localization.

If you are working on 2D image-based approaches, e.g. instance level recognition, image retrieval，geometric/spatial verification (re-ranking) and want to evaluate the performance of ranked list (shortlist) of the best matching images, please prepare the results in the 2D image-based method format.

If you are working on 3D structure-based approaches, e.g. image-based localization, 2D3D/3D2D matching, SfM, and want to evaluate the peformance of the pose estimation, please prepare the results in the 3D structure-based method format.

2D image-based method (shortlist)

The shortlist should contain each query image followed by the N best matching database images as follows: 

<N: Number of database images for each query>
<Query name>
<Score> <Database image name>
<Score> <Database image name>
...
<Query name>
<Score> <Database image name>
<Score> <Database image name>
...

3D-structure based method (pose)

The pose format is similar to the shortlist format. It should include the estimated camera pose <Camera pose> in the vectorized form after <Database image name>:

<N: Number of database images for each query>
<Query name>
<Score> <Database image name (optional)> <Camera pose>
<Score> <Database image name (optional)> <Camera pose>
...
<Query name>
<Score> <Database image name (optional)> <Camera pose>
<Score> <Database image name (optional)> <Camera pose>
...

where <Camera pose> is the vectorized form 

<Camera pose> = R11 R21 R31 R12 R22 R32 R13 R23 R33 t1 t2 t3

of the exrinsic camera parameters

                 ┌ R11 R12 R13 t1 ┐
   P = [R | t] = │ R21 R22 R23 t2 │
                 └ R31 R32 R33 t3 ┘

If your method does not return any database image, give 'NaN' instead of <Database image name>. For more details, please find some examples, in the input/ directory, for the 2D image-based method (SForiginal.txt, SForiginal_rerank.txt) and the 3D structure-based method (shortlist_3D_sample.txt).

References

[1] D. Chen, G. Baatz, K. Koeser, S. Tsai, R. Vedantham, T. Pylvanainen, K. Roimela, X. Chen, J. Bach, M. Pollefeys, B. Girod, and R. Grzeszczuk: City-scale landmark identification on mobile devices. CVPR 2011.

[2] San Francisco Landmark Dataset. https://purl.stanford.edu/vn158kj2087

[3] T, Sattler, A. Torii, J. Sivic, M. Pollefeys, H. Taira, M. Okutomi, T. Pajdla: Are Large-Scale 3D Models Really Necessary for Accurate Visual Localization? CVPR 2017.

LICENSE

Copyright (c) 2017 Hajime Taira

Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.