The structure-from-motion reconstruction pipeline – a survey with focus on short image sequences

Klaus Häming; Gabriele Peters

Kybernetika (2010)

  • Volume: 46, Issue: 5, page 926-937
  • ISSN: 0023-5954

Abstract

top
The problem addressed in this paper is the reconstruction of an object in the form of a realistically textured 3D model from images taken with an uncalibrated camera. We especially focus on reconstructions from short image sequences. By means of a description of an easy to use system, which is able to accomplish this in a fast and reliable way, we give a survey of all steps of the reconstruction pipeline. For the purpose of developing a coherent reconstruction system it is necessary to integrate a number of different techniques such as feature detection, algorithms of the RANSAC-family, and methods for auto-calibration. We describe and review recent developments of distinct strands of these techniques. While developing our system the necessity of improvements of several steps of the state-of-the-art reconstruction pipeline emerged. Two of these innovations are introduced in detail in this paper: an advanced SIFT-based feature detector and a two-stage RANSAC process facilitating a faster selection of relevant object points. In addition, we give a recommendation regarding auto-calibration for short image sequences.

How to cite

top

Häming, Klaus, and Peters, Gabriele. "The structure-from-motion reconstruction pipeline – a survey with focus on short image sequences." Kybernetika 46.5 (2010): 926-937. <http://eudml.org/doc/197165>.

@article{Häming2010,
abstract = {The problem addressed in this paper is the reconstruction of an object in the form of a realistically textured 3D model from images taken with an uncalibrated camera. We especially focus on reconstructions from short image sequences. By means of a description of an easy to use system, which is able to accomplish this in a fast and reliable way, we give a survey of all steps of the reconstruction pipeline. For the purpose of developing a coherent reconstruction system it is necessary to integrate a number of different techniques such as feature detection, algorithms of the RANSAC-family, and methods for auto-calibration. We describe and review recent developments of distinct strands of these techniques. While developing our system the necessity of improvements of several steps of the state-of-the-art reconstruction pipeline emerged. Two of these innovations are introduced in detail in this paper: an advanced SIFT-based feature detector and a two-stage RANSAC process facilitating a faster selection of relevant object points. In addition, we give a recommendation regarding auto-calibration for short image sequences.},
author = {Häming, Klaus, Peters, Gabriele},
journal = {Kybernetika},
keywords = {structure from motion; feature detection; RANSAC; auto-calibration; structure from motion; feature detection; RANSAC; auto-calibration},
language = {eng},
number = {5},
pages = {926-937},
publisher = {Institute of Information Theory and Automation AS CR},
title = {The structure-from-motion reconstruction pipeline – a survey with focus on short image sequences},
url = {http://eudml.org/doc/197165},
volume = {46},
year = {2010},
}

TY - JOUR
AU - Häming, Klaus
AU - Peters, Gabriele
TI - The structure-from-motion reconstruction pipeline – a survey with focus on short image sequences
JO - Kybernetika
PY - 2010
PB - Institute of Information Theory and Automation AS CR
VL - 46
IS - 5
SP - 926
EP - 937
AB - The problem addressed in this paper is the reconstruction of an object in the form of a realistically textured 3D model from images taken with an uncalibrated camera. We especially focus on reconstructions from short image sequences. By means of a description of an easy to use system, which is able to accomplish this in a fast and reliable way, we give a survey of all steps of the reconstruction pipeline. For the purpose of developing a coherent reconstruction system it is necessary to integrate a number of different techniques such as feature detection, algorithms of the RANSAC-family, and methods for auto-calibration. We describe and review recent developments of distinct strands of these techniques. While developing our system the necessity of improvements of several steps of the state-of-the-art reconstruction pipeline emerged. Two of these innovations are introduced in detail in this paper: an advanced SIFT-based feature detector and a two-stage RANSAC process facilitating a faster selection of relevant object points. In addition, we give a recommendation regarding auto-calibration for short image sequences.
LA - eng
KW - structure from motion; feature detection; RANSAC; auto-calibration; structure from motion; feature detection; RANSAC; auto-calibration
UR - http://eudml.org/doc/197165
ER -

References

top
  1. Baumberg, A., Reliable feature matching across widely separated views, In: IEEE Conf. on Computer Vision and Pattern Recognition 2000, Vol. 01, pp. 1774–1781. (2000) 
  2. Bay, H., Tuytelaars, T., Gool, L. Van, Surf: Speeded up robust features, In: 9th European Conference on Computer Vision, Graz 2006. (2006) 
  3. Beardsley, P. A., Torr, P. H. S., Zisserman, A., 3d model acquisition from extended image sequences, In: ECCV ’96: Proc. 4th European Conference on Computer Vision-Volume II, Springer, London 1996, pp. 683–695. (1996) 
  4. Beis, J. S., Lowe, D. G., Shape indexing using approximate nearest-neighbour search in high-dimensional spaces, In: Proc. IEEE Conf. Comp. Vision Patt. Recog 1997, pp. 1000–1006. (1997) 
  5. Birchfield, S., Tomasi, C., 10.1023/A:1008160311296, Internat. J. Comput. Vision 3 (1999), 269–293. (1999) DOI10.1023/A:1008160311296
  6. Canny, J., 10.1109/TPAMI.1986.4767851, IEEE Trans. Pattern Anal. Mach. Intell. 8 (1986), 6, 679–698. (1986) DOI10.1109/TPAMI.1986.4767851
  7. Chum, O., Matas, J., Matching with PROSAC – progressive sample consensus, In: Proc. Conference on Computer Vision and Pattern Recognition (C. Schmid, S. Soatto, and C. Tomasi, eds.), Vol. 1, Los Alamitos 2005, IEEE Computer Society, pp. 220–226. (2005) 
  8. Chum, O., Matas, J., Kittler, J., Locally optimized ransac, In: DAGM-Symposium 2003, pp. 236–243. (2003) 
  9. Chum, O., Matas, J., Obdržálek, Š., Enhancing RANSAC by generalized model optimization, In: Proc. Asian Conference on Computer Vision (ACCV) (K.-S. Hong and Z. Zhang, eds.), Vol. 2, Seoul 2004, Asian Federation of Computer Vision Societies, pp. 812–817. (2004) 
  10. Cox, I. J., Hingorani, S. L., Rao, S. B., Maggs, B. M., 10.1006/cviu.1996.0040, Comput. Vis. Image Underst. 63 (1996), 3, 542–567. (1996) DOI10.1006/cviu.1996.0040
  11. Dellaert, F., Seitz, S. M., Thorpe, Ch. E., Thrun, S., Structure from motion without correspondence, In: IEEE Conf. on Computer Vision and Pattern Recognition 2000, pp. 557–564. (2000) 
  12. Dempster, A. P., Laird, N. M., Rubin, D. B., Maximum likelihood from incomplete data via the em algorithm, J. Roy. Statist. Soc. Ser. B 39 (1977), 1, 1–38. (1977) Zbl0364.62022MR0501537
  13. Fischler, M. A., Bolles, R. C., 10.1145/358669.358692, Commun. ACM 24 (1981), 6, 381–395. (1981) MR0618158DOI10.1145/358669.358692
  14. Fitzgibbon, A. W., Zisserman, A., Automatic 3D model acquisition and generation of new images from video sequences, In: Proc.European Signal Processing Conference (EUSIPCO ’98), Rhodes 1998, pp. 1261–1269. (1998) 
  15. Fitzgibbon, A. W., Zisserman, A., Automatic camera recovery for closed or open image sequences, In: Proc. European Conference on Computer Vision 1998, pp. 311–326. (1998) 
  16. Frahm, J.-M., Pollefeys, M., Ransac for (quasi-)degenerate data (qdegsac), In: CVPR ’06: Proc. 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, Washington 2006, IEEE Computer Society, pp. 453–460. (2006) 
  17. Friedman, J. H., Bentley, J. L., Finkel, R. A., An algorithm for finding best matches in logarithmic expected time, ACM Trans. Math. Software 3 (1997), 3, 209–226. (1997) 
  18. Pollefeys, M., Gool, L J. Van, Meerbergen, G. Van, Vergauwen, M., 10.1023/A:1014562312225, Internat. J. Comput. Vision 47 (2002), 275–285. (2002) DOI10.1023/A:1014562312225
  19. Häming, K., Peters, G., Extension of the generalized image rectification – Catching the infinity cases, In: Proc. 4th International Conference on Informatics in Control, Automation, and Robotics (ICINCO 2007) (J. Zaytoon, J.-L. Ferrier, J. A. Cetto, and J. Filipe, eds.), Vol. RA-2, Angers 2007, Institute for Systems and Technologies of Information, Control and Communication, pp. 275–279. (2007) 
  20. Harris, Ch., Stephens, M., A combined Corner and Edge detector, In: 4th ALVEY Vision Conference 1988, pp. 147–151. (1988) 
  21. Hartley, R. I., Zisserman, A., Multiple View Geometry in Computer Vision, Second edition. Cambridge University Press 2004. (2004) Zbl1072.68104MR2059248
  22. Koch, R., Pollefeys, M., Gool, L. J. Van, 10.1002/1099-1778(200007)11:3<115::AID-VIS228>3.0.CO;2-2, J. Visualization and Computer Animation 11 (2000), 3, 115–127. (2000) DOI10.1002/1099-1778(200007)11:3<115::AID-VIS228>3.0.CO;2-2
  23. Lhuillier, M., Quan, L., 10.1109/TPAMI.2005.44, IEEE Trans. Pattern Analysis and Machine Intelligence 27 (2005), 3, 418–433. (2005) DOI10.1109/TPAMI.2005.44
  24. Lindeberg, T., Feature detection with automatic scale selection, Internat. J. Comput. Vision 30 (1998), 2, 77–116. (1998) 
  25. Lindeberg, T., Bretzner, L., Real-time scale selection in hybrid multi-scale representations, In: Proc. Scale-Space, Lect. Notes in Comput. Sci. 2695, Springer 2003, pp. 148–163. (2003) Zbl1067.68753
  26. Lowe, D. G., 10.1023/B:VISI.0000029664.99615.94, Internat. J. Comput. Vision 60 (2004), 2, 91–110. (2004) DOI10.1023/B:VISI.0000029664.99615.94
  27. Lucas, B. D., Kanade, T., An iterative image registration technique with an application to stereo vision, In: IJCAI81, pp. 674–679. 
  28. Mikolajczyk, K., Schmid, C., 10.1023/B:VISI.0000027790.02288.f2, Internat. J. Comput. Vision 60 (2004), 1, 63–86. (2004) DOI10.1023/B:VISI.0000027790.02288.f2
  29. Peters, G., Häming, K., Fast freehand acquisition of 3d objects and their visualization, J. Commun. Comput. 7 (2010), 2–3. (2010) 
  30. Pollefeys, M., Gool, L. Van, Vergauwen, M., Verbiest, F., Cornelis, K., Tops, J., Koch, R., 10.1023/B:VISI.0000025798.50602.3a, Internat. J. Comput. Vision 59 (2004), 3, 207–232. (2004) DOI10.1023/B:VISI.0000025798.50602.3a
  31. Pollefeys, M., Koch, R., Gool, L. J. van, Self-calibration and metric reconstruction in spite of varying and unknown internal camera parameters, In: ICCV 1998, pp. 90–95. (1998) 
  32. Pollefeys, M., Koch, R., Gool, L. J. van, A simple and efficient rectification method for general motion, In: Proc. Internat. Conference on Computer Vision (ICCV 1999), pp. 496–501. (1999) 
  33. Pollefeys, M., Verbiest, F., Gool, L. Van, Surviving dominant planes in uncalibrated structure and motion recovery, In: Computer Vision – ECCV 2002, 7th European Conference on Computer Vision (Johansen, ed.). Lect. Notes Comput. Sci. 2351, Springer-Verlag 2002, pp. 837–851. (2002) 
  34. Pollefeys, M., Vergauwen, M., Cornelis, K., Tops, J., Verbiest, F., Structure, L. Van Gool., In, motion from image sequences., Proc, Conference on Optical 3-D Measurement Techniques V (K. Gruen, ed.), Vienna 2001. pp. 251–258. (2001) 
  35. Ponce, J., Papadopoulo, T., Teillaud, M., Triggs, B., On the absolute quadratic complex and its application to autocalibration, In: IEEE Conference on Computer Vision & Pattern Recognition 2005, Vol. I., pp. 780–787. (2005) 
  36. Prasad, M., Fitzgibbon, A. W., Single view reconstruction of curved surfaces, In: IEEE Conf. on Computer Vision and Pattern Recognition 2006, Vol. 02, pp. ,1345–1354. (2006) 
  37. Saxena, A., Sun, M., Ng, A. Y., Make3d: Depth perception from a single still image, In: AAAI (D. Fox and C. P. Gomes, eds.), AAAI Press 2008, pp. 1571–1576. (2008) 
  38. Schaffalitzky, F., Zisserman, A., Multi-view matching for unordered image sets, or “How do I organize my holiday snaps?”, In: Proc. 7th European Conference on Computer Vision, Copenhagen 2002, Springer, Vol. 1, pp. 414–431. (2002) Zbl1034.68662
  39. Schaffalitzky, F., Zisserman, A., Hartley, R. I., Torr, P. H. S., A six point solution for structure and motion, In: ECCV ’00: Proc. 6th European Conference on Computer Vision, Vol. I, London 2000, Springer, pp. 632–648. (2000) 
  40. Shen, F., Wang, H., A local edge detector used for finding corners, Proc. ICICS, 2001. (2001) 
  41. Shi, J., Tomasi, C., Good features to track, In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR’94), Seattle 1994. (1994) 
  42. Snavely, N., Seitz, S. M., Szeliski, R., 10.1145/1141911.1141964, ACM Trans. on Graphics (SIGGRAPH Proc.), 25 (2006), 3, 835–846. (2006) DOI10.1145/1141911.1141964
  43. Torr, P. H. S., Zisserman, A., 10.1006/cviu.1999.0832, Comput. Vis. Image Underst. 78 (2000), 1, 138–156. (2000) DOI10.1006/cviu.1999.0832
  44. Triggs, B., Autocalibration and the absolute quadric, In: Proc. IEEE Conference on Computer Vision and Pattern Recognition, Puerto Rico 1977, IEEE Computer Society Press, pp. 609–614. (1977) 
  45. Tsai, R. Y., A versatile camera calibration technique for high-accuracy 3D machine vision metrology using off-the-shelf TV cameras and lenses, In: Radiometry (L. B. Wolff, S. A. Shafer, and G. Healey, eds.), Jones and Bartlett Publishers, Inc., pp. 221–244, 1992. (1992) 
  46. Vergauwen, M., Gool, L. Van, 10.1007/s00138-006-0027-1, Mach. Vision Appl. 17 (2006), 6, 411–426. (2006) DOI10.1007/s00138-006-0027-1
  47. Viola, P., Jones, M., Rapid object detection using a boosted cascade of simple features, IEEE Computer Society Conference on Computer Vision and Pattern Recognition 2001, Vol. 1, 511. (2001) 
  48. Woodford, O. J., Torr, P. H. S., Reid, I. D., Fitzgibbon, A. W., Global stereo reconstruction under second order smoothness priors, In: Proc. IEEE Conference on Computer Vision and Pattern Recognition, Anchorage 2008. (2008) 
  49. Zhang, Z., 10.1109/34.888718, IEEE Trans. Pattern Analysis and Machine Intelligence 22 (1998), 1330–1334. (1998) DOI10.1109/34.888718

NotesEmbed ?

top

You must be logged in to post comments.

To embed these notes on your page include the following JavaScript code on your page where you want the notes to appear.

Only the controls for the widget will be shown in your chosen language. Notes will be shown in their authored language.

Tells the widget how many notes to show per page. You can cycle through additional notes using the next and previous controls.

    
                

                
Note: Best practice suggests putting the JavaScript code just before the closing </body> tag.