[go: up one dir, main page]
Skip to main content
SpringerLink
Log in

Hyper-parameter optimization tools comparison for multiple object tracking applications

  • Original paper
  • Published:
Machine Vision and Applications Aims and scope Submit manuscript

Abstract

Commonly, an algorithm needs a certain number of variables that control its behavior. The optimal values result in a better performance that could generate profits for companies, make the algorithm stands out from similar applications or improve its ranking in algorithm competitions. However, finding this values is not straightforward because manual tuning could be a stressful and difficult task even for expert users. This paper presents, evaluates and compares 4 tools in the literature for hyper-parameter optimization, selected due to their number of citations, code availability and impact on literature: MCMC, SMAC, TPE and Spearmint. We evaluate these tools using the publicly available source code provided by the authors in a computer vision application: multiple object tracking (MOT), which is a challenging topic and the strategies rely on a set of tunable parameters. This analysis considers the impact of hyper-parameter optimization tools in terms of stability, performance, usability, among others. The evaluations are carried out using public benchmarks such as PETS09 and ETH and the results show how these tools change the performance of a MOT framework and how this would affect the results of real ranked competitions. Our goal is (1) to encourage the reader to use these tools and (2) to provide some guidelines that helps anyone when tuning his/her methods.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11

Similar content being viewed by others

Explore related subjects

Discover the latest articles, news and stories from top researchers in related subjects.

Notes

  1. http://fastml.com/tuning-hyperparams-automatically-with-spearmint/

  2. http://www.nersc.gov/users/data-analytics/data-analytics-2/deep-learning/hyperparameter-o/spearmint-bayesian/.

  3. https://github.com/HIPS/Spearmint.

  4. http://www.cs.ubc.ca/labs/beta/Projects/SMAC/v2.10.03/quickstart.html.

  5. http://hyperopt.github.io/hyperopt/.

  6. http://optunity.readthedocs.io/.

  7. https://motchallenge.net/results/2D_MOT_2015/

References

  1. Berclaz, J., Turetken, E., Fleuret, F., Fua, P.: Multiple object tracking using \(k\)-shortest paths optimization. IEEE Trans. Pattern Anal. Mach. Intell. 23, 54–72 (2011)

    Google Scholar 

  2. Bergstra, J., Bengio, Y.: Random search for hyper-parameter optimization. J. Mach. Learn. Res. 13, 281–305 (2012). http://dl.acm.org/citation.cfm?id=2188385.2188395

  3. Bergstra, J.S., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Shawe-Taylor, J., Zemel, R.S., Bartlett, P.L., Pereira, F., Weinberger, K.Q. (eds.) Advances in Neural Information Processing Systems, vol. 24, pp. 2546–2554 (2011)

  4. Bergstra, J.S., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Shawe-Taylor, J., Zemel, R.S., Bartlett, P.L., Pereira, F., Weinberger, K.Q. (eds.) Advances in Neural Information Processing Systems, vol. 24, pp. 2546–2554. Curran Associates, Inc., Red Hook (2011)

    Google Scholar 

  5. Bernardin, K., Stiefelhagen, R.: Evaluating multiple object tracking performance: the clear mot metrics. EURASIP J. Image Video Process. 2008(1), 1–10 (2008)

    Article  Google Scholar 

  6. Bewley, A., Ge, Z., Ott, L., Ramos, F., Upcroft, B.: Simple online and realtime tracking. In: 2016 IEEE International Conference on Image Processing (ICIP), pp. 3464–3468 (2016). https://doi.org/10.1109/ICIP.2016.7533003

  7. Binitha, S., Sathya, S.: A survey of bio inspired optimization algorithms. Comput. Eng. 2, 137–151 (2012)

    Google Scholar 

  8. Bishop, C.M.: Pattern Recognition and Machine Learning (Information Science and Statistics). Springer, Secaucus (2006)

    MATH  Google Scholar 

  9. Boussad, I., Lepagnot, J., Siarry, P.: A survey on optimization metaheuristics. Inf. Sci. 237, 82–117 (2013). (Prediction, control and diagnosis using advanced neural computations)

    Article  MathSciNet  MATH  Google Scholar 

  10. Breitenstein, M.D., Reichlin, F., Leibe, B., Koller-Meier, E., Van Gool, L.: Online multiperson tracking-by-detection from a single, uncalibrated camera. IEEE Trans. Pattern Anal. Mach. Intell. 33(9), 1820–1833 (2011)

    Article  Google Scholar 

  11. Brunetti, A., Buongiorno, D., Trotta, G.F., Bevilacqua, V.: Computer vision and deep learning techniques for pedestrian detection and tracking: a survey. Neurocomputing 300, 17–33 (2018). https://doi.org/10.1016/j.neucom.2018.01.092

    Article  Google Scholar 

  12. Burgard, W., Brock, O., Stachniss, C.: Active Policy Learning for Robot Planning and Exploration under Uncertainty, p. 352. MIT Press, Cambridge (2008). http://ieeexplore.ieee.org/xpl/articleDetails.jsp?arnumber=6280104

  13. Chodorow, K., Dirolf, M.: MongoDB: The Definitive Guide, 1st edn. O’Reilly Media Inc., Newton (2010)

    Google Scholar 

  14. Collins, R.T., Carr, P.: Hybrid stochastic/deterministic optimization for tracking sports players and pedestrians. In: Fleet, D., Pajdla, T., Schiele, B., Tuytelaars, T. (eds.) Computer Vision—ECCV 2014, pp. 298–313. Springer, Cham (2014)

    Chapter  Google Scholar 

  15. Dalal, N., Triggs, B.: Histograms of oriented gradients for human detection. In: CVPR, pp. 886–893 (2005)

  16. Dollr, P., Appel, R., Belongie, S., Perona, P.: Fast feature pyramids for object detection. IEEE Trans. Pattern Anal. Mach. Intell. 36(8), 1532–1545 (2014). https://doi.org/10.1109/TPAMI.2014.2300479

    Article  Google Scholar 

  17. Domhan, T., Springenberg, J.T., Hutter, F.: Speeding up automatic hyperparameter optimization of deep neural networks by extrapolation of learning curves. In: Proceedings of the 24th International Conference on Artificial Intelligence, IJCAI’15, pp. 3460–3468. AAAI Press, New York (2015). http://dl.acm.org/citation.cfm?id=2832581.2832731

  18. Dong, X., Shen, J., Wang, W., Liu, Y., Shao, L., Porikli, F.: Hyperparameter optimization for tracking with continuous deep \(q\)-learning. In: The IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2018)

  19. Dubuisson, S., Gonzales, C.: A survey of datasets for visual tracking. Mach. Vis. Appl. 27(1), 23–52 (2016). https://doi.org/10.1007/s00138-015-0713-y

    Article  Google Scholar 

  20. Ess, A., Leibe, B., Gool, L.V.: Depth and appearance for mobile scene analysis. In: 2007 IEEE 11th International Conference on Computer Vision, pp. 1–8 (2007). https://doi.org/10.1109/ICCV.2007.4409092

  21. Ferryman, J., Ellis, A.L.: Performance evaluation of crowd image analysis using the pets2009 dataset. Pattern Recognit. Lett. 44, 3–15 (2014). https://doi.org/10.1016/j.patrec.2014.01.005. (Pattern recognition and crowd analysis)

    Article  Google Scholar 

  22. Ferryman, J., Shahrokni, A.: Pets2009: dataset and challenge. In: Twelfth IEEE International Workshop on Performance Evaluation of Tracking and Surveillance, pp. 1–6 (2009). https://doi.org/10.1109/PETS-WINTER.2009.5399556

  23. Floudas, C.A., Gounaris, C.E.: A review of recent advances in global optimization. J. Glob. Optim. 45(1), 3 (2008). https://doi.org/10.1007/s10898-008-9332-8

    Article  MathSciNet  MATH  Google Scholar 

  24. Franceschi, L., Donini, M., Frasconi, P., Pontil, M.: Forward and reverse gradient-based hyperparameter optimization. In: Precup, D., Teh, Y.W., (eds.) Proceedings of the 34th International Conference on Machine Learning, Proceedings of Machine Learning Research, vol. 70, pp. 1165–1173. PMLR, International Convention Centre, Sydney (2017)

  25. Guo, L.: Stability of recursive stochastic tracking algorithms. SIAM J. Control Optim. 32(5), 1195–1225 (1994). https://doi.org/10.1137/S0363012992225606

    Article  MathSciNet  MATH  Google Scholar 

  26. Haftka, R.T., Villanueva, D., Chaudhuri, A.: Parallel surrogate-assisted global optimization with expensive functions—a survey. Struct. Multidiscip. Optim. 54(1), 3–13 (2016). https://doi.org/10.1007/s00158-016-1432-3

    Article  MathSciNet  Google Scholar 

  27. Hutter, F., Hoos, H., Leyton-Brown, K.: Sequential model-based optimization for general algorithm configuration. In: Proceeding of LION-5, pp. 507–523 (2011)

  28. Hutter, F., Hoos, H., Leyton-Brown, K.: An efficient approach for assessing hyperparameter importance. In: Proceedings of the 31st International Conference on International Conference on Machine Learning, ICML’14, vol. 32, pp. I-754–I-762 (2014)

  29. Kim, K., Davis, L.: Multi-camera tracking and segmentation of occluded people on ground plane using search-guided particle filtering. Comput. Vis. ECCV 2006, 98–109 (2006)

    Google Scholar 

  30. Leal-Taixé, L., Milan, A., Reid, I., Roth, S., Schindler, K.: MOTChallenge 2015: towards a benchmark for multi-target tracking (2015). arXiv:1504.01942

  31. Lizotte, D., Wang, T., Bowling, M., Schuurmans, D.: Automatic gait optimization with Gaussian process regression. In: Proceedings of the 20th International Joint Conference on Artifical Intelligence, IJCAI’07, pp. 944–949. Morgan Kaufmann Publishers Inc., San Francisco (2007). http://dl.acm.org/citation.cfm?id=1625275.1625428

  32. Luo, G.: A review of automatic selection methods for machine learning algorithms and hyper-parameter values. Netw. Model. Anal. Health Inf. Bioinform. 5(1), 18 (2016). https://doi.org/10.1007/s13721-016-0125-6

    Article  Google Scholar 

  33. Luo, W., Zhao, X., Kim, T.: Multiple object tracking: a literature review (2014). arXiv:1409.7618

  34. Maggio, E., Taj, M., Cavallaro, A.: Efficient multitarget visual tracking using random finite sets. IEEE Trans. Circuits Syst. Video Technol. 18(8), 1016–1027 (2008). https://doi.org/10.1109/TCSVT.2008.928221

    Article  Google Scholar 

  35. Maurice, C., Madrigal, F., Lerasle, F.: Hyper-optimization tools comparison for parameter tuning applications. In: 2017 14th IEEE International Conference on Advanced Video and Signal Based Surveillance (AVSS), pp. 1–6 (2017). https://doi.org/10.1109/AVSS.2017.8078499

  36. Minton, S., Johnston, M.D., Philips, A.B., Laird, P.: Minimizing conflicts: a heuristic repair method for constraint satisfaction and scheduling problems. Artif. Intell. 58(1–3), 161–205 (1992)

    Article  MathSciNet  MATH  Google Scholar 

  37. Nocedal, J., Wright, S.J.: Numerical Optimization, 2nd edn. World Scientific, Singapore (2006)

    MATH  Google Scholar 

  38. Pazhaniraja, N., Paul, P.V., Roja, G., Shanmugapriya, K., Sonali, B.: A study on recent bio-inspired optimization algorithms. In: 2017 4th International Conference on Signal Processing, Communication and Networking (ICSCN), pp. 1–6 (2017). https://doi.org/10.1109/ICSCN.2017.8085674

  39. Qin, Z., Shelton, C.R.: Improving multi-target tracking via social grouping. In: 2012 IEEE Conference on Computer Vision and Pattern Recognition, pp. 1972–1978 (2012). https://doi.org/10.1109/CVPR.2012.6247899

  40. Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., de Freitas, N.: Taking the human out of the loop: a review of Bayesian optimization. Proc. IEEE 104(1), 148–175 (2016). https://doi.org/10.1109/JPROC.2015.2494218

    Article  Google Scholar 

  41. Shan, S., Wang, G.G.: Survey of modeling and optimization strategies to solve high-dimensional design problems with computationally-expensive black-box functions. Struct. Multidiscip. Optim. 41(2), 219–241 (2010). https://doi.org/10.1007/s00158-009-0420-2

    Article  MathSciNet  MATH  Google Scholar 

  42. Shen, J., Liang, Z., Liu, J., Sun, H., Shao, L., Tao, D.: Multiobject tracking by submodular optimization. IEEE Trans. Cybern. (2018). https://doi.org/10.1109/TCYB.2018.2803217

    Article  Google Scholar 

  43. Shen, J., Yu, D., Deng, L., Dong, X.: Fast online tracking with detection refinement. IEEE Trans. Intell. Transp. Syst. 19(1), 162–173 (2018). https://doi.org/10.1109/TITS.2017.2750082

    Article  Google Scholar 

  44. Smeulders, A.W.M., Chu, D.M., Cucchiara, R., Calderara, S., Dehghan, A., Shah, M.: Visual tracking: an experimental survey. IEEE Trans. Pattern Anal. Mach. Intell. 36(7), 1442–1468 (2014). https://doi.org/10.1109/TPAMI.2013.230

    Article  Google Scholar 

  45. Snoek, J., Larochelle, H., Adams, R.P.: Practical Bayesian optimization of machine learning algorithms. In: Proceedings of the 25th International Conference on Neural Information Processing Systems—Volume 2, NIPS’12, pp. 2951–2959. Curran Associates Inc., Red Hook (2012)

  46. Snoek, J., Rippel, O., Swersky, K., Kiros, R., Satish, N., Sundaram, N., Patwary, M., Prabhat, M., Adams, R.: Scalable Bayesian optimization using deep neural networks. In: Bach, F., Blei, D. (eds.) Proceedings of the 32nd International Conference on Machine Learning, Proceedings of Machine Learning Research, vol. 37, pp. 2171–2180. PMLR, Lille (2015)

  47. Tsamardinos, I., Rakhshani, A., Lagani, V.: Performance-estimation properties of cross-validation-based protocols with simultaneous hyper-parameter optimization. In: Likas, A., Blekas, K., Kalles, D. (eds.) Artificial Intelligence: Methods and Applications, pp. 1–14. Springer, Cham (2014)

    Google Scholar 

  48. Wang, Z., Zoghi, M., Hutter, F., Matheson, D., de Freitas, N.: Bayesian optimization in high dimensions via random embeddings. In: International Joint Conferences on Artificial Intelligence (IJCAI)—Distinguished Paper Award (2013). http://www.cs.ubc.ca/~hutter/papers/13-IJCAI-BO-highdim.pdf

  49. Watada, J., Musa, Z., Jain, L.C., Fulcher, J.: Human tracking: a state-of-art survey. In: Setchi, R., Jordanov, I., Howlett, R.J., Jain, L.C. (eds.) Knowledge-Based and Intelligent Information and Engineering Systems, pp. 454–463. Springer, Berlin (2010)

    Chapter  Google Scholar 

  50. Weise, T.: Global Optimization Algorithms—Theory and Application, 2nd edn. Self-Published (2009). http://www.it-weise.de/

  51. Wu, Y., Lim, J., Yang, M.H.: Object tracking benchmark. IEEE Trans. Pattern Anal. Mach. Intell. 37(9), 1834–1848 (2015). https://doi.org/10.1109/TPAMI.2014.2388226

    Article  Google Scholar 

  52. Yan, C., Xie, H., Chen, J., Zha, Z.J., Hao, X., Zhang, Y., Dai, Q.: An effective Uyghur text detector for complex background images. IEEE Trans. Multimed. (2018). https://doi.org/10.1109/TMM.2018.2838320

    Article  Google Scholar 

  53. Yan, C., Xie, H., Liu, S., Yin, J., Zhang, Y., Dai, Q.: Effective Uyghur language text detection in complex background images for traffic prompt identification. IEEE Trans. Intell. Transp. Syst. 19(1), 220–229 (2018). https://doi.org/10.1109/TITS.2017.2749977

    Article  Google Scholar 

  54. Yan, C., Xie, H., Yang, D., Yin, J., Zhang, Y., Dai, Q.: Supervised hash coding with deep neural network for environment perception of intelligent vehicles. IEEE Trans. Intell. Transp. Syst. 19(1), 284–295 (2018). https://doi.org/10.1109/TITS.2017.2749965

    Article  Google Scholar 

  55. Yan, C., Zhang, Y., Xu, J., Dai, F., Li, L., Dai, Q., Wu, F.: A highly parallel framework for HEVC coding unit partitioning tree decision on many-core processors. IEEE Signal Process. Lett. 21(5), 573–576 (2014). https://doi.org/10.1109/LSP.2014.2310494

    Article  Google Scholar 

  56. Yan, C., Zhang, Y., Xu, J., Dai, F., Zhang, J., Dai, Q., Wu, F.: Efficient parallel framework for HEVC motion estimation on many-core processors. IEEE Trans. Circuits Syst. Video Technol. 24(12), 2077–2089 (2014). https://doi.org/10.1109/TCSVT.2014.2335852

    Article  Google Scholar 

  57. Yaseen, M.U., Anjum, A., Rana, O., Antonopoulos, N.: Deep learning hyper-parameter optimization for video analytics in clouds. IEEE Trans. Syst. Man Cybern. Syst. (2018). https://doi.org/10.1109/TSMC.2018.2840341

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. LAAS-CNRS, 7 av. du Colonel Roche, 31400, Toulouse, France

    Francisco Madrigal, Camille Maurice & Frédéric Lerasle

Authors
  1. Francisco Madrigal
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Camille Maurice
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Frédéric Lerasle
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Francisco Madrigal.

Additional information

This work was carried at LAAS-CNRS and supported by the DGA-RAPID Project SERVAT: Suivi Et Reconnaissance Visuels, Adaptatif, Temps-réel.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Madrigal, F., Maurice, C. & Lerasle, F. Hyper-parameter optimization tools comparison for multiple object tracking applications. Machine Vision and Applications 30, 269–289 (2019). https://doi.org/10.1007/s00138-018-0984-1

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00138-018-0984-1

Keywords

Search

Navigation