[go: up one dir, main page]
Skip to main content
Springer Nature Link
Log in

Multi-objective Reinforcement Learning for Responsive Grids

  • Published:
Journal of Grid Computing Aims and scope Submit manuscript

Abstract

Grids organize resource sharing, a fundamental requirement of large scientific collaborations. Seamless integration of Grids into everyday use requires responsiveness, which can be provided by elastic Clouds, in the Infrastructure as a Service (IaaS) paradigm. This paper proposes a model-free resource provisioning strategy supporting both requirements. Provisioning is modeled as a continuous action-state space, multi-objective reinforcement learning (RL) problem, under realistic hypotheses; simple utility functions capture the high level goals of users, administrators, and shareholders. The model-free approach falls under the general program of autonomic computing, where the incremental learning of the value function associated with the RL model provides the so-called feedback loop. The RL model includes an approximation of the value function through an Echo State Network. Experimental validation on a real data-set from the EGEE Grid shows that introducing a moderate level of elasticity is critical to ensure a high level of user satisfaction.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  1. The Grid Observatory Portal. www.grid-observatory.org. Accessed 24 May 2010

  2. Baird, L.: Residual algorithms: reinforcement learning with function approximation. In: 12th Int. Conf. on Machine Learning, pp. 30–37. Morgan Kaufmann, San Francisco, CA (1995)

    Google Scholar 

  3. Beckman, P., Nadella, S., Trebon, N., Beschastnikh, I.: SPRUCE: a system for supporting urgent high-performance computing. IFIP Series 239, 295–311 (2007)

    Google Scholar 

  4. Blanchet, C., Combet, C., Deleage, G.: Integrating bioinformatics resources on the EGEE Grid platform. In: 6th ACM/IEEE Int. Symp. on Cluster Computing and the Grid, p. 48 (2006)

  5. Blanchet, C., Mollon, R., Thain, D., Deleage, G.: Grid deployment of legacy bioinformatics applications with transparent data access. In: 7th IEEE/ACM International Conference on Grid Computing, pp. 120–127 (2006)

  6. Boyan, J.A., Moore, A.W.: Generalization in reinforcement learning: safely approximating the value function. In: Advances in Neural Information Processing Systems, vol. 7, pp. 369–376. MIT Press, Cambridge, MA (1995)

  7. Colling, D.J., McGough, A.S.: The GRIDCC project. In: 1st Int. Conf. on Communication System Software and Middleware, pp. 1–4 (2006)

  8. Doya, K.: Reinforcement learning in continuous time and space. Neural Comput. 12, 219–245 (2000)

    Article  Google Scholar 

  9. Weissenbach, D., Clévédé, E., Gotab, B.: Distributed jobs on EGEE Grid infrastructure for an Earth science application: moment tensor computation at the centroid of an earthquake. Earth Science Informatics 7(1–2), 97–106 (2009)

    Google Scholar 

  10. Iosup, A., et al.: The Grid workloads archive. Future Gener. Comput. Syst. 24(7), 672–686 (2008)

    Article  Google Scholar 

  11. Germain-Renaud, C., et al.: Grid analysis of radiological data. In: Cannataro, M. (ed.) Handbook of Research on Computational Grid Technologies for Life Sciences, Biomedicine and Healthcare. IGI Press, Hershey, PA (2009)

  12. Laure, E., et al.: Programming the Grid with gLite. Comput. Methods Sci. Technol. 12(1), 33–45 (2006)

    Google Scholar 

  13. Gagliardi, F., et al.: Building an infrastructure for scientific Grid computing: status and goals of the EGEE project. Philos. Trans. R. Soc. A 1833, 1729–1742 (2005)

    Article  Google Scholar 

  14. Montagnat, J., et al.: Workflow-based data parallel applications on the EGEE production Grid infrastructure. Journal of Grid Computing 6(4), 369–383 (2008)

    Article  Google Scholar 

  15. Foster, I., Kesselman, C., Tuecke, S.: The anatomy of the Grid: enabling scalable virtual organizations. Int. J. Supercomput. Appl. 15(3), 200–222 (2001)

    Article  Google Scholar 

  16. Foster, I., Zhao, Y., Raicu, I., Lu, S.: Cloud computing and Grid computing 360-degree compared. In: Grid Computing Environments Workshop, pp. 1–10. IEEE, Austin, TX (2008)

  17. Germain-Renaud, C., Loomis, C., Moscicki, J., Texier, R.: Scheduling for responsive Grids. Journal of Grid Computing 6(1), 15–27 (2008)

    Article  Google Scholar 

  18. Germain-Renaud, C., Perez, J., Kégl, B., Loomis, C.: Grid differentiated services: a reinforcement learning approach. In: 8th IEEE International Symposium on Cluster Computing and the Grid, Lyon France (2008)

  19. Germain-Renaud, C., Rana, O.F.: The convergence of clouds, Grids, and autonomics. Internet Computing 13(6), 9 (2009)

    Article  Google Scholar 

  20. Gordon, G.J.: Reinforcement learning with function approximation converges to a region. In: Advances in Neural Information Processing Systems, pp. 1040–1046. MIT Press, Denver, CO (2001)

  21. Jaeger, H.: Adaptive nonlinear system identification with Echo State Networks. In: Advances in Neural Information Processing Systems, vol. 15, pp. 593–600. MIT Press, Cambridge, MA (2003)

  22. Jaeger, H., Haas, H.: Harnessing nonlinearity: predicting chaotic systems and saving energy in wireless communication. Science 304(5667), 78–80 (2004)

    Article  Google Scholar 

  23. Douglas Jensen, E., Douglas Locke, C., Tokuda, H.: A time-driven scheduling model for real-time operating systems. In: IEEE Real-Time Systems Symposium, pp. 112–122 (1985)

  24. Amar, L., Barak, A., Levy, E., Okun, M.: An on-line algorithm for fair-share node allocations in a cluster. In: 7th IEEE/ACM Int. Symp. on Cluster Computing and the Grid, pp. 83–91 (2007)

  25. Li, H., Muskulus, M.: Analysis and modeling of job arrivals in a production Grid. SIGMETRICS Perform. Eval. Rev. 34(4), 59–70 (2007)

    Article  Google Scholar 

  26. Li, H., Muskulus, M., Wolters, L.: Modeling correlated workloads by combining model based clustering and a localized sampling algorithm. In: 21st Int. Conf. on Supercomputing, pp. 64–72 (2007)

  27. Llorente, I.M.: Researching cloud resource management and use: the StratusLab initiative. The Cloud Computing Journal. http://cloudcomputing.sys-con.com/node/856815 (2009). Accessed 24 May 2010

  28. Loomis, C.: The Grid observatory. In: Grids Meet Autonomic Computing workshop at ICAC’09. ACM, New York, NY (2009)

  29. Luckow, A., Lacinski, L., Jha, S.: SAGA BigJob: an extensible and interoperable pilot-job abstraction for distributed applications and systems. In: 10th ACM/IEEE Int. Symp. on Cluster, Cloud and Grid Computing (2010)

  30. Perez, J., Germain Renaud, C., Kégl, B., Loomis, C.: Utility-based reinforcement learning for reactive Grids. In: 5th IEEE International Conference on Autonomic Computing (2008). Short paper

  31. Rasmusen, C.E., Williams, C.: Gaussian Processes for Machine Learning. MIT Press, Cambridge, MA (2006)

  32. Sakellariou, R., Yarmolenko, V.: Job scheduling on the Grid: towards SLA-based scheduling. In: Grandinetti, L. (ed.) High Performance Computing and Grids in Action. IOS Press, Amsterdam (2008)

  33. Snell, Q., Clement, M.J., Jackson, D.B., Gregory, C.: The performance impact of advance reservation meta-scheduling. In: IPDPS ’00/JSSPP ’00, pp. 137–153. Springer, Berlin/Heidelberg (2000)

  34. Sutton, R.S., Barto, A.G.: Reinforcement Learning: An Introduction. MIT Press, Cambridge, MA (1998)

  35. Szita, I., Gyenes, V., Lorincz, A.: Reinforcement learning with echo state networks. In: Artificial Neural Networks, ICANN 2006, pp. 830–839. Springer, Berlin/Heidelberg (2006)

  36. Tesauro, G., Jong, N.K., Das, R., Bennani, M.N.: On the use of hybrid reinforcement learning for autonomic resource allocation. Cluster Comput. 10(3), 287–299 (2007)

    Google Scholar 

  37. Tesauro, G., Sejnowski, T.J.: A parallel network that learns to play backgammon. Artif. Intell. 39(3), 357–390 (1989)

    Article  MATH  Google Scholar 

  38. Tesauro, G.J., Walsh, W.E., Kephart, J.O.: Utility-function-driven resource allocation in autonomic systems. In: Int. Conf. Autonomic computing and Communications, pp. 342–343 (2005)

  39. Wu, H., Ravindran, B., Jensen, E.D., Balli, U.: Utility accrual scheduling under arbitrary time/utility functions and multiunit resource constraints. In: IEEE Real-Time and Embedded Computing Systems and Applications, pp. 80–98 (2004)

  40. Vengerov, D.: A reinforcement learning approach to dynamic resource allocation. Eng. Appl. Artif. Intell. 20(3) (2007)

Download references

Author information

Authors and Affiliations

  1. Laboratoire de Recherche en Informatique (LRI)/CNRS, Université Paris-Sud, 91405, Orsay Cedex, France

    Julien Perez & Cécile Germain-Renaud

  2. Laboratoire de l’ Accélérateur Linéaire (LAL)/CNRS, Université Paris-Sud, Orsay, France

    Balazs Kégl & Charles Loomis

Authors
  1. Julien Perez
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Cécile Germain-Renaud
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Balazs Kégl
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Charles Loomis
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Cécile Germain-Renaud.

Additional information

This work has been partially supported by the EGEE-III project funded by the European Union INFSO-RI-222667 and by the NeuroLOG project ANR-06-TLOG-024 and by DIMLSC grant 2008-17D.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Perez, J., Germain-Renaud, C., Kégl, B. et al. Multi-objective Reinforcement Learning for Responsive Grids. J Grid Computing 8, 473–492 (2010). https://doi.org/10.1007/s10723-010-9161-0

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10723-010-9161-0

Keywords

Search

Navigation