Abstract
The offline evaluation of recommender systems is typically based on accuracy metrics such as the Mean Absolute Error (MAE) and the Root Mean Squared Error (RMSE), while on the other hand Precision and Recall is used to measure the quality of the top-N recommendations. However, it is difficult to reproduce the results since there are different libraries that can be used for running experiments and also within the same library there are many different settings that if not taken into consideration when replicating the result might vary. In this paper, we show that it is challenging to reproduce results using a different library but with the use of the same library an explanation based approach can be used to assist in the reproducibility of experiments. Our proposed approach has been experimentally evaluated using a real dataset and the results show that it is both practical and effective.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
References
Polatidis, N., Kapetanakis, S., Pimenidis, E., Kosmidis, K.: Reproducibility of experiments in recommender systems evaluation. In: Iliadis, L., Maglogiannis, I., Plagianakos, V. (eds.) AIAI 2018. IAICT, vol. 519, pp. 401–409. Springer, Cham (2018). https://doi.org/10.1007/978-3-319-92007-8_34
Said, A., Bellogín, A.: Comparative recommender system evaluation. In: Proceedings of the 8th ACM Conference on Recommendation Systems - RecSys 2014, pp. 129–136 (2014)
Herlocker, J.L., Konstan, J.A., Terveen, L.G., Riedl, J.T.: Evaluating collaborative filtering recommender systems. ACM Trans. Inf. Syst. 22, 5–53 (2004)
Jannach, D., Lerche, L., Gedikli, F., Bonnin, G.: What recommenders recommend – an analysis of accuracy, popularity, and sales diversity effects. In: Carberry, S., Weibelzahl, S., Micarelli, A., Semeraro, G. (eds.) UMAP 2013. LNCS, vol. 7899, pp. 25–37. Springer, Heidelberg (2013). https://doi.org/10.1007/978-3-642-38844-6_3
Konstan, J.A., Adomavicius, G.: Toward identification and adoption of best practices in algorithmic recommender systems research. In: Proceedings of the International Workshop on Reproducibility and Replication in Recommender Systems Evaluation, pp. 23–28. ACM, New York (2013)
Bellogin, A., Castells, P., Said, A., Tikk, D.: Workshop on reproducibility and replication in recommender systems evaluation. In: Proceedings of the 7th ACM conference on Recommender systems - RecSys 2013, pp. 485–486 (2013)
Bellogin, A., Castells, P., Said, A., Tikk, D.: Report on the workshop on reproducibility and replication in recommender systems evaluation (RepSys). SIGIR Forum. 48, 29–35 (2014)
Košir, A., Odić, A., Tkalčič, M.: How to improve the statistical power of the 10-fold cross validation scheme in recommender systems. In: RecSys RepSys 2013: Proceedings of the International Workshop on Reproducibility and Replication in Recommender Systems Evaluation, pp. 3–6 (2013)
Said, A., Bellogín, A.: RiVal – a toolkit to foster reproducibility in recommender system evaluation. In: RecSys 2014 Proceedings of the 8th ACM Conference on Recommendation Systems, pp. 371–372 (2014)
Hernández del Olmo, F., Gaudioso, E.: Evaluation of recommender systems: a new approach. Expert Syst. Appl. 35, 790–804 (2008)
Peker, S., Kocyigit, A.: mRHR: a modified reciprocal hit rank metric for ranking evaluation of multiple preferences in Top-N recommender systems. In: Dichev, C., Agre, G. (eds.) AIMSA 2016. LNCS (LNAI), vol. 9883, pp. 320–329. Springer, Cham (2016). https://doi.org/10.1007/978-3-319-44748-3_31
Harper, F.M., Konstan, J.A.: The MovieLens datasets. ACM Trans. Interact. Intell. Syst. 5, 1–19 (2015)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2018 Springer Nature Switzerland AG
About this paper
Cite this paper
Polatidis, N., Pimenidis, E. (2018). Reproduction of Experiments in Recommender Systems Evaluation Based on Explanations. In: Pimenidis, E., Jayne, C. (eds) Engineering Applications of Neural Networks. EANN 2018. Communications in Computer and Information Science, vol 893. Springer, Cham. https://doi.org/10.1007/978-3-319-98204-5_16
Download citation
DOI: https://doi.org/10.1007/978-3-319-98204-5_16
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-319-98203-8
Online ISBN: 978-3-319-98204-5
eBook Packages: Computer ScienceComputer Science (R0)