Abstract
Nowadays, there is a growing interest in the use of Natural-Language Processing (NLP) for supporting software test automation. This paper investigates the adoption of NLP in web testing. To this aim, a case study has been conducted to compare the cost of the adoption of a NLP testing approach, with respect to more consolidated approaches, i.e., programmable testing and capture and replay testing, in two testing tasks: test cases development and test case evolution/maintenance. Even if preliminary, results show that NLP testing is quite competitive with respect to the more consolidated approaches since the cumulative testing effort of a NLP testing approach, computed considering both development and evolution efforts, is almost always lower than the one of programmable testing and capture &replay testing.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
Notes
- 1.
- 2.
- 3.
- 4.
- 5.
- 6.
- 7.
Being a commercial tool, we think it is better not to disclose its identity. In all cases, the other NLP tools considered were also very similar to the one chosen.
- 8.
https://expresscart.markmoffat.com/documentation.html (last access: February 2022).
- 9.
https://shopizer-ecommerce.github.io/documentation/#/starting (last access: February 2022).
- 10.
https://bigprof.com/appgini/applications/online-inventory-manager (last access: February 2022).
- 11.
- 12.
References
Carvalho, G., Falcão, D., Barros, F., Sampaio, A., Mota, A., Motta, L., Blackburnc, M.: Nat2testscr: Test case generation from natural language requirements based on scr specifications. Sci. Comput. Program. 95, 275–297 (2014). https://doi.org/10.1016/j.scico.2014.06.007
Cauchi, A., Colombo, C., Francalanza, A., Micallef, M., Pace, G.: Using gherkin to extract tests and monitors for safer medical device interaction design. In: 8th Symposium on Engineering Interactive Computing Systems (SIGCHI), ACM, June 2016. https://doi.org/10.1145/2933242.2935868
Colombo, C., Micallef, M., Scerri, M.: Verifying web applications: from business level specifications to automated model-based testing. Electron. Proc. Theor. Comput. Sci. 141, 14–28 (2014). https://doi.org/10.4204/eptcs.141.2
Ebert, C., Gallardo, G., Hernantes, J., Serrano, N.: DevOps. IEEE Softw. 33(3), 94–100 (2016). https://doi.org/10.1109/ms.2016.68
Fischbach, J., Vogelsang, A., Spies, D., Wehrle, A., Junker, M., Freudenstein, D.: SPECMATE: automated creation of test cases from acceptance criteria. In: 13th International Conference on Software Testing, Validation and Verification (ICST), IEEE, October 2020. https://doi.org/10.1109/icst46399.2020.00040
García, B., Gallego, M., Gortázar, F., Organero, M.: A survey of the selenium ecosystem. Electronics 9, 1067 (2020). https://doi.org/10.3390/electronics9071067
Garousi, V., Bauer, S., Felderer, M.: NLP-assisted software testing: a systematic mapping of the literature. Inf. Softw. Technol. 126, 106321 (2020). https://doi.org/10.1016/j.infsof.2020.106321
Gupta, A., Mahapatra, R.P.: A circumstantial methodological analysis of recent studies on NLP-driven test automation approaches. In: Udgata, S.K., Sethi, S., Srirama, S.N. (eds.) Intelligent Systems. LNNS, vol. 185, pp. 155–167. Springer, Singapore (2021). https://doi.org/10.1007/978-981-33-6081-5_14
Leotta, M., Clerissi, D., Ricca, F., Tonella, P.: Capture-replay vs. Programmable web testing: an empirical assessment during test case evolution. In: Proceedings of 20th Working Conference on Reverse Engineering (WCRE 2013), pp. 272–281. IEEE (2013). https://doi.org/10.1109/WCRE.2013.6671302
Leotta, M., Clerissi, D., Ricca, F., Tonella, P.: Visual vs. DOM-based web locators: an empirical study. In: Casteleyn, S., Rossi, G., Winckler, M. (eds.) ICWE 2014. LNCS, vol. 8541, pp. 322–340. Springer, Cham (2014). https://doi.org/10.1007/978-3-319-08245-5_19
Leotta, M., Clerissi, D., Ricca, F., Tonella, P.: Approaches and tools for automated end-to-end web testing. Adv. Comput. 101, 193–237 (2016). https://doi.org/10.1016/bs.adcom.2015.11.007
Li, L., et al.: Clustering test steps in natural language toward automating test automation. In: 28th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering, ACM, November 2020. https://doi.org/10.1145/3368089.3417067
Longo, D.H., Vilain, P., da Silva, L.P.: Measuring test data uniformity in acceptance tests for the FitNesse and gherkin notations. J. Comput. Sci. 17(2), 135–155 (2021). https://doi.org/10.3844/jcssp.2021.135.155
Malik, M., Sindhu, M., Abbasi, R.: Test oracle using semantic analysis from natural language requirements. In: 22nd International Conference on Enterprise Information Systems. SCITEPRESS (2020). https://doi.org/10.5220/0009471903450352
Marchetto, A., Ricca, F., Torchiano, M.: Comparing “traditional” and web specific fit tables in maintenance tasks: a preliminary empirical study. In: 12th European Conference on Software Maintenance and Reengineering. IEEE, April 2008. https://doi.org/10.1109/csmr.2008.4493327
Pribisalic, M.: Automatic generation of test cases from use-case specification using natural language processing. In: 33rd Bled eConference - Enabling Technology for a Sustainable Society, pp. 725–734 (2020). https://doi.org/10.18690/978-961-286-362-3.52
Ricca, F., Marchetto, A., Stocco, A.: Ai-based test automation: a grey literature analysis. In: IEEE International Conference on Software Testing, Verification and Validation Workshops (ICSTW), pp. 263–270 (2021). https://doi.org/10.1109/ICSTW52544.2021.00051
Tahvili, S., Hatvani, L., Ramentol, E., Pimentel, R., Afzal, W., Herrera, F.: A novel methodology to classify test cases using natural language processing and imbalanced learning. Eng. Appl. Artif. Intell. 95, 103878 (2020). https://doi.org/10.1016/j.engappai.2020.103878
Wang, C., Pastore, F., Goknil, A., Briand, L.C.: Automatic generation of acceptance test cases from use case specifications: an NLP-based approach. IEEE Trans. Softw. Eng. 48(2), 585–616 (2022). https://doi.org/10.1109/tse.2020.2998503
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2022 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Leotta, M., Ricca, F., Stoppa, S., Marchetto, A. (2022). Is NLP-based Test Automation Cheaper Than Programmable and Capture &Replay?. In: Vallecillo, A., Visser, J., Pérez-Castillo, R. (eds) Quality of Information and Communications Technology. QUATIC 2022. Communications in Computer and Information Science, vol 1621. Springer, Cham. https://doi.org/10.1007/978-3-031-14179-9_6
Download citation
DOI: https://doi.org/10.1007/978-3-031-14179-9_6
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-14178-2
Online ISBN: 978-3-031-14179-9
eBook Packages: Computer ScienceComputer Science (R0)