USER STORY QUALITY EVALUATION: ANALYZING FRAMEWORKS AND APPLICATION METHODS
DOI:
https://doi.org/10.11113/aej.v15.24306Keywords:
user stories, quality, INVEST, Quality User Story, AQUSAAbstract
Quality issues in User Stories (US) frequently arise during implementation, with common problems including ambiguity, integration challenges, and inconsistencies in templates or formal structure. These challenges emphasize the critical need to evaluate the quality of the US, prompting extensive research focused on developing methods and innovations to improve the requirement elicitation process. This study aims to analyze the frameworks, application methods, and tools employed in previous research to assess US quality. Utilizing the Kitchenham framework, 26 relevant studies were systematically reviewed. Next, the analysis identified six prominent quality evaluation frameworks: INVEST, Quality User Story (QUS), ISO 25010, IEEE 830, USQM, and linguistic-based approaches. The application of these frameworks often involves experts, practitioners, and artificial intelligence (AI). Meanwhile, among the tools reviewed, AQUSA emerged as the most frequently used due to its alignment with QUS standards. These findings highlight the adaptability of existing frameworks and tools while underscoring the potential for further integrating generative AI to enhance the accuracy and efficiency of US quality evaluations. Subsequently, future research should explore innovative AI-based methods to advance this critical area of requirement engineering.
References
Y. A. Kustiawan and T. Y. Lim, 2023 “User Stories in Requirements Elicitation: A Systematic Literature Review,” in 2023 IEEE 8th International Conference On Software Engineering and Computer Systems (ICSECS), IEEE, Aug., 211–216. DOI: 10.1109/ICSECS58457.2023.10256364.
C. O’hEocha and K. Conboy, 2010 “The Role of the User Story Agile Practice in Innovation,” in International Conference on Lean Enterprise Software and Systems, Oct., 20–30. DOI: 10.1007/978-3-642-16416-3_3.
Rupali M. Chopade and Nikhil S. Dhavase, 2017 “Agile Software Development: Positive and Negative User Stories,” in 2017 2nd International Conference for Convergence in Technology (I2CT), Apr., 297-299. DOI: 10.1109/I2CT.2017.8226139
A. Ananjeva, J. S. Persson, and A. Bruun, 2020 “Integrating UX work with agile development through user stories: An action research study in a small software company,” Journal of Systems and Software, 170: 2-10. DOI: 10.1016/j.jss.2020.110785
F. Dalpiaz, I. van der Schalk, S. Brinkkemper, F. B. Aydemir, and G. Lucassen, 2019 “Detecting terminological ambiguity in user stories: Tool and experimentation,” Journal of Information and Software Technology, 110: 3–16. DOI: 10.1016/j.infsof.2018.12.007.
[6] A. R. Amna and G. Poels, 2022 “Ambiguity in user stories: A systematic literature review,” Journal of Information and Software Technology, 145: 1-14. Elsevier B.V., DOI: 10.1016/j.infsof.2022.106824.
[7] C. A. Peláez and A. Solano, 2024 “A practice for specifying user stories in multimedia system design: An approach to reduce ambiguity.” Interaction Design and Architecture(s) Journal, 60: 214–236. DOI: 10.55612/s-5002-060-009.
S. Nasiri, Y. Rhazali, M. Lahmer, and N. Chenfour, 2020 “Towards a Generation of Class Diagram from User Stories in Agile Methods,” in Procedia Computer Science, 170: 831–837 Elsevier B.V,. DOI: 10.1016/j.procs.2020.03.148.
M. Urbieta, L. Antonelli, G. Rossi, and J. C. S. do Prado Leite, 2020 “The impact of using a domain language for an agile requirement management,” Journal of Information and Software Technology, 127: 1-16. DOI: 10.1016/j.infsof.2020.106375.
M. I. Zul, S. M. Yasin, and D. S. S. Sahid, 2024 “Exploring Requirement Engineering Challenges in Software Development: Insights from Global and Indonesian Landscape,” in 2024 4th International Conference on Electrical Engineering and Informatics (ICon EEI), IEEE, Oct., 136–141. DOI: 10.1109/IConEEI64414.2024.10748069.
[11] H. Gardner, A. F. Blackwell, and L. Church, 2020 “The patterns of user experience for sticky-note diagrams in software requirements workshops,” in Journal of Computer Languages, 61: 1-9. Elsevier Ltd., DOI: 10.1016/j.cola.2020.100997.
M. Trkman, J. Mendling, P. Trkman, and M. Krisper, 2019 “Impact of the conceptual model’s representation format on identifying and understanding user stories,” Journal of Information and Software Technology, 116: 1-17. DOI: 10.1016/j.infsof.2019.08.001.
Y. Wautelet, S. Heng, M. Kolp, and I. Mirbel, 2014 “Unifying and Extending User Story Models,” in CAiSE 2014: Advanced Information Systems Engineering. 211–225. DOI: 10.1007/978-3-319-07881-6_15.
B. Wake, 2025 “INVEST in Good Stories, and SMART Tasks,” XP123: Exploring Extreme Programming. https://xp123.com/invest-in-good-stories-and-smart-tasks/. Retrieved date: 27 January 2025
L. Buglione and A. Abran, 2013 “Improving the user story Agile technique using the INVEST criteria,” in Proceedings - Joint Conference of the 23rd International Workshop on Software Measurement and the 8th International Conference on Software Process and Product Measurement, IWSM-MENSURA 2013, IEEE Computer Society, Oct., 49–53. DOI: 10.1109/IWSM-Mensura.2013.18.
T. Tamai and M. I. Kamata, 2009 “Impact of Requirements Quality on Project Success or Failure,” in Design Requirements Engineering: A Ten-Year Perspective. Lecture Notes in Business Information Processing, 14: 258–275 Springer, Berlin, Heidelberg,. DOI: 10.1007/978-3-540-92966-6_15.
T. Wang, C. Li, C. Wang, T. Li, and Y. Zhai, 2023 “A Deep Learning-Based Method for Identifying User Story Semantic Conflicts,” in 2023 IEEE 23rd International Conference on Software Quality, Reliability, and Security Companion (QRS-C), IEEE, Oct., 220–229. DOI: 10.1109/QRS-C60940.2023.00063.
G. B. Herwanto, G. Quirchmayr, and A. M. Tjoa, 2024 “Leveraging NLP Techniques for Privacy Requirements Engineering in User Stories,” in IEEE Access, 12: 22167–22189. DOI: 10.1109/ACCESS.2024.3364533.
A. R. Amna and G. Poels, 2022 “Systematic Literature Mapping of User Story Research,” in IEEE Access, IEEE, 10: 51723-51746. DOI: 10.1109/ACCESS.2022.3173745.
A. Hendriana, T. Raharjo, and A. Nurfitriani, 2024 “Approaches in Determining User Story Quality through Requirement Elicitation: A Systematic Literature Review,” Indonesian Journal of Computer Science, vol. 12(6): 3599-3614. DOI: 10.33022/ijcs.v12i6.3639.
I. K. Raharjana, D. Siahaan, and C. Fatichah, 2021 “User Stories and Natural Language Processing: A Systematic Literature Review,” IEEE Access, 9: 53811–53826. DOI: 10.1109/ACCESS.2021.3070606.
B. Kitchenham, 2024 “Procedures for Performing Systematic Reviews,” Keele University Technical Report TR/SE-0401, July.
B. Kitchenham and P. Brereton, 2013 “A systematic review of systematic review process research in software engineering,” Journal of Information and Software Technology, 55(12): 2049–2075. DOI: 10.1016/j.infsof.2013.07.010.
D. Hallmann, 2020 “‘I Don’t Understand!’: Toward a Model to Evaluate the Role of User Story Quality,” in 21st International Conference on Agile Software Development, XP 2020, Springer, Cham, Jun., 103–112. DOI: 10.1007/978-3-030-49392-9_7.
X. Xu, Y. Dou, L. Qian, Z. Zhang, Y. Ma, and Y. Tan, 2023 “A Requirement Quality Assessment Method Based on User Stories,” Electronics (Basel), 2(10): 2155-2171. DOI: 10.3390/electronics12102155.
P. Pokharel and P. Vaidya, 2020 “A Study of User Story in Practice,” in 2020 International Conference on Data Analytics for Business and Industry: Way Towards a Sustainable Economy, ICDABI 2020, IEEE, Oct,. 1-5. DOI: 10.1109/ICDABI51230.2020.9325670.
S. S. do Nascimento, J. M. Abe, L. R. Forçan, C. C. de Oliveira, K. Nakamatsu, and A. Ari, 2022 “Improving the Process of Evaluating User Stories Using the Paraconsistent Annotated Evidential Logic Eτ,” in New Approaches for Multidimensional Signal Processing. NAMSP 2022. Smart Innovation, Systems and Technologies, 332: 133–142. Springer, Singapore, DOI: 10.1007/978-981-19-7842-5_12.
Z. Zhang, M. Rayhan, T. Herda, M. Goisauf, and P. Abrahamsson, 2024 “LLM-Based Agents for Automating the Enhancement of User Story Quality: An Early Report,” in 25th International Conference on Agile Software Development XP 2024, W. van der Aalst, S. Ram, M. Rosemann, C. Szyperski, and G. Guizzardi, Eds., Bozen Bolzano: Springer, Jun., 117–126. DOI: 10.1007/978-3-031-61154-4_8.
X. Xu, Y. Dou, L. Qian, J. Jiang, K. Yang, and Y. Tan, 2023 “Quality improvement method for high-end equipment’s functional requirements based on user stories,” Advanced Engineering Informatics, 56, DOI: 10.1016/j.aei.2023.102017.
Y. Li, J. Keung, Z. Yang, X. Ma, J. Zhang, and S. Liu, 2024 “SimAC: simulating agile collaboration to generate acceptance criteria in user story elaboration,” Automated Software Engineering, 31(2): 55. DOI: 10.1007/s10515-024-00448-7.
B. Kumar, U. Tiwari, and D. C. Dobhal, 2022 “User Story Splitting in Agile Software Development using Machine Learning Approach,” in 2022 Seventh International Conference on Parallel, Distributed and Grid Computing (PDGC), Nov., 167–171. DOI: 10.1109/PDGC56933.2022.10053226.
A. Brockenbrough and D. Salinas, 2024 “Using Generative AI to Create User Stories in the Software Engineering Classroom,” in 2024 36th International Conference on Software Engineering Education and Training (CSEET), 1–5. DOI: 10.1109/CSEET62301.2024.10662994.
A. Ferreira, A. Rodrigues da Silva, and A. Paiva, 2022 “Towards the Art of Writing Agile Requirements with User Stories, Acceptance Criteria, and Related Constructs,” in Proceedings of the 17th International Conference on Evaluation of Novel Approaches to Software Engineering, SCITEPRESS - Science and Technology Publications, 477–484. DOI: 10.5220/0011082000003176.
G. Lucassen, F. Dalpiaz, J. M. E. M. van der Werf, and S. Brinkkemper, 2015 “Forging high-quality User Stories: Towards a discipline for Agile Requirements,” in 2015 IEEE 23rd International Requirements Engineering Conference (RE), IEEE, Aug., 126–135. DOI: 10.1109/RE.2015.7320415.
G. Lucassen, F. Dalpiaz, J. M. E. M. van der Werf, and S. Brinkkemper, 2016 “Improving agile requirements: the Quality User Story framework and tool,” Requirement Engineering, 21(3): 383–403. DOI: 10.1007/s00766-016-0250-x.
O. Abed, K. Nebe, and A. B. Abdellatif, 2024 “AI-Generated User Stories Supporting Human-Centred Development: An Investigation on Quality,” in HCI International 2024 Posters. HCII 2024. Communications in Computer and Information Science, 2120: 3–13. Springer, Cham. DOI: 10.1007/978-3-031-62110-9_1.
E. Scott, T. Tõemets, and D. Pfahl, 2021 “An Empirical Study of User Story Quality and Its Impact on Open Source Project Performance,” in Software Quality: Future Perspectives on Software Engineering Quality. SWQD 2021, 404: 119–138. Springer, Cham, DOI: 10.1007/978-3-030-65854-0_10.
E. Trisnawati, I. K. Raharjana, T. Taufik, A. H. Basori, A. B. F. Mansur, and N. A. Alghanmi, 2024 “Analyzing Variances in User Story Characteristics: A Comparative Study of Stakeholders with Diverse Domain and Technical Knowledge in Software Requirements Elicitation,” Journal of Information Systems Engineering and Business Intelligence, 10(1): 110–125. DOI: 10.20473/jisebi.10.1.110-125.
K. Ronanki, B. Cabrero-Daniel, and C. Berger, 2024 “ChatGPT as a Tool for User Story Quality Evaluation: Trustworthy Out of the Box?,” in Agile Processes in Software Engineering and Extreme Programming - Workshops, Springer, Cham, ch. AI-assisted Agile, Dec., 173–181. DOI: 10.1007/978-3-031-48550-3_17.
S. N. F. N. B. Mustaffa, J. Bin Sallim, and R. B. Mohamed, 2021 “Enhancing High-Quality User Stories with AQUSA: An Overview Study of Data Cleaning Process,” in 2021 International Conference on Software Engineering & Computer Systems and 4th International Conference on Computational Science and Information Management (ICSECS-ICOCSIM), IEEE, Aug., 295–300. DOI: 10.1109/ICSECS52883.2021.00060.
C. Ankora and A. D., “Integrating User Stories in the Design of Augmented Reality Application,” International Journal of Information Technologies and Systems Approach, 15(1): 1–19. DOI: 10.4018/IJITSA.304809.
T. Wang, C. Wang, T. Li, Z. Liu, and Y. Zhai, 2022 “User Story Quality Assessment Based on Multi-dimensional Perspective: A Preliminary Framework,” in CEUR Workshop Proceedings: 15th International iStar Workshop, Hyderabad, India: ceur-ws.org, Oct., 7–13.
S. Jiménez, A. Alanis, C. Beltrán, R. Juárez‐Ramírez, A. Ramírez‐Noriega, and C. Tona, 2023 “USQA: A User Story Quality Analyzer prototype for supporting software engineering students,” Computer Applications in Engineering Education, 31(4): 1014–1024. DOI: 10.1002/cae.22620.
Z. Xuan, T. Wang, C. Wang, and T. Li, 2024 “A Tool for Automatically Identifying Semantic Conflicts in User Stories by Combining NLP and BERT Model,” in 2024 IEEE 32nd International Requirements Engineering Conference (RE), 484–487. DOI: 10.1109/RE59067.2024.00057.
P. Kamthan and N. Shahmir, 2020 “A Framework for the Semiotic Quality of User Stories,” in 27th International Conference on Systems Engineering, ICSEng 2020, Las Vegas, USA: Springer Nature, Dec., 413–422. DOI: 10.1007/978-3-030-65796-3_40.
EEE, 1998 “IEEE Recommended Practice for Software Requirements Specifications,” Jun., IEEE, Piscataway, NJ, USA. doi: 10.1109/IEEESTD.1998.88286.
M. A. Kuhail and S. Lauesen, 2022 “User Story Quality in Practice: A Case Study,” Software, 1(3): 223–243. DOI: 10.3390/software1030010.
ISO, 2025 “Systems and software Quality Requirements and Evaluation (SQuaRE) — Product quality model,” 2023, https://www.iso.org/standard/78176.html. Retrieved Date: 27 January 2025
K. A. Alam, H. Asif, I. Inayat, and S.-U.-R. Khan, 2024 “Automated Quality Concerns Extraction from User Stories and Acceptance Criteria for Early Architectural Decisions,” in Software Architecture. ECSA 2024. Lecture Notes in Computer Science, 14889: 359–367. Springer, Cham, DOI: 10.1007/978-3-031-70797-1_24.
S.-T. Lai, 2017 “A User Story Quality Measurement Model for Reducing Agile Software Development Risk,” International Journal of Software Engineering & Applications, 8(2): 75–86. DOI: 10.5121/ijsea.2017.8205.
E. Jharko, 2024 “Some Issues in Using the Model of Determining the User Stories Quality to Reduce Software Development Risks,” in 2024 International Conference on Industrial Engineering, Applications and Manufacturing (ICIEAM), 658–663. DOI: 10.1109/ICIEAM60818.2024.10553873.
A. Adewumi, S. Misra, and N. Omoregbe, 2015 “Evaluating Open Source Software Quality Models Against ISO 25010,” in 2015 IEEE International Conference on Computer and Information Technology; Ubiquitous Computing and Communications; Dependable, Autonomic and Secure Computing; Pervasive Intelligence and Computing, 872–877. DOI: 10.1109/CIT/IUCC/DASC/PICOM.2015.130.
E. Stephen and E. Mit, 2020 “Evaluation of Software Requirement Specification Based on IEEE 830 Quality Properties,” International Journal on Advanced Science, Engineering and Information Technology, 10(4): 1396–1402. DOI: 10.18517/ijaseit.10.4.10186.
T. Tõemets, 2025 “Analysing the Quality of User Stories in Open Source Projects,” PhD Thesis, University of Tartu, Estonia, 2020. Retrieved Date: Jan., [Online]. Available: https://hdl.handle.net/10062/93991
