A Combination of Three Scales for Measuring User-Perceived Usability of a Clinical Information System: Which Approach Produces the Most Informative Results?

Shahabedin Rahmatizadeh, Jurek Kirakowski, Saeideh Valizadeh-Haghi, Maryam Taheri, Sanaz Tavasoli



Introduction: For the first time in Iran, a kidney stone clinical information system (CIS) has been designed to manage and calculate and visualize patients’ kidney stone risk profiles. Nevertheless, the usability of this system has not been evaluated yet. Medically, this study aims to evaluate the user-perceived usability of the kidney stone CIS. Technically, the current study aims to determine which user-perceived usability testing approach produces the most informative results about the usability of this system.

Material and Methods: Three questionnaires, including system usability scale (SUS), software usability measurement inventory (SUMI), and post-study system usability questionnaire, were applied to carry out the study. A total of 15 users of the kidney stone CIS participated.

Results: The findings revealed that the system is of medium usability. Moreover, of the three methods used, the SUMI echoes the comments of the end-users. Despite the medium usability of the system, it was comprehensive in terms of proper data collection and storage, as well as reporting. The interface design, the lack of appropriate guidance, the time-consuming data entry, and the slow reporting system were aspects needed improvement.

Conclusion: Using a combination of tools is recommended for usability evaluation. Since there is much space in the Global and each sub-scale, whereby measures may improve with SUMI, this research recommends its use for future evaluation of the kidney stone CIS.


Clinical Information System; User-Perceived Usability; Medical Informatics Applications; Usability Evaluation; Health Information Systems;


Nabovati E, Vakili Arki H, Eslami S, Khajouei R. Usability evaluation of laboratory and radiology information systems integrated into a hospital information system. J Med Syst. 2014; 38(4): 35. PMID: 24682671 DOI: 10.1007/s10916-014-0035-z

Kaushal R, Shojania KG, Bates DW. Effects of computerized physician order entry and clinical decision support systems on medication safety. Arch Intern Med. 2003; 163(12): 1409-16. PMID: 12824090 DOI: 10.1001/archinte.163.12.1409

Su K-W, Liu C-L. A mobile nursing information system based on human-computer interaction design for improving quality of nursing. J Med Syst. 2012; 36(3): 1139-53. PMID: 20827569 DOI: 10.1007/s10916-010-9576-y

Kern R, Haase R, Eisele JC, Thomas K, Ziemssen T. Designing an electronic patient management system for multiple sclerosis: Building a next generation multiple sclerosis documentation system. Interact J Med Res. 2016; 5(1): e2. PMID: 26746977 DOI: 10.2196/ijmr.4549

Price M, Yuen EK, Goetter EM, Herbert JD, Forman EM, Acierno R, et al. mHealth: A mechanism to deliver more accessible, more effective mental health care. Clin Psychol Psychother. 2014; 21(5): 427-36. PMID: 23918764 DOI: 10.1002/cpp.1855

Khajouei R, Jaspers MWM. The impact of CPOE medication systems’ design aspects on usability, workflow and medication orders: A systematic review. Methods Inf Med. 2010; 49(1): 3-19. PMID: 19582333 DOI: 10.3414/ME0630

Thyvalikakath TP, Monaco V, Thambuganipalle HB, Schleyer T. A usability evaluation of four commercial dental computer-based patient record systems. J Am Dent Assoc. 2008; 139(12): 1632-42. PMID: 19047669 DOI: 10.14219/jada.archive.2008.0105

Kushniruk AW, Triola MM, Borycki EM, Stein B, Kannry JL. Technology induced error and usability: The relationship between usability problems and prescription errors when using a handheld application. Int J Med Inform. 2005; 74(7-8): 519-26. PMID: 16043081 DOI: 10.1016/j.ijmedinf.2005.01.003

Khajouei R, Peek N, Wierenga PC, Kersten MJ, Jaspers MWM. Effect of predefined order sets and usability problems on efficiency of computerized medication ordering. Int J Med Inform. 2010; 79(10): 690-8. PMID: 20833104 DOI: 10.1016/j.ijmedinf.2010.08.001

Horsky J, Kaufman DR, Patel VL. The cognitive complexity of a provider order entry interface. AMIA Annu Symp Proc. 2003; 2003: 294-8. PMID: 14728181 PMCID: PMC1480200

Lewis JR. Usability testing. In: Salvendy G (Editor). Handbook of human factors and ergonomics. 4th ed. John Wiley & Sons, Inc.; 2012.

Issa T, Turk A. Applying usability and HCI principles in developing marketing websites. International Journal of Computer Information Systems and Industrial Management Applications. 2012; 4: 76–82.

Shah SGS, Robinson I. Benefits of and barriers to involving users in medical device technology development and evaluation. Int J Technol Assess Health Care. 2007; 23(1): 131-7. PMID: 17234027 DOI: 10.1017/S0266462307051677

Koppel R, Metlay JP, Cohen A, Abaluck B, Localio AR, Kimmel SE, et al. Role of computerized physician order entry systems in facilitating medication errors. JAMA. 2005; 293(10): 1197-203. PMID: 15755942 DOI: 10.1001/jama.293.10.1197

Jaspers MWM. A comparison of usability methods for testing interactive health technologies: Methodological aspects and empirical evidence. Int J Med Inform. 2009; 78(5): 340-53. PMID: 19046928 DOI: 10.1016/j.ijmedinf.2008.10.002

Freire LL, Arezes PM, Campos JC. A literature review about usability evaluation methods for e-learning platforms. Work. 2012; 41(Suppl 1): 1038-44. PMID: 22316857 DOI: 10.3233/WOR-2012-0281-1038

Lewis JR. Usability: Lessons Learned. and Yet to Be Learned. International Journal of Human–Computer Interaction. 2014; 30: 663–84.

Brooke J. SUS: A quick and dirty usability scale. In: Jordan PW, Thomas B, McClelland IL, Weerdmeester B (Eds). Usability evaluation in industry. CRC Press; 1996.

Appleton KM, Bray J, Price S, Liebchen G, Jiang N, Mavridis I, et al. A mobile phone app for the provision of personalized food-based information in an eating-out situation: Development and initial evaluation. JMIR Form Res. 2019; 3(4): e12966. PMID: 31682575 DOI: 10.2196/12966

Wellard-Cole L, Potter M, Jung JJ, Chen J, Kay J, Allman-Farinelli M. A tool to measure young adults’ food intake: Design and development of an Australian database of foods for the eat and track smartphone app. JMIR Mhealth Uhealth. 2018; 6(11): e12136. PMID: 30404768 DOI: 10.2196/12136

Kirakowski J, Corbett M. SUMI: The software usability measurement inventory. British Journal of Educational Technology. 1993; 24: 210–2.

ISO. Ergonomic requirements for office work with visual display terminals (VDTs), Part 11: Guidance on usability specification and measures [Internet]. 1998 [cited: 28 Sep 2023]. Available from: https://www.iso.org/standard/16883.html

Kulkarni R, Padmanabham P, Sagare V, Maheshwari V. Usability evaluation of PS using SUMI (Software Usability Measurement Inventory). International Conference on Advances in Computing, Communications and Informatics. IEEE; 2013.

van der Weegen S, Verwey R, Tange HJ, Spreeuwenberg MD, de Witte LP. Usability testing of a monitoring and feedback tool to stimulate physical activity. Patient Prefer Adherence. 2014; 8: 311-22. PMID: 24669188 DOI: 10.2147/PPA.S57961

Hirasawa H, Murata H, Mayama C, Araie M, Asaoka R. Evaluation of various machine learning methods to predict vision-related quality of life from visual field data and visual acuity in patients with glaucoma. Br J Ophthalmol. 2014; 98(9): 1230-5. PMID: 24795333 DOI: 10.1136/bjophthalmol-2013-304319

Sauro J, Lewis JR. Quantifying the user experience: Practical statistics for user research. Morgan Kaufmann; 2012.

Señor IC, Fernández-Alemán JL, Toval A. Are personal health records safe? A review of free web-accessible personal health record privacy policies. J Med Internet Res. 2012; 14(4): e114. PMID: 22917868 DOI: 10.2196/jmir.1904

Tullis TS, Stetson JN. A comparison of questionnaires for assessing website usability. UPA Presentation; 2004.

Kirakowski J. What is SUMI? [Internet]. 1999 [cited: 17 Nov 2020]. Available from: http://sumi.uxp.ie/about/whatis.html

System Usability Scale (SUS) [Internet]. 1999 [cited: 12 Jul 2019]. Available from: https://www.usability.gov/how-to-and-tools/methods/system-usability-scale.html

De Nunzio C, Lombardo R, Tema G, Cancrini F, Russo GI, Chacon R, et al. Mobile phone apps for the prediction of prostate cancer: External validation of the Coral and Rotterdam apps. Eur J Surg Oncol. 2019; 45(3): 471-6. PMID: 30343993 DOI: 10.1016/j.ejso.2018.09.018

Pereira-Azevedo N, Osório L, Fraga A, Roobol MJ. Rotterdam prostate cancer risk calculator: Development and usability testing of the mobile phone app. JMIR Cancer. 2017; 3(1): e1. PMID: 28410180 DOI: 10.2196/cancer.6750

Sandhu H, Wilson K, Reed N, Mihailidis A. A mobile phone app for the self-management of pediatric concussion: Development and usability testing. JMIR Hum Factors. 2019; 6(2): e12135. PMID: 31152527 DOI: 10.2196/12135

Dowding D, Merrill JA, Barrón Y, Onorato N, Jonas K, Russell D. Usability evaluation of a dashboard for home care nurses. Comput Inform Nurs. 2019; 37(1): 11-9. PMID: 30394879 DOI: 10.1097/CIN.0000000000000484

Yuan M, Powell G, Lavigne M, Okhmatovskaia A, Buckeridge DL. Initial usability evaluation of a knowledge-based population health information system: The population health record (PopHR). AMIA Annu Symp Proc. 2018; 2017: 1878-84. PMID: 29854259

Ferreira JJ, Godinho C, Santos AT, Domingos J, Abreu D, Lobo R, et al. Quantitative home-based assessment of Parkinson’s symptoms: The SENSE-PARK feasibility and usability study. BMC Neurol. 2015; 15: 89. PMID: 26059091 DOI: 10.1186/s12883-015-0343-z

Pressman RS. Software engineering: A practitioner’s approach. 7th ed. ‎ McGraw-Hill; 2009.

Danino N. Human-computer interaction and your site [Internet]. 2001 [cited: 15 Jun 2019]. Available from: https://www.sitepoint.com/computer-interaction-site/

Law ELC, Hvannberg E, Cockton G (Eds). Maturing usability: Quality in software, interaction and value. Springer; 2008.

Lewis JR. Using the PSSUQ and CSUQ: In user experience research and practice. MeasuringU Press; 2019.

DOI: https://doi.org/10.30699/fhi.v13i0.569


  • There are currently no refbacks.