Skip to main content

Usability assessment of an electronic health record in a comprehensive dental clinic

Abstract

In this paper we present the development and usability of an electronic health record (EHR) system in a comprehensive dental clinic.The graphic user interface of the system was designed to consider the concept of cognitive ergonomics.The cognitive task analysis was used to evaluate the user interface of the EHR by identifying all sub-tasks and classifying them into mental or physical operators, and to predict task execution time required to perform the given task. We randomly selected 30 cases that had oral examinations for routine clinical care in a comprehensive dental clinic. The results were based on the analysis of 4 prototypical tasks performed by ten EHR users. The results showed that on average a user needed to go through 27 steps to complete all tasks for one case. To perform all 4 tasks of 30 cases, they spent about 91 min (independent of system response time) for data entry, of which 51.8 min were spent on more effortful mental operators. In conclusion, the user interface can be improved by reducing the percentage of mental effort required for the tasks.

Introduction

Electronic health record (EHR) is a generic term for integrated, computer-based, health information systems, accessible at the point of care. EHRs were classified on the basis of the International Organization for Standardization (ISO) definition (ISO/DTR 20514 2004). According to this definition, the EHR means a repository of patient data in digital form, stored and exchanged securely, and accessible by multiple authorized users. It contains retrospective, concurrent, and prospective information and its primary purpose is to support continuing, efficient and quality integrated health care. Some of the major advances in EHR use in health care during the past four and a half decades have dealt with relatively mundane matters such as approaches to capturing and storing information, communicating it, retrieving it, and producing and distributing reports (Mantas 2002). These capabilities have greatly reduced transcription errors, improved legibility of reports, eliminated redundancy, facilitated billing and financial functions, and provided a wide variety of other benefits, which indirectly do affect patient safety, health care quality, and efficacy (Poissant et al. 2005;Weiskopf & Weng 2013). Despite their promise, reports of EHR-related safety hazards are now emerging (Singh et al. 2011). Many currently unanswered, legal, ethical, and financial questions threaten the widespread adoption and use of EHRs. Key legal dilemmas that must be addressed in the near-term pertain to the extent of clinicians’ responsibilities for reviewing the entire computer-accessible clinical synopsis from multiple clinicians and institutions, the liabilities posed by overriding clinical decisions support warnings and alerts, and mechanisms for clinicians to publically report potential EHR safety issues (Sittig & Singh 2011).

The main thrust of information technology efforts recently has been to encourage broad adoption of EHRs, and clinical systems and applications based on the EHR that provide various kinds of functionality (Takian et al. 2012). Dentists are increasingly adopting EHRs in their practices (Zvárová et al. 2008;Cederberg & Valenza 2012;Tokede et al. 2013). Adoption rate is high among academic centers, but less so at community hospitals and in office practices. Estimates of adoption vary, depending on what one considers the benefits of an EHR are, ranging from the ability to review laboratory and radiology results or other limited functionality to that of a system that includes computerized physician order entry. Dentists are increasingly adopting EHRs, and are using structured data entry interfaces to enter data such that the data can be easily retrieved and exchanged (Walji et al. 2013). Although some studies have discussed the benefits of EHR in dentistry (Zvárová et al. 2008;Tokede et al. 2013;Hippmann et al. 2010), Poor usability is one of the major barriers against optimum usage of electronic health records (EHRs). It is common for user resistance to challenge implementation efforts. As more dental clinics undertake EHR implementations, a solid understanding of how to foster acceptance of EHR is necessary to reap the benefits of clinical error reduction, improved quality of care, and decreased healthcare costs (Jenkings & Wilson 2007). The principles and practices of human-computer interactions can be used to bolster user satisfaction and increase usability, thereby increasing the chances of being successful in the implementation of EHR (Garrard 2000).

In this paper we present the development and usability of an electronic health record (EHR) system in a comprehensive dental clinic. The graphic user interface of the system has been designed to consider the concept of cognitive ergonomics. The user interface of the EHR was analyzed using the Cognitive Task Analysis (CTA) method called “Goals, Operators, Methods, and Selection rules” (GOMS) (Saitwal et al. 2010) and an associated technique called “Keystroke Level Model” (KLM) (Chi & Chung 1996).The GOMS method was used to evaluate the user interface of the EHR by identifying all sub-tasks of a given task and classifying them into mental (internal) or physical (external) operators. KLM was used to predict task execution time required to perform the given task through the application of its standard set of operators. Our goal is to suggest areas of improvement to promote flexibility in the user interface.

Materials and methods

Electronic oral health record system

In this study, we focus on the uses of EHR in a comprehensive dental clinic of final year undergraduates. The dental students take care of the patients in the comprehensive care clinic under supervisions of the faculty members. The dental clinic EHR connects with the hospital EHR. All related medical information, e.g. past medical history, laboratory tests, can be retrieved from the hospital EHR. It integrates specifications of all aspects of oral health care, including history, signs and complaints, prescriptions and procedures for holistic oral health care. The dental section of the system is an open source that can plug in to the hospital EHR. The hospital EHR is the open source program and operates in client–server architecture. The hospital information system is able to connect to the infrastructures and networks of Community hospitals, Province hospitals and health care centers. The data is recorded in a Database Server and used in client/server applications.

Usually, existing electronic oral health records allow the practitioner to electronically document patient care, allow claims transactions to be more quickly and reliably processed as well as being able to communicate with records from other healthcare disciplines. The EHR developed in this study has additional functions. The graphic user interface of each oral health status item has been designed to consider the concept of cognitive ergonomics. Cognitive ergonomics (Hoc 2008), as defined by the International Ergonomics Association, is concerned with mental processes, such as perception, memory, reasoning, and motor response, as these processes affect the interactions among humans and other elements in a system. The graphic user interface of each oral health status item has been designed to follow steps in status recording, from chief complaint to hygiene and periodontal condition, defect and restoration. Graphics for each status item have no ambiguity and are easy to remember.

The status and intervention (SI) index and decision support has been developed to provide the basis for a completely numerical recording system that can cover all of the data on oral status treatment needs, records of planned and completed procedures, clinic organization and scheduling of patients. As for the decision support, the appropriate intervention, care provider, time, setting and cost have been provided for a given oral status which can be altered according to the dentist and patient preferences. The SI index can be used for detailed identification of the treatment needed and which conditions should be referred to the community or province hospitals for treatment. In addition, SI score is an indicator of individual or community health status and of types of intervention in holistic views. SI score can be used for identify the needs of the population to specify the tasks to be accomplished in oral health care, plan sufficient resources for the workplace and community thus allowing for effective performance without unnecessary or excess facilities. The system also enables the epidemiological evaluation of community status and the quantity, quality and effectiveness of care provided; the data can be rapidly and economically summarized, either by hand or by computer. An example of an oral health status user interface is shown in Figure 1. The diagram shows how the program works from users entering data to when they get the decision support message.

Figure 1
figure 1

An example of an oral health status and decision support user interface design after patient’s oral health status recorded in the EHR.

Usability evaluation

To evaluate the user interface of the EHR, we randomly selected 30 paper based cases that had oral examinations for routine clinical care by final year undergraduates. Ten final year undergraduates were recruited and instructed on the use of the EHR and the requirements of the oral health status recording. All participants gave their written informed consent which was approved by the Ethical Review Board of the University. The students had no experience of using EHR and familiarized themselves with the systems interface for 1 week, but they did not familiarize themselves with the task. After the familiarization process, the students recorded the oral health status of the 30 case. Inter-rater reliability was then calculated to determine agreement among students who independently conducted GOMS analysis for each task.

Analyzing a task into Goals, Operators, Methods, and Selection rules is an established method for characterizing a user’s procedural knowledge. When combined with additional theoretical mechanisms, the resulting GOMS model provides a way to quantitatively predict human learning and performance for an interface design, in addition to serving as a useful qualitative description of how the user will use a computer system to perform a task. A GOMS model is a description of the procedural knowledge that a user must have in order to carry out tasks on a device or system; it is a representation of the “how to do it” knowledge that is required by a system in order to get the intended tasks accomplished. Briefly, a GOMS model consists of descriptions of the Methods needed to accomplish specified Goals. The Methods are a series of steps consisting of Operators that the user performs. A Method may call for sub-goals to be accomplished, so the Methods have a hierarchical structure. GOMS is a usability technique that helps to identify lower level perceptual motor issues, quantify the complexity and efficiency of an interface, and evaluate the interface as a whole rather than in isolation. Goals are what the user intends to accomplish (e.g. “locate the patient”). Often goals consist of sub-goals. Operators consist of actions performed to achieve the goal (e.g. “extend hand towards mouse”). Methods are sequences of operators that accomplish a goal (Table 1). Selection rules are used to identify a method in cases where multiple methods may accomplish the same goal.

Table 1 “Locating the patient” task using GOMS along with KLM technique

GOMS was performed on a set of 4 common tasks that were identified by users of the EHR (Table 2). Two evaluators independently conducted GOMS on each of the 4 tasks. Inter-rater reliability was calculated (using SPSS: Statistical Package for the Social Sciences 16.0) to see the agreement between the total number of steps required to achieve the given task and cognitive distribution (as mental or physical operator) of those tasks.

Table 2 A GOMS analysis for 4 tasks and their subtask

The KLM model was used to estimate the time required to accomplish each of the 4 tasks. The EHR is a standard set of eight operators in KLM with their execution time estimated from experimental data (Saitwal et al. 2010). We did not take into consideration the time taken for the system to respond to the (W(t)) operator, so this analysis is essentially system-independent. In this study, four of those operators are used; Point the mouse to target on display (P), Double click (BB), Type a sequence of characters on keyboard (T) and Mental operator (M). Of these, the first three operators are exclusively physical operators and are estimated to have fixed execution times, except T operator. For T operator, we designed the study to fix all 30 cases to have the same chief complaint. It meant that two evaluators typed the same characters on the keyboard, but may have shown a different duration of execution time. Hence, their total average execution time (T) for a given task can be given by

T = i = 1 a n i t i N

wherei= 1, 2, 3, … a ; a is the number of steps on each taskst i is the time required to perform ith operator, n i is the number of times ith operator is used, and

N = i = 1 a n i

As far as forth operator – mental operator (M) is concerned, it is estimated that those operators take 1.2 s on average.

Results

As show in Table 3, the total number of steps for a given task ranged from a minimum of 3 for “Task 4 - Save the data” to a maximum of 10 for “Task3 - Document coding of tooth status”.

Table 3 Summary results of 4 prototypical tasks showing total number of steps, distribution of mental and physical operators, execution times, and inter-rater reliability

Steps for each of the 4 tasks were further classified as either mental [M] or physical [P] operators depending upon their cognitive distributions based on the GOMS classification. For 30 cases, the results show that of the total operators, 35.30 % of the steps were mental.

The second half of Table 3 estimates the total amount of time the user would take to execute each task in all 30 cases. These values reflect the time a user has spend interacting with the EHR and do not include the time in examining a patient. Further, these estimates are based on the assumption that the users are expert computer users. Execution time for the 4 tasks ranged from a low of 69 s to high of 1911.6 s. The mental operators accounted for about 56.89% of the total time.

Inter-rater reliability was also assessed between the mental/ physical operator classification results by two evaluators (see Table 3). These values range between 0.997 (substantial agreement) and 1 (almost perfect agreement) with the average of 0.9, indicating good reliability of the evaluation method.

Figure 2 shows the average time required for mental and physical operators in a given task. In particular, the average time for physical steps in each task was calculated for all 4 tasks using Eqs. Note that the average physical operator time vary between 0.36 s and 0.56 s for all tasks because of a different set of clicks and menus in those tasks. Figure 2 also plots the average time for all mental steps, which is always estimated to be 1.2 s. This figure along with the results from Table 3 demonstrates that on average mental steps accounts for 35% of total operators.

Figure 2
figure 2

The average execution times of the participants for all mental (internal) and physical (external) operators for all 4 tasks listed in Table  3 .

Discussion

EHR systems have the potential to address many limitations of paper-based records and have been described by the Institute of Medicine as an essential technology for medical care (Kassirer 1995).Dental schools have recently begun an ambitious program of converting undergraduate, graduate, and faculty clinics from paper to EHRs (Cederberg & Valenza 2012;Tokede et al. 2013). The functional requirements of EHRs in dental schools are different from those in medicine and nursing. Surveys of students, faculty, and staff before and after implementation indicated that users had mixed feelings about the system in terms of efficiency and time required compared with paper charts. As reported by Walji et al. (2009), the users found that the electronic patient record improved patient care and that they would recommend such a system to dentists starting a new practice. It is essential that the dentists understand the potential benefits of using EHRs in their practices not only for patient care but also for outcome measurements (when linked with other health and social care datasets), quality improvement, public health surveillance, and research.

Researchers in medicine, ergonomics, health economics, cognitive science, and biomedical informatics have conducted several studies on the design of EHRs (Liu et al. 2009;Collins et al. 2011;Cho et al. 2010;Blobel et al. 2003). The general approach that has been used in analyzing the basis for human performance is known as cognitive task analysis (CTA) (Militello & Hutton 1998). Its purpose is to capture the way the mind works―to capture cognition. CTA could describe the basis for skilled performance that is being studied. In using CTA, cognitive scientist try to capture what people are thinking about, what they are paying attention to, the strategies they are using in making decisions, what they are trying to accomplish, what information they discard, and what they know about the way a process works. In studying system usability, there are several possible task analysis techniques that are used to detail the interactions between users and systems. This is done to facilitate and enhanced system design, procedures, training, and support. Task analysis is ideally used when designing a system (Shachak et al. 2009). Using task analysis early in the design process is one way to integrate the study of human processes, including user capabilities and limitations, into the final product. After changes have been made based on task analysis findings, it is important to re-analyze the resulting system to ensure no unforeseen consequences developed as a result of the process. Task analysis could have assisted several authors mentioned in the literature review. The studies by Murff and Kannry (Murff & Kannry 2001) and Payne et al. (2003) investigated various ways that design affects how physicians carry out their work and what physician preferences are. The knowledge gained from these studies is in hindsight. In contrast, a thorough task analysis, by improving the design of a system, can save time and money upfront by reducing the need for revisions. Payne et al. (2003) mentioned that more research in the design stages could have prevented the creation of numerous unnecessary order sets. In this study, we used the cognitive task analysis method called “Goals, Operators, Methods, and Selection rules” and an associated technique called “Keystroke Level Model”. The results suggest that there are many opportunities to improve the efficiency of information delivery and task performance to reduce system complexity. For example, designers could try to simplify the interface and improve the organization to integrate relevant information and data using less windows and screens, or redesign some tasks and functions. Mental workload could be reduced by providing information reminders and recognition-based assistance. For example, some functions like “fill in the blank” could be replaced with “select from a droplist”. Furthermore, we can use colour to classify each main menu on the user interface to reduce mental recall, and also improve the user-friendly interface. For future work, we plan to evaluate the usability of the EHR system on behalf of the user, as in function and representation.

Conclusions

This study investigated the current user interface of the EHR in the dental student clinic using the cognitive task analysis. The study reveals that the total number of steps required for doing all given tasks. On the other hand, the study also shows that the mental operators are the main part of the total step operators. Further analysis of the execution time shows that more than half of the time is spent in performing mental operators, which can lead to mental fatigue for the users due to extended mental workload for long periods of time. The user interface can be improved by reducing the percentage of mental effort required for the tasks.

References

  • Blobel B, Stassinopoulos G, Pharow P: Model-based design and implementation of secure, interoperable EHR systems. AMIA Annu Symp Proc 2003, 96-100.

    Google Scholar 

  • Cederberg RA, Valenza JA: Ethics and the electronic health record in dental school clinics. J Dent Educ 2012, 76: 584-589.

    Google Scholar 

  • Chi CF, Chung KL: Task analysis for computer-aided design (CAD) at a keystroke level. Appl Ergon 1996, 27: 255-265. 10.1016/0003-6870(96)00012-9

    Article  Google Scholar 

  • Cho I, Kim J, Kim JH, Kim HY, Kim Y: Design and implementation of a standards-based interoperable clinical decision support architecture in the context of the Korean EHR. Int J Med Inform 2010, 79: 611-622. 10.1016/j.ijmedinf.2010.06.002

    Article  Google Scholar 

  • Collins SA, Bakken S, Vawdrey DK, Coiera E, Currie L: Model development for EHR interdisciplinary information exchange of ICU common goals. Int J Med Inform 2011, 80: e141-149. 10.1016/j.ijmedinf.2010.09.009

    Article  Google Scholar 

  • Garrard CS: Human-computer interactions: can computers improve the way doctors work? Schweiz Med Wochenschr 2000, 130: 1557-1563.

    Google Scholar 

  • Hippmann R, Dostálová T, Zvárová J: Voice-supported electronic health record for temporomandibular joint disorders. Methods Inf Med 2010, 49: 168-172.

    Article  Google Scholar 

  • Hoc JM: Cognitive ergonomics: a multidisciplinary venture. Ergonomics 2008, 51: 71-5. 10.1080/00140130701801124

    Article  Google Scholar 

  • ISO/DTR 20514: Health Informatics – Electronic Health Record – Definition, Scope, and Context. 2004. http://www.iso.org/iso/home/store/catalogue_tc/catalogue_detail.htm?csnumber=39525

    Google Scholar 

  • Jenkings KN, Wilson RG: The challenge of electronic health records (EHRs) design and implementation: responses of health workers to drawing a ‘big and rich picture’ of a future EHR programme using animated tools. Inform Prim Care 2007, 15: 93-101.

    Google Scholar 

  • Kassirer JP: The next transformation in the delivery of health care. N Engl J Med 1995, 332: 52-54. 10.1056/NEJM199501053320110

    Article  Google Scholar 

  • Liu H, Hou XQ, Hu G, Li J, Ding YQ: Development of an EHR system for sharing - a semantic perspective. Stud Health Technol Inform 2009, 150: 113-117.

    Google Scholar 

  • Mantas J: Electronic health record. Stud Health Technol Inform 2002, 65: 250-257.

    Google Scholar 

  • Militello LG, Hutton RJ: Applied cognitive task analysis (ACTA): a practitioner’s toolkit for understanding cognitive task demands. Ergonomics 1998, 41: 1618-1641. 10.1080/001401398186108

    Article  Google Scholar 

  • Murff HJ, Kannry J: Physician satisfaction with two order entry systems. J Am Med Inform Assoc 2001, 8: 499-509. 10.1136/jamia.2001.0080499

    Article  Google Scholar 

  • Payne TH, Hoey PJ, Nichol P, Lovis C: Preparation and use of preconstructed orders, order sets, and order menus in a computerized provider order entry system. J Am Med Inform Assoc 2003, 10: 322-329. 10.1197/jamia.M1090

    Article  Google Scholar 

  • Poissant L, Pereira J, Tamblyn R, Kawasumi Y: The impact of electronic health records on time efficiency of physicians and nurses: a systematic review. J Am Med Inform Assoc 2005, 12: 505-516. 10.1197/jamia.M1700

    Article  Google Scholar 

  • Saitwal H, Feng X, Walji M, Patel V, Zhang J: Assessing performance of an Electronic Health Record (EHR) using Cognitive Task Analysis. Int J Med Inform 2010, 79: 501-506. 10.1016/j.ijmedinf.2010.04.001

    Article  Google Scholar 

  • Shachak A, Hadas-Dayagi M, Ziv A, Reis S: Primary care physicians’ use of an electronic medical record system: a cognitive task analysis. J Gen Intern Med 2009, 24: 341-348. 10.1007/s11606-008-0892-6

    Article  Google Scholar 

  • Singh H, Classen DC, Sittig DF: Creating an oversight infrastructure for electronic health record-related patient safety hazards. J Patient Saf 2011, 7: 169-174. 10.1097/PTS.0b013e31823d8df0

    Article  Google Scholar 

  • Sittig DF, Singh H: Legal, ethical, and financial dilemmas in electronic health record adoption and use. Pediatrics 2011, 127: e1042-1047. 10.1542/peds.2010-2184

    Article  Google Scholar 

  • Takian A, Petrakaki D, Cornford T: Building a house on shifting sand: methodological considerations when evaluating the implementation and adoption of national electronic health record systems. BMC Health Serv Res 2012, 30: 105.

    Article  Google Scholar 

  • Tokede O, Walji M, Ramoni R: Treatment planning in dentistry using an electronic health record: implications for undergraduate education. Eur J Dent Educ 2013, 17: e34-e43. 10.1111/j.1600-0579.2012.00759.x

    Article  Google Scholar 

  • Walji MF, Taylor D, Langabeer JR 2nd, Valenza JA: Factors influencing implementation and outcomes of a dental electronic patient record system. J Dent Educ 2009, 73: 589-600.

    Google Scholar 

  • Walji MF, Kalenderian E, Tran D, Kookal KK, Nguyen V, Tokede O, White JM, Vaderhobli R, Ramoni R, Stark PC, Kimmes NS, Schoonheim-Klein ME, Patel VL: Detection and characterization of usability problems in structured data entry interfaces in dentistry. Int J Med Inform 2013, 82: 128-138. 10.1016/j.ijmedinf.2012.05.018

    Article  Google Scholar 

  • Weiskopf NG, Weng C: Methods and dimensions of electronic health record data quality assessment: enabling reuse for clinical research. J Am Med Inform Assoc 2013, 20: 144-151. 10.1136/amiajnl-2011-000681

    Article  Google Scholar 

  • Zvárová J, Dostálová T, Hanzlícek P: Electronic health record for forensic dentistry. Methods Inf Med 2008, 47: 8-13.

    Google Scholar 

Download references

Author information

Authors and Affiliations

Authors

Corresponding author

Correspondence to Siriwan Suebnukarn.

Additional information

Competing interest

The authors declare that they have no competing of interest.

Authors’ contributions

SS analysed and designed the electronic health record program, PP impremented the program, PR, BT, KB, MW participated in the evaluation process. All authors read and approved the final manuscript.

Authors’ original submitted files for images

Below are the links to the authors’ original submitted files for images.

Authors’ original file for figure 1

Authors’ original file for figure 2

Rights and permissions

Open Access This article is distributed under the terms of the Creative Commons Attribution 2.0 International License (https://creativecommons.org/licenses/by/2.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.

Reprints and permissions

About this article

Cite this article

Suebnukarn, S., Rittipakorn, P., Thongyoi, B. et al. Usability assessment of an electronic health record in a comprehensive dental clinic. SpringerPlus 2, 220 (2013). https://doi.org/10.1186/2193-1801-2-220

Download citation

  • Received:

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1186/2193-1801-2-220

Keywords