Digital video analysis of health professionals' interactions with an electronic whiteboard: A longitudinal, naturalistic study of changes to user interactions

As hospital departments continue to introduce electronic whiteboards in real clinical settings a range of human factor issues have emerged and it has become clear that there is a need for improved methods for designing and testing these systems. In this study, we employed a longitudinal and naturalistic method in the usability evaluation of an electronic whiteboard system. The goal of the evaluation was to explore the extent to which usability issues experienced by users change as they gain more experience with the system. In addition, the paper explores the use of a new approach to collection and analysis of continuous digital video recordings of naturalistic "live" user interactions. The method developed and employed in the study included recording the users' interactions with system during actual use using screen-capturing software and analyzing these recordings for usability issues. In this paper we describe and discuss both the method and the results of the evaluation. We found that the electronic whiteboard system contains system-related usability issues that did not change over time as the clinicians collectively gained more experience with the system. Furthermore, we also found user-related issues that seemed to change as the users gained more experience and we discuss the underlying reasons for these changes. We also found that the method used in the study has certain advantages over traditional usability evaluation methods, including the ability to collect analyze live user data over time. However, challenges and drawbacks to using the method (including the time taken for analysis and logistical issues in doing live recordings) should be considered before utilizing a similar approach. In conclusion we summarize our findings and call for an increased focus on longitudinal and naturalistic evaluations of health information systems and encourage others to apply and refine the method utilized in this study.