Understanding Preference: A Meta-Analysis of User Studies

A user’s preference for one system over another is probably the most basic user experience (UX) measure, yet user studies often focus on performance and treat preference as supplementary. This meta-analysis of 144 studies shows that while users in general prefer systems with which they achieve lower task time and error rate, they more consistently and more strongly prefer systems that impose lower workload. In only 2% of the studies a preferred system imposes significantly higher workload than a nonpreferred system. Across the studies, a stronger preference coincides with a larger difference in workload, task time, and error rate. This correlation is strongest for workload, lower for task time, and lowest for error rate. That is, workload is a stronger predictor of preference than performance is, even for the near exclusively utilitarian tasks covered by this meta-analysis. The implications of these findings include that workload should be more fully integrated in research on usability, UX, and design and that it is risky for practitioners to infer preference from performance, or vice versa.