What you get is what you see

Revisiting the evaluator effect in usability tests

Morten Hertzum, Rolf Molich, Niels Ebbe Jacobsen

    Publikation: Bidrag til tidsskriftTidsskriftartikelForskningpeer review

    Resumé

    Usability evaluation is essential to user-centred design; yet, evaluators who analyse the same usability test sessions have been found to identify substantially different sets of usability problems. We revisit this evaluator effect by having 19 experienced usability professionals analyse video-recorded test sessions with five users. Nine participants analysed moderated sessions; 10 participants analysed unmoderated sessions. For the moderated sessions, participants reported an average of 33% of the problems reported by all nine of these participants and 50% of the subset of problems reported as critical or serious by at least one participant. For the unmoderated sessions, the percentages were 32% and 40%. Thus, the evaluator effect was similar for moderated and unmoderated sessions, and it was substantial for the full set of problems and still present for the most severe problems. In addition, participants disagreed in their severity ratings. As much as 24% (moderated) and 30% (unmoderated) of the problems reported by multiple participants were rated as critical by one participant and minor by another. The majority of the participants perceived an evaluator effect when merging their individual findings into group evaluations. We discuss reasons for the evaluator effect and recommend ways of managing it
    OriginalsprogEngelsk
    TidsskriftBehaviour and Information Technology
    Vol/bind33
    Udgave nummer2
    Sider (fra-til)143-161
    ISSN0144-929X
    DOI
    StatusUdgivet - 2014

    Citer dette

    Hertzum, Morten ; Molich, Rolf ; Jacobsen, Niels Ebbe. / What you get is what you see : Revisiting the evaluator effect in usability tests. I: Behaviour and Information Technology. 2014 ; Bind 33, Nr. 2. s. 143-161.
    @article{b5026e97e2ea43ffbefb0884b8b95a5a,
    title = "What you get is what you see: Revisiting the evaluator effect in usability tests",
    abstract = "Usability evaluation is essential to user-centred design; yet, evaluators who analyse the same usability test sessions have been found to identify substantially different sets of usability problems. We revisit this evaluator effect by having 19 experienced usability professionals analyse video-recorded test sessions with five users. Nine participants analysed moderated sessions; 10 participants analysed unmoderated sessions. For the moderated sessions, participants reported an average of 33{\%} of the problems reported by all nine of these participants and 50{\%} of the subset of problems reported as critical or serious by at least one participant. For the unmoderated sessions, the percentages were 32{\%} and 40{\%}. Thus, the evaluator effect was similar for moderated and unmoderated sessions, and it was substantial for the full set of problems and still present for the most severe problems. In addition, participants disagreed in their severity ratings. As much as 24{\%} (moderated) and 30{\%} (unmoderated) of the problems reported by multiple participants were rated as critical by one participant and minor by another. The majority of the participants perceived an evaluator effect when merging their individual findings into group evaluations. We discuss reasons for the evaluator effect and recommend ways of managing it",
    author = "Morten Hertzum and Rolf Molich and Jacobsen, {Niels Ebbe}",
    year = "2014",
    doi = "10.1080/0144929X.2013.783114",
    language = "English",
    volume = "33",
    pages = "143--161",
    journal = "Behaviour and Information Technology",
    issn = "0144-929X",
    publisher = "Taylor & Francis",
    number = "2",

    }

    What you get is what you see : Revisiting the evaluator effect in usability tests. / Hertzum, Morten; Molich, Rolf; Jacobsen, Niels Ebbe.

    I: Behaviour and Information Technology, Bind 33, Nr. 2, 2014, s. 143-161.

    Publikation: Bidrag til tidsskriftTidsskriftartikelForskningpeer review

    TY - JOUR

    T1 - What you get is what you see

    T2 - Revisiting the evaluator effect in usability tests

    AU - Hertzum, Morten

    AU - Molich, Rolf

    AU - Jacobsen, Niels Ebbe

    PY - 2014

    Y1 - 2014

    N2 - Usability evaluation is essential to user-centred design; yet, evaluators who analyse the same usability test sessions have been found to identify substantially different sets of usability problems. We revisit this evaluator effect by having 19 experienced usability professionals analyse video-recorded test sessions with five users. Nine participants analysed moderated sessions; 10 participants analysed unmoderated sessions. For the moderated sessions, participants reported an average of 33% of the problems reported by all nine of these participants and 50% of the subset of problems reported as critical or serious by at least one participant. For the unmoderated sessions, the percentages were 32% and 40%. Thus, the evaluator effect was similar for moderated and unmoderated sessions, and it was substantial for the full set of problems and still present for the most severe problems. In addition, participants disagreed in their severity ratings. As much as 24% (moderated) and 30% (unmoderated) of the problems reported by multiple participants were rated as critical by one participant and minor by another. The majority of the participants perceived an evaluator effect when merging their individual findings into group evaluations. We discuss reasons for the evaluator effect and recommend ways of managing it

    AB - Usability evaluation is essential to user-centred design; yet, evaluators who analyse the same usability test sessions have been found to identify substantially different sets of usability problems. We revisit this evaluator effect by having 19 experienced usability professionals analyse video-recorded test sessions with five users. Nine participants analysed moderated sessions; 10 participants analysed unmoderated sessions. For the moderated sessions, participants reported an average of 33% of the problems reported by all nine of these participants and 50% of the subset of problems reported as critical or serious by at least one participant. For the unmoderated sessions, the percentages were 32% and 40%. Thus, the evaluator effect was similar for moderated and unmoderated sessions, and it was substantial for the full set of problems and still present for the most severe problems. In addition, participants disagreed in their severity ratings. As much as 24% (moderated) and 30% (unmoderated) of the problems reported by multiple participants were rated as critical by one participant and minor by another. The majority of the participants perceived an evaluator effect when merging their individual findings into group evaluations. We discuss reasons for the evaluator effect and recommend ways of managing it

    U2 - 10.1080/0144929X.2013.783114

    DO - 10.1080/0144929X.2013.783114

    M3 - Journal article

    VL - 33

    SP - 143

    EP - 161

    JO - Behaviour and Information Technology

    JF - Behaviour and Information Technology

    SN - 0144-929X

    IS - 2

    ER -