Quality of maps: measuring communication

How can we assess quality of the cartography of a map with regard to communicating map information to the user? How can we assess the quality of achievements and product development if we can’t measure the quality? Map producers may need to carry out quality measurements of map prototypes before release to users as it is common practice in other industries, e.g. a car manufacturer carries out ‘customer clinics’; new drugs are tested in user trials in order to test their effectiveness.
A series of tests were carried out with the purpose of developing and assessing a methodology for measuring the quality of communication on maps. The aim was to develop a standard method, which can be used to test maps, map prototypes, and other forms of geographical information, in large-scale tests with representative user groups.
The method consists of four parts: 1) tasks to be solved with the aid of a map, 2) eye tracking recording, 3) think aloud method, 4) interview after completing the tasks.
The study was based on ten subjects/users with no specialist map knowledge. These subjects were asked to answer 22 questions with the help of two real topographic maps that differed in design but covered the same area and had the same content. A video camera was used to observe the subjects and record their answers and verbalised thought processes. Eye movements were recorded using a helmet mounted eye-tracking device.
The result was a measure of the time taken to solve each task, number of words used and whether the answers were correct or not. The eye movements were used to assess the type of fixation, length and number of fixation and saccades. Marks were also given for time taken, strategy and overall performance. Correlation between quantitative factors was sought. The different measurements used to assess the subject also acted as a ‘control’ to highlight conflicts between what was said by the subject and what the subject did. Uniform/similar behaviour or statements by the subjects were also noted. Part of the study was for example to investigate the number of eye fixations between map and legend to see if one map design requires more consultation in the legend than the other design.
The results were in general remarkable. Overall answer correctness (based on 220 answers) was 60% with standard deviation 4%. Time used per question averaged 78 sec with a standard deviation 11 sec. In the markings for performance (66 marks per subject given by the researcher), the standard deviation was also 4%. Other results show the same clear, uniform tendency. The overall conclusion was that the method is useful because the performance of the subjects was uniform.