Crowdsourcing Based Evaluation of Ranking Approaches for Spatial Keyword Querying

Spatial keyword querying has attracted considerable research efforts in the past few years. A prototypical query takes a location and keywords as arguments and returns the k objects that score the highest according to a ranking function. While different scoring functions have been used, how to compare different ranking functions for spatial keyword querying still remains an open question with little investigation. We propose a crowdsourcing-based approach to evaluate and compare ranking functions for spatial keyword search. Given two ranking functions f1 and f2, we use a matrix to model all possible binary questions regarding the different results produced by f1 and f2. We propose a multi-step process to reduce the number of binary questions, identifying the most important questions to ask. Further, we design a crowdsourcing model that obtains the answers to those important binary questions from crowd workers. We also devise a global evaluation process that is able to quantitatively compare f1 and f2 based on a multitude of answers received. According to the results of empirical studies using real data, the proposed approach is efficient and able to draw reliable conclusions in comparing ranking functions for spatial keyword search.