Interpreting Survey Findings: Can Survey Results Be Compared across Organizations and Countries?

With the rise in worldwide efforts to understand public administration by surveying civil servants, issues of survey question comparability become paramount Surveys can rarely be understood in a void but rather require benchmarks and points of reference However, it is not clear whether survey questions, even when phrased and structured in the same manner, measure the same concepts in the same way and, therefore, can be compared For multiple reasons, including work environment, adaptive expectations, and cultural factors, different people might understand the same question in different
ways and adjust their answers accordingly This might make survey results incomparable, not only across countries but also across different groups of civil servants within a national public administration This chapter uses results from seven public service surveys from across Europe, Latin America, and South Asia to investigate the extent to which the same survey questions measure the same concepts similarly—that is, are measurement invariant—using as an example questions related to transformational leadership To ascertain measurement invariance, models of a hypothesized relationship between questions measuring transformational leadership are compared across countries, as well as along
gender, educational, and organizational lines within countries Solid evidence of metric invariance and tentative evidence of scalar invariance is found in cross-country comparisons Moreover, factor loadings can be judged equal (metric invariance) across gender, education level, and organization in most countries, as can latent factor means (scalar invariance) Our results suggest that groups of public servants within countries delineated, for instance, by gender, education, or organization—can typically be benchmarked without invariance concerns Across countries, evidence for valid benchmarking that is, scalar invariance is strongest for countries in similar regions and at similar income levels It is
weaker—though still suggestive—when comparing all countries in the sample Our chapter concludes that less culturally contingent concepts may be plausibly benchmarked with care across countries.