Multimodality and Design of Interactive Virtual Environments for Creative Collaboration: A Social Semiotic Perspective

Developments in digital interaction technologies provide new interdisciplinary fields for creative work at the intersection of design, art and technology. These technologies also present new possibilities to integrate physical spaces with interactive layers of digital information for multi-user interaction, customization and interdisciplinary collaboration. These spaces accommodate new forms of spatial and social practices, provide multimodal communication resources in physical and virtual environments, and allow individuals (or groups) to actively engage with collaborative creative experiences. The three-dimensional representation of space and the resources for non-verbal communication enable the users to interact with the digital content in more complex yet engaging ways. However, understanding the communicative resources in virtual spaces with the theoretical tools that are conventionally used to describe physical spaces become problematic, especially when we consider the affordances and constraints for mediating spatial practices with digital tools.

This article proposes a multimodal social semiotic approach to the design of interactive virtual environments and virtually augmented physical spaces as new forms of technologically mediated collaborative spaces. It presents the findings of multiple case studies conducted by different 3D virtual environment technologies and interaction contexts, including screen-based virtual worlds and virtually- augmented physical spaces. The multimodal analytical perspective particularly emphasizes the role of audio-visual resources in co-creating representations for effective collaboration, and the socio-cultural factors in construction of meaningful virtual environments.