Course description
Language use is inherently multimodal as it involves not only (para) verbal expressions, but also images, sounds, gestures, gaze, posture, bodily and sensorimotor actions, objects etc. as semiotic resources in the process of meaning making (Feyaerts et al. 2016). This course focuses on the multimodal nature of communication with a specific emphasis on face-to-face interaction. It comprises both a theoretical and a practical part.
The theoretical part provides an overview of the major definitions and research lines involving the concept of multimodality. A large part of this overview, first, deals with onsets focusing on multiple types of text-image-sound interplay as apparent in different formats and genres, whereas in the context of this course, second, most attention will be devoted to the analysis of different semiotic resources in face-to-face interaction. Following a brief historic and cultural overview featuring writings and artistic work by early predecessors of modern gesture studies (Quintilian (Institutionis oratoriae), Bonifacio, Bulwer, De Lairesse, Vico, Condillac, Diderot, Efron), the course will then zoom in on modern gesture analysis following Calbris (2011: 6) in her definition of gesture as ‘the visible movement of any body part consciously or unconsciously made with the intention of communicating while speech is being produced’. Within this view, we will discuss both formal and functional aspects of co-speech gestures dealing with such formal descriptive features as hand shape, orientation of the hand, movement, position in gesture space (Bressem 2013). For the analysis of functional gesture types, which is the major topic of the theoretical part, we adopt the typology described by McNeill (1992, 2005) and which consists of six mutually non-exclusive gesture types: symbolic, iconic, metaphoric, pointing and pragmatic gestures as well as beats. Every single type, along with complex gesture clusters, will be analyzed and discussed on the basis of video excerpts from different types of interaction. Along with gestures, we will also look into other social resources such as gaze, posture, materialities (such as clothing, face masks), spatial positioning etc., which make up the multimodal complexity of the process of interactional meaning making.

Throughout this theoretical part, much attention will be paid to two communicative phenomena, which are of major local and actual importance for the analysis of multimodal interaction:
-The use of gestures and other resources in (intercultural) interactions among different social groups in an urban (Venetian) context: tourists and local (Italian) residents, buyers and sellers on the market place, authorities (policemen, …) and citizens, etc.
-The impact of the corona pandemic and its socially restrictive health measures (distancing, lockdown, face masks etc.) on the multimodal realization of urban communication. We will analyze commonalities and differences in the ways, in which different cities, societies, religions and cultures implemented and elaborated the restrictive communicative guidelines that were issued to tackle the outspread off the covid-19 virus.

In the practical part of this course, students will learn to identify and analyze multimodal aspects of various types of interaction (presented in pictures, sound files or videos).
Students will also learn how to use the ELAN editing tool (Sloetjes & Wittenburg 2008; freeware developed at the Max Planck Institute Nijmegen) for adequately and flexibly annotating and analyzing video and audio data. For that purpose, brief video recordings will be provided in the e-learning platform. Students are also required to make two or three brief smartphone recordings (backed by informed consent) of spontaneous face-to-face interactions among people in the urban (Venetian) public space. Students not staying in Venice and participating online are required to make recordings in an urban setting at home. The analytical focus of the recordings are the co-speech gestures (taken in the broadest sense), which are being used.

The interaction in the recorded clips will first be discussed in two types of individual feedback sessions. Students discuss their recordings among each other, but also with the professor. In the final weeks of the semester, all individual recordings are presented and discussed in the plenary group. Finally, every student hands in an empirical report, which combines an analysis in ELAN with a concisely written report (max. 3000 words) elaborating the major empirical findings along with relevant theoretical and methodological issues.
Detailed information, guidelines and useful materials will be available during the semester in the e-learning platform.

Covid-19 measure: course in hybrid format
Due to the covid-19 pandemic, this course takes the form of a hybrid course making it accessible to all students regardless their place of residence (on the VIU-campus or elsewhere). The course materials are all digital (powerpoint presentations, pictures, videos, reading materials, etc.), which will be made available in the e-learning platform. In line with the general VIU-guidelines, this course will be given online during the first two weeks of the semester. After that, each session of the course will be given simultaneously on the VIU-campus in Venice and streamed via Zoom. Every session will be recorded so anyone can listen to it again afterwards. With regard to assignments and empirical analyzes, the course offers equivalent alternatives for Venice-specific assignments for students who take the course online. Presentations, discussions and feedback sessions can be done either on-campus or online.

Learning outcomes
- Students gain awareness that both production and interpretation of language are no inherent processes, in which meaning is confined to verbal forms alone. Language is multimodal in the sense that next to verbal, also embodied and all sorts of locally determined material resources take active part in the interactive process of meaning making.
- Students acquire knowledge and skills about co-speech gestures as an important resource for optimizing their (intercultural) communication abilities.
- Students become aware of both the systematicity and the dynamics of gesturing in order to overcome language- and culture-bound difficulties in interaction.
- Students learn to profitably use the ELAN-editing tool as a standard open-access software tool for collecting, annotating and analyzing multimodal interaction data.

Teaching methods
As far as teaching is concerned, this course will mix both theoretical and practical methods, with a clear increase of the latter towards the end of the semester. During the first weeks, students mainly prepare classes by doing preparatory reading. Students (alone or pairwise) will also be asked to prepare annotation exercises and case studies, some of which by using the ELAN editing tool. In the second half of the semester, students are required to make brief smartphone recordings annex ELAN-analysis of a few face-to-face interactions in the (Venetian) public urban space. In the final weeks, every student gives a presentation of their findings to the seminar. With regard to the optimization of this assignment, intermediate individual feedback sessions about the use of ELAN as well as the annotation and analysis of the data will be organized.

Evaluation methods
- Attendance and participation (10%)
- Preparation of exercise materials (multimodal aspects of interactions) (20%)
- Individual feedback & discussion session (10%)
- Oral presentation of recorded interactions (20%)
- Empirical report (40%)

Bibliography
Bateman, J., J. Wildfeuer & T. Hiippala. (2017). Multimodality. Part I: Working your way into ‘Multimodality’. Mouton De Gruyter.
Bressem, J. (2013). A linguistic perspective on the notation of form features in gestures. In C. Müller, A. Cienki, E. Fricke, S.H. Ladewig, D. McNeill & S. Teßendorf (Red.), Body-Language-Communication: An International Handbook on Multimodality in Human Interaction. Berlin, Boston: Mouton De Gruyter, 1079-1098.
Bressem, J. & C. Müller (2014). A repertoire of recurrent gestures of German In C. Müller, A. Cienki, E. Fricke, S.H. Ladewig, D. McNeill & J. Bressem (Red.), Body-Language-Communication: An International Handbook on Multimodality in Human Interaction. Berlin, Boston: Mouton De Gruyter, 1575-1591.
Cienki, A. (1998). Metaphoric gestures and some of their relations to verbal metaphoric expressions. In J.-P. Koenig (Red.), Discourse and Cognition: Bridging the Gap. Stanford, CA: Center for the Study of Language and Information, 189-204.
Feyaerts K., G. Brône & B. Oben (2017). Multimodality in Interaction. In B. Dancygier. (Red.), The Cambridge Handbook of Cognitive Linguistics. Cambridge: Cambridge University Press, 135-156.
Jewitt, C., J. Bezemer & K. O’Halloran. (2016). Introducing Multimodality. London: Routledge.
Kendon, A. (1980) Gesture and speech: two aspects of the process of utterance. In M.R. Key (Red.), Nonverbal Communication and Language. The Hague: Mouton, 207-227.
Kendon, A. (2004). Gesture. Visible Action as Utterance. Cambridge: Cambridge University Press.
Ladewig, S.H. (2014). Recurrent gestures. In C. Müller, A. Cienki, E. Fricke, S.H. Ladewig, D. McNeill & J. Bressem (Red.), Body-Language-Communication: An International Hand- book on Multimodality in Human Interaction. Berlin, Boston: Mouton De Gruyter, 1558- 1575.
McNeill, D. (1992). Hand and Mind: What Gestures Reveal about Thought. Chicago: University of Chicago Press.
McNeill, D. (2005). Gesture and Thought. Chicago: University of Chicago Press.
Müller, C. (2013). Gestures and speech from a linguistic perspective: a new field and its history. In C. Müller, A. Cienki, E. Fricke, S.H. Ladewig, D. McNeill & S. Teßendorf (Red.), Body—Language—Communication. Berlin: Mouton De Gruyter, 55-81.
Sloetjes, H., & Wittenburg, P. (2008). Annotation by category - ELAN and ISO DCR. In: Proceedings of the 6th International Conference on Language Resources and Evaluation (LREC 2008). Max Planck Institute for Psycholinguistics, The Language Archive, Nijmegen, The Netherlands. undefined