Space and Time in Our Hands
This dissertation is about the relationships that connect verbal language to the typical hand movements that accompany speech. The two communicative channels of speech and hand gestures present a number of striking regular alignment patterns. The thesis focuses in particular on two forms of alignment: the alignment between the prosodic contour of speech and the observable kinetic efforts of gestures, and the regular `alignment’ (or better compatibility) between the information conveyed by speech and the one expressed with gestures, with a particular focus on iconic gestures.
The alignment between prosodic contour and hands movements is investigated with two experiments. The experiments show that people are naturally sensitive to small variations (offsets of 250 milliseconds) in the relative placement of gestures and prosodic peaks. The experiments also show that prosody alone is a reliable predictor of the preferred temporal location of gestures with respect to speech.
The second type of alignment is analyzed in the framework of a theory of meaning representation in iconic gestures. The main hypothesis is that the meaning of this class of gestures can be modeled as a form of information dependent spatial similarity between the virtual space depicted by the gesture and an abstract representation of the spatial extensions of the discourse referents introduced by verbal language. The hypothesis is precisely expressed in the form of a spatial logic equipped with a notion of model similarity. This logic forms the basis of a computational model of gesture generation which is in turn used in a set of experiments that confirm the original hypothesis.
