0% found this document useful (0 votes)
98 views29 pagesAn Introduction To Multimodailty
Uploaded by
Agnes KonraderCopyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF or read online on Scribd
0% found this document useful (0 votes)
98 views29 pagesAn Introduction To Multimodailty
Uploaded by
Agnes KonraderCopyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF or read online on Scribd
/ 29
An introduction to multimodality
Carey Jewitt
nodality, describes its scope, sind
This chapter introduces the theoretical assumptions of mult
borates on seven core concepts for multimodality: mode, materiality, modal affordance, me.
ing potential or metafunction, intersemiotic or intermodal relationships, discourse and genre,
which inform the handbook chapters.
Theoretical assumptions that underpin multimodality
tion to be
jon and represen!
Multimodality describes approaches chat understand communic
nnd which attend to the full range of communicational forms peo-
ple se — image, gesture, gaze, posture, and so on ~ and the relationships between these
Four interconnected theoretical assumptions underpin multimodality as it is broadly con
ceived.
more than about language,
hese are brietly introduced and discussed below.
The first assumption underlying multimodality is that lang
1d different perspectives on mul-
eis part of a multimodal
ensemble. This weaves across all the chapters in the volume
timodality: Language is widely wken to be the most significant mode of communications this is
ly so in coments of learning and teaching. Muldmodality, however, proceeds on the
assumption that representation and communication always draw on a multiplicity of modes, all
of which have the potential to contribute equally to meaning. The basic assumption that runs
particula
through multimodality is that meanings are made, distributed, received, interpreted and remade
in interpretation through many representational and communicative modes ~ not just through
“all interactions are multimodal
language ~ whether as speech or as writing. As Norris asserts:
J multimodality “steps away from the notion that hinguage always plays the central role in
imeraction, without denying that it often does" (Norris, 20004: 3)
Gaze, gesture and posture, for instance, tend to be considered a support to speech; rein=
Je: speech but not providing communication in its own right,
. Multimodal research
forcing or otherwise modif
and image is often thought to be in a supportive relation to writing
_ i. . i. _(s#
on practice for people to use demonstrations and
across a rang
place, for instance, it iy entirely com
25 and 26), images and models (e.g. Chapters 4,7, 15 17. 18,
gestures (e.g, Chapters 6, CAREY JEWITT
32 and 33) to explain phenomena, or to set and engage in tasks that require 3 response using
visual or other non-linguistic means,
‘As the chapters in this volume demonstrate, multimodal rese:
ing and describing the full repertoire of meaning-making revourc
communicate and represent (e.g, students and teachers bring to the classroony
are ‘organized’ to make 1 From a multimodal perspective, language is therefore only
ever one mode nestled among a multimodal ensemble of modes. Others have looked before
actnon-verbal” modes; however, multimodality differs in that it does not take language asits
ting point or as providing a prototypical model of all modes of communication Ge. where
mework and processes for understanding all forms
in Chapter 1-4, multimodality offers
nd ‘is not a simple rephrasinget
rch provides tools for anal
‘es which people use
) and how these
language provides the conceptual tools, fr
of comm ion). That is, as Scollon and Scollon argue
a new perspective and methods for understanding language 3
nonverbal communication’
sumption central to multimodal research is that each mode in. multimodd
s realizing different communicative work. Multimodality assumes the
weural
Pacts to be constituted of a
‘on specific roles in a spe
and situated, Por example,
classrooms was thit
‘The second
ensemble is understood
all modes have, like language, been shaped through their cu
al functions. Multimodality takes all communicational
we and other non-linguistic modes take
1, historical and social uses (©
na
if
realize soc
through the social. Im
context and moment in time. These roles are not fixed but articulated
a key finding of Kress et al.'s (2001) in-depth study of London school science
how teachers’ choice and use of different modes of representation led to radica
i =.
Blin the science classroom as an image or through writing, in colour of black and white, ors
3D model or an animated sequence on a CD-ROM or webpage lable and foregrounts
Jifteront aspects of the concept of cell, Each of these representational forms makes differene
“There was alo evidence that different modes have differential potenti]
fleets for learning, the shaping of learner identities and how learners create reading palhwax
through texts, The choice of mode, then, ix a central aspect of the epistemological shaping ot
knowledge and ideological design, What can be done and thought with image, writing, oF
through action differs in ways dhat are significant for learning, Inv this regard, the longstanding
focus on language as the principal, if not sole medium of instruction, can at best offer a very
partial view of the work oF communicating in the chissroom and beyond: In the examples dis
iineedd in this volume the resources of gesture (¢4 Chapters 25 and 26), for instance the spacitl
extent of a gesture; th sth of a gaze are all under
Stood as part of the resources for making meaning,
“This connects with the third assumption underpinning multimodality
trate meaning through their selection and configuration of modes. Thus th
hodes is significant for meaning-making, While multimodal communication is not
jtself new, Ventola et al. (2004; 1) have suggested that:
demands on the learner,
intonational range of voice; the direction and I
— that people orches
interaction betweet
tin and oF
“The various possibilities of combining communication modes in the ‘new’ media, like the
computer and the Internet, have forced scholars to think about the partic Jar character
istics of these modes and the way they semiotically finetion and combine in the modern
discourse worlds
ings in any mode are always interwoven with the meanings made with those of al
The ny
other modes co-present and ‘co-operating’ in the communicative event. The interaction
between modes is itself a part of the production of meaning — a point this chapter returns to.
16 AN INTRODUCTION TO MULTIMODALITY
pioned from
norms and
ally, multimodality is built on the assumption that the meanings of signs
semiotic resources are, like speech, social. That is, they are shaped by th
naking, influenced by the motivations and interests of
multimod
rules operating at the moment of sign
sign-maker in a spec hat is,
throuugh the process of reading/interpretation of the si
adapt and refashion meanings
ic social context’
gn-muakers se
“These effvet and shape the sign dhat
iy made.
st section sets ont to indicate
te, chapters in this volume are
Before introducing the central concepts of multimodality, the
and where appropri
fuller discussion
something of the scope of multimodality
introduced to point the reader in the direction of.
The scope of multimodality
rather
es the
This see ch can do and what it is good fo
1 gives a sense of what multimodal res
than providing an exhaustive account of the literature on multimodal research, I indicat
pingemaking and inter
ange of top=
tiotic resources for me:
potential of multimodal research to describe se
semiotic relations, to dey
ics oF contests including technology.
nd literacy, as Well as the production of identity:’Phiy brief introduction is supported
by the analysis and discussion across the chapters in this volume, which include key references
and suggested
dlility in more depth.
across a
Jop research tools, and its successtial application
nediated interaction, questions of knowledge, pedigogic
practices
adings to support readers who want to investigate a specific aspect of multino
Describing semiotic resources
Mal
muterials and artefacts people communicate with) that modes make available to people in pa
Gicular phices and times (what is meant by mode is discussed in detail later in this chapter and
by Gunther Kress in Chapter 4 and ‘Theo Van Lecuwen in Chapter 5).The work by O"foole
(1994) and Kress and Van Leeuwen (1996, 2006) has contributed to mapping the semiotic
resources of art and visual images. Dewailed studies have
wdlality can be used to build inventories of the semiotic resources (that is, the actions,
so been undertaken to describe the
Zing principles, and cultural references of many other modes. These
jotic modes of colour (Kress and Vane Leenwen, 2002; Van Leeuwen, 2010),
gesture and movement (Kress ¢f al., 2001, 2004; Martinee, 2000; Bezemer, Chapter 25 this vol~
ume; Bezemer ef al. 2011; Mondada, 2011), gaze (Lancaster, 2001; Bezemer, 2005),
music (Van Leeuwen, 1999; West, 2007, Chapter 30 this volume) and space (O” loole,
Stenglin, 2009, Chapter 31 this volume:Van Leeuwen, 20054), to name a few.
Numerous studies have been conducted that set out to understind how semiotic resources
are used to articulate discourses across a variety of contests. The semiotic resources, finetions
and systems of multiple modes have been described with respect to buildings and rooms, for
instance how the design of a child's bedroom connects with notions of gender identity (Kress
and Van Leeuwen, 2001), how framing — disconnection and connection of elements ~ is real~
ized through the use of spatial rhy aration of elements in
school and office buildings (Van Leeuwen, 20053), the analysis of nniks and mecafunctional
systems in the Sydney Opera House to understand its meanings (Toole, 2004), ay well as
the design and experience of muscum exhibitions (c.g. Ravelli, 2005; Diamantopoulou, 2008;
Stenglin, Chapter 31 this volume). Multimodal studies have focused on how the resources of
nized on the page or screen of textbooks,
7
semiotic resources, or
include the s
voice and
0045
ne OF ContRASt, Segregation OF se}
image and writing as well as other modes are org CAREY Jewitr
websites and ot
Jones, 2005, Ch,
Smith, 2011)
Chapter 8 this
her digital learnin
ie
en:
nd Bezeneh”
Ree utees (eg, Unsworth, 2001; Kress an
"Hallo
O'Hal "soot
es Guo, 2004; Lim F, 2004; Norris, 20 vi ellos |
new media texts (e.g. C
volumes Br:
Burn, 2009, Chapter 27 this vo ources
ext. As this brie!
multimodal research 8 with is wide
Chapters 24-33, whi
and are briefly in
The dimensior
ss
Preview suggests, the scope of fe
“ranging, analysis of ™
AETV ofthis handbook, provide case study a weil
. g ig describ
nS of tables as three-climensional objects as mode is Fey i
¢ relation between practical dararee oF gest! 6
Piects. Bezemer focuses in ve the semiotic Thurow in Cho?
ical operations in Chapter 25, Jaworski and rier ours a
a is central modes i On ;
‘entities. The next three chapter jmage
discusses the mode of moving within or
ns and multimodal relaionhips wt Lee
O'Halloran in Chapter 28, wer west of
in Chapter 29. In Chapter ody more
‘ch, gesture (including gaze, cations! og
id music within the specific edu Chapter
a the focus of Stenglin’ case study snd ema ea
© Use of the modal resources of writing os one in
°r analysis of children’s multimodal mapping. nd
nally, Knox focuses
es.
c resoure :
on the semiotic re: o
a
Pewspaper texts, sts 0 00
5 Able of diagramming called system me
ecu
996;Van Leer
& Kress and Van Leeuwen, 199 12000,
cit use of the
rer
15
Products and
Bjorkvall in Chapter 24, whi
ning potential of such o
me;
corms
gas a system a
als for making meaning * erally 1 lat
‘emiotic the metafunctions are ge potenti
y 5
conceptual tool to describe g explore the Semiotic resources and meaning,
People use to make ‘meaning rather than to Map the system itself
5 se
Investigating intersemiotic relations 3
The relationships across ond between modes in molto
of multimodal research
chm
h. Substantial theo
Tange ahd language have bean off
(1996) and Lemke’ work on sci
Bezemer and K
ca cent ag
vodal texts and interaction are 8 COM
oretical descriptions of;
"ed, for example, by the
‘ence textbooks (1998p)
POY curriculum, n
M!eoserbles-This work
S and En,
oration bee
the dynamics of int on Le 9
carly work of Kress and
er and Kres © ins
and work by Bezemer a ai Fy
gate the le:
Fess examine coy materials to investigate ng resoure fo
earning
and losses of differey ws on a corpus of learn
ind 0 Oo
islish from the 1930s, the 1980s a
deca
Kle of the twenty-fi
_
8
ping resour tt
Stally represented and online le : E ages to EE yt
Prelect provides a social Semiotic account of the chang
—DCr—S. their epistemological
18
how
ce and sl
ficance 3
and Social/pedagogic significan AN INTRODUCTION TO MULTIMODALITY
image a
ined ee nes meshed in the construction of content. O'Halloran has also
ros a ations Benen lng nage ad mathematical symbolism in mathematics
Tonics ogee this volun) and mahal csroons (2000). Research on the multimodal
ie ee eee finding: in complex multimodal texts, the
Reenter mods blur and meh in new conigurons (Clot 20,27,8,29,33 and 34).
eee eee ier Gu) emu relationship beeween image and text. They
cope de® functional grammar to csi the reo between image and text in terms of
fc Tnee RY Rains betwen image and wort (equal or unequal) and the nature of the seman-
hia, Reteee elements They csi hes eatin ay caborgon, excesion or enhancement.
Adin © 8 lation where one cluse restates oF caries another; in extension one clause
Un or ea t® other and in ean one clase provides information such as how, when,
tee indo 2 ation to the other use Martine and alway have identified examples ofthese
relations py kage between images and text septs anu offer a senso syston of image-text
Siege nines A Vn Lec (2008) ty of intersemiotic relations in new miedia texts
Som niga ations betwen image and exe are potenti ly remade through their reconfigura-
on fa ta athongh dese relations are not (=) fully established oF stable
stain able dry on diferent modes when they communicate has also been the subject of
orchestra tultinodaity Bourne and Jewiee 2003), for example, analysed how a teacher's
and sre Of modes in a classroom drew together the semiotic nd social resources of texts
(2006 ate create 8 Benulered debate, Shing the focus from teachers to students, Flewite’s
beter model study of preschool classroom interaction deaonsraes the strong link
‘oumunicative demands ofa context (in this case home and preschool playgroup)
and ¢f
Ne modes
ize gag Os it use, Focusing on all modes of communication (speech, gesture, movennenty
and so, . p ere!
Modalities young children’s multifunctional uses of different
dry
th
©n) she is able to scrutinize
argue any cS intentional, socially organized
Sains ‘pathologizing the absence of talk’ (Flewitt, 2006: 47). These studies show the
Pote
locating oe multimodal research to offer a different account of classroom communication by
The oe of speech in the broader context of children’s total nnultimodal resources,
concep ig me ation of intersemiotie resources i key aspect of multimodality. One associated
modal cohesion (Van Lecuwen, 20052; Adami, 2010); this explores how differ~
ent se
cIiotic re
Thythin, oe ceSOUECES are integrated to form multimodal texts and communicative events via
ctivity in the construction of meaning’ and to
any oF sions information linking, and dialogue.
Nottis in cp sheer in this volume are concerned with the interac
Modal deny ae 6 introduces two concepts for thinking about these
fon, teamed Modal intensity. Newel in Chapter 7 outlines the concepts of transforma~
ACTOS texts, co and the transmodal moment as tools for analysing the interaction of modes
mage wi PO ane Cleirigh discuss image text relations in Chapter 12 ally
Nd Tanga ty Work emphasizing the distinctively different semiotic affordances of image
lige and propose a semiotic framework designed to address the synergistic nature of
imag
ehingua r
Tom a Fae interaction in meaning construction. Approaching intersemiotic relations
lindscape gent trtng point in Chapter 8, O'Halloran takes @ histori 1 view of the semiotic
° 1 mathematical symbolic notation.
1 of semiotic modes.
relationships, namely
nd crit
in mathe
mathematics, with a particular focus on
Matti
, .
‘MOdality and technology
he Multimodal f
“Ommunic. 1
configura
nd and movement to enter the
and features of multimodal
cilities of digital technologies enable image, so"
“ational c 7. . A
ational landscape in new andl significant ways. The facilities
ions e¢ "
Ns can impact on design and text production and interpretative practices.
1 CAREY JEWNTT.
Multimodal research has been conducted on the technologization of practices and communic
tion and intera aenore generally (e4, Marsh,2005:Alvermann, 2001 der, 2007; Unsworth
6 al, 2005; Lankshear and Knobel, 2003; Cope and Kalantis, 2000). Much of this work explors
and theorizes the nature of image and writing relations in narratives, relationships between book
and computer-based versions of texts,and the role of on-line communities of various kinds in the
nen tits well as the interpretation and generation of new forms of multimodal and digital na
rantt and ltericy practices. This work ofien deverilee new forms of literacy in an attempe 0
Fae the territory of communication in a contemporary context and the kinds of practices tht
hiclp move across it stich as blogging and culture jamming (Lankshear and Knobel, 2003; Sefton-
Green and Sinker, 2000), Recent studies suggest that conventional print literacy pedagogy pro-
ceeds independently of the everyday multimodal social and commumicative worlds, of many chil
don (e.g. Marsh, 20006). Marsh's studies (2006), for example, investigate young childre
{0 4 years) mediascapes to identify the complex mukimodal
Ae engiged with in the home. Her focus ig on understanding the fi
éSPressions have in maintaining the soc 1 relations of the fin
ath and the development of literacy skills, She
‘nd se mata designs in creative pay
The visual character of writing com
objects of lite
Jewite’s (2002)
example,
(age 25
picative practices that they
ctions that these digital media
ly,accessing knowledge, pres:
documents how migrant students reappropriate
inily life and home/school transitions.
mi
Nes to the fore on screen, for instance, to function *
cy in findamentally different ways than it does on the page (Jewite, 2002, 2008).
Goxe Study on the transformation from printed novel to novel ne CD-ROM, for
Mawes dhat the visual character of writing on se Teen, combined with the dominance
Strves to restructure texts, and fragment forms of writing. A modularization that ean
Tafa uty Contexts as information is reorganized across the sere enand elsewhere, co
Produce information in bite-size chunks regardless of media and mode. Another potential
Mute OF digital technologies is the mode of hypertext which embeds writing, image (and
other modes) into webslike patterns ant layers of information and genres that make meaning-
making a process of navigation and choice, and create new resources (and demands) on mean=
‘necanaking (Luke, 2003; Lemke, 2002b: 7
Zammit and Callow, 1998; amit and Downes, 2002
Technology and its impact on nniltinoxte Fepresentation and interaction isa theme thae rams
Tass hanulbook, and which isforeyrounded manent ore chapters.
{0 Chapter 8, for example, "Hh, ‘xamiines the impact of the printing press and com
puter technology on the ndseape of mathenmatics. Rodney Jones in Chapter 9
invest play and its impact on multimedal communication. In
has affected the ways people use their bodies as a come
al interaction through the analysis of how gay men use an internet
ys of their bodies. Kevin Leander and Lalitha Vasudevan approach the
and wnultimodality through the lens of mobile and video tech jology in Chap-
ter 10. fn Chapter 11, Lemke explores issues of how technologies produce meaning eflects across
male timescales, sing the illustrative example ofthe era video genes a tans
tneals franchises, which are typically experienced cect rie ie 3850 hou oer periods
‘weeks or months, while in Chapter 27, 28,29 nd 39 dhe ee igurations of digital tech-
fon in the context of film, games and online
oF imag
be seen a
loran
semiotic |
ites technology as a site of d
Particular he explore
s how technology
muunicative resource in soci
Webvite to create disp
thee of culty
nologies are a focus of investig
Knowledge, pedagogy and literacy
x es Gee, Gunther
When first made by the New London Group (1996) (Courtney Cazden, Jam < comet
Kress, Allan Luke and others) the call to unddenstonct Ksowede andl pedaony 2 oe
Talical.A key design element ofa future pedagogy war herald as lenin fore
20 AN INTRODUCTION TO MULTIMODALITY
of mei
ning’ (New London Group, 1996). In part this call way a response to the soc!
cultural reshaping of the communicational landscape (related to globalization,»
gies, and new den
Hand
w technolo-
ands for work), The conclusion that reading this ‘new’ multimedia, multi-
evitable. This. work
eh and pedagogic models within which multimodality
(broadly conceived) isa key factor (Cope and Kalantzis, 2000).
A spec ication on multimodality was one of the first publications
to provide tools for educational researchers wanting to undertake multimodal research (Lemke,
modal landscape for its linguistic meanings alone is not enough was
spawned a strand of educational res
issue of Linguistics and
1998), Brom early 2000 there hay been an explosion of interest in multimodality within
ly T y
ich and this perspective has been actively taken up by educational researchers leading. to
g-tuaking across a wide range of sites. Sub=
substantial work that looks at nvultimodal mea
stantial multimodal research has b
a focus on meaning multimodal making practices (e.g. Bearne and Kress, 2001; Kress, 1997a¢
Kenner, 2004; Mavers, 2003; Pahl, 1999; Lancaster, 2001; Marsh, 2006). Science education has
proven to be a productive site for multimodal investigations into the construction of knowledge
across a range of resources (c.g. Kress ef dl, 2001
(O'Halloran, 2000, 2005), musie education (West, 2007) and school English and Media educa
tion (Goodwyn, 2005; Kress et al, 2005; Burn and Durran, 2007), Multimodal studies in Higher
Education have also examined pedagogic strate 20008)
Taken as a whole, these multimodal studies show: that significant pedagogic work is realized
through a range of modes. The Multimodal Production of School English project (Kress et ul..2004)
showed that thiy holds true even in a curriculum context such as English where talk. and writing,
dominate the classroom. The complex ways in which image, gesture.
body posture, writin, ssroomn production of school subject knowle
Ic highlighted how students and teachers co-produce notions of ability.
undert
on in pre-school and early years contests, with
Reis ef al. 2007).as has mathematics education
xa7e, interaction with objects,
al speech interact in the
resistance and identity in
the classroom through their multimodal interaction. The project also analysed how classroom dis
phys.space,
that the work of interpreting
furniture and artefiets were designed to realize versions of English. Phis research showed
school English is beyond.
guage and rea)
jonships bewween hens, Ie
senve of a ringe of modes and the 1 ko |
raultimodal identity work that stud
Gunther Kress, in Chapter 4, explores how modes shape knowledge and practices and the
role of layout in the social and ontological consequences of the designed snultimodal clays
tion of information. The need to rethink what it means to learn and to be literate is a thread
that runs through much multimodal research This raises numerous research questions in relation
to learning, including how representations impact on thinking and learning, as well as what
lite Hand multilingual communicational landscape
to study it, These issues are picked up across the handbook, and are central o the arguments of
Newtield (Chapter 7) and Archer (Chapter 13) that explore literacy, knowledge and pedagogic
reet, Pahl and Rowsell, who
l with in the classroom,
ns are eng
nd how
wey is and could be ina mul
power through a multimodal lens, as well ay Chapter 16 by $
explore the potential of merging of New Literacy Studies with multimodality
Identity practices
How identities are articulated through multimodal means is an area that hay attracted some
tention within multimodal research, Norris (2004a) hay explored this by examining how
people shift modes, and modal awareness to manage multiple idemtity roles in interaction («-g:
ging with their child, in the context of another interaction). Bezemer and Kress
(2008) explore how pupil identities are negotiated in the claysroom through the use of gaze, and
4 mother eng CAREY JEWITT
2s (Chapter 9 this volume) explores how sexual identist
orientation to texts and teachers. Jone
are performed and managed through the disp thholding of representations of the bod
es from a multimodal per
Much of the work on what might broadly be ¢
spective is also concerned with the production of idemities (c., Pahl and Pollan, 2008; Stei
2003, 2008; Stein and Mamabolo, 2005). For example, Stein (2008) explores how students #
South African educational contexts express complex narratives of identities and culture throug
inuktimodal texts, highlighting the links between representational means, the production of der
tities, and social justice. Her work explores how multimodal pedagogy ean reconnect linguistical
disenfranchised learners — through the use of performance, semiotic artefacts, visual representatio
and so on. In the Olifantsvlei fresh stories project, Stein (2003) undertook a literacy project ovt
nd two at a Johannesburg primary schol
inilics living in informal settlements. St
a systematic use of differet
“The project explored tk
Student case stue
writing, three
and wi
led literacy practic
six months with teachers and students of grades one a
which serves children of unemployed and migrant
worked with multimodal literacy practices and pedagogy through
semiotic modes in order to develop forms of learning beyond language:
relations between creativity, multimodal pedagogy, representation and learnin
ies involved observation and interviews, students’ use of ewo-dimensional drawing
dimensional figures, spoken dialogues, multimodal play and performance to create narratives
identity and culture. The focus was on the representation of doll and child Ggures and their syur
otic meanings, Stein describes the children’s transformation and recontestualization of cultaraly
Ily situated practices of these representations. Stein argues that multimodal pedagog:
to enter the schod
and histori
enables the assertion of student
context in ways that are signific:
Identity is another theme that weaves its way across several cha
scrongly than others. For instance, how young men interact with mobile technologies in the pe-
es ig a then chapter 10 by Leander and Vausdevan, Lemke, in Chay
nd make sense of multimedia is strongly related &
jentity, cultural practices and community
nt for literacy and teaching
apters in this book, some mot
formitivity of their ident
‘gues that how people experienc
ing a different starting point for thinking about identiy
4, while Jaworski and Thurlos
er 26,and Burn develo
ter 11, who
their identities, values and desires. T
Bjérkvall touches on the able as a marker of identity in Chapter 2
explore gesture and gaze as part of the identity practices of tourists in Chapt
them in different ways through his discussion of games and film in Chapter 27,
The following section discusses seven core concepts for multimodality:
function, intersemiotic or intermodal relationship.
approaches to mut
mode, materialit
1 affordance, meaning potential or met
Phe nuances and differences in emphasis beeween the
here that these concepts are ne
mod
genre and discourse.
timodality are discussed in Chapter 2, but it is worth not
given the same import or attention within cach approach
Core concepts for multimodal analysis
‘The following core concepts are in a state of change and fluidity, and are continuously taken uy
and shaped in different ways by different approaches to multimodal research. The description
that follow are therefore intended to provide the reader with useful (working) definitions,
Mode
Within social semioties,a mode, its organizing principles and resources, is understood as an out
come ofthe cuba shaping of a material The resources come to diphy regulars rough
the ways in which people use them. In other words in a specific context (time and place) mode AN INTRODUCTION TO MULTIMODALITY
are shaped by the daily social interaction of people. Its these that multimodal analysts call modes
Kress (Chapter 4 this volume) sets out the central issues of the category of mode, as seen fiom a
social semiotic multimodal perspective. The chapter includes discussion of the ‘logics’ of modes;
issties Of mode, representation and knowledge: sign and mode, including the notion of modal
choice and epistemological commitment; as well as the notion of sign, genre and discourse
O'Halloran and O'Toole have a slightly different conception of ‘semiotic resource’ and
“mode” does not feature in the same way as for Kress and Van Lecuwen, From this perspective
uw
(and following trom Halliday (1978), Lamenage as Social Semiotid) language, mathensatical sym
holism and images, for example, are understood as semiotic resources rather than modes.
ning that realize different functions, and there-
emiotic resources Consist of systems of mn
fore, meming becomes a matter of choice from the systems of meaning from different semi~
ate in multimodal phenomena, From this starting,
otic resource, and how these choices integ
point, Language can be realized through written text (a visual mode) and spoken language
(an oral mode). As O'Halloran argues in Chapter 8 the meanings of modern mathemat
symbolism, for example, change when developed in the written mode ay opposed to when it
is realized in the oral mode.
Multimodal interactional analysis focuses on action and therefore does not have the same
although all actions are understood as‘mediated by the systems of representation
draw on’ (Norris, 2004a; 12-13). However, the focus is on the situated interplay
focus on mode
that the
between modes at a given moment in social interaction (see Chapter 6).
The regular pattern of using a set of resources has traditionally been called ‘grammar’
Machin has questioned the possibility and usefulness of applying a linguistic term such as
*grammar’ to image and other modes (Chapter 15 this volume,and Machin, 2007). However,
a general principle is that in order for something to ‘be a mode" there needs to be a shared
cultural sense of a set of resources and how these can be organized to realize meaning, A
number of detailed studies on specific modes have helped to describe theye semiotic
resources, material affordances, organizing principles, and cultural references (e.g. the work
on images by O'Toole (1994) and Kress and Vani Lecuwen (1996) set out to map the semiotic
resources of image). The chapters within this volume examine a variety of modal resources
a
mnunic, image, mathematical symbolism, written and spoken language, and three-dimensional
7e, body posture and movement, sound, voice
and their use including spac
KeStUUTe, £5
objects such as tables; as well as examining how these modal choices and configurations vary
across a range of social settings and media.
The purpose of multimodal investigations is to understand the principles of use and modal
resources available ina multimodal representation ( multimodal text) or the situated commu
nicative moment, rather than to seck to establish a universal inventory for a mode, What is
considered a mode and interaction between modes is inextricably shaped and construed by
andl mediate how modes are take
social, cultural and historical fietors. These factors influes
and the production of modal conventions. People draw upon
ny given
up and used for meaning-making
the available modal resources to make meaning in specific contests. Consequently,
mode is conting, nig resources of meaning, rather than static skill repli
upon fluid and dy
cation and use. Its in this way that modes are constantly transformed by their users in response
ommiunities, instittitions and societies: new modes are created,
to the communicative needs of
and existing modes are transformed.
nd culture are central
The influences of timescales, technologies and sites of display, histor
to the idea of mode, issues that inform many of the chapters; in particular, see O'Halloran
s used to realize are also shaped
(Chapter 8). The resources of modes and the finetions they
23 CAREY JeWITT
ys by changing facilities of technologies, itsell'a part of the social ndscape (see
ader and Vausdevan, Chapter 10; Lemke, Chapter 11; Chouliaraki, S hapter
in important x
Jones, Chapter 9; Le:
18; Knox, Chapter 33).
Semiotic resource
Scmiotic resource is centeal to multimodality, although it features slightly differently within par.
ticular approaches. Kress and Van Leeuwen suggest semiotic resource can be thought Of as. the
connection between representational resources and what people do with them. Van Leeuwen
describes semiotic resource as follows:
Semiotic resources are the actions, materials and artifacts we use for communica tive
purposes, whether produced physiologically ~ for example, with our vocal appara tus,
the muscles we use to make facial expressions and gestures — or technologically ~ fy
example, with pen and ink, or computer hardware and software ~ together with te
ways in which these resources can be organized. Semiotic resources have a mearnixy gx
potential, based on their past uses, and a set of affordances based on their possible Uses,
and these will be actualized in concrete social contexts where their use is subject: eG
some form of semiotic regime.
(Van Leeuwe' 85)
lier, O'Toole (1994) and O'Halloran (2005) view semiotic resourees (BA Soya ee,
image, etc.) as systems of meaning that people have at their disposal. People make choic cross
different semiotic resources simultancously and these combine in multimodal phenomena |
The emphasis on rules within social semiotics is on rules as socially made and chan see aye
through social interaction (Van Leeuwen, 2005a). This stands in contrast to traditional sera jg ee
Understanding of rules (or codes) as fixed and resistant to modification Saussure, 1974 Barey oo
19776). Traditional semiotics sees language and other semiotic systems as a code oF sets © py
for connecting signs and meanings.This means that once two or more people have unde rs, 7
As indicated e:
the same code they can connect the same meanings to the same sounds or
Understand each other. This suggests that the semiotic system is simply ‘there’. Ie can be
but it cannot be changed in any way. The sign is viewed as a pre-e jn
hifier and signified, an element in a code, to be understood and used. This view of seray 3 OS
passive role to the production of meaning and establishes lan ge OS
tion of
“ystems places people in a
and other semiotic systems as entirely stable,
The concept of semiotic resource offers a different starting point for thinking about sera je
syste : : tic
stems and the role of the sign-maker in the process of making meaning, In this perspec gy 1S
SEIS area product ofa social process of sign-making.A person (sign-maker) ‘chooses'a Ser 3 YS
y bring together a semiotic resource (
FLOUECE from an available system of resources. TH
they want to express. In other words, people es
fer) with the meaning (the signified) th
Meanings through their selection from the semiotic resources that are available to then
ing is choice from a system. But this choice is always soc 7
ailable to who,and the discs.
Particular moments
dea
here are various kinds of norm
resources are 1
snd regulated, both with respect to wh
that regulate and shape how modes are used by people
dliscourses for how we use semiotic resources — sometimes
of different kinds, but nonetheless they do provide ‘rules’ for their use. Discourses of gy
Social class, race, generatior norms and other articulations of power shape
institutio
24 AN INTRODUCTION TO MULTIMODALITY
not ‘codes’ in the sense that they cannot
Lecuwen, 20058)
regulate people’ use of semiotic resources. These
pe changed and that they ‘are simply there’ ~ but they are social rules (Va
In addition to mapping the semiotic resources available to people to make meaning with,
jotie resources.
nulti~
mukimodality can also contribute to the development of new ways of using s
Focusing (through historical analysis) on how ser
modality can ask why the
ity’ comes to be represented and offering the potential to imugine it differently and to
ly
jotic resour
es come to be as they 3
are as they are. This isa powertul way of enabling people to see how
itis ary
redesign it. This shifis the primary focus from descriptive accounts to connect more exp
with macro social, political and cultural concerns within design.
Metafunctions
‘The turn to social explanations within linguistics, for instance in the work of McDerrmot,
to name a few shifted attention onto how kinguage is shaped by how
aye are put to, This turn to the
(1978) theorization of the social
Hymes and Halli
people use it — the social fimetions that the resources of lan
social provides the context for the linguist Michae! [alli
fanctions of language as realizing three mctufiautions.
People, Halliday theorized, construct representations of what goes on in the world’ and their
experience of the world through the ideational resources of a mode. (Mdeational meaning. is also
ning, and sometimes called experiential meaning or logical
ys, including the words choyen
ion of different kinds of relation-
referred to as presentational me:
meaning.) fn Tanguage this may
to represent people, places and things in the world; or the cre:
ships between these ‘participants’ by positioning them as active, passive or reactive. Social re
tions between the person who makes a sign, the person who engages with it.and the thing that
is represented are also realized in a sign. These relations are constituted and enacted by the
erred (0 as‘orientational’ meaning).‘These mean
be achieved ina number of x
interpersonal resources of a mode (sometimes re
ings need to be orginized into texts and this draws on the fextual resources of a mode (some-
times called organizational meaning). Language provides many different resources for realizing
nid striteture of a test. Halliday’s work explores how these three kinds of meaning
the cohesion
potentials are ‘held by’ the grammar
Mulimodal approaches take up the concept of the metafimetions and apply them to alf modal
nd clements of hang
netafanctions are viewed as a higher order of meaning rather than
Gemiotic) resources (that is th
specific to language). Another way of thinking,
about the metalinctions is as meaning potential that
with a particular set of modal (6
piotic) resources.
is what can be meant’ or what can be don
Jewitt 2006), for example, employed the metafanctions to analyse the how technology mediated
alum knowledge in the chystoom with reypect to image,
jonal metalianetion enabled questions to be asked
learning influences the shapes of cv
so on. The ide.
xive, gesture, writing, spec
‘on how the multimodal computer applications
excluded and how what was disphyed shaped curriculum knowledge, Using the interpersonal
nictafimnetion made it possible to explore how learners were positioned to knowledge through the
rhe textual metafinetion provided a tool with
cen orgginized the text
use presented “the world’: what was included and
design of the multimodal applications they used.
which to get at how the arringement of multimodal clementy on se
Modal affordance, meaning potential and materiality
The term affordances is contested and continuously debated within multimodal research, It has
y in social semiotic approaches to multimodality: It originated
particular emphasis and curren
25 CAREY JEWITT
in the work on cognitive perception of Gibson (1977). It was later ken up by Nom jp
1 (1988, 1990). Norman's view of affordance considers the material and ci)
ibson 3
relation to desi
aspects of design. Van Leeuwen uses affordance following al uses the term “mesg
potential’ to refer to the material and the cultural aspects of modes. Taking a slightly ditrert
emphasis, modal affordance, is used by Kress (1993) to refer to what it is possible to expre,
‘sant
nce as a complex concept connected & het
represent easily with a mode. He positions afforda
the material and the cultural,and the social historical use of a mode (cach of which is intity eels
connected), while Jewitt argues that neither Gibson nor Norman’ notion of alfordare,
ptual and material objects) are shaped by peopl
cn
adequately acknowledge how tools (con
of them in specific social situations (Jewitt, 2008a).
fordance in Kress’s terms, or the meaning potential of a mode Ryyin
used, what it has been repeatedly used 1
es ust
In other words, the
Leeuwen, is shaped by how a mode has bee
and do, and the social conventions that inform its use in context. Wh
mest
COMES frou!
nce, becomes a part of its affordance oF meaning PO ren tise
camode
its history of cultural work, its prov
Ie thus follows, that image, in the form of graphic marks on a two-dimensional surface (ie. is
material) as well as how it has been socially shaped through its use, combine to offer ditfgrent
meaning potentials for the expression and representation of meaning than speech in he form
of sounds over time and space.
‘The particular semiotic resources of a mode (or the particular systems of semiotic res \yrcese
from the perspective of O'Toole and O'Halloran) have come to be shaped through these di
ferent histories and so has people's use of them. Each mode (as itis realized in a particu gyi
context) possesses a specific logic and provides different communicational and representayjond
potentials, referred to as modal logic. The sounds of speech for instance usually happen eros
time, and this sequence in time shapes what can be done with (peech) sounds, The Ig
sequence in time is unavoidable for speech: one sound is uttered after another, one wor afer
another, one syntactic and textual clement after another. This sequence becomes an aflordance
or meaning potential: it produces the possibilities for putting thingy first oF last, or some\gher
ele in a sequence. The mode of speech is therefore strongly governed by the logic of tye. 0
contrast, (till) i y the logic of space and simultaneity, Lie
all governing principles they do not hold in all contexts and are realized through the comple
ges are more strongly governed by
interaction of the social as material and vice versa — in this sense the material constitutes th?
social and vice
ersa,
nd na
y is the focu
The connection between culture ure, meaning and material of Chap~
ter 5 by Theo Van Leeuwen who argues for their inextricable connection’ through an aGeount
of the semiotics of voice quality and is a theme taken up across many of the chapters in this
handbook,
The repeated use of modes to represent and communicate particular me
ngs in specific
social contexts has resulted in the specialization of modes. From this perspective not every mod
is equally ‘useable’ for a particular task. Some things can be signified in an image as Well &
through talk, while some others can only be realized in an image and others only in tall THE
introduces the importance of the concept of intersemiotic relations, also referred to as inter
modal relations, or multimodal ensembles.
Intersemiotic relationships
Modal affordance or meaning potential raises the question of what image is*best’ for and wv
words, and other modes and their arrangements are ‘best’ for ticular context (La
Ms
26 AN INTRODUCTION TO MULTIMODALITY
201). When several modes are involved int a communicative event (e.g. text, at website, a spo-
ken interchange) all of the modes combine to represent a message’ meaning (e.g. Kress ef al.
2001, 2004), The meaning of any message is, however, distributed across all of these modes and
not necessarily evenly. The different aspects of meaning are carried in different ways by each of
2.1 part of the message
the modes in the ensemble, Any one mode in that ensemble is carn
x and speech and
only: each mode is therefore partial in relation to the whole of the meat
(012). Multimodal research
ch mode and how each mode interacts with
ends to the interplay
writing are no exception (Jewitt and Kress,
between modes to look at the specific work of
and contributes to the others in the multimodal ensemble. At times the meaning realized by
two modes can be ‘aligned’, at other times they may be complementary and at other dines each
be wed to refer to distinet aspects of meaning and be contradictory, of in tension
(Lemke, 1998a). As Lemke has stated (2002: 303)
No [written] text is an image. No text or visual representation means in all and only the
same ways that text can mean, Itis this essential incommensurability that enables genuine
new meanings to be made from the combinations of modalities.
onships between modes ay they are orchestrated in interactions (and texts) may
1 the aspects of meaning, in a text. This kind of tension can itself be
The structure of a text and
ens, These contribute to the
hat to attend to, what
The re
realize tensions betw
me.
ful and a means for encouraging reflection and critiqu
hyperlinks realizes connections and disconnections beww
pansion of meaning relations between elements. The question of w
to"make n
aningful’ isa significant aspect of the work of making meaning. In other words,
the task of what to attend to and to select ay salient to the task at I
nd is amplified by a
multimodal focus,
As mentioned earlier, the question of how modes interact is dealt with by authors throu
this volume, in particular by Nortis (Chapter 6), Jones (Chapter 9), Unsworth and Cléirigh
(Chapter 12), Stick) (Chapter 19), Jaworski and Thurlow (Chapter 26), Burn, (Chapter 27),
O'Halloran ( ’) and Knox (Chapter 33).
ighout
apter 23
Multimodal discourse
Multimodal discourse is a term used by many of the authors throughout this handbook and one
that is fully engaged with by Choutiaraki in Chapter 18 on critical discourse analysis and mul-
timodatity. It is an important yet contested term rooted in different disciplines and used tn a
n be understood as language in use — everyday
variety of ways. Ina narrow sense, discourse ¢:
ways of talking ~ what James Gee in his book Social Linguisties and Litenwies: Mdeology in Discourses
(1990) refers to as ‘little dl discourse. Ina broader sense it can be used to refer to a system of
ing practices (c.g. behaviour, dress and customary practices/
ul reality — what Gee refers to as ‘big D’ Discourse
jon, commercial Discourse, medical Discourse, oF legal
language use and other meaning
habits) that form ways of talking about so
Fores
ample the Discourse of traffic reg
dled stretches of
Discourse. In socio-linguistics, discourse tends to be used to refer to ext
speech or writing and to draw attention to the uses and onginization of language in its social
contest. In sociology and philosophy, the writings of Foucault have been particularly important,
bringing into focus not only the social origins but abo the social effects of power that discourse
has on social practices. Critical discourse analysis (CDA) is usually associated with the work of
Chouliaraki and Faitclough, which provides a method of analysing texts (© complement
27 é
a
ne
ne sey of af
2s for the
1 Well. offering examples fo we
Fal CONC EPLS, a8 Well as of wot,
in é
‘md interactions] discourse. sd many work eof cs
: es ae f
Mt term for multimodality ene sl
cemed with ndertanding the ase and effects of disco il sine 8,
eraeemnent in mots enetnbles. The assumption is sd aac all modes. ne isco
ed, and eren
OmMUnicatiy SREMtS ate always digg Unsively shaped, an setive, differ m3
MPFeSSION Of digg OUrses, From this perspe
“YY Modal
be we! ” 5!
therefore, the choice of modes ces ih
ss fic: texts. pace: We’
Presence Of different discourses in specific on el sp i
‘ingly Media are multimedia forms that occupy 1
(of
he wor 2) f
by notably in the i
Mas strong links with multimods
aud Ci So ie
“Mediation: New 4 feddia, Citizenship d rergence ne en
1 Mulimodality Of new Media discourses, such as conv S is be
18 © explore how, these
and change the
it
Ways in whieh
cl
yy itse
ie
pundaries bent
discourses blur the bout ets
We understand and enac
Muttimoday genre
The term, ‘genre’
of: SeMiotic che
PA
eg
4
a
er
Jo describe eT 9
THY studies and has been extended
Ce
TALES fron Hg ts that ve pat in ;
$¢s in multimodal Communicative objects and seein this a ned
communiti a cultures. johny 1 MM (Chapter 17) has led roses
Mita an i)
pectin ge
With a OCUS On extend;
USS Within fing © Provide a theo
Tetation of,
ally
ng genre to the form: vcs bod
retical mechani th bees
cant activity HE
Sete as a socially signifi a
* Onventional; ed and structures
e task among
finctive communicative task
ately tied to th
Mukimoda)
.
ei
ya
vailable by
he modes made avail
UrPase,
ema.
sens ar according 40 Ba !
vi
odes Pe
_
emiotic tes
: ¢ the sem ese strat eve
I strategies deploy eee achie™ pe
N COmmunication i, being ena in order © 7 on
‘lization over Aime and (cooperate (i) ene rec08
Me 28d Gi) in onder Support the SD 2008:
dicate Chae the wen is being enacte oy
onte
c
ocial
tween $0
50 understand the Tink betwe
8
oy
hy
jes in! 7 ee
and filme
Mukimod,
‘ticated Th,
interproty
nd docuny
Semres change
analysis Joh,
6
nod \
combine mou | esohe
lal texts that combi of es
tion and effective ema word
on. Ba a
ent interpretation is for
Over time and why th
hol!
pother sere
Knox (Chapter 33) is an A
eorizil
5 Multimodal genre in theore se
SPB genres of “WSPapers, His work out! to the
Wspaper home merging in Cah
Ws reporting (Knox, 2 LITY
‘AN INTRODUCTION TO MULTIMODA\
s volume.
esearch in thi
nodal resea _
J above are common across oa!
