Author:
Kun Zhu School of International Studies, Zhejiang University, Hangzhou, China

Search for other papers by Kun Zhu in
Current site
Google Scholar
PubMed
Close
https://orcid.org/0000-0003-3030-9212
Restricted access

Abstract

In recent decades, there has been a growing interest in multimodal translation studies. However, there is currently no suitable framework available for analyzing how the original meaning is reconstructed through the interaction of various semiotic resources in the multimodal target text. Under this context, this article proposes a systemic functional framework that integrates the systems of the meaning potential of semiotic resources, including language, sound (human voice), objects, space, and semiotic ensembles of the human body, and a tripartite classification of the relationships between semiotic resources. To exemplify the analytical potential of the framework, this article presents an example of analysis, taking an excerpt from Fences as the object of analysis.

  • Adami, E. (2015). What’s in a click? A social semiotic framework for the multimodal analysis of website interactivity. Visual Communication, 14(2), 133153.

    • Search Google Scholar
    • Export Citation
  • Bezemer, J., & Kress, G. (2008). Writing in multimodal texts: A social semiotic account of designs for learning. Written Communication, 25(2), 165195.

    • Search Google Scholar
    • Export Citation
  • Bonvillian, J. D., Lee, N. K., Dooley, T. T., & Loncke, F. T. (Eds.) (2020). Simplified signs: A manual sign-communication system for special populations (Vol. 2). Open Book Publishers.

    • Search Google Scholar
    • Export Citation
  • Borba, M. C., O’Halloran, K. L., & Neves, L. X. (2022). Multimodality, systemic functional-multimodal discourse analysis and production of videos in mathematics education. In M. Danesi (Ed.), Handbook of cognitive mathematics (pp. 909938). Springer.

    • Search Google Scholar
    • Export Citation
  • Eggins, S., & Slade, D. (2006). Analysing casual conversation. Equinox Publishing Ltd.

  • Fan, W. F. (2001). A systemic-functional approach to grammatical metaphor. Foreign Language Teaching and Research Press.

  • Frank, T. (2019). Determinism and self-organization of human perception and performance. Springer.

  • Fricke, E. (2013). Towards a unified grammar of gesture and speech: A multimodal approach. In C. Müller, A. Cienki, E. Fricke, S. H. Ladewig, D. McNeill, & S. Teßendorf (Eds.), Body - language - communication: An international handbook on multimodality in human interaction (pp. 733754). De Gruyter Mouton.

    • Search Google Scholar
    • Export Citation
  • Gottlieb, H. (2018). Semiotics and translation. In K. Malmkjaer (Ed.), The Routledge handbook of translation studies and linguistics (pp. 4563). Routledge.

    • Search Google Scholar
    • Export Citation
  • Gu, Q. B. (2015). Multimodal discourse analysis of business presentation: A case study based on Steve Jobs product launch presentations (Doctoral dissertation). Available from China Dissertations Database.

    • Search Google Scholar
    • Export Citation
  • Guijarro, A. J. M., & Sanz, M. J. P. (2009). On interaction of image and verbal text in a picture book: A multimodal and systemic functional study. In E. Ventola, & A. J. M. Guijarro (Eds.), The world told and the world shown (pp. 107123). Palgrave Macmillan.

    • Search Google Scholar
    • Export Citation
  • Halliday, M. A. K. (1978). Language as social semiotic: The social interpretation of language and meaning. Edward Arnold.

  • Halliday, M. A. K. (1985). Systemic background. In J. D. Benson, & W. S. Greaves (Eds.), Systemic perspectives on discourse (Vol. 1, pp. 115). Ablex Publishing Corporation.

    • Search Google Scholar
    • Export Citation
  • Halliday, M. A. K. (1994). An introduction to functional grammar (2nd ed.). Edward Arnold.

  • Harley, J. M. (2016). Measuring emotions: A survey of cutting edge methodologies used in computer-based learning environment research. In S. Y. Tettegah, & M. Gartmeier (Eds.), Emotions, technology, design, and learning (pp. 89114). Academic Press.

    • Search Google Scholar
    • Export Citation
  • Jewitt, C. (2006). Technology, literacy, learning: A multimodal approach. Routledge.

  • Jewitt, C. (2009a). Introduction to part Ⅰ. In C. Jewitt (Ed.), The Routledge handbook of multimodal analysis (pp. 1113). Routledge.

    • Search Google Scholar
    • Export Citation
  • Jewitt, C. (2009b). An introduction to multimodality. In C. Jewitt (Ed.), The Routledge handbook of multimodal analysis (pp. 1427). Routledge.

    • Search Google Scholar
    • Export Citation
  • Jewitt, C. (2009c). Different approaches to multimodality. In C. Jewitt (Ed.), The Routledge handbook of multimodal analysis (pp. 2839). Routledge.

    • Search Google Scholar
    • Export Citation
  • Jewitt, C. (2013). Multimodal methods for researching digital technologies. In S. Price, C. Jewitt, & B. Brown (Eds.), The Sage handbook of digital technology research (pp. 250265). Sage.

    • Search Google Scholar
    • Export Citation
  • Jewitt, C. (2017). Towards a multimodal social semiotic agenda for touch. In S. Zhao, E. Djonov, A. Bjorkvall, & M. Boerils (Eds.), Advancing multimodal and critical discourse studies: Interdisciplinary research (pp. 7993). Routledge.

    • Search Google Scholar
    • Export Citation
  • Kress, G., & Van Leeuwen, T. (1996). Reading images: The grammar of visual design. Routledge.

  • Kress, G., & Van Leeuwen, T. (2002). Colour as a semiotic mode: Notes for a grammar of colour. Visual Communication, 1(3), 343368.

  • Martinec, R. (2001). Interpersonal resources in action. Semiotica, (135), 117145.

  • Martinec, R., & Salway, A. (2005). A system for image–text relations in new (and old) media. Visual Communication, 4(3), 337371.

  • Norris, S. (2004). Analyzing multimodal interaction: A methodological framework. Routledge.

  • Norris, S. (2007). The micropolitics of personal national and ethnicity identity. Discourse & Society, 18(5), 653674.

  • Norris, S. (2009). Modal density and modal configurations: Multimodal actions. In C. Jewitt (Ed.), The Routledge handbook of multimodal analysis (pp. 7890). Routledge.

    • Search Google Scholar
    • Export Citation
  • Norris, S. (2011). Three hierarchical positions of deictic gesture in relation to spoken language: A multimodal interaction analysis. Visual Communication, 10(2), 129147.

    • Search Google Scholar
    • Export Citation
  • O’Halloran, K. L. (2004). Visual semiosis in film. In K. L. O’Halloran (Ed.), Multimodal discouse analysis: Systemic functional perspectives (pp. 109130). Continuum.

    • Search Google Scholar
    • Export Citation
  • O’Halloran, K. L. (2005). Mathematical discourse: Language, symbolism and visual images. Continuum.

  • O’Halloran, K. L. (2009). Historical changes in the semiotic landscape: From calculation to computation. In C. Jewitt (Ed.), The Routledge handbook of multimodal analysis (pp. 98113). Routledge.

    • Search Google Scholar
    • Export Citation
  • O’Toole, M. (1994). The language of displayed art. Leicester University Press.

  • O’Toole, M. (2011). The language of displayed art (2nd ed.). Routledge.

  • Ranker, J. (2014). The emergence of semiotic resource complexes in the composing processes of young students in a literacy classroom context. Linguistics and Education, 25, 129144.

    • Search Google Scholar
    • Export Citation
  • Rossi, F., & Sindoni, M. G. (2017). The phantoms of the opera: Toward a multidimensional interpretative framework of analysis. In M. C. Sindoni, J. Wildfeuer, & K. L. O’Halloran (Eds.), Mapping multimodal performance studies (pp. 6184). Routledge.

    • Search Google Scholar
    • Export Citation
  • Stenglin, M. K. (2009). Space odyssey: Towards a social semiotic model of three-dimensional space. Visual Communication, 8(1), 3564.

  • Van Leeuwen, T. (1999). Speech, music, sound. Macmillan.

  • Wignell, P., O’Halloran, K. L., Tan, S., Lange, R., & Chai, K. (2018). Image and text relations in ISIS materials and the new relations established through recontextualisation in online media. Discourse & Communication, 12(5), 535559.

    • Search Google Scholar
    • Export Citation
  • Wilmes, S., & Siry, C. (2021). Multimodal interaction analysis: A powerful tool for examining plurilingual students’ engagement in science practices. Research in Science Education, 51, 7191.

    • Search Google Scholar
    • Export Citation
  • Wilson, A. (1986). Fences. Penguin Group.

  • Wu, Y. (2021). The medial turn and multimodal translation studies. Journal of Foreign Languages, 44(1), 115123.

  • Zhu, S. C. (2002). Value of modality and modulation: Mirror of communicative interpersonal relations. Journal of Shanghai Teachers University (Social Science), 31(5), 117124.

    • Search Google Scholar
    • Export Citation
  • Zhu, Y. S. (2007). Theory and methodology of multimodal discouse analysis. Foreign Language Research, (138), 8286.

  • Collapse
  • Expand

Editor-in-Chief: Krisztina KÁROLY (Eötvös Loránd University, Hungary)

Consulting Editor: Dániel MÁNY  (Semmelweis University, Hungary)

Managing Editor: Réka ESZENYI (Eötvös Loránd University, Hungary)

Founding Editor-in-Chief: Kinga KLAUDY (Eötvös Loránd University, Hungary)

EDITORIAL BOARD

  • Andrew CHESTERMAN (University of Helsinki, Finland)
  • Kirsten MALMKJÆR (University of Leicester, UK)
  • Christiane NORD (University of Free State, Bloemfontein, South Africa)
  • Anthony PYM (Universitat Rovira i Virgili, Tarragona, Spain, University of Melbourne, Australia)
  • Mary SNELL-HORNBY (University of Vienna, Austria)
  • Sonja TIRKKONEN-CONDIT (University of Eastern Finland, Joensuu, Finland)

ADVISORY BOARD

  • Mona BAKER (Shanghai International Studies University, China, University of Oslo, Norway)
  • Łucja BIEL (University of Warsaw, Poland)
  • Gloria CORPAS PASTOR (University of Malaga, Spain; University of Wolverhampton, UK)
  • Rodica DIMITRIU (Universitatea „Alexandru Ioan Cuza” Iasi, Romania)
  • Birgitta Englund DIMITROVA (Stockholm University, Sweden)
  • Sylvia KALINA (Cologne Technical University, Germany)
  • Haidee KOTZE (Utrecht University, The Netherlands)
  • Sara LAVIOSA (Università degli Studi di Bari Aldo Moro, Italy)
  • Brian MOSSOP (York University, Toronto, Canada)
  • Orero PILAR (Universidad Autónoma de Barcelona, Spain)
  • Gábor PRÓSZÉKY (Hungarian Research Institute for Linguistics, Hungary)
  • Alessandra RICCARDI (University of Trieste, Italy)
  • Edina ROBIN (Eötvös Loránd University, Hungary)
  • Myriam SALAMA-CARR (University of Manchester, UK)
  • Mohammad Saleh SANATIFAR (independent researcher, Iran)
  • Sanjun SUN (Beijing Foreign Studies University, China)
  • Anikó SOHÁR (Pázmány Péter Catholic University,  Hungary)
  • Sonia VANDEPITTE (University of Gent, Belgium)
  • Albert VERMES (Eszterházy Károly University, Hungary)
  • Yifan ZHU (Shanghai Jiao Tong Univeristy, China)

Prof. Dr. Krisztina KÁROLY 
School of English and American Studies, Eötvös Loránd University
H-1088 Budapest, Rákóczi út 5., Hungary 
E-mail: 

  • WoS Arts & Humanities Citation Index
  • Wos Social Sciences Citation Index
  • WoS Essential Science Indicators
  • Scopus
  • Linguistics Abstracts
  • Linguistics and Language Behaviour Abstracts
  • Translation Studies Abstractst
  • CABELLS Journalytics

2023  
Web of Science  
Journal Impact Factor 1.0
Rank by Impact Factor Q2 (Linguistics)
Journal Citation Indicator 0.76
Scopus  
CiteScore 1.7
CiteScore rank Q1 (Language and Linguistics)
SNIP 1.223
Scimago  
SJR index 0.671
SJR Q rank Q1

Across Languages and Cultures
Publication Model Hybrid
Submission Fee

none

Article Processing Charge 900 EUR/article
Printed Color Illustrations 40 EUR (or 10 000 HUF) + VAT / piece
Regional discounts on country of the funding agency World Bank Lower-middle-income economies: 50%
World Bank Low-income economies: 100%
Further Discounts Editorial Board / Advisory Board members: 50%
Corresponding authors, affiliated to an EISZ member institution subscribing to the journal package of Akadémiai Kiadó: 100%
Subscription fee 2025 Online subsscription: 362 EUR / 398 USD
Print + online subscription: 420 EUR / 462 USD
Subscription Information Online subscribers are entitled access to all back issues published by Akadémiai Kiadó for each title for the duration of the subscription, as well as Online First content for the subscribed content.
Purchase per Title Individual articles are sold on the displayed price.

Across Languages and Cultures
Language English
Size B5
Year of
Foundation
1999
Volumes
per Year
1
Issues
per Year
2
Founder Akadémiai Kiadó
Founder's
Address
H-1117 Budapest, Hungary 1516 Budapest, PO Box 245.
Publisher Akadémiai Kiadó
Publisher's
Address
H-1117 Budapest, Hungary 1516 Budapest, PO Box 245.
Responsible
Publisher
Chief Executive Officer, Akadémiai Kiadó
ISSN 1585-1923 (Print)
ISSN 1588-2519 (Online)

Monthly Content Usage

Abstract Views Full Text Views PDF Downloads
Oct 2024 323 5 6
Nov 2024 105 2 4
Dec 2024 73 5 7
Jan 2025 69 3 4
Feb 2025 114 2 4
Mar 2025 48 1 3
Apr 2025 0 0 0