In recent decades, there has been a growing interest in multimodal translation studies. However, there is currently no suitable framework available for analyzing how the original meaning is reconstructed through the interaction of various semiotic resources in the multimodal target text. Under this context, this article proposes a systemic functional framework that integrates the systems of the meaning potential of semiotic resources, including language, sound (human voice), objects, space, and semiotic ensembles of the human body, and a tripartite classification of the relationships between semiotic resources. To exemplify the analytical potential of the framework, this article presents an example of analysis, taking an excerpt from Fences as the object of analysis.
Adami, E. (2015). What’s in a click? A social semiotic framework for the multimodal analysis of website interactivity. Visual Communication, 14(2), 133–153.
Bezemer, J., & Kress, G. (2008). Writing in multimodal texts: A social semiotic account of designs for learning. Written Communication, 25(2), 165–195.
Bonvillian, J. D., Lee, N. K., Dooley, T. T., & Loncke, F. T. (Eds.) (2020). Simplified signs: A manual sign-communication system for special populations (Vol. 2). Open Book Publishers.
Borba, M. C., O’Halloran, K. L., & Neves, L. X. (2022). Multimodality, systemic functional-multimodal discourse analysis and production of videos in mathematics education. In M. Danesi (Ed.), Handbook of cognitive mathematics (pp. 909–938). Springer.
Eggins, S., & Slade, D. (2006). Analysing casual conversation. Equinox Publishing Ltd.
Fan, W. F. (2001). A systemic-functional approach to grammatical metaphor. Foreign Language Teaching and Research Press.
Frank, T. (2019). Determinism and self-organization of human perception and performance. Springer.
Fricke, E. (2013). Towards a unified grammar of gesture and speech: A multimodal approach. In C. Müller, A. Cienki, E. Fricke, S. H. Ladewig, D. McNeill, & S. Teßendorf (Eds.), Body - language - communication: An international handbook on multimodality in human interaction (pp. 733–754). De Gruyter Mouton.
Gottlieb, H. (2018). Semiotics and translation. In K. Malmkjaer (Ed.), The Routledge handbook of translation studies and linguistics (pp. 45–63). Routledge.
Gu, Q. B. (2015). Multimodal discourse analysis of business presentation: A case study based on Steve Jobs product launch presentations (Doctoral dissertation). Available from China Dissertations Database.
Guijarro, A. J. M., & Sanz, M. J. P. (2009). On interaction of image and verbal text in a picture book: A multimodal and systemic functional study. In E. Ventola, & A. J. M. Guijarro (Eds.), The world told and the world shown (pp. 107–123). Palgrave Macmillan.
Halliday, M. A. K. (1978). Language as social semiotic: The social interpretation of language and meaning. Edward Arnold.
Halliday, M. A. K. (1985). Systemic background. In J. D. Benson, & W. S. Greaves (Eds.), Systemic perspectives on discourse (Vol. 1, pp. 1–15). Ablex Publishing Corporation.
Halliday, M. A. K. (1994). An introduction to functional grammar (2nd ed.). Edward Arnold.
Harley, J. M. (2016). Measuring emotions: A survey of cutting edge methodologies used in computer-based learning environment research. In S. Y. Tettegah, & M. Gartmeier (Eds.), Emotions, technology, design, and learning (pp. 89–114). Academic Press.
Jewitt, C. (2006). Technology, literacy, learning: A multimodal approach. Routledge.
Jewitt, C. (2009a). Introduction to part Ⅰ. In C. Jewitt (Ed.), The Routledge handbook of multimodal analysis (pp. 11–13). Routledge.
Jewitt, C. (2009b). An introduction to multimodality. In C. Jewitt (Ed.), The Routledge handbook of multimodal analysis (pp. 14–27). Routledge.
Jewitt, C. (2009c). Different approaches to multimodality. In C. Jewitt (Ed.), The Routledge handbook of multimodal analysis (pp. 28–39). Routledge.
Jewitt, C. (2013). Multimodal methods for researching digital technologies. In S. Price, C. Jewitt, & B. Brown (Eds.), The Sage handbook of digital technology research (pp. 250–265). Sage.
Jewitt, C. (2017). Towards a multimodal social semiotic agenda for touch. In S. Zhao, E. Djonov, A. Bjorkvall, & M. Boerils (Eds.), Advancing multimodal and critical discourse studies: Interdisciplinary research (pp. 79–93). Routledge.
Kress, G., & Van Leeuwen, T. (1996). Reading images: The grammar of visual design. Routledge.
Kress, G., & Van Leeuwen, T. (2002). Colour as a semiotic mode: Notes for a grammar of colour. Visual Communication, 1(3), 343–368.
Martinec, R. (2001). Interpersonal resources in action. Semiotica, (135), 117–145.
Martinec, R., & Salway, A. (2005). A system for image–text relations in new (and old) media. Visual Communication, 4(3), 337–371.
Norris, S. (2004). Analyzing multimodal interaction: A methodological framework. Routledge.
Norris, S. (2007). The micropolitics of personal national and ethnicity identity. Discourse & Society, 18(5), 653–674.
Norris, S. (2009). Modal density and modal configurations: Multimodal actions. In C. Jewitt (Ed.), The Routledge handbook of multimodal analysis (pp. 78–90). Routledge.
Norris, S. (2011). Three hierarchical positions of deictic gesture in relation to spoken language: A multimodal interaction analysis. Visual Communication, 10(2), 129–147.
O’Halloran, K. L. (2004). Visual semiosis in film. In K. L. O’Halloran (Ed.), Multimodal discouse analysis: Systemic functional perspectives (pp. 109–130). Continuum.
O’Halloran, K. L. (2005). Mathematical discourse: Language, symbolism and visual images. Continuum.
O’Halloran, K. L. (2009). Historical changes in the semiotic landscape: From calculation to computation. In C. Jewitt (Ed.), The Routledge handbook of multimodal analysis (pp. 98–113). Routledge.
O’Toole, M. (1994). The language of displayed art. Leicester University Press.
O’Toole, M. (2011). The language of displayed art (2nd ed.). Routledge.
Ranker, J. (2014). The emergence of semiotic resource complexes in the composing processes of young students in a literacy classroom context. Linguistics and Education, 25, 129–144.
Rossi, F., & Sindoni, M. G. (2017). The phantoms of the opera: Toward a multidimensional interpretative framework of analysis. In M. C. Sindoni, J. Wildfeuer, & K. L. O’Halloran (Eds.), Mapping multimodal performance studies (pp. 61–84). Routledge.
Stenglin, M. K. (2009). Space odyssey: Towards a social semiotic model of three-dimensional space. Visual Communication, 8(1), 35–64.
Van Leeuwen, T. (1999). Speech, music, sound. Macmillan.
Wignell, P., O’Halloran, K. L., Tan, S., Lange, R., & Chai, K. (2018). Image and text relations in ISIS materials and the new relations established through recontextualisation in online media. Discourse & Communication, 12(5), 535–559.
Wilmes, S., & Siry, C. (2021). Multimodal interaction analysis: A powerful tool for examining plurilingual students’ engagement in science practices. Research in Science Education, 51, 71–91.
Wilson, A. (1986). Fences. Penguin Group.
Wu, Y. (2021). The medial turn and multimodal translation studies. Journal of Foreign Languages, 44(1), 115–123.
Zhu, S. C. (2002). Value of modality and modulation: Mirror of communicative interpersonal relations. Journal of Shanghai Teachers University (Social Science), 31(5), 117–124.
Zhu, Y. S. (2007). Theory and methodology of multimodal discouse analysis. Foreign Language Research, (138), 82–86.