多模態在語言教育中扮演著至關重要的角色，可協助學習者傳遞知識或獲取新訊息。研究顯示，學習者對各種不同模態的使用抱持正面評價，但也擔憂此種多模態創作無益於語言發展。根據理論（Krashen, 1982; Mayer, 2001; Paivio, 1986; Schmidt, 1990），語言和非語言模態的結合其實有助學習，特別是在字彙發展方面，英語教室內也常見多模態學習資源，但很少相關研究分析著重於英語教師的多模態課程設計。因此，本研究採用深度式的單案例研究方法來分析一位教師的多模態字彙教學。
選定的研究對象在台灣北部的一所高職任教，其課堂錄影、課程包和半結構化訪談結果為此深度式研究的資料集。多模態分析的語料庫包含49個字彙的教學錄影，本研究參考七種模態（Cope & Kalantzis, 2020）、多媒體學習的常見學習資源（Mayer, 2001）和單字學習內容（Nation, 2001）進行影片編碼，辨別出教師所使用的多模態種類後，進一步利用滯後序列分析法（Bakeman & Gottman, 1997）來分析模態之間的轉換顯著性、並繪製事件轉移圖以呈現單字教學的序列。
將統計結果與教師的陳述進行了交叉分析後，本研究發現，教師傾向（1）改變文本的字體和佈局來減輕單字學習負擔，（2）使用圖形或實物例子來則增進學生對於單字意思的掌握，以及（3）提供多種聲音資源來協助學生熟悉單字發音。此外，教師教學中兩個最顯著的序列，即在展示圖片後重複定義和在播放錄音後重複發音，可以提高學習者對於多模態理解性。而兩個教師提及的序列未能顯著，原因在於教師會考量單字種類、多變課堂環境，彈性調整教學順序。總而言之，希望本研究能提供多模態設計參考方向也希望研究結果能強調一些潛在影響因素。

Multimodality plays a crucial role in language education. Modes can serve either as learners' output for knowledge representation or input for information acquisition. Researchers of multimodal composing have revealed learners' positive feedback toward using various modes, while concerns about linguistic development have been raised. According to many theories (Krashen, 1982; Mayer, 2001; Paivio, 1986; Schmidt, 1990), combining verbal and non-verbal modes facilitates learning, especially vocabulary development. English teachers tend to integrate multimodality in the classroom, but the examination of their lesson designs is scant. Therefore, this study adopts an embedded single-case study to analyze a teacher's multimodal vocabulary teaching.
The selected teacher works at a vocational high school in northern Taiwan. The study collected three datasets: classroom recordings, lesson packages, and semi-structured interview results. Video recordings of 49 vocabulary lessons were collected as a corpus for Multimodal Analysis. The presented modes were encoded based on seven meaning forms in Transpositional Grammar (Cope & Kalantzis, 2020), common input for multimedia learning (Mayer, 2001), and Nation's (2001) framework of vocabulary knowledge. Lag Sequential Analysis (Bakeman & Gottman, 1997) further analyzed and visualized the transitional patterns between modes.
We triangulated the statistical results with the teacher's statements. We found that the teacher tended to (1) change the text's fonts and layouts to alleviate learning burdens, (2) use graphics or physical examples to enhance meaning acquisition, and (3) provide multiple audio resources to familiarize learners with spoken forms. In addition, the two most significant sequences, repeating the definition after presenting pictures and repeating the pronunciation after playing the recordings, could increase the comprehensibility of input available to learners. Two teacher-reported sequences failed to be significant due to dynamic classroom conditions and the nature of the target word. To conclude, this study is important in providing teachers with a reference for multimodal designs and in highlighting some latent influential factors.

Abraham, L. B. (2008). Computer-mediated glosses in second language reading comprehension and vocabulary learning: A meta-analysis. Computer Assisted Language Learning, 21(3), 199-226.
Acha, J. (2009). The effectiveness of multimedia programmes in children's vocabulary learning. British Journal of Educational Technology, 40(1), 23–31.
Akbulut, Y. (2007). Effects of multimedia annotations on incidental vocabulary learning and reading comprehension of advanced learners of English as a foreign language. Instructional Science, 35, 499–517.
Allison, P. D., & Liker, J. K. (1982). Analyzing sequential categorical data on dyadic interaction: Acomment on Gottman. Psychological Bulletin, 91, 393–403.
Al-Seghayer, K. (2001). The effect of multimedia annotation modes on L2 vocabulary acquisition: A comparative study. Language Learning and Technology, 5(1), 202–232.
Apel, K. (2014). A comprehensive definition of morphological awareness. Topics in Language Disorders, 34(3), 197–209.
Bakeman, R., & Gottman, J. M. (1997). Observing interaction: An introduction to sequential analysis. Cambridge University Press.
Baldry, A., & Thibault, P. J. (2006). Multimodal transcription and text analysis: A multimedia toolkit and coursebook. Equinox.
Barcroft, J. (2015). Lexical input processing and vocabulary learning. John Benjamins.
Bateman, J. (2014). Using multimodal corpora for multimodal research. In C. Jewitt (Ed.), The routledge handbook of multimodal analysis (pp. 238–252). Routledge.
Bateman, J., Wildfeuer, J., & Hiippala, T. (2017). Multimodality: Foundations, research and analysis – A problem-oriented introduction. De Gruyter Mouton.
Belcher, D. (2017). On becoming facilitators of multimodal composing and digital design. Journal of Second Language Writing, 38, 80–85.
Bezemer, J. J., & Kress, G. R. (2016). Multimodality, learning and communication: A social semiotic frame. Routledge.
Boers, F., Warren, P., He, L., & Deconinck, J. (2017). Does adding pictures to glosses enhance vocabulary uptake from reading? System, 66, 113–129.
Chen, C., Chen, H. H., Tseng, W., & Liu, Y. (2022). Efficacy of different presentation modes for L2 video comprehension: Full versus partial display of verbal and nonverbal input. ReCALL, 35(1), 105–121.
Choi, J., & Yi, Y. (2015). Teachers’ integration of multimodality into classroom practices for English language learners. TESOL Journal, 7(2), 304–327.
Chun, D. M., & Plass, J. L. (1996). Effects of multimedia annotations on vocabulary acquisition. The Modern Language Journal, 80(2), 183–198.
Cope, B., & Kalantzis, M. (2000). Multiliteracies: Literacy learning and the design of social futures. Routledge.
Cope, B., & Kalantzis, M. (2009). New media, new learning. In D. R. Cole & D. L. Pullen (Eds.), Multiliteracies in motion: Current theory and practice (pp. 87-104). Routledge.
Cope, B., & Kalantzis, M. (2020). Making sense: reference, agency, and structure in a grammar of multimodal meaning. Cambridge University Press.
Cope, B., & Kalantzis, M. (2023). Towards education justice: Multiliteracies revisited. In G. C. Zapata, M. Kalantzis and B. Cope (Eds.), Multiliteracies in international educational contexts: Towards education justice (pp. 1-26). Routledge.
Diamantopoulou, S., & Ørevik, S. (2021). Multimodality in English language learning: The case of EAL. In S. Diamantopoulou & Ørevik S. (Eds.), Multimodality in English language learning (pp. 3-15). Routledge.
Douglas Fir Group. (2016). A transdisciplinary framework for SLA in a multilingual world. The Modern Language Journal, 100, 19–47.
Dressman, M. (2019). Multimodality and language learning. In M. Dressman & R. W. Sadler (Eds.), The handbook of informal language learning (pp. 39-55). John Wiley & Sons.
Hafner, C. A. (2013). Embedding digital literacies in English language teaching: Students’ digital video projects as multimodal ensembles. TESOL Quarterly, 48(4), 655-685.
Halliday, M. A. K. (1985). An introduction to functional grammar. Edward Arnold.
Hellmich, E. A., Castek, J., Smith, B. E., Floyd, R., & Wen, W. (2021). Student perspectives on multimodal composing in the L2 classroom: Tensions with audience, media, learning and sharing. English Teaching, 20(2), 210-226.
Henriksen, B. (1999). Three dimensions of vocabulary development. Studies in Second Language Acquisition, 21(2), 303–317.
Huang, X., Kim, N., & Christianson, K. (2019). Gesture and vocabulary learning in a second language. Language Learning, 69(1). 177-197.
Hulstijn, J. H. (2003). Incidental and intentional learning. In C. J. Doughty & M. H. Long (Eds.), The handbook of second language acquisition (pp. 349-381). Blackwell Publishing.
Iverson, J. M., & Goldin-Meadow, S. (2005). Gesture paves the way for language development. Psychological Science, 16, 368–371.
Jewitt, C. (2008). Multimodality and literacy in school classrooms. Review of Research in Education, 32, 241-267.
Jewitt, C. (2009). An introduction to multimodality. In C. Jewitt (Ed.), The Routledge handbook of multimodal analysis (pp. 14–27). Routledge.
Jewitt, C. (2015). Multimodal analysis. In A. Georgakopoulou & T. Spilioti (Eds.), The Routledge handbook of language and digital communication (pp.69-84), Routledge.
Jewitt, C., Bezemer, J., & O'Halloran, K. (2016). Introducing multimodality. [eBook edition]. Routledge.
Jiang, L., & Luk, J. (2016). Multimodal composing as a learning activity in English classrooms: Inquiring into the sources of its motivational capacity. System, 59, 1–11.
Kalantzis, M., & Cope, B. (2022). After language: A grammar of multimodal transposition. In C. Lütge (Ed.), Foreign language learning in the digital age: Theory and pedagogy for developing literacies. Routledge.
Kaplan-Rakowski, R., & Loranc-Paszylk, B. (2019). The impact of verbal and nonverbal auditory resources on explicit foreign language vocabulary learning. System, 85, 102-114.
Kim, Y., & Belcher, D. (2020). Multimodal Composing and Traditional Essays: Linguistic performance and learner Perceptions. RELC Journal, 51(1), 86–100.
Kim, Y., & Kang, S. (2020). Writing to make meaning through collaborative multimodal composing among Korean EFL Learners: Writing processes, writing quality and student perception. Computer & Compositions, 58.
Kim, D., & Kim, D. J. (2012). Effect of screen size on multimedia vocabulary learning. British Journal of Educational Technology, 43(1), 62–70.
Krathwohl, D. R., Bloom, B. S., & Masia, B. B. (1964). Taxonomy of educational objectives: Handbook II: Affective domain. Longman Inc.
Krashen, S. (1982). Principles and practice in second language acquisition. Pergamon Press.
Kress, G. (2010). Multimodality: A Social Semiotic Approach to Contemporary Communication. Routledge.
Kress, G., Jewitt, C., Ogborn, J., & Tsatsarelis, C. (2000). Multimodal teaching and learning: The rhetorics of the science classroom. Bloomsbury Academic.
Kress, G., & Van Leeuwen, T. (2001). Multimodal discourse: The modes and media of contemporary communication. Oxford University Press.
Lee, S., Lo, Y., & Chin, T. (2021). Practicing multiliteracies to enhance EFL learners’ meaning making process and language development: a multimodal Problem-based approach. Computer Assisted Language Learning, 34(1–2), 66–91.
Li, M. (2020). Multimodal pedagogy in TESOL teacher education: Students’ perspectives. System, 94.
Liang, W. J., & Lim, F. V. (2020). A pedagogical framework for digital multimodal composing in the English Language classroom. Innovation in Language Learning and Teaching, 15(4), 306-320.
Lim, F. V., Cope, B., & Kalantzis, M. (2022). Metalanguage for learning: Rebalancing the cognitive with the social-material. Frontiers in Communication, 7.
Lin, C. (2009). Learning action verbs with animation. JALT CALL Journal, 5(3), 23–40.
Lin, C., & Tseng, Y. (2012). Videos and animations for vocabulary learning: A study on difficult words. Turkish Online Journal of Educational Technology, 11, 346–355.
Lindstromberg, S. (2019). Intentional L2 vocabulary learning. In S. Webb (Ed.), The routledge handbook of vocabulary studies (pp. 240-254). Routledge.
Lo, Y. H. G., Lee, S. Y., & Tien, W. C. F. (2020). Transforming a magazine into a video involving a target audience: A multiliteracies case study in an EFL context. Educational Technology & Society, 23(3), 27–39.
Lomicka, L. L. (1998). “To gloss or not to gloss”: An investigation of reading comprehension online. Language Learning & Technology, 1(2), 41-50.
Mackey, A., & Gass, S. M. (2015). Second language research. Routledge.
Malone, T. W., & Lepper, M. R. (1987). Making learning fun: a taxonomy of intrinsic motivations for learning. In R. E. Snow, & M. J. Farr (Eds.), Aptitude, learning, and instruction: III. Conative and affective process analyses (pp. 223-253). Routledge.
Manchón, R. M. (2017). The potential impact of multimodal composition on language learning. Journal of Second Language Writing, 38, 94-95.
Mayer, R. E. (2001). Multimedia learning. Cambridge University Press.
Mayer, R. E., & Fiorella, L. (2021). The Cambridge handbook of multimedia learning. Cambridge University Press.
Mayer, R., & Moreno, R. (2002). Aids to computer-based multimedia learning. Learning and Instruction, 12, 107-119.
McKinley, J., & Thompson, G. (2018). Washback effect in teaching English as an international language. In J. I. Liontas, M. DelliCarpini, & S. Abrar-ul-Hassan (Eds.), The TESOL encyclopedia of English language teaching (pp. 1–12). Wiley.
McNeill, D. (1992). Hand and mind: What gestures reveal about thought. University of Chicago Press.
Miller, G. A., & Fellbaum, C. (1991). Semantic networks of English. Cognition, 41(1-3), 197–229.
Mohsen, M. A., & Balakumar, M. (2011). A review of multimedia glosses and their effects on L2 vocabulary acquisition in CALL literature. ReCALL, 23(2), 135–159.
Montero-Perez, M., Van Den Noortgate, W., & Desmet, P. (2013). Captioned video for L2 listening and vocabulary learning: A meta-analysis. System, 41(3), 720–739.
Moreno, R., & Mayer, R. E. (2007). Interactive multimodal learning environments. Educational Psychology Review, 19(3), 309-326.
Nakata, T. (2019). Learning words with flash cards and word cards. In S. Webb (Ed.), The routledge handbook of vocabulary studies (pp. 304-319). Routledge.
Nation, I. S. P. (2001). Learning vocabulary in another language. Cambridge University Press.
Nation, P. (2007). The four strands. Innovation in Language Learning and Teaching, 1(1), 2–13.
New London Group. (1996). A pedagogy of multiliteracies: Designing social factors. Harvard Educational Review, 66(1), 60-92.
O’Halloran, K. L., & Lim, F. V. (2014). Systemic functional multimodal discourse analysis. In S. Norris & C. Maier (Eds.), Texts, images and interactions: A reader in multimodality (pp. 137–154). Mouton de Gruyter.
Paivio, A. (1986). Mental representations. Oxford University Press.
Paivio, A. (1990). Mental representation: A dual-coding approach. Oxford University Press.
Perez, M. M. (2020). Multimodal input in SLA research. Studies in Second Language Acquisition, 42(3), 653-663.
Peters, E., & Muñoz, C. (2020). Introduction to the special issue: Language learning from multimodal input. Studies in Second Language Acquisition, 42, 489-497.
Pi, Z., Zhu, F., Zhang, Y., & Yang, J. (2021). An instructor’s beat gestures facilitate second language vocabulary learning from instructional videos: Behavioral and neural evidence. Language Teaching Research, 1-29.
Qu, W. (2017). For L2 writers, it is always the problem of the language. Journal of Second Language Writing, 38, 92-93.
Ramezanali, N., & Faez, F. (2019). Vocabulary learning and retention through multimedia glossing. Language Learning and Technology, 23(2), 105–124. 10125/44685
Ramezanali, N., Uchihara, T., & Faez, F. (2020). Efficacy of multimodal glossing on second language vocabulary learning: A meta‐analysis. TESOL Quarterly, 55(1), 105–133.
Read, J. (2004). Plumbing the depths: How should the construct of vocabulary knowledge be defined? In P. Bogaards & B. Laufer (Eds.), Vocabulary in a second language (pp. 209–227). John Benjamins.
Sadeghi, K., Khezrlou, S., & Modirkhameneh, S. (2016). Calling Iranian learners of L2 English: Effect of gloss type on lexical retention and reading performance under different learning conditions. Journal of Research in Reading, 39(1), 1–21.
Schmidt, R. (1990). The role of consciousness in second Language learning. Applied Linguistics, 11(2), 129-158.
Stein, P., & Newfield, D. (2006). Multiliteracies and multimodality in English in education in Africa: Mapping the terrain. English Studies in Africa, 49(1). 1-21.
Teng, M. F. (2023). The effectiveness of multimedia input on vocabulary learning and retention. Innovation in Language Learning and Teaching, 17(3), 738–754.
TESOL International Association. (2014). TESOL research agenda 2014.
Türk, E., & Erçetin, G. (2014). Effects of interactive versus simultaneous display of multimedia glosses on L2 reading comprehension and incidental vocabulary learning. Computer Assisted Language Learning, 27(1), 1–26.
Van Leeuwen, T. (2005). Introducing social semiotics. Psychology Press.
Vilkaitė-Lozdienė, L., & Schmitt, N. (2019). Frequency as a guide for vocabulary usefulness. In S. Webb (Ed.), The routledge handbook of vocabulary studies (pp. 81-96). Routledge.
Warren, P., Boers, F., Grimshaw, G., & Siyanova-Chanturia, A. (2018). The effect of gloss type on learners’ intake of new words during reading: Evidence from eye-tracking. Studies in Second Language Acquisition, 40, 883–906.
Webb, S. (2019). The routledge handbook of vocabulary studies. Routledge.
Xu, J. (2010). Using multimedia vocabulary annotations in L2 reading and listening activities. The CALICO Journal, 27(2), 311–327.
Yang, Y. C., & Wu, W. (2012). Digital storytelling for enhancing student academic achievement, critical thinking, and learning motivation: A year-long experimental study. Computers & Education, 59(2), 339-352.
Yeh, Y., & Wang, C. (2003). Effects of multimedia vocabulary annotations and learning styles on vocabulary learning. CALICO Journal, 21(1), 131–144.
Yin, R. (2008). Case study research: Design and methods. SAGE Publication.
Yoshii, M. (2006). L1 and L2 glosses: Their effects on incidental vocabulary learning. Language Learning and Technology, 10(3), 85–101.
Yoshii, M., & Flaitz, J. (2002). Second language incidental vocabulary retention: The effect of picture and annotation types. CALICO Journal, 20(1), 33–58.
Zhang, Y. (2020). Preparing teachers to work with English learners: A multimodal vocabulary mini‐lesson project. TESOL Journal, 12(2).
Zhang, M., Akoto, M., & Li., M. (2021). Digital multimodal composing in post-secondary L2 settings: A review of the empirical landscape. Computer Assisted Language Learning.
Zhang, R., & Zou, D. (2022). A state-of-the-art review of the modes and effectiveness of multimedia input for second and foreign language learning. Computer Assisted Language Learning, 35(9), 2790–2816.