Touch design and narrative interpretation: a social semiotic approach to picture book apps

Book chapter

Zhao, Sumin and Unsworth, L.. (2017). Touch design and narrative interpretation: a social semiotic approach to picture book apps. In In N. Kucirkova and G. Falloon (Ed.). Apps, Technology and Younger Learners: International evidence for teaching pp. 89 - 101 Routledge.
AuthorsZhao, Sumin and Unsworth, L.
EditorsN. Kucirkova and G. Falloon
Abstract

Interactivity is often considered as the defining feature of a book app. However, what constitutes interactivity in the context of apps is not always clear. We postulate in this chapter a multimodal social semiotic theory of interactivity, which considers interactivity as not simply a function of technology, but also a resource for meaning-making. We distinguish two basic types of interactivity-intra-text and extra-text-incorporated in the touch design, and explore the different functions they perform in a broad range of picture book apps. In particular, we look at the app version of The Heart and the Bottle in depth, and illustrate how interactive design elements help to create an interpretative possibility of the story. We suggest that a better understanding of interactive touch design would promote more effective adult-child interactions around mobile applications.

KeywordsInteractivity; multimodal; e-book; literacy; semiotic; meaning-making
Page range89 - 101
Year2017
Book titleApps, Technology and Younger Learners: International evidence for teaching
PublisherRoutledge
Place of publicationUnited Kingdom
ISBN9781138927872
Research GroupInstitute for Learning Sciences and Teacher Education (ILSTE)
Publisher's version
File Access Level
Controlled
Permalink -

https://acuresearchbank.acu.edu.au/item/873qx/touch-design-and-narrative-interpretation-a-social-semiotic-approach-to-picture-book-apps

Log in to edit

Restricted files

Publisher's version

  • 181
    total views
  • 0
    total downloads
  • 0
    views this month
  • 0
    downloads this month
These values are for the period from 19th October 2020, when this repository was created.

Export as

Related outputs

Reading Images for Knowledge Building : Analysing Infographics in School Science
Martin, J. R. and Unsworth, Leonard Charles. (2023). Reading Images for Knowledge Building : Analysing Infographics in School Science Routledge. https://doi.org/10.4324/9781003164586
Evaluating Students’ Coding of Animated Narratives as Contemporary Multimodal Authoring in the Middle School English Curriculum
Unsworth, Len. (2023). Evaluating Students’ Coding of Animated Narratives as Contemporary Multimodal Authoring in the Middle School English Curriculum [Dataset]. Australian Catholic University. https://doi.org/10.26199/acu.90y6z
Multimodal composing in the English classroom : Recontextualizing the curriculum to learning
Lim, Fei Victor and Unsworth, Len. (2023). Multimodal composing in the English classroom : Recontextualizing the curriculum to learning. English in Education. 57(2), pp. 102-119. https://doi.org/10.1080/04250494.2023.2187696
Multimodal literacy in a new era of educational technology : Comparing points of view in animations of children's and adult literature
Unsworth, Len. (2023). Multimodal literacy in a new era of educational technology : Comparing points of view in animations of children's and adult literature. ECNU Review of Education. pp. 1-22. https://doi.org/10.1177/20965311231179738
Including visual representations within senior high school biology assessment : Considerations of grammatical complexity
Fenwick, Lisl and Unsworth, Len. (2023). Including visual representations within senior high school biology assessment : Considerations of grammatical complexity. Curriculum Journal. 34(3), pp. 412-436. https://doi.org/10.1002/curj.181
Visualization type and frequency in final year high school science examinations
Unsworth, Len and Herrington, Michele. (2023). Visualization type and frequency in final year high school science examinations. Research in Science Education. 53(4), pp. 707-725. https://doi.org/10.1007/s11165-022-10094-2
Literacy for digital futures : Mind, body, text
Mills, Kathy A., Unsworth, Len and Scholes, Laura. (2023). Literacy for digital futures : Mind, body, text Routledge. https://doi.org/10.4324/9781003137368
Multimodal literacy in school science
Unsworth, Len, Tytler, Russell, Fenwick, Lisl, Humphrey, Sally, Chandler, Paul, Herrington, Michele and Pham, Lam. (2022). Multimodal literacy in school science Routledge. https://doi.org/10.4324/9781003150718
Research vignette 8 : Animated movie adaptations of literary picture books
Unsworth, Len. (2022). Research vignette 8 : Animated movie adaptations of literary picture books. In In Serafini, Frank (Ed.). Beyond the visual : An introduction to researching multimodal phenomena pp. 125-129 Teachers College Press.
Teacher orchestration of language and gesture in explaining science concepts in images
Ngo, Thu, Unsworth, Len and Herrington, Michele. (2022). Teacher orchestration of language and gesture in explaining science concepts in images. Research in Science Education. 52(3), pp. 1013-1030. https://doi.org/10.1007/s11165-021-10011-z
Methodological challenges for collaborative research in senior science classrooms
Unsworth, Len, Tytler, Russell and Fenwick, Lisl. (2021). Methodological challenges for collaborative research in senior science classrooms. In In White, Peta J., Tytler, Russell, Ferguson, Joseph Paul and Cripps Clark, John (Ed.). Methodological approaches to STEM education research ; volume 2 pp. 66-84 Cambridge Scholars Publishing.
High school science infographics : Multimodal meaning complexes in composite image-language ensembles
Unsworth, Len. (2021). High school science infographics : Multimodal meaning complexes in composite image-language ensembles. Pensamiento Educativo, Revista de Investigacion Educacional Latinoamericana . 58(2), pp. 1-18. https://doi.org/10.7764/PEL.58.2.2021.9
Multiliteracies in the Singapore English language classroom : Perceptions and practices
Lim, Fei Victor, Weninger, Csilla, Chia, Alexius, Nguyen, Thi Thu Ha, Tan, Jia Min, Adams, Jonathon, Tan-Chia, Lydia, Peters, Charles Matthew, Towndrow, Phillip Alexander and Unsworth, Len. (2020). Multiliteracies in the Singapore English language classroom : Perceptions and practices Singapore, Singapore: National Institute of Education, Nanyang Technological University.
Developing the multimodal language of emotions of low SES primary students
Mills, Kathy, Unsworth, Len and Stone, Bessie. (2020). Developing the multimodal language of emotions of low SES primary students Brisbane, QLD: Institute for Learning Sciences and Teacher Education, Australian Catholic University. https://doi.org/10.13140/RG.2.2.16361.03682
Multimodal language of attitude in digital composition
Mills, Kathy A., Stone, Bessie G., Unsworth, Len and Friend, Lesley. (2020). Multimodal language of attitude in digital composition. Written Communication. 37(2), pp. 135-166. https://doi.org/10.1177/0741088319897978
English language teaching of attitude and emotion in digital multimodal composition
Unsworth, Len and Mills, Kathy A.. (2020). English language teaching of attitude and emotion in digital multimodal composition. Journal of Second Language Writing. 47, p. Article 100712. https://doi.org/10.1016/j.jslw.2020.100712
Infusing pro-environmental values in science education: A multimodal analysis of ecology animations for children
Mandy Hoi Man Yu, Dezheng (William) Feng and Len Unsworth. (2020). Infusing pro-environmental values in science education: A multimodal analysis of ecology animations for children. In In Len Unsworth (Ed.). Learning from Animations in Science Education pp. 55-74 Springer Nature. https://doi.org/10.1007/978-3-030-56047-8_3
A Multidisciplinary Perspective on Animation Design and Use in Science Education
Unsworth, Len. (2020). A Multidisciplinary Perspective on Animation Design and Use in Science Education. In In Unsworth, Len (Ed.). Learning from Animations in Science Education pp. 3-22 Springer Nature Switzerland AG. https://doi.org/10.1007/978-3-030-56047-8_1
Intermodal relations, mass and presence in school science explanation genres
Unsworth, Len. (2020). Intermodal relations, mass and presence in school science explanation genres. In In Zappavigna, Michele and Dreyfus, Shoshana (Ed.). Discourses of hope and reconciliation : On J. R. Martin's contribution to systemic functional linguistics pp. 131-151 Bloomsbury Academic. https://doi.org/10.5040/9781350116092.ch-007
Developing the multimodal language of emotions of low SES primary students (LP150100030, 2016-2019)
Mills, Kathy, Unsworth, Len, Williamson, Mark, Ferguson, Angela, Quinn, M., Clay, Rosalind, Trembath, Melissa and Maksoud, Tony. (2019). Developing the multimodal language of emotions of low SES primary students (LP150100030, 2016-2019) [Dataset]. https://doi.org/10.26199/5dfbfbaffd787
The digital mediation of emotions in late modernity
Mills, Kathy A., Unsworth, Len and Barton, Georgina. (2019). The digital mediation of emotions in late modernity. In In Patulny, Roger, Bellocchi, Alberto, Olson, Rebecca E., Khorana, Sukhmani, McKenzie, Jordan and Peterie, Michelle (Ed.). Emotions in late modernity pp. 190-208 Routledge. https://doi.org/10.4324/9781351133319-17
The multimodal construction of race : A review of critical race theory research
Mills, Kathy A. and Unsworth, Len. (2018). The multimodal construction of race : A review of critical race theory research. Language and Education. 32(4), pp. 313-332. https://doi.org/10.1080/09500782.2018.1434787
What images reveal: A comparative study of science images between Australian and Taiwanese junior high school textbooks
Ge, Y. P., Unsworth, Len, Wang, Kuo-Hua and Chang, Huey-Por. (2018). What images reveal: A comparative study of science images between Australian and Taiwanese junior high school textbooks. Research in Science Education. 48(6), pp. 1409 - 1431. https://doi.org/10.1007/s11165-016-9608-9
Sensory literacies, the body, and digital media
Mills, Kathy A., Unsworth, Len and Exley, Beryl. (2018). Sensory literacies, the body, and digital media. In In Mills, Kathy A., Stornaiuolo, Amy, Smith, Anna and Zaher Pandya, Jessica (Ed.). Handbook of writing, literacies, and education in digital cultures pp. 26-36 Routledge. https://doi.org/10.4324/9781315465258-4
iPad animations : Powerful multimodal practices for adolescent literacy and emotional language
Mills, Kathy A. and Unsworth, Len. (2018). iPad animations : Powerful multimodal practices for adolescent literacy and emotional language. Journal of Adolescent and Adult Literacy. 61(6), pp. 609-620. https://doi.org/10.1002/jaal.717
Image design for enhancing science learning: helping students build taxonomic meanings with salient tree structure images
Ge, Yun-Ping, Unsworth, Len, Wang, Kuo-Hua and Chang, Huey-Por. (2018). Image design for enhancing science learning: helping students build taxonomic meanings with salient tree structure images. In In K.S Tang and K. Danielsson (Ed.). Global Developments in Literacy Research for Science Education pp. 237 - 258 Springer International Publishing. https://doi.org/10.1007/978-3-319-69197-8_15
Image-language interaction in text comprehension: reading reality and national reading tests
Unsworth, L.. (2017). Image-language interaction in text comprehension: reading reality and national reading tests. In In C. Ng and B. Bartlett (Ed.). Improving Reading and Reading Engagement in the 21st Century: International Research and Innovation pp. 99 - 118 Springer. https://doi.org/10.1007/978-981-10-4331-4_5
The effects of explicit visual cues in reading biological diagrams
Ge, YP, Unsworth, Len and Wang, KH. (2017). The effects of explicit visual cues in reading biological diagrams. International Journal of Science Education. 39(5), pp. 605 - 626. https://doi.org/10.1080/09500693.2017.1297549
Multimodal literacy
Mills, Kathy A. and Unsworth, Len. (2017). Multimodal literacy. In Oxford research encyclopedia of education pp. 1-29 Oxford University Press. https://doi.org/10.1093/acrefore/9780190264093.013.232
Functional grammatics: reconceptualizing knowledge about language and image for school English
Macken-Horarik, Mary, Love, Kristina, Sandiford, Carmel and Unsworth, L.. (2017). Functional grammatics: reconceptualizing knowledge about language and image for school English Routledge.
Stance-taking as negotiating academic conflict in applied linguistics research article discussion sections
Cheng, Fei-Wen and Unsworth, Len. (2016). Stance-taking as negotiating academic conflict in applied linguistics research article discussion sections. Journal of English for Academic Purposes. 24, pp. 43 - 57. https://doi.org/10.1016/j.jeap.2016.09.001
The literacy curriculum : A critical review
Mills, Kathy A. and Unsworth, Len. (2016). The literacy curriculum : A critical review. In In Wyse, Dominic, Hayward, Louise and Pandya, Jessica (Ed.). The SAGE handbook of curriculum, pedagogy and assessment pp. 621-637 SAGE Publications Inc..
Interpretive responses to images in picture books by primary and secondary school students : Exploring curriculum expectations of a 'visual grammatics'
Unsworth, Len and Macken-Horarik, Mary. (2015). Interpretive responses to images in picture books by primary and secondary school students : Exploring curriculum expectations of a 'visual grammatics'. English in Education (online version). 49(1), pp. 56 - 79. https://doi.org/10.1111/eie.12047
New ways of working 'with grammar in mind' in school English : Insights from systemic functional grammatics
Macken-Horarik, Mary, Sandiford, Carmel, Love, Kristina and Unsworth, Leonard. (2015). New ways of working 'with grammar in mind' in school English : Insights from systemic functional grammatics. Linguistics and Education. 31, pp. 145 - 158. https://doi.org/10.1016/j.linged.2015.07.004
New ways of working 'with grammar in mind' in school English : Insights from systemic functional grammatics
Macken-Horarik, Mary, Sandiford, Carmel, Love, Kristina and Unsworth, Len. (2015). New ways of working 'with grammar in mind' in school English : Insights from systemic functional grammatics. Linguistics and Education. 31, pp. 145 - 158. https://doi.org/10.1016/j.linged.2015.07.004
Reworking the appraisal framework in ESL research : Refining attitude resources
Ngo, Thu and Unsworth, Len. (2015). Reworking the appraisal framework in ESL research : Refining attitude resources. Functional Linguistics (online). 2(1), pp. 1 - 24. https://doi.org/10.1186/s40554-015-0013-x
Persuasive narratives : Evaluative images in picture books and animated movies
Unsworth, Len. (2015). Persuasive narratives : Evaluative images in picture books and animated movies. Visual Communication. 14(1), pp. 73 - 96. https://doi.org/10.1177/1470357214541762
Curriculum literacies: Accessing disciplinary discourses
Unsworth, Len. (2015). Curriculum literacies: Accessing disciplinary discourses. In In J. Turnbill, G. Barton and C. Brock (Ed.). Teaching writing in today's classrooms: Looking back to look forward pp. 262 - 280 Australian Literacy Educators' Association.
From 'bored witless' to 'rhetorical nous': Teacher orientation to knowledge about language and strengthening student persuasive writing
Love, Kristina, Sandiford, Carmel, Macken-Horarik, Mary and Unsworth, Len. (2014). From 'bored witless' to 'rhetorical nous': Teacher orientation to knowledge about language and strengthening student persuasive writing. English in Australia. 49(3), pp. 43 - 56.
The logogenesis of writing to learn: A systemic functional perspective
Klein, Perry D. and Unsworth, Len. (2014). The logogenesis of writing to learn: A systemic functional perspective. Linguistics and Education. 26, pp. 1 - 17. https://doi.org/10.1016/j.linged.2013.12.003
New challenges for literature study in primary school English : Building teacher knowledge and know-how through systemic functional theory
Macken-Horarik, Mary Rose and Unsworth, Len. (2014). New challenges for literature study in primary school English : Building teacher knowledge and know-how through systemic functional theory. Onomázein. https://doi.org/10.7764/onomazein.alsfal.1
Music, multiliteracies and multimodality : Exploring the book and movie versions of Shaun Tan's The Lost Thing
Barton, Georgina and Unsworth, Len. (2014). Music, multiliteracies and multimodality : Exploring the book and movie versions of Shaun Tan's The Lost Thing. Australian Journal of Language and Literacy. 37(1), pp. 3 - 20.
Interfacing visual and verbal narrative art in paper and digital media: Recontextualising literature and literacies
Unsworth, L.. (2014). Interfacing visual and verbal narrative art in paper and digital media: Recontextualising literature and literacies. In In G. Barton (Ed.). Literacy in the arts: Retheorising learning and teaching pp. 55 - 76 Springer. https://doi.org/10.1007/978-3-319-04846-8_4
Comparing the images in Taiwanese and Australian science textbooks by grammar of visual design : An Example of biological classification
Ge, YP, Wang, KH, Chung, CH, Chang, HP and Unsworth, Len. (2014). Comparing the images in Taiwanese and Australian science textbooks by grammar of visual design : An Example of biological classification. Chinese Journal of Science Education. 22(2), pp. 109 - 134. https://doi.org/10.6173/CJSE.2014.2202.01
Point of view in picture books and animated film adaptations: Informing critical multimodal comprehension and composition pedagogy
Unsworth, Len. (2014). Point of view in picture books and animated film adaptations: Informing critical multimodal comprehension and composition pedagogy. In In E. Djonov and S. Zhao (Ed.). pp. 202 - 216 Routledge.
From 'bored witless' to 'rhetorical nous' : Teacher orientation to knowledge about language and strengthening student persuasive writing
Love, Kristina, Sandiford, Carmel, Macken-Horarik, Mary and Unsworth, Leonard. (2014). From 'bored witless' to 'rhetorical nous' : Teacher orientation to knowledge about language and strengthening student persuasive writing. English in Australia. 49(3), pp. 43 - 56.
Multimodal reading comprehension: Curriculum expectations and large-scale literacy testing practices
Unsworth, Len. (2014). Multimodal reading comprehension: Curriculum expectations and large-scale literacy testing practices. Pedagogies: An international journal. 9(1), pp. 26 - 44. https://doi.org/10.1080/1554480X.2014.878968
Children's emotions and multimodal appraisal of places : Walking with the camera
Mills, Kathy, Unsworth, Len, Bellochi, Alberto, Park, Ji Yong and Ritchie, Stephen. (2014). Children's emotions and multimodal appraisal of places : Walking with the camera. Australian Journal of Language and Literacy. 37(3), pp. 171-181.
Multimodality and reading: The construction of meaning through image-text interaction
Unsworth, Len and Cléirigh, Chris. (2014). Multimodality and reading: The construction of meaning through image-text interaction. In In C. Jewitt (Ed.). The Routledge handbook of multimodal analysis pp. 176 - 188 Routledge.
Analysing the semiotic potential of typographic resources in picture books in english and in translation
Unsworth, Len, Meneses, Alejandra, Ow Gonzalez, Maili and Castillo, Guillermo. (2014). Analysing the semiotic potential of typographic resources in picture books in english and in translation. International Research in Children's Literature. 7(2), pp. 117 - 135. https://doi.org/10.3366/ircl.2014.0127
Reading Visual Narratives: Image Analysis of Children's Picture Books
Painter, Clare, Martin, J. R. and Unsworth, Len. (2013). Reading Visual Narratives: Image Analysis of Children's Picture Books Equinox Publishing Ltd.
Re-configuring image-language relations and interpretive possibilities in picture books as animated movies: A site for developing multi-modal literacy pedagogy
Unsworth, Len. (2013). Re-configuring image-language relations and interpretive possibilities in picture books as animated movies: A site for developing multi-modal literacy pedagogy. Ilha do Desterro: a journal of English language, literatures in English, and cultural studies. 64, pp. 15 - 47. https://doi.org/10.5007/2175-8026.2013n64p15
Digital natives? New and old media and children's language acquisition
Bittman, M, Rutherford, Leonie, Brown, Jude and Unsworth, Len. (2012). Digital natives? New and old media and children's language acquisition. Family Matters.
Enhancing expressions of attitudes: Achieving equity for international students in everyday communication
Unsworth, Len, Feez, Susan and Ngo, Thu. (2012). Enhancing expressions of attitudes: Achieving equity for international students in everyday communication. TESOL in Context. S3, pp. 1 - 10.
Evaluation of students' digital animated multimodal narratives and the indentification of high-performing classrooms
Chandler, Paul D., Unsworth, Len and O'Brien, Annemaree. (2012). Evaluation of students' digital animated multimodal narratives and the indentification of high-performing classrooms. The Journal of Literacy and Technology: an international online academic journal. 13(3), pp. 80 - 127.
Image-language interaction in online reading environments: Challenges for students' reading comprehension
Chan, Eveline and Unsworth, Len. (2011). Image-language interaction in online reading environments: Challenges for students' reading comprehension. The Australian Educational Researcher. 38(2), pp. 181 - 202. https://doi.org/10.1007/s13384-011-0023-y
A grammatics 'good enough' for school English in the 21st century: Four challenges in realising the potential
Macken-Horarik, Mary Rose, Love, Kristina and Unsworth, Len. (2011). A grammatics 'good enough' for school English in the 21st century: Four challenges in realising the potential. Australian Journal of Language and Literacy. 34(1), pp. 9 - 23.
Analysis and comprehension of multimodal texts
Daly, Ann and Unsworth, Len. (2011). Analysis and comprehension of multimodal texts. Australian Journal of Language and Literacy. 34(1), pp. 61 - 80.
Digital Natives? :New and Old Media and Children's Outcomes
Bittman, M, Rutherford, Leonie, Brown, Jude and Unsworth, Len. (2011). Digital Natives? :New and Old Media and Children's Outcomes. Australian Journal of Education. 55(2), pp. 161 - 175.
Organizing Visual Meaning: Framing and Balance in Picture-Book Images
Painter, Clare, Martin, J, J. and Unsworth, Len. (2010). Organizing Visual Meaning: Framing and Balance in Picture-Book Images. In In S Dreyfus, Susan Hood and Maree Stenglin (Ed.). pp. 125 - 143 Continuum International Publishing Group Ltd.
Multiliteracies, e-literature and English teaching
Unsworth, Len. (2010). Multiliteracies, e-literature and English teaching. Language and Education. 22(1), pp. 62 - 75. https://doi.org/10.2167/le726.0
Introducing Multimodal Literacy to Young Children Learning English as a Second Language (ESL)
Unsworth, Leonard and Bush, Rosemary. (2010). Introducing Multimodal Literacy to Young Children Learning English as a Second Language (ESL). In In David R Cole and Darren L Pullen (Ed.). pp. 59 - 83 Routledge.
Multiliteracies and metalanguage: Describing image/text relations as a resource for negotiating multimodal texts
Unsworth, L.. (2009). Multiliteracies and metalanguage: Describing image/text relations as a resource for negotiating multimodal texts. In In J. Coiro, M. Knobel and C. Lankshear et al. (Ed.). Handbook of Research on New Literacies pp. 377 - 405 Routledge. https://doi.org/10.4324/9781410618894
Bridging multimodal literacies and national assessment programs in literacy
Unsworth, Lem and Chan, Eveline. (2009). Bridging multimodal literacies and national assessment programs in literacy. Australian Journal of Language and Literacy.
Exploring the Narrative Art of David Weisner: Using a Grammar of Visual Design and Learning Experiences on the World Wide Web
Unsworth, Len and Ortigas, Isable. (2008). Exploring the Narrative Art of David Weisner: Using a Grammar of Visual Design and Learning Experiences on the World Wide Web. L1 Educational Studies in Language and Literature.
Explicating inter-modal meaning-making in media and literary texts: Towards a metalanguage of image/language relations
Unsworth, Len. (2008). Explicating inter-modal meaning-making in media and literary texts: Towards a metalanguage of image/language relations. In In A. Burn and C. Durrant (Ed.). Media Teaching: Language, Audience and Production pp. 48 - 80 Wakefield Press.
Comparing and composing digital re-presentations of literature: Multimedia authoring and meta-communicative knowledge
Unsworth, Len. (2008). Comparing and composing digital re-presentations of literature: Multimedia authoring and meta-communicative knowledge. In In L. Unsworth (Ed.). New Literacies and the English Curriculum : Multimodal Perspectives pp. 186 - 211 Continuum International Publishing Group Ltd.
Assessing integrative reading of images and text in group reading comprehension tests
Unsworth, Len and Chan, Eveline. (2008). Assessing integrative reading of images and text in group reading comprehension tests. Curriculum Perspectives.