Conference Dinner

Time: 19:00 August 13, 2018

Location: Den Gamle Kro (Overgade 23, 5000 Odense C)

Registration (no later than August 9, pay at the restaurant):


Yao Du

University of California, Irvine

Tracking Children’s Speech and Language Development: A Pilot Study Using Interactive Mobile Storytime

Early identification and intervention of speech and language delays in children contribute to better communication and literacy skills for school readiness and are protective against behavioural and mental health problems, as well as academic failure. Speech and language delay affects 5 to 12 percent of US children between the ages of 2 and 5 years. Identification of these delays in primary care settings can be inefficient and is particularly challenging for children with low medical resources. These issues can lead to delays in treatment, resulting in detrimental effects on the child’s development. With more children growing up using digital devices on a regular basis, mobile applications offer unique opportunities to identify and monitor their development. Despite this, there are few cost-effective, technology-mediated means of assessment for communication impairments in children. Shared storybook reading, an evidence-based intervention technique, is a common activity used by many families and speech therapists for improving language and literacy skills in children. While e-books for children have become ubiquitous, research that has attempted to use digital storybooks to mediate screening and monitoring of speech and language skills remains limited.

In collaboration with a digital health startup company Cognoa, we designed “Storytime,” an interactive video that uses a virtual avatar for storytelling to mediate autonomous speech and language assessment between children and parents in the home setting. Our pilot study collects audio and video recordings of 76 pairs of parents and children ages 4 to 6 years old, with and without communication impairments. Children’s speech and language production is manually transcribed and analyzed to evaluate how they engage with and respond to this mobile-mediated interaction. In this talk, we present the initial findings from this study and discuss the implications on advancing existing research methods and participatory design between academic and industry collaborators.



Yao Du is a doctoral student in Informatics at the University of California, Irvine, working with advisor Katie Salen Tekinbas. Her research lies at the intersections of mobile HCI, instructional technology, and assistive technology, and her work focuses on designing and evaluating interactive tools for children with communication impairments, their caregivers, and healthcare providers. She has previously worked as a bilingual Mandarin-English speech-language pathologist in both educational and medical settings (e.g., schools, hospitals, clinics) with children and adults with disabilities.




Thomas Enemark Lundtofte

University of Southern Denmark

Points-of-View: Providing a video ethnographic method for studying young children’s play practices with tablet computers

Video observation methods have become increasingly common in ethnographic research with young children. However, when young children’s practices with digital toys are investigated using video cameras, it is often difficult to provide equal representation of the child and the object. As argued by Jackie Marsh (2017), following a post-humanist perspective, agency and affordances are dynamic when it comes to the relationship between humans and technology. When we study practices, we should be equally mindful of both human and non-human actors. The Points-of-View (POV) method (Lundtofte & Johansen, forthcoming) provides up-close access to sociomaterial practices between children and digital toys.

The POV method uses a rigged dual-camera set-up and has been employed in research on how seven young Danish children played with tablet computers. This setup yielded video data that was highly comparable across the different informants and settings. Importantly, as this fieldwork was conducted in Denmark, issues of consent were between the researcher, the children and their parents. In addition to this, the POV method provided means for demonstrating the video recording process to the children, which made the consensual aspect of ethical considerations easier to align with the methodology.

Through a number of examples, this presentation will show how the POV method provides some nuances on the subject of young children’s play practices with tablet computers. As this data provides visual information from both sides of an interaction, we are able to understand how the interactions change or end. On the other hand, there are some disadvantages to the method. Aside from the task of introducing research gear into people’s private space, object mobility, which is a key affordance of tablet computers, was somewhat restricted. Therefore, some reflections on future innovations of the POV method will be touched upon.


Lundtofte, T. E., & Johansen, S. L. (forthcoming). Video Methodology: Researching Sociomaterial Points-of-View in Children’s Play Practices with IoToys. In G. Mascheroni & D. Holloway (Eds.), The Internet of Toys: Practices, Affordances and the Political Economy of Children’s Play. London: Palgrave Macmillan.
Marsh, J. (2017). The Internet of Toys: A Posthuman and Multimodal Analysis of Connected Play. Teachers College record, 119(15).


Thomas Enemark Lundtofte is a  Ph.D. fellow in Media Studies at the University of Southern Denmark. He researches the characteristics of young children’s sociomaterial play practices with tablet computers via the case of DR Ramasjang – a highly popular app in Denmark provided by the national broadcast corporation. He is an affiliated researcher with the Centre for Children’s Literature and Media at Aarhus University as well as an adjunct faculty member at the San Diego State University National Center for the Study of Children’s Literature.

Svein T. Heddeland

University of Agder, Norway

Digital Documentation Practices in ECEC

The aim of this study is to explore and describe ethical considerations in the on-going practice of visual documentation of children in early childhood education and care (ECEC) in a digital context. This study is based on the literature in the new sociology of childhood (Corsaro, 2005) which emphasises that children are competent and active agents, and the field of social semiotics (Halliday, 1978, van Leeuwen, 2005) to inform how early childhood practitioners use and make sense of visual documentation in preschools. To understand how the mutual relationship between technical objects, the natural environment and social practice I look to the field of Science, Technology and Society (STS) (Van House, 2011) and mediatization studies (Hjarvard, 2016). The theoretical and conceptual framework is structured as a critical discourse analysis (Fairclough, 1992) and draws on theoretical perspectives including visual grammar (Kress & van Leeuwen, 2006), pedagogical documentation (Taguchi, 1998), and ICT in ECEC (Stephen & Plowman, 2002; Bølgan, 2008).

The study is based on a social semiotic epistemology and adopts a qualitative research design using document analysis. The research was conducted in three Norwegian preschools. The data constructed consists of interview data (practitioners and parents) and multimodal texts that the early childhood practitioners shared online with parents for an entire year of preschool (210 multimodal texts including 2833 images). In the analysis I use the software UamImage-tool to annotate all 2833 images. Through the examination of the analytical categories vertical perspective, framing and salience, I will analyze how digital images of children in preschool convey certain representations and social interactions. This investigation will be useful in further discussion of children´s rights and autonomy in visual documentation practices in a ECEC-context.



Participation in the study was voluntary and informed consent was obtained from all participants. Anonymity was given to all individuals and preschools in the study to protect their identities.The study is reported to NSD (Data Protection Official for Research for all the Norwegian universities).

There are many critical issues to raise about power relations and whose voices are represented in visual documentations. In the Norwegian ECEC-context young children are viewed as competent and rational beings. Preliminary findings suggest that digital visual documentation practices are not informed by this view on childhood, which may lead to less emphasis of children´s point of view and perspectives in pedagogical documentation.

This study has implications on how early childhood practitioners document pedagogical practices in a digital context and, and in a broader sense; how early childhood education students (should) acquire skills and knowledge related to digital documentation.



Svein T. Heddeland is a research fellow at the University of Agder. His research areas are critical discourse analysis, multimodality, social semiotics and documentation and communication practices in ECEC. You can read about his research here:

Kaela Zhang (9ICOM)

The Hong Kong Polytechnic University

Public health education through posters in two world cities: A multimodal corpus-based analysis

In this paper, I will attempt to create multimodal understanding of public health posters used in two global cities – New York City and Hong Kong. In contrast to prior research that is either the case study that has focused on so few public health posters which is quite difficult to unravel the more representative patterns of such data and is a threat to the analysis’s validity, or the exclusively qualitative or quantitative exploration of a pool of data, I draw upon both qualitative and quantitative research methods to conduct a more holistic study. The paper examines both linguistic and non-linguistic resources that 60 public health posters make use of in the construction of health-related messages for public health education from three different vantage points of below, roundabout and above. In the first, following systemic-functional semiotics, I investigate the semiotic labour performed by each of the individual semiotic systems (i.e. language and image) on the page of the public health posters. I analyse experiential meaning and interpersonal meaning that the different semiotic resources make. In the second area of investigation, I annotate the content, layout structure, and rhetorical organisation of each poster and build an XML-based multimodal corpus. The annotated corpus provides me with a reliable empirical basis to analyse the various semiotic resources for realising logico-semantic relations as tactic patterns, to explore the possible effect of matching/mismatching hierarchical rhetorical and layout organisations. Subsequently, these two areas are complemented by a further contextual analysis, which as a whole explores how the public health posters educate the general public in New York City and Hong Kong. The results of the study might also be used to improve the information design of health education materials and to propose web-based annotation tools applied to enhancing the multimodal corpus building.



Kaela Peijia ZHANG, currently working as a research associate on Professor Christian Matthiessen’s healthcare communication research project at The Hong Kong Polytechnic University (PolyU), is interested in the application of functional theory and description in multimodal discourse interpretation, multiliteracy, health education and communication, etc. Being supervised by Professor Matthiessen at the PolyU, being ‘mentored’ by Professor John Bateman at Universität Bremen and Dr Tuomo Hiippala at the University of Helsinki, she built a multimodal corpus of public health posters as part of her PhD project. During the past years, Kaela attended the summer school regarding multimodal methodologies at the UCL IOE; she studied at UIUC on an attachment programme with Professor Bill Cope; she presented in several conferences in France, Germany, etc., and worked as research assistant for a couple of research projects at PolyU. Her papers include “Multimodal corpus research based on rhetorical structure theory”, etc.

Bursary Winners

We are very excited to announce that we are able to offer a total of five bursaries to support PhD students and ECRs to attend the Researching Multimodal Childhood Symposium (August 13-14) and the 9th International Conference on Multimodality (August 15-17).

You can read about the winners and their diverse research here:

Carlsberg Bursary 

Nataly Gonzalez-Acevedo (Spain), Universitat Autònoma de Barcelona

Aline Frederico (Brazil), University of Cambridge

Ben Burbank (UK), Oxford Brooks University


Halliday-Hasan International Fund Bursary 

Yao Du,  University of California, Irvine

Kaela Peijia Zhang, The Hong Kong Polytechnic University


We thank Carlsberg Foundation and Halliday-Hasan International Fund for supporting emerging researchers/scholars and their research.