Deriving Design Principles for Educational Chatbots from Empirical Studies on Human–Chatbot Interaction

2-1 Expectations and Roles of Chatbots

A chatbot is a computer program to simulate human conversation via text or voice interaction [19]. Other terms for chatbots include talkbots, chatterbots, conversational agents, artificial conversational entities, and a conversational system. Efforts have been made to introduce chatbots or similar technologies in the education field, and related terms include a pedagogical agent or intelligent pedagogical agent (IPA), intelligent tutoring systems (ITS), and Artificial Intelligence Markup Language (AIML) -based chatbot. In the context of technology-mediated learning [2], chatbot-mediated learning (CML) contributes to motivation, self-directed learning, and individual learning by providing learners with individual learning environments that enhance the learning process and its outcomes. More specifically, chatbots can influence the learner’s learning process – the way in which information is found and communicated. In other words, rather than being provided with the contents passively, learners can support themselves to ask questions and lead the way. Second, learners can effectively support the learning process in large classrooms or in large online courses such as MOOCs. This may contribute to lowering the dissatisfaction experienced by learners and lowering the dropout rate. Third, learners can help them to make the right judgment by providing optimal information at the right time, and can provide continuous feedback to learners / teachers.

In general, chatbots are responsible for providing guidance, answering questions, or facilitating specific actions as coaches or colleagues (Table 1). In the educational context, the role of the chatbot can be set in various ways, which can be divided into five roles (see table 2). They are: tutors who guide and support the learning process of individual learners; evaluators who check the learner’s progress and diagnose performance; respondents who answer learners’ questions; communicators who mediate instructors and learners through interaction with learners; and fellow learners who exchange everyday conversations.

Table 1.

An empirical study of a chatbot

Table 2.

Educational roles of a chatbot

2-2 Principles of Chatbot Design

The following should be considered when designing chomps derived from Facebook (bot) [8], interoperability [12], and Microsoft [17] design and development principles.

The principles in table 3 provide guidelines on how to interact with chatbots from the UI or UX standpoint, but do not provide a standard on the purpose for which it should be used. In order to actively use a chatbot in an educational context, design and development guidelines should be prepared from the viewpoint of teaching and learning.

Table 3.

Chatbot design principles (example)

Hints for deriving chatbot design principles can be found in the Conversational Agents (CA) study. Traditional research was mainly on agent support, voice, and appearance (see table 4). The research that is required for the future is empirical and qualitative study of the change due to the agent’s participation, and research into the role of the agent.

Table 4.

Research related to conversational agents

4-1 Research question 1: How have chatbots been incorporated into empirical studies on human-chatbot interaction?

To answer the research question, we organized the review findings into two sets; one sorted by chatbot feature and the other by research variables and results. Basic information on each study was included in the first set (see table 5). Of the seven studies reviewed, all the researches were conducted under a higher education setting except for that of Corti and Gillespie (2016) [6], which was in an open setting, and that of van der Meij, van der Meij, and Harnsen (2015) [21], at a secondary school. The articles covered target knowledges in a varied range of disciplines such as healthy eating behavior [3], the circulatory system [9], instructional planning [14], and kinematics [21]. The chatbots used in the studies also differed from each other.

Table 5.

Articles reviewed sorted by chatbot features

The chatbot features examined in the studies were mostly variations of delivery types (or representation types). They included expressions made by chatbots (e.g., facial expression, emotional expression, empathetic expression), the gender of the chatbots (i.e., male and female), modality (e.g., voice, text), and other representation types (e.g., head movement). A few studies incorporated instructional features into chatbots by providing prompts and feedback [9] and motivational scaffolding [21].

The major findings of the studies are listed in table 6. Overall, the results showed a tendency for participants to project their human-to-human interaction practices to their human-to-chatbot interaction, especially when the chatbot was designed to be more human-like. In detail, participants report more positive outcomes when the chatbots express or represent emotion than when they interact with chatbots designed to exhibit neutral emotion [3, 14, 18]. They also exhibited social stereotyping towards a gendered chatbot [14]. In cases of modality, though the results were not perfectly consistent, participants seemed to better understand a text-based chatbot than a speaking chatbot [3], while they showed more human-like interaction with the latter [6, 18].

Table 6.

Summary of results in articles

4-2 Research question 2: What implications for educational chatbots can be derived from the studies?

From the review, we reorganized the findings with similar attributes and characteristics. Explanations for each attribute were then elaborated in the learning context. The implications are as follows.

• Live emotion – chatbots are better when designed to display consistent facial expressions or positive emotional expressions.
• Neutral emotion – a chatbot with a neutral emotional expression is more acceptable for persuasion.
• Modality – written text is better for delivering information or a guiding process; spoken text is better for affective support.
• Extraneous – too many animated or visual graphics have a detrimental effect on performance.
• Gender – people project social gender stereotyping according to the chatbot’s gender; people value information from a chatbot differently, depending on its gender representation.
• Bot effect – a chatbot can perform works that are redundant and require accuracy better than a human can.
• Imitation – more human-like chatbots drive more human-like interactions and establish a trusting relationship when giving information.

After extracting the implications, they were matched with each role of the educational chatbot (i.e., tutor, assessment, question and answer, communication, common dialogue); see table 7.

Table 7.

Implication from review for educational chatbot

5-1 Key result

In order to derive design principles for educational chatbots, the seven studies examined in this study examined how appearance characteristics such as facial expressions, gender, and style of chatbot affect the learning process and performance. As a result, when the chatbot expresses emotionally rather than neutrally, text-based rather than speech-based human interactions contribute more to learning. The design principles derived from this are the Live emotion principle, Neutral emotion principle, Modality principle, Extraneous principle, Gender principle, Bot effect principle, Imitation principle, and so on. In addition, this study matched design principles to be considered according to the role of chatbot when designing an educational chatbot. When designing a chatbot that plays the role of a tutor, it is necessary to consider the Live emotion principle, Modality principle, and Extraneous principle. When designing a chatbot that acts as an evaluator, the Bot effect principle should be considered. When developing a chatbot that acts as a responder, the Gender principle and Modality principle should be considered. In the case of a chatbot that plays the role of a moderator, it is necessary to consider the Neutral emotion principle, and in the case of a chatbot that plays the role of peer learner, the Modality principle (voice), the Imitation principle, and the Neutral emotion principle should be considered. In this study, we explored some principles for educational chatbots based on previous studies, but most of them were related to the appearance characteristics of chatbots. In the future, research is needed on the contents presentation method of chatbots and differentiated roles.

5-2 Areas for further study

As mentioned above, there are relatively few studies on the principles to be considered in the design of educational chatbots and the appropriate design principles according to the role of the chatbots. Related research needs to be actively conducted in the future, and research on suitable design principles is required according to the purpose and role of the chatbot.

Prior studies have found that it is difficult to find consensus on the characteristics of educationally effective chatbots, but learners want to learn with more human and emotional chatbots. Although this may be beneficial in terms of motivation, further research is needed to determine whether it will have significant effects on learning outcomes. In addition, it is necessary to study the differences between education through chatbots and through other educational methods, and in short- and long-term settings.

It is also necessary to study how the role of the instructor and how the interaction between the instructor and the learner is changed by the educational use of the chatbot. Research is also required on the side effects of using chatbots and the degree of acceptance according to learners’ characteristics; for example, study of how the chatbot’s performance varies according to a learner’s ability to use a computer, propensity to cooperate, learning style, and learning level. There is also a need for research on the cost-effectiveness of educational use. It is also necessary to discuss which educational context is the most effective when a chatbot is used for any educational purpose, and that from a cost-effectiveness analysis it is worth introducing a chatbot.

5-3 Limitation

This study has some limitations. First of all, although some papers have educational contexts, they include cases that are not for educational purposes, so it is hard to say that they derive principles entirely for educational chatbots. Since this study did not examine the gray literature, such as theses, current research, academic journals, and research reports, there is a possibility of publication bias. It is also difficult to avoid language bias because it includes papers in English only. However, this study attempted to study the special area of the educational chatbot, which was not sufficiently examined in the past, and it is considered to have sufficient advantages because it tried to derive differentiated principles. In order to develop a chatbot with various purposes and roles for educational purposes, it is necessary to make various efforts with various experts.