Métricas alternativas
Keywords
tutoría
educación superior
agente conversacional virtual
Escala de Usabilidad de un Sistema Artificial intelligence
tutoring
University education
embodied conversational agents
System Usability Scale
How to Cite
Abstract
This paper delves into the possibilities of conversational virtual agents as a tool to tutor university students' work. A quantitative methodology with a descriptive, correlational and differential design was used to evaluate the usability of the conversational agent in a sample of 303 university students. For this, a virtual conversational agent was designed and evaluated to support the End-of-Degree Project tutorials with the SUS Scale (System Usability Scale). The results indicate that the scale has a satisfactory metric quality and good model goodness, aspects that are verified in the empirical structure and in the favorable internal consistency of the questionnaire. The data also show that there are significant differences (99.95% CI) in the variables gender, grade, level of knowledge, and the grade of chatbot use. It was completed with the record of the actual use of the agent, within a period of six months, by 589 students from three different degrees, answering 3025 questions in six months. In conclusion, the results allow to establish explanatory criteria on the use of chatbots. It is necessary to continue deepening in this type of tools for the monitoring and evaluation of students.
References
Babu, S. V., Suma, E., Hodges, L. F., & Barnes, T. (2011). Learning cultural conversational protocols with immersive interactive virtual humans. International Journal of Virtual Reality, 10(4), 25-35. http://dx.doi.org/10.20870/IJVR.2011.10.4.2826
Bangor, A., Kortum, P. T., & Miller, J. T. (2008). An Empirical Evaluation of the System Usability Scale. International Journal of Human-Computer Interaction, 24, 574-594. http://dx.doi.org/10.1080/10447310802205776
Brooke, J. (1986). SUS: A “quick and dirty” usability scale. In Jordan, P. W., Thomas, B., Weerdmeester, B. A., McClelland (Ed.) Usability Evaluation in Industry (pp. 189-194). Taylor & Francis.
Brooke, J. (1996). SUS-A quick and dirty usability scale. Usability evaluation in industry, 189(194), 4-7.
Brooke, J. (2013). SUS: a retrospective. Journal of usability studies, 8(2), 29-40.
Bunge, M. (2010). Las pseudociencias. Laetoli.
Catalán, C., Delgado, C., Alario-Hoyos, C., & Muñoz-Merino, P. J. (2018, September). Supporting a MOOC through a conversational agent. Design of a first prototype. In 2018 International Symposium on Computers in Education (SIIE) (pp. 1-6). IEEE. https://doi.org/10.1109/siie.2018.8586694
Cochran, W. G., & Bouclier, A. S. (1980). Técnicas de muestreo. Continental.
Colace, F., De Santo, M., Lombardi, M., Pascale, F., Pietrosanto, A., & Lemma, S. (2018). Chatbot for e-learning: A case of study. International Journal of Mechanical Engineering and Robotics Research, 7(5), 528-533. https://doi.org/10.18178/ijmerr.7.5.528-533
Coperich, K., Cudney, E., & Nembhard, H. (2017). Continuous improvement study of chatbot technologies using a human factors methodology. In Proceedings of the 2017 Industrial and Systems Engineering Conference.
Dale, R. (2016). The return of the chatbots. Natural Language Engineering, 22(5), 811-817. https://doi.org/10.1017/s1351324916000243
Fadhil, A., & Gabrielli, S. (2017, May). Addressing challenges in promoting healthy lifestyles: the al-chatbot approach. In Proceedings of the 11th EAI international conference on pervasive computing technologies for healthcare (pp. 261-265). https://doi.org/10.1145/3154862.3154914
Fast, E., Chen, B., Mendelsohn, J., Bassen, J., & Bernstein, M. S. (2018, April). Iris: A conversational agent for complex tasks. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems (pp. 1-12). https://doi.org/10.1145/3173574.3174047
Feine, J., Gnewuch, U., Morana, S., & Maedche, A. (2019). A Taxonomy of Social Cues for Conversational Agents. International Journal of Human-Computer Studies, 132, 138-161. https://doi.org/10.1016/j.ijhcs.2019.07.009
Finstad, K. (2010). The usability metric for user experience. Interacting with Computers, 22(5), 323-327. https://doi.org/10.1016/j.intcom.2010.04.004
Fornell, C & Larcker, D.F. (1981). Evaluating structural equiations modelswith unoservable variables and measurement error. Journal of Marketing Research, 18(1), 39-50.
Ghosh, D., Foong, P. S., Zhang, S., & Zhao, S. (2018, April). Assessing the Utility of the System Usability Scale for Evaluating Voice-based User Interfaces. In Proceedings of the Sixth International Symposium of Chinese CHI (pp. 11-15). https://doi.org/10.1145/3202667.3204844
Hair, F., Anderson, R.E, Tatham, R.L. & Black, W.C. (1998). Multivariate data analysis with readings. Prentice Hall.
Hair, F., Anderson, R.E., Tatham, R.L. & Black, W.C. (2005). Multivariate data analysis. Prentice Hall.
Herbert, D., & Kang, B. H. (2018). Intelligent conversation system using multiple classification ripple down rules and conversational context. Expert Systems with Applications, 112, 342-352. https://doi.org/10.1016/j.eswa.2018.06.049
Hernández R.; Fernández, C. y Baptista, P. (2010). Metodología de la investigación. McGraw-Hill. (5ª Ed.)
Hobert, S. & Meyer von Wolff, R. (2019). Say Hello to Your New Automated Tutor – A Structured Literature Review on Pedagogical Conversational Agents. In Proceedings of the 14th International Conference on Wirtschaftsinformatik (S. 301–315). Siege.
Jung, S. (2013). Exploratory factor analysis with small sample sizes: A comparison of three approaches. Behavioural Processes, 97, 90–95.
Kaplan, R. M., & Saccuzzo, D. P. (2009). Standardized tests in education, civil service, and the military. Psychological testing: Principles, applications, and Issues, 7, 325-327.
Kerlinger, F.N. (1979). Enfoque conceptual de la investigación del comportamiento. Nueva Editorial Interamericana.
Kerly, A., Ellis, R., & Bull, S. (2008). Conversational agents in E-Learning. Proceedings of Artificial Intelligence 2008 (pp. 169–182).
Kirakowski, J. (1994). The use of questionnaire methods for usability assessment. http://sumi.ucc.ie/sumipapp.html
Landauer, T. K. (1997). Behavioral Research Methods in Human-Computer Interaction. In Helander, M., Landauer, T., Prabhu, P. (Ed.) Handbook of Human-Computer Interaction pp. 203-227. Elsevier
Lewis, J. (2018) The System Usability Scale: Past, Present, and Future. International Journal of Human–Computer Interaction, 34(7), 577-590. https://doi.org/10.1080/10447318.2018.1455307
Lewis, J. R., & Sauro, J. (2009). The factor structure of the system usability scale. In International conference on human centered design (pp. 94-103). Springer, Berlin, Heidelberg. http://dx.doi.org/10.1007/978-3-642-02806-9_12
Lucey, N. M. (1991). More than Meets the I: User-Satisfaction of Computer Systems. Unpublished thesis for Diploma in Applied Psychology, University College Cork.
Neumann, A. T., de Lange, P., & Klamma, R. (2019, December). Collaborative Creation and Training of Social Bots in Learning Communities. In 2019 IEEE 5th International Conference on Collaboration and Internet Computing (CIC) (pp. 11-19). IEEE. https://doi.org/10.1109/cic48465.2019.00011
Nunnally, J.C. (1978). Psychometric theory. McGraw-Hill Book.
Park, M., Aiken, M., & Salvador, L. (2019). How do humans interact with chatbots? An analysis of transcripts. International Journal of Management & Information Technology, 14, 3338-3350. https://doi.org/10.24297/ijmit.v14i0.7921
Ren, R., Castro, J.W., Acuña, S.T., & de Lara, J. (2019). Usability of Chatbots: A Systematic Mapping Study. In Proceedings of 31st International Conference on Software Engineering & Knowledge Engineering (SEKE’19) (pp. 479-484). https://doi.org/10.18293/seke2019-029
Saenz, J., Burgess, W., Gustitis, E., Mena, A., & Sasangohar, F. (2017). The usability analysis of chatbot technologies for internal personnel communications. In IIE Annual Conference. Proceedings (pp. 1357-1362). Institute of Industrial and Systems Engineers (IISE).
Sauro, J., & Lewis, J. R. (2009). Correlations among Prototypical Usability Metrics: Evidence for the Construct of Usability. In Proceedings of the SIGCHI conference on human factors in computing systems (pp. 1609-1618). https://doi.org/10.1145/1518701.1518947
Schroeder, J., Wilkes, C., Rowan, K., Toledo, A., Paradiso, A., Czerwinski, M., Mark, G., & Linehan, M. M. (2018, April). Pocket skills: A conversational mobile web app to support dialectical behavioral therapy. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems (pp. 1-15). https://doi.org/10.1145/3173574.3173972
Sinoo, C., van Der Pal, S., Henkemans, O. A. B., Keizer, A., Bierman, B. P., Looije, R., & Neerincx, M. A. (2018). Friendship with a robot: Children’s perception of similarity between a robot’s physical and virtual embodiment that supports diabetes self-management. Patient Education and Counseling, 101(7), 1248-1255. https://doi.org/10.1016/j.pec.2018.02.008
Sumikawa, Y., Fujiyoshi, M., Hatakeyama, H., & Nagai, M. (2020). Supporting creation of FAQ dataset for E-learning chatbot. In Intelligent Decision Technologies 2019 (pp. 3-13). Springer. https://doi.org/10.1007/978-981-13-8311-3_1
Tabachnik, B.G., & Fidell, L.S., (1989). Using Multivariate Statistics. Harper Collins Publishers.
Tegos, S., & Demetriadis, S. (2017). Conversational agents improve peer learning through building on prior knowledge. Journal of Educational Technology & Society, 20(1), 99-111.
Tegos, S., Demetriadis, S., & Tsiatsos, T. (2014). A configurable conversational agent to trigger students’ productive dialogue: a pilot study in the CALL domain. International Journal of Artificial Intelligence in Education, 24(1), 62-91. https://doi.org/10.1007/s40593-013-0007-3
Tielman, M.L., Neerincx, M.A., Bidarra, R., Kybartas, B., & Brinkman, W.P. (2017). A therapy system for post-traumatic stress disorder using a virtual agent and virtual storytelling to reconstruct traumatic memories. Journal of medical systems, 41(8), 125-145. https://doi.org/10.1007/s10916-017-0771-y
Tullis, T. S., & Stetson, J. N. (2004). A Comparison of Questionnaires for Assessing Website Usability, unpublished presentation given at the UPA Annual Conference. http://home.comcast.net/~tomtullis/publications/UPA2004TullisStetson.pdf
Wargnier, P., Malaisé, A., Jacquemot, J., Benveniste, S., Jouvelot, P., Pino, M., & Rigaud, A. S. (2015, March). Towards attention monitoring of older adults with cognitive impairment during interaction with an embodied conversational agent. In 2015 3rd IEEE VR International Workshop on Virtual and Augmented Assistive Technology (VAAT) (pp. 23-28). IEEE. https://doi.org/10.1109/vaat.2015.7155406
Wellnhammer, N., Dolata, M., Steigler, S., & Schwabe, G. (2020, January). Studying with the Help of Digital Tutors: Design Aspects of Conversational Agents that Influence the Learning Process. In Proceedings of the 53rd (pp. 146-155). Hawaii International Conference on System Sciences. https://doi.org/10.24251/hicss.2020.019
Winkler, R., & Söllner, M. (2020, January). Towards Empowering Educators to Create their own Smart Personal Assistants. In Proceedings of the 53rd (pp. 22-31). Hawaii International Conference on System Sciences. https://doi.org/10.24251/hicss.2020.005
Zeni, J., da Costa, C. & da Rosa Righi, R. (2019). Survey of conversational agents in health. Expert Systems with Applications, 129, 56-67. https://doi.org/10.1016/j.eswa.2019.03.054
This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License.
Copyright (c) 2021 Pixel-Bit. Media and Education Journal