Métricas alternativas
Palabras clave
tutoría
educación superior
agente conversacional virtual
Escala de Usabilidad de un Sistema Artificial intelligence
tutoring
University education
embodied conversational agents
System Usability Scale
Cómo citar
Resumen
Este trabajo es resultado del proyecto PIE2020-11 “Diseño didáctico y experiencia en el aprendizaje digital en la enseñanza de distintas áreas de conocimiento: evaluar gamificando, tecnologías analíticas e inteligencia artificial” de la Universidad de Las Palmas de Gran Canaria. Se diseño, evaluó e implementó un agente conversacional virtual para apoyar las tutorías de Trabajos de Fin de Grado. Se utilizó una metodología cuantitativa con diseño descriptivo, correlacional y diferencial, para evaluar su usabilidad del agente conversacional en una muestra de 303 estudiantes universitarios. Se completó con el registro de uso real del agente, en un plazo de seis meses. Los datos indican que la escala posee una satisfactoria calidad métrica y una buena bondad del modelo, aspectos que se constatan en la estructura empírica y en la consistencia interna favorables del cuestionario. Los resultados también muestran que existen diferencias significativas (IC 99,95%) en las variables género, grado, nivel de conocimiento y el grado de uso de chatbots. En conclusión, los datos permiten establecer criterios explicativos sobre el uso de chatbots. Se dan valoraciones positivas del alumnado sobre la usabilidad del chatbot que apoyan el uso posterior del agente, por 589 estudiantes de tres titulaciones diferentes, contestando a 3025 preguntas en seis meses.
Citas
Babu, S. V., Suma, E., Hodges, L. F., & Barnes, T. (2011). Learning cultural conversational protocols with immersive interactive virtual humans. International Journal of Virtual Reality, 10(4), 25-35. http://dx.doi.org/10.20870/IJVR.2011.10.4.2826
Bangor, A., Kortum, P. T., & Miller, J. T. (2008). An Empirical Evaluation of the System Usability Scale. International Journal of Human-Computer Interaction, 24, 574-594. http://dx.doi.org/10.1080/10447310802205776
Brooke, J. (1986). SUS: A “quick and dirty” usability scale. In Jordan, P. W., Thomas, B., Weerdmeester, B. A., McClelland (Ed.) Usability Evaluation in Industry (pp. 189-194). Taylor & Francis.
Brooke, J. (1996). SUS-A quick and dirty usability scale. Usability evaluation in industry, 189(194), 4-7.
Brooke, J. (2013). SUS: a retrospective. Journal of usability studies, 8(2), 29-40.
Bunge, M. (2010). Las pseudociencias. Laetoli.
Catalán, C., Delgado, C., Alario-Hoyos, C., & Muñoz-Merino, P. J. (2018, September). Supporting a MOOC through a conversational agent. Design of a first prototype. In 2018 International Symposium on Computers in Education (SIIE) (pp. 1-6). IEEE. https://doi.org/10.1109/siie.2018.8586694
Cochran, W. G., & Bouclier, A. S. (1980). Técnicas de muestreo. Continental.
Colace, F., De Santo, M., Lombardi, M., Pascale, F., Pietrosanto, A., & Lemma, S. (2018). Chatbot for e-learning: A case of study. International Journal of Mechanical Engineering and Robotics Research, 7(5), 528-533. https://doi.org/10.18178/ijmerr.7.5.528-533
Coperich, K., Cudney, E., & Nembhard, H. (2017). Continuous improvement study of chatbot technologies using a human factors methodology. In Proceedings of the 2017 Industrial and Systems Engineering Conference.
Dale, R. (2016). The return of the chatbots. Natural Language Engineering, 22(5), 811-817. https://doi.org/10.1017/s1351324916000243
Fadhil, A., & Gabrielli, S. (2017, May). Addressing challenges in promoting healthy lifestyles: the al-chatbot approach. In Proceedings of the 11th EAI international conference on pervasive computing technologies for healthcare (pp. 261-265). https://doi.org/10.1145/3154862.3154914
Fast, E., Chen, B., Mendelsohn, J., Bassen, J., & Bernstein, M. S. (2018, April). Iris: A conversational agent for complex tasks. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems (pp. 1-12). https://doi.org/10.1145/3173574.3174047
Feine, J., Gnewuch, U., Morana, S., & Maedche, A. (2019). A Taxonomy of Social Cues for Conversational Agents. International Journal of Human-Computer Studies, 132, 138-161. https://doi.org/10.1016/j.ijhcs.2019.07.009
Finstad, K. (2010). The usability metric for user experience. Interacting with Computers, 22(5), 323-327. https://doi.org/10.1016/j.intcom.2010.04.004
Fornell, C & Larcker, D.F. (1981). Evaluating structural equiations modelswith unoservable variables and measurement error. Journal of Marketing Research, 18(1), 39-50.
Ghosh, D., Foong, P. S., Zhang, S., & Zhao, S. (2018, April). Assessing the Utility of the System Usability Scale for Evaluating Voice-based User Interfaces. In Proceedings of the Sixth International Symposium of Chinese CHI (pp. 11-15). https://doi.org/10.1145/3202667.3204844
Hair, F., Anderson, R.E, Tatham, R.L. & Black, W.C. (1998). Multivariate data analysis with readings. Prentice Hall.
Hair, F., Anderson, R.E., Tatham, R.L. & Black, W.C. (2005). Multivariate data analysis. Prentice Hall.
Herbert, D., & Kang, B. H. (2018). Intelligent conversation system using multiple classification ripple down rules and conversational context. Expert Systems with Applications, 112, 342-352. https://doi.org/10.1016/j.eswa.2018.06.049
Hernández R.; Fernández, C. y Baptista, P. (2010). Metodología de la investigación. McGraw-Hill. (5ª Ed.)
Hobert, S. & Meyer von Wolff, R. (2019). Say Hello to Your New Automated Tutor – A Structured Literature Review on Pedagogical Conversational Agents. In Proceedings of the 14th International Conference on Wirtschaftsinformatik (S. 301–315). Siege.
Jung, S. (2013). Exploratory factor analysis with small sample sizes: A comparison of three approaches. Behavioural Processes, 97, 90–95.
Kaplan, R. M., & Saccuzzo, D. P. (2009). Standardized tests in education, civil service, and the military. Psychological testing: Principles, applications, and Issues, 7, 325-327.
Kerlinger, F.N. (1979). Enfoque conceptual de la investigación del comportamiento. Nueva Editorial Interamericana.
Kerly, A., Ellis, R., & Bull, S. (2008). Conversational agents in E-Learning. Proceedings of Artificial Intelligence 2008 (pp. 169–182).
Kirakowski, J. (1994). The use of questionnaire methods for usability assessment. http://sumi.ucc.ie/sumipapp.html
Landauer, T. K. (1997). Behavioral Research Methods in Human-Computer Interaction. In Helander, M., Landauer, T., Prabhu, P. (Ed.) Handbook of Human-Computer Interaction pp. 203-227. Elsevier
Lewis, J. (2018) The System Usability Scale: Past, Present, and Future. International Journal of Human–Computer Interaction, 34(7), 577-590. https://doi.org/10.1080/10447318.2018.1455307
Lewis, J. R., & Sauro, J. (2009). The factor structure of the system usability scale. In International conference on human centered design (pp. 94-103). Springer, Berlin, Heidelberg. http://dx.doi.org/10.1007/978-3-642-02806-9_12
Lucey, N. M. (1991). More than Meets the I: User-Satisfaction of Computer Systems. Unpublished thesis for Diploma in Applied Psychology, University College Cork.
Neumann, A. T., de Lange, P., & Klamma, R. (2019, December). Collaborative Creation and Training of Social Bots in Learning Communities. In 2019 IEEE 5th International Conference on Collaboration and Internet Computing (CIC) (pp. 11-19). IEEE. https://doi.org/10.1109/cic48465.2019.00011
Nunnally, J.C. (1978). Psychometric theory. McGraw-Hill Book.
Park, M., Aiken, M., & Salvador, L. (2019). How do humans interact with chatbots? An analysis of transcripts. International Journal of Management & Information Technology, 14, 3338-3350. https://doi.org/10.24297/ijmit.v14i0.7921
Ren, R., Castro, J.W., Acuña, S.T., & de Lara, J. (2019). Usability of Chatbots: A Systematic Mapping Study. In Proceedings of 31st International Conference on Software Engineering & Knowledge Engineering (SEKE’19) (pp. 479-484). https://doi.org/10.18293/seke2019-029
Saenz, J., Burgess, W., Gustitis, E., Mena, A., & Sasangohar, F. (2017). The usability analysis of chatbot technologies for internal personnel communications. In IIE Annual Conference. Proceedings (pp. 1357-1362). Institute of Industrial and Systems Engineers (IISE).
Sauro, J., & Lewis, J. R. (2009). Correlations among Prototypical Usability Metrics: Evidence for the Construct of Usability. In Proceedings of the SIGCHI conference on human factors in computing systems (pp. 1609-1618). https://doi.org/10.1145/1518701.1518947
Schroeder, J., Wilkes, C., Rowan, K., Toledo, A., Paradiso, A., Czerwinski, M., Mark, G., & Linehan, M. M. (2018, April). Pocket skills: A conversational mobile web app to support dialectical behavioral therapy. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems (pp. 1-15). https://doi.org/10.1145/3173574.3173972
Sinoo, C., van Der Pal, S., Henkemans, O. A. B., Keizer, A., Bierman, B. P., Looije, R., & Neerincx, M. A. (2018). Friendship with a robot: Children’s perception of similarity between a robot’s physical and virtual embodiment that supports diabetes self-management. Patient Education and Counseling, 101(7), 1248-1255. https://doi.org/10.1016/j.pec.2018.02.008
Sumikawa, Y., Fujiyoshi, M., Hatakeyama, H., & Nagai, M. (2020). Supporting creation of FAQ dataset for E-learning chatbot. In Intelligent Decision Technologies 2019 (pp. 3-13). Springer. https://doi.org/10.1007/978-981-13-8311-3_1
Tabachnik, B.G., & Fidell, L.S., (1989). Using Multivariate Statistics. Harper Collins Publishers.
Tegos, S., & Demetriadis, S. (2017). Conversational agents improve peer learning through building on prior knowledge. Journal of Educational Technology & Society, 20(1), 99-111.
Tegos, S., Demetriadis, S., & Tsiatsos, T. (2014). A configurable conversational agent to trigger students’ productive dialogue: a pilot study in the CALL domain. International Journal of Artificial Intelligence in Education, 24(1), 62-91. https://doi.org/10.1007/s40593-013-0007-3
Tielman, M.L., Neerincx, M.A., Bidarra, R., Kybartas, B., & Brinkman, W.P. (2017). A therapy system for post-traumatic stress disorder using a virtual agent and virtual storytelling to reconstruct traumatic memories. Journal of medical systems, 41(8), 125-145. https://doi.org/10.1007/s10916-017-0771-y
Tullis, T. S., & Stetson, J. N. (2004). A Comparison of Questionnaires for Assessing Website Usability, unpublished presentation given at the UPA Annual Conference. http://home.comcast.net/~tomtullis/publications/UPA2004TullisStetson.pdf
Wargnier, P., Malaisé, A., Jacquemot, J., Benveniste, S., Jouvelot, P., Pino, M., & Rigaud, A. S. (2015, March). Towards attention monitoring of older adults with cognitive impairment during interaction with an embodied conversational agent. In 2015 3rd IEEE VR International Workshop on Virtual and Augmented Assistive Technology (VAAT) (pp. 23-28). IEEE. https://doi.org/10.1109/vaat.2015.7155406
Wellnhammer, N., Dolata, M., Steigler, S., & Schwabe, G. (2020, January). Studying with the Help of Digital Tutors: Design Aspects of Conversational Agents that Influence the Learning Process. In Proceedings of the 53rd (pp. 146-155). Hawaii International Conference on System Sciences. https://doi.org/10.24251/hicss.2020.019
Winkler, R., & Söllner, M. (2020, January). Towards Empowering Educators to Create their own Smart Personal Assistants. In Proceedings of the 53rd (pp. 22-31). Hawaii International Conference on System Sciences. https://doi.org/10.24251/hicss.2020.005
Zeni, J., da Costa, C. & da Rosa Righi, R. (2019). Survey of conversational agents in health. Expert Systems with Applications, 129, 56-67. https://doi.org/10.1016/j.eswa.2019.03.054
Esta obra está bajo una licencia internacional Creative Commons Atribución-NoComercial-SinDerivadas 4.0.
Derechos de autor 2021 Píxel-Bit. Revista de Medios y Educación