Natural language processing by AI in the assessment of open-ended student responses in the DOCENTIA-UCM program
Article Sidebar
Main Article Content
Abstract
INTRODUCTION. In the context of evaluating the quality of teaching staff at the Complutense University of Madrid, a questionnaire is administered to students as part of the DOCENTIA program. In programs of this magnitude, the analysis of qualitative responses through traditional methods faces limitations such as errors, biases, and resource consumption. METHOD. In this study, artificial intelligence specialized in natural language processing is used to analyze the 27,290 student comments in the DOCENTIA questionnaire. The process involves cleaning the comments, lemmatizing words, and applying vector space models, followed by clustering techniques to categorize them into clusters. RESULTS. The results reveal three significant clusters in the students’ responses. The “Good Professor” cluster represents 14.2% of the responses and highlights qualities such as kindness, accessibility, competence, and teachers’ passion. Cluster 1, called “Good Teaching,” comprises 26.1% of the responses and reflects students’ perceptions regarding aspects like the quality of explanations, teacher motivation, and logical structuring of the subject, among others. Cluster 2, focused on “Clear objectives and standards of the subject”, encompasses 59.7% of the responses and centers on the quality of objectives, transparency in evaluation, and expectations regarding the fulfilment of teaching duties. DISCUSSION. These clusters and their associated terms reflect students’ perception of teaching quality and its relationship with the objectives outlined in the standards of the Academic Teaching Development Framework. Automated analysis of student comments proves useful, but it is suggested to investigate providing more specific details, such as analyzing the sentiment of the comments, due to the importance that this type of analysis can have in relation to educational quality standards.
Downloads
Article Details
This work is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License.
References
Abiodun, O. I., Jantan, A., Omolara, A. E., Dada, K. V., Umar, A. M., Linus, O. U., Arshad, H., Kazaure, A. A., Gana, U. y Kiru, M. U. (2019). Comprehensive review of artificial neural network applications to pattern recognition. IEEE Access, 7, 158820-158846.
ANECA (2015). Criterios y directrices para el aseguramiento de Calidad en el EEES 2015. http://www.aneca.es/Internacional/Documentos-internacionales-de-referencia/Criterios-y-estandares/Criterios-y-directrices-para-el-aseguramiento-de-Calidad-en-el-EEES-2015
Ausubel, D. (1963). The Psychology of Meaningful Verbal Learning. Grune & Stratton.
Ausubel, D. (1968). Educational Psychology: A Cognitive View. Holt, Rinehart & Winston.
Baumer, E., Mimno, D., Guha, S., Quan, E. y Gay, G. (2017). Comparing Grounded Theory and Topic Modeling: Extreme Divergence or Unlikely Convergence? Journal of the Association for Information Science and Technology, 68(6), 1397-1410.
Bengio, Y., Ducharme, R., Vincent, P. y Janvin, C. (2003). A Neural Probabilistic Language Model. Journal of Machine Learning Research, 3, 1137-1155.
Biggs, J. B. y Tang, C. (1999). Teaching for Quality Learning at University: What the Student Does. Society for Research into Higher Education & Open University Press.
Bird, S., Klein, E. y Loper, E. (2009). Natural language processing with Python: analyzing text with the natural language toolkit. O‘Reilly Media, Inc.
Bowman, N. A., Seifert, T. A., Mayhew, M. J., Wolniak, G. C., Rockenbach, A. N., Pascarella, E. T. y Terenzini, P. T. (2016). How College Affects Students: 21st Century Evidence That Higher Education Works. Jossey-Bass.
Broomfield, D. y Bligh, J. (1998). An evaluation of theshort form‘course experience questionnaire with medical students. Medical Education, 32(4), 367-369.
Cherven, K. (2013). Network Graph Analysis and Visualization with Gephi. Packt Publishing Ltd. ISBN 978-1-78328-013-1.
Chesebro, J. L. y McCroskey, J. C. (2001). The relationship of teacher clarity and immediacy with student state receiver apprehension, affect and cognitive learning. Communication Education, 50(1), 59-68. https://doi.org/10.1080/03634520109379232
Curtis, D. D. y Keeves, J. P. (2000). The Course Experience Questionnaire as an institutional performance indicator. International Education Journal, 1(2).
Estepp, C. M. y Roberts, T. G. (2015). Teacher Immediacy and Professor/Student Rapport as Predictors of Motivation and Engagement. NACTA Journal, 59, 155-163.
Gibbs, G. y Simpson, C. (2009). Condiciones para una evaluación continuada favorecedora del aprendizaje. ICE, Universitat de Barcelona y Ediciones Octaedro.
Ginns, P., Prosser, M. y Barrie, S. (2007). Students’ perceptions of teaching quality in higher education: The perspective of currently enrolled students. Studies in Higher Education, 32(5), 603-615.
Glasserman Morales, L. D. y Ruiz Ramírez, J. A. (2021). Características del aseguramiento de la calidad educativa: un mapeo sistemático 2016-2020. Revista Complutense de Educación.
González, C. M., Varela, S. y Miguel, S. (2017). Aplicación de algoritmos no supervisados para la detección de tópicos de investigación. Presentado en V Jornadas de Intercambio y Reflexión acerca de la Investigación en Bibliotecología. Universidad Nacional de La Plata, Facultad de Humanidades y Ciencias de la Educación, La Plata.
Günther, F., Rinaldi, L. y Marelli, M. (2019). Vector-space models of semantic representation from a cognitive perspective: A discussion of common misconceptions. Perspect Psychol Sci., 14(6), 1006-1033.
Harris, C. y Kloubec, J. (2014). Assessment of student experience in a problem-based learning course using the course experience questionnaire. Journal of Nutrition Education and Behavior, 46(4), 315-319.
Hirschberg, J. y Lye, J. (2016). The influence of student experiences on post-graduation surveys. Assessment & Evaluation in Higher Education, 41(2), 265-285.
Huybers, T. (2017). Exploring the use of best-worst scaling to elicit course experience questionnaire responses. Assessment & Evaluation in higher education, 42(8), 1306-1318.
Jaramillo, C. L. (2019). El proceso de la encuesta online. Más poder local, 39, 30-33.
Kaye, B. K. y Johnson, T. J. (1999). Research methodology: Taming the cyber frontier: Techniques for improving online surveys. Social Science Computer Review, 17(3), 323-337.
Kurilovas, E. (2020). On data-driven decision-making for quality education. Computers in Human Behavior, 107, 105774.
Lyon, P. M. y Hendry, G. D. (2002). The use of the Course Experience Questionnaire as a monitoring evaluation tool in a problem-based medical programme. Assessment & Evaluation in Higher Education, 27(4), 339-352.
Manarbek, G., Zhakupova, G., Kaliyeva, A. y Hezi, H. (2020). The university-industry cooperation: The role of employers in quality assurance of Education. En E3S Web of Conferences (vol. 159, p. 09010).
Markowitsch, J. (2018). Is there such a thing as school quality culture? In search of conceptual clarity and empirical evidence. Quality Assurance in Education, 26(1), 25-43.
Martín, L. M. (2000). The relationship of college experiences to psychosocial outcomes in students. Journal of College Student Development, 41(3), 292-301.
Paricio, J. (2019). La calidad de “lo que el estudiante hace”: aprendizaje activo y constructivo. En J. Paricio, A. Fernández e I. Fernández (eds.), Cartografía de la buena docencia universitaria. Un marco para el desarrollo del profesorado basado en la investigación (pp. 57-88). Narcea.
Paricio, J., Fernández March, A. y Fernández Fernández, I. (2019). Marco de desarrollo académico docente. Un mapa de la buena docencia universitaria basado en la investigación.
Paricio, J. y Fernández March (2023). Seis estándares para la evaluación del profesorado desde el Nivel 1 del MDAD. Red de Docencia Universitaria (REDU).
Pascarella, E. T. y Terenzini, P. T. (1991). How college affects students: findings and insights from twenty years of research. Jossey-Bass Inc.
Pascarella, E. T. y Terenzini, P. T. (2005). How college affects students: a third decade of research (2nd ed.). Jossey-Bass Inc.
Piza Burgos, N. D., Amaiquema Márquez, F. A. y Beltrán Baquerizo, G. E. (2019). Métodos y técnicas en la investigación cualitativa. Algunas precisiones necesarias. Conrado, 15(70), 455-459.
Pulido, A. (2005). Indicadores de calidad en la evaluación del profesorado universitario. Estudios de Economía Aplicada, 23, 667684.
Ramsden, P. (1991). A Performance Indicator of Teaching Quality in Higher Education: The Course Experience Questionnaire. Studies in Higher Education, 16, 129-150.
Richardson, J. T. (2005). Students’ approaches to learning and teachers’ approaches to teaching in higher education. Educational Psychology, 25(6), 673-680.
Richardson, J. T. E., Slater, J. B. y Jane (2007). The National Student Survey: development, findings and implications. Studies in Higher Education, 32(5), 557-580.
Richardson, W. (2012). Why school: How education must change when learning and information are everywhere. TED.