Evaluating the response quality of LLM-driven university guidance applications

Abstract

Recent technological advances are creating new educational challenges. The rise of Artificial Intelligence is enabling the implementation of new tools useful for education. In the case of university guidance services, applications such as Copilot or ChatGPT, based on multimodal language models, stand out. The aim of this study is to analyse the quality and reliability of the answers provided by Copilot and ChatGPT to questions posed by students in informal networks. The method was based on a qualitative and analytical validation approach to assess the accuracy of the answers. An observation tool consisting of 48 items, divided into four thematic blocks: access, management, difficulty of studies and employability, was applied in Copilot and ChatGPT-4 for fifteen public universities. A sufficient degree of fit of 100% was determined for all thematic blocks, except for management. Two items were found to have an insufficient degree of fit. Both were implemented in the new multimodal language model ChatGPT-4o and an improvement in the degree of fit was detected. Subsequently, the answers provided by ChatGPT-4o and the information found on the websites were described,
highlighting the confusion regarding the credit price information on the websites and the difficulty in finding the maximum limit of credits to be taken while studying at the same time. It is concluded that Copilot and ChatGPT have potential as university guidance services. The effectiveness of these AI assistants will depend on the quality and accessibility of information on university websites. It is essential that universities organise and update the information on their websites to improve the effectiveness of AI-based applications.

Keywords: artificial intelligence, guidance, recognition of studies, education technology, university.