Métricas alternativas
Keywords
GPT-3
análisis textual
investigación cualitativa
inteligencia artificial natural language processing
GPT-3
text analysis
qualitative research
artificial intelligence
How to Cite
Abstract
Assigning meaning to segments of information through analysis of open texts in qualitative research requires considerable investment of time. Natural Language Processing tools can be a valuable resource for qualitative researchers, as their algorithms allow for faster, qualitative interpretation of texts. However, this requires testing these tools’ levels of verbal comprehension beforehand. The introduction of OpenAI's GPT-3 model has marked a qualitative leap forward compared to previous Natural Language Processing models. The study objective was to analyse this tool’s verbal comprehension ability. The tests from the verbal comprehension index of the WAIS-IV IQ battery were applied. The results of the reliability tests were satisfactory. The responses put GPT-3 higher than the 99th percentile of human standards of verbal comprehension. These results demonstrate that it is possible to use this model as a tool to analyse open texts, opening up enormous possibilities for qualitative research, although its use must be based on precise, specific utilization for each analysis process.
References
Aspers, P., & Corte, U. (2019). What is qualitative in qualitative research. Qualitative Sociology, 42(2), 139–160. https://doi.org/10.1007/s11133-019-9413-7
Bail, C. (2014). The cultural environment: Measuring culture with big data. Theory and Society, 43(3), 465–482. https://doi.org/10.1007/s11186-014-9216-5
Bajaj, D., Goel, A., Gupta, S.C., & Batra, H. (2022). MUCE: A multilingual use case model extractor using GPT-3. International Journal of Information Technology, 14, 1543–1554. https://doi.org/10.1007/s41870-022-00884-2
Binz, M., & Schulz, E. (2023). Using cognitive psychology to understand GPT-3. PNAS. Proceedings of the National Academy of Sciences, 120(6), e2218523120. https://doi.org/10.1073/pnas.2218523120
Brown, T.B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., Agarwal, S., Herbert-Voss, A., Krueger, G., Henighan, T., Child, R., Ramesh, A., Ziegler, D.M., Wu, J., Winter, C., … Amodei, D. (2020). Language models are few-shot learners. In H. Larochelle, M.A. Ranzato, R. Hadsell, M.F. Balcan, & H.T. Lin (Eds.), Advances in Neural Information Processing Systems 33 (NeurIPS 2020). https://bit.ly/3ZmRPQG
Carriere, J., Shafi, H., Brehon, K., Pohar, M.K., Churchill, K., Ho, C., & Tavakoli, M. (2021). Case report: Utilizing AI and NLP to assist with healthcare and rehabilitation during the COVID-19 pandemic. Frontiers in Artificial Intelligence, 4, 613637. https://doi.org/10.3389/frai.2021.613637
Chang, T., DeJonckheere, M., Vydiswaran, V.G.V., Li, J., Buis, L.R., & Guetterman, T.C. (2021). Accelerating mixed methods research with natural language processing of big text data. Journal of Mixed Methods Research, 15(3), 398–412. https://doi.org/10.1177/15586898211021196
Cohen, L., Manion, L., & Morrison, K. (2018). Research methods in education. Routledge. https://doi.org/10.4324/9780203224342
Cope, B., & Kalantzis, M. (2015). Sources of evidence-of-learning: Learning and assessment in the era of big data. Open Review of Educational Research, 2(1), 194–217. https://doi.org/10.1080/23265507.2015.1074869
Cypress, B.S. (2019). Data analysis software in qualitative research. Preconceptions, expectations, and adoption. Dimensions of Critical Care Nursing, 38(4), 213-220. https://doi.org/10.1097/DCC.0000000000000363
Detterman, D. (2011). A challenge to Watson. Intelligence, 39(2-3), 77-78. https://doi.org/10.1016/j.intell.2011.02.006
Dobrev, D. (2019). The IQ of artificial intelligence. Serdica Journal of Computing, 13(1-2), 41-70. https://doi.org/10.55630/sjc.2019.13.41-70
Dowe, D.L., & Hernández-Orallo, J. (2012). IQ tests are not for machines, yet. Intelligence, 40(2), 77–81. https://doi.org/10.1016/j.intell.2011.12.001
Flick, U. (2018). Designing qualitative research. SAGE.
García-Peñalvo, F.J. (2023). La percepción de la inteligencia artificial en contextos educativos tras el lanzamiento de ChatGPT: Discrupción o pánico. Education in the Knowledge Society, 24, e31279. https://doi.org/10.14201/eks.31279
García Peñalvo, F.J., Llorens-Largo, F., & Vidal, J. (2023). La nueva realidad de la educación ante los avances de la inteligencia artificial generativa. RIED-Revista Iberoamericana de Educación a Distancia, 27(1). https://doi.org/10.5944/ried.27.1.37716
Golder, S.A., & Macy, M.W. (2011). Diurnal and seasonal mood vary with work, sleep, and day length across diverse cultures. Science, 333(6051), 1878–1881. https://doi.org/10.1126/science.1202775
Grodal, S., Anteby, M., & Holm, A.L. (2021). Achieving rigor in qualitative analysis: The role of active categorization in theory building. Academy of Management Review, 46(3), 591-612. https://doi.org/10.5465/amr.2018.0482
Guetterman, T.C., Chang, T., DeJonckheere, M., Basu, T., Scruggs, E., & Vydiswaran, V. (2018). Augmenting qualitative text analysis with natural language processing: Methodological study. Journal of Medical Internet Research, 20(6), e231. https://doi.org/10.2196/jmir.9702
Hammersley, M. (2013). What is qualitative research? Bloomsbury Academic.
Hernández-Orallo, J., & Minaya-Collado, N. (1998). A formal definition of intelligence based on an intensional variant of Kolmogorov complexity. In Proceedings of the international symposium of engineering of intelligent systems (EIS’98) (pp. 146–163). ICSC Press. https://bit.ly/3JdQTbK
Longo, L. (2020). Empowering qualitative research methods in education with artificial intelligence. In A.P. Costa, L.P. Reis, & A. Moreira (Eds.), Computer supported qualitative research. New trends on qualitative research (WCQR2019). (pp. 1-21). Springer. https://doi.org/10.1007/978-3-030-31787-4
Mills, K. A. (2019). Big data for qualitative research. Routledge Focus. https://doi.org/10.4324/9780429056413
Oussalah, M. (2022). AI explainability. A bridge between machine vision and natural language processing. In A. Del Bimbo, R. Cucchiara, S. Sclaroff, G.M. Farinella, T. Mei, M. Bertini, H.J. Escalante, & R. Vezzani (Eds.), Pattern recognition. ICPR International workshops and challenges (pp. 257-273). Springer. https://doi.org/10.1007/978-3-030-68790-8
Rezaeenour, J., Ahmadi, M., Jelodar, H., & Shahrooei, R. (2022). Systematic review of content analysis algorithms based on deep neural networks. Multimedia Tools & Applications. https://doi.org/10.1007/s11042-022-14043-z
Shlomo, N., & Goldstein, H. (2015). Big data in social research. Journal of the Royal Statistical Society, 178(4), 787–790. https://doi.org/10.1111/rssa.12144
Si, C., Gan, Z., Yang, Z., Wang, S., Wang, J., Boyd-Graber, J., & Wang, L. (2022). Prompting GPT-3 to be reliable. arXiv. https://doi.org/10.48550/arXiv.2210.09150
Silver, C., & Lewins, A. (2014). Using software in qualitative research: A step-by-step guide. SAGE.
Tack, A. & Piech, C. (2022). The AI teacher test: Measuring the pedagogical ability of blender and GPT-3 in educational dialogues. arXiv. https://doi.org/10.48550/arXiv.2205.07540
Towler, L., Bondaronek, P., Papakonstantinou, T., Amlôt, R., Chadborn, T., Ainsworth, B., & Yardley, L. (2022). Applying machine-learning to rapidly analyse large qualitative text datasets to inform the COVID-19 pandemic response: Comparing human and machine-assisted topic analysis techniques. MedRxiv. https://doi.org/10.1101/2022.05.12.22274993
Turing, A.M.I. (1950). Computing machinery and intelligence. Mind, 236(LIX), 433–460. https://doi.org/10.1093/mind/LIX.236.433
Wang, H., Tian, F., Gao, B., Bian, J., & Liu, T.Y. (2016). Solving verbal comprehension questions in IQ test by knowledge-powered word embedding. arXiv. https://doi.org/10.48550/arXiv.1505.07909
Wechsler, D. (2012a). WAIS-IV. Escala de inteligencia de Wechsler para adultos-IV. Manual de aplicación y corrección. Pearson.
Wechsler, D. (2012b). WAIS-IV. Escala de inteligencia de Wechsler para adultos-IV. Manual técnico y de interpretación. Pearson.
Williams, E.N., & Morrow, S.L. (2009). Achieving trustworthiness in qualitative research: A pan-paradigmatic perspective. Psychotherapy Research, 19(4-5), 576-582. https://doi.org/10.1080/10503300802702113
Williams, M., & Moser, T. (2019). The art of coding and thematic exploration in qualitative research. International Management Review, 15(1), 45-55. https://bit.ly/3mt4BPh
Xie, Y., Vosoughi, S., & Hassanpour, S. (2022). Interpretation quality score for measuring the quality of interpretability methods. arXiv. https://doi.org/10.48550/arXiv.2205.12254
This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License.
Copyright (c) 2023 Pixel-Bit. Media and Education Journal