Admission Test Model for Pre-service Teacher Training
Courses Based on a Participatory Design Approach

Modelo para las pruebas de admisión a los estudios de
maestro a partir de un proceso de diseño participativo

DOI: 10.4438/1988-592X-RE-2021-393-493

Miquel F. Oliver-Trobat

Maria Rosa Rosselló-Ramon

Rubén Comas-Forgas

Aina Calvo-Sastre

Jaume Sureda-Negre

Universidad de las Islas Baleares

Abstract

This paper describes and presents the design, development and implementation process of an ad-hoc test for student admission to pre-service teacher training bachelor’s degrees (both early childhood and primary education). Admission tests for teaching training courses were designed and validated over several years through a participatory methodology involving over 500 participants. The tests were proposed within the framework of a programme aimed at establishing and applying various measures to improve initial teacher training at the University of the Balearic Islands. The final result of the process is a model for the design and implementation of admission tests from a participatory perspective that may be useful for the development of similar tests at other universities.

Keywords: Teacher education, initial training, admission criteria, participatory research, teacher recruitment, teaching profession.

Resumen

En este trabajo se presenta el proceso de diseño, elaboración e implementación de unas pruebas de admisión ad-hoc para el ingreso de alumnado a los Grados de Maestro. A lo largo de varios cursos, mediante una metodología participativa y con la implicación de más de 500 participantes, se proyectaron y validaron unas pruebas de admisión para acceder a los Grados de Educación Primaria e Infantil. Estas pruebas se plantearon en el marco de un programa orientado a establecer y aplicar diversas medidas para la mejora de la formación inicial docente en la Universidad de las Islas Baleares. El resultado final del proceso es un modelo para el diseño y la implementación de unas pruebas de admisión desde una perspectiva participativa que puede ser de utilidad para la elaboración de pruebas similares en otras universidades.

Palabras clave: Formación de profesores, formación inicial, criterios de admisión, investigación participativa, selección del profesor, profesión docente.

Introduction

Nearly all countries set criteria for admission to teacher training studies, whether through a general system used for all courses (mainly based on high-school grades and university entrance exams) or through a specific procedure with supplementary requirements to the general admission criteria (Egido-Gálvez, 2020; Eurydice, 2018). The first option is the most common both internationally and in Spain. However, 21 Spanish universities (five public and 16 private) use some type of specific test. In terms of what these tests cover, ten universities exclusively focus on cognitive aspects (five public and five private); two solely run non-cognitive tests; and nine private universities combine cognitive and non-cognitive tests. At this time, no public university assesses non-cognitive abilities (Manso 2019, Generalitat de Catalunya, 2021 and Universidad de Deusto, 2021).

The main reasons given to justify setting selective criteria for admission to teacher training degrees are equity, social justice, market saturation and improving course quality (Holden and Kitchen, 2017). The latter argument is the most common in Spain. It appears repeatedly in highly diverse documents which, in recent years, have agreed on the need to re-think the entry system for initial teacher training. These documents have emerged from reflections by different academic institutions (Conferencia de Decanos/as de Educación, 2017), analyses by professional associations and groups (Foro de Sevilla, 2014; REDE, 2018; Grupo Palma, 2018) and results from government-led forums (Marina, Pellicer and Manso, 2015; Ministerio de Educación y Formación Profesional, 2018).

With regard to research on the topic, current reviews (Menter, Hulme, Elliot and Lewin, 2010; Parker, 2018) highlight that the issue of Admission Tests (AT) is one of the least studied in the area of initial teacher training. Generally, these reviews have focused on analysing admission models in different countries and putting recommendations forward for their implementation (Casey and Childs, 2007; Martínez-Martin, Prats-Gil and Marín-Blanco, 2015; Klassen, Dolan and Afzal, 2015; Childs and Ferguson, 2015; The Australian Secondary Principals’ Association, 2015; Darmody and Smyth, 2016; Holden and Kitchen, 2017; Egido-Gálvez, 2020; Pérez-Granados, 2015, 2018). Further guiding principles include how to set criteria for the tests, how they are implemented and their effectiveness or predictive ability for professional success (Corcoran, 2000; Corcoran and O’Flaherty, 2018; Katz and Frish, 2016; Mikitovics and Crehan, 2002; Wright, 2015).

Selecting candidates for initial teacher training continues to be a challenge both in terms of policy and educational research: it is not easy to identify applicants whose personal attributes enable us to foresee them having greater potential for success in their initial training and subsequent professional career. In turn, establishing selective tests or not is an axiological issue that lies outside the evidence that educational research can provide.

In this context, it should be highlighted that the production process for this type of test from a participatory design perspective has not been undertaken previously. This approach is likely to improve efficiency and effectiveness in educational change processes (Janssen, Könings and van Merriënboer, 2017), and has been shown to be relevant in different areas of education (Simonsen and Robertson, 2013; Bustamante, Brendel, Degbelo and Kray, 2018; Könings, Bovill and Woolner, 2017).

Thus, with the goal of improving bachelor’s degree courses in early childhood and primary education (GIP), the Faculty of Education at the University of the Balearic Islands (UIB) began a participatory process in 2013 that focuses on designing, validating and implementing an ad hoc AT for these courses. This article analyses this process and its results.

Method

The general aim of this work is to describe and analyse the AT production process on the Early Childhood and Primary Education Degrees at the UIB from a participatory design approach (Abu-Amsha, Gordon, Benton, Vasalou and Webster, 2019; Engelbertink, Kelders, Woudt-Mittendorff and Westerhof, 2020). Different procedures were undertaken throughout this process aimed at: (1) exploring and identifying problems; (2) reflection and action; (3) planning based on critical contributions and, finally, (4) designing prototypes. The specific aims of the work are set out in Table I. All these aims led to a three-phase strategy: pre-design, design and post-design (see Table I).

With regard to the type and number of participatory strategies used throughout the process, we should highlight that two Delphi panels (Diamond, Grant, Feldman, Pencharz, Ling, Moore, Wales, 2014), four workshops (Lee, Leong and Chan, 2015; Nickelsen y Bal, 2021) and three work teams (WT) (Bayona and Heredia, 2012) were organised. Finally, two validation processes were performed: a context analysis and a literature review from a systematised perspective (Alexander 2020; Codina, 2018).

TABLE I. Objectives, strategies and participants in each of the design process phases

PHASES	SUB-PHASES	OBJECTIVES	STRATEGIES	PARTICIPANTS
Phase 1. Pre-design	1.1. Exploring and identifying problems	- Analysing the status of initial teacher training in the Balearics	- Context analysis of initial teacher training	- Research team
		- Determining measures to improve initial teacher training in the Balearics	- Delphi panel-1 (5 rounds)	- 28 lecturers from the Department of Pedagogy at the UIB that teach most GIP classes
			- Delphi panel-2 (3 rounds)	- 128 lecturers from different departments that teach on the GIP and 157 teachers-placement tutors at schools
	1.2. Agreed strategic design based on reflection	- Designing a collaborative programme to improve initial teacher training	- Work team-1: producing an intervention proposals document	- 28 lecturers from the Department of Pedagogy at the UIB who teach most on the GIP and consultation with all course lecturers
Phase 2. Design.	2.1. AT planning	- Producing a scientific literature review on the topic	- Systematic review of admission tests and criteria	- Research team
		- Determining the purpose, attributes and assessment tools for the AT	- Workshop 1 and 2 on analysing two key experiences	- Research team with: a team from the Finnish Institute for Educational Research (Workshop-1) and a team from the MIF programme in Catalonia (Workshop-2)
			- Workshop-3 on the purpose and attributes of the test	- 41 specialists from five Spanish universities; teachers and head teachers from early childhood and primary education centres, and representatives from government education bodies
	2.2. AT design and validation	- Designing and validating the AT	- Work team-2: initial and final test design	- 12 university lecturers and 2 teachers
			- Validation-1 (internal and external)	- 13 experts (university lecturers and teachers) - 10 teachers, counsellors and heads from secondary education centres - 107 baccalaureate and VET students
Phase 3. Post-design	3.1. Planning and designing supplementary measures to attract the best candidates	- Planning and designing supplementary measures to attract the best prepared and most motivated candidates	- Workshop-4 on ‘Defining supplementary measures for admission tests’	- 39 specialists (university lecturers and working teachers)
	3.2. Designing a longitudinal assessment study of the skills shown in the tests and their evolution during the GIP degree programmes	- Designing an assessment system of student skills to assess the predictive validity of the AT	- Work team-3: designing the assessment system	- Research team
			- Validation-2 by experts	- 13 experts (university lecturers and teachers)

Source: Prepared by the authors

Results

This section sets out the dynamic and main results of each of the three phases for the process followed.

FIGURE I. AT planning cycle and main results from the phases

Source: Prepared by the authors

Phase 1. AT pre-design (2013-2017)

Exploring and identifying problems

After a highly detailed analysis of the status of initial teacher training in the education system of the Balearic Islands and at the UIB (Sureda-Negre and Oliver-Trobat, 2015), and with a view to identifying agreed measures to improve courses, Delphi Panel-1 was organised with 28 participating department lecturers at the UIB who taught the most on the early childhood and primary education degree courses. This resulted in 14 proposed measures. ‘Establishing selective admission criteria for courses’ (Sureda-Negre, Oliver-Trobat, Comas-Forgas, 2016, p. 158) received the widest agreement.

Afterwards, Delphi Panel-2 was organised with all UIB lecturers who had at least three years of teaching experience on the GIP courses: in total, 128 university lecturers from 11 departments and 157 teachers-placement tutors were invited to take part (Calvo-Sastre, Sureda-Negre and Oliver-Trobat, 2018). Consequently, 170 took part in the first round, 129 in the second and 149 in the third. This led to agreed improvement measures including ‘that establishing more selective admission criteria would be a measure for greatly improving future teachers’ training’ (Sureda-Negre, Oliver-Trobat and Comas-Forgas, 2020, p. 85).

Agreed strategic design based on reflection

Based on the state-of-play in initial training within the context of experience, and after identifying agreed improvement measures (amongst university lecturers and teachers-placement tutors at schools), the Office of the Vice Chancellor at the UIB was called on to participate in the debate. This led to the creation of the Improvement Programme for Initial Training on Primary and Secondary Teacher Training Courses (PM_FIP)1 (UIB, 2017b; Oliver-Trobat, Sureda-Negre and Calvo-Sastre, 2018).

With the framework of the PM_FIP, Work Team-1 drafted an initial ‘Strategic and Intervention Proposals Document’ (PM_FIP, 2017; UIB, 2017a) with the participation of 28 lecturers from a department on the early childhood and primary education degree courses. Six objectives were set out alongside their respective action lines and assessment indicators. Objective three included improving the admission system for the GIP degrees, which proposed designing and implementing an experimental student selection system.

Thus, the pre-design phase agreed on the suitability of producing ATs for initial teacher training as an improvement strategy for the early childhood and primary education degree courses.

Phase 2. AT design (2017-2019)

AT planning

Determining the purpose, attributes and assessment tools for the AT was approached in three workshops. Workshop-1, held in mid-2017, focused on analysing the Finnish model whilst Workshop-2, held in late 2017, analysed the Teacher Training Improvement and Innovation Programme (MIF) experience in Catalonia (Martínez-Martin et al., 2015).

Workshop-3 was held in May 2018 and was organised into four working groups. It focused on discussing the purposes of the ATs, their attributes and the assessment strategies. The participants included 41 specialists from five Spanish universities, teachers and head teachers from early childhood and primary education centres, and representatives from government education bodies.

A bibliographic review had been undertaken previously with a systematised research focus into programme admissions. Up to 22 databases were consulted and, based on different criteria (including only focusing on empirical studies published from 1980 onwards), 137 documents were selected. The resulting document was made available to the Workshop-3 participants so that they had an overview of the state-of-play for the topic when making decisions.2

AT purpose

It should firstly be noted that the ultimate purpose of the entire process was established in the pre-design phase. Both the results from the Delphi panels and the production of the PM_FIP made it clear that establishing selective criteria for admission to the GIP degree programmes was an important strategy for improving the courses. Nonetheless, it was deemed appropriate for Workshop-3 participants3—who had viewed the systematised literature review from the previous phase—to reflect on this purpose. The aim was to ascertain the reasons why implementing ATs could lead to an improvement in the degree programmes. The analysis of the conclusions from the workshop working groups highlighted four broad core reasons:

■ The need to identify and recruit the best candidates since places are limited (setting value)

■ The advisability of not accepting unsuitable students for the programmes (suitability value)

■ The need to design ATs based on academic (in terms of skills) and non-academic attributes (attribution value)

■ The advisability of knowing candidate strengths and weaknesses for optimum future development in initial training and in their professional career (predictive value).4

AT attributes and assessment strategies

Based on the consensus achieved in Workshop-3, it was decided that ATs should focus on three cognitive or academic attributes (communication skills in both official languages of the Balearics, logic and mathematical skills, and oral communication) and several non-cognitive skills (interpersonal abilities, motivations, prior experience, etc.). In turn, the assessment strategies for these attributes were also agreed.

AT design

Once the attributes to be included and the assessment strategies to be followed were decided on, WT-2 comprising UIB lecturers and current schoolteachers was set up to undertake and specify the AT design, based on the scientific literature review on the topic and the consensus reached in Workshop-3. The result was a test prototype split into three phases: (a) cognitive selection through written tests; (b) mixed selection (cognitive and non-cognitive) through oral tests; and (c) eligibility verification.5 The cut-off mark for all tests (after a logical analysis of the items included in them) was set at five on a scale of 0 to 10 (González and Jornet, 2009).

FIGURE II AT phases

Source: Compiled by authors

Cognitive selection phase (written)

The prototype for this phase was structured into two tests aimed at assessing Communicative Competence and Critical Thinking (CCCT) and Logic and Mathematical Competence (CLOM). Assessment for the CCCT was via a test comprising: (a) understanding and interpreting texts (a commentary on two texts: one in Spanish and one in Catalan with a similar content); (b) writing skills (two essay questions); and (c) language system mastery (three exercises on basic language knowledge). With regard to assessing the CLOM skills, a test with ten exercises and five scenarios from different content blocks was drafted: numbers and equations, geometry, algebra, probability, statistics and measurement.

Candidates would have to pass these two tests in order to move on to the oral selection phase.

Mixed selection phase: cognitive/non-cognitive (oral)

In order to score attributes from the mixed selection stage6, two complementary tests were chosen: a Video Presentation (VP) and a Group Interview (GI).

The VP comprised individual production of a so-called written profile but in video format.7 Candidates were requested to make an ‘introductory letter-style’ video responding to a series of questions: Who am I? What am I like? What are my interests and hobbies? Why do I want to train to be a teacher?, etc.

With regard to the GI, applicant groups were formed who were given a news item on education and discussed, reasoned and argued their views until they reached a common position. Each interview was led and assessed by a university lecturer and a current schoolteacher.

To attain objective results from the interview, WT-2 designed a score sheet with six indicators for cognitive attributes (communication skills) and three non-cognitive attributes (interpersonal skills). In addition to the assessment indicators, it also highlighted their performance levels (with four attainment levels) and the weighting of each test in the final grade (30% for the VP and 70% for the GI). Interviewer training activities were also run, including role-play instruction.

Table II. Cognitive and non-cognitive attributes of the ATs

Phase 1. Cognitive tests (written)	Phase 2. Mixed tests (oral)
Cognitive attributes	Cognitive attributes	Non-cognitive attributes
- Understanding the meaning of written texts on useful topics, the ability to summarise and incorporate information, expressing thinking in a clear and concise manner, and mastery of the language system enabling communication at an appropriate language level in the two official languages of the Autonomous Region - The ability to apply scientific and technical thinking and knowledge about the social setting to interpret information received and to predict and take decisions independently and with initiative, as well as differentiate and assess scientific knowledge by contrasting it with other forms of knowledge - The ability to formulate, use and interpret mathematics in different contexts, solving problems that require basic calculation operations, basic geometric knowledge, estimates that require data processing, probability and chance	- The ability to discuss and argue, and knowing how to orally express ideas fluently and clearly, using correct language, varied vocabulary and a variety of grammatical structures	- Interpersonal skills - Psychological adaptation - Intrinsic motivation - Professional development orientation - Prior experiences, beliefs and motivations

Source: Compiled by authors

Eligibility verification stage

This phase verified whether the participating students had attained the University Admission Grade (CAU), thus ensuring they fulfilled the eligibility criteria. This would be the equivalent of the ‘eligibility checks’ stage proposed by Klassen and Kim (2017, p.19).8

AT validation

The plausibility of the tests (Ebel and Frisbie, 1991; Bachman, 1990 and López, 2009) was ensured in one external and three internal validation processes. Internal validation was performed by consulting: (a) experts not involved in producing the test; (b) secondary education teachers, counsellors and head; and (c) the candidates. External validation was performed by applying statistical methods to determine likeness between the results obtained by students in the pilot test and those in the assessment tests performed at their home school centre during the school year.

Internal validation

Validation by experts

With regard to the expert validation (Callejo-Gallego and Viedma-Rojas, 2016; Díez-Gutierrez, 2020), a group of 13 specialists was established who had experience in the topics in the different tests (see Table III). These experts provided advice throughout the process whilst suggesting proposals to improve the prototypes and the assessment criteria, in line with indicators on clarity; importance; coverage; relevance; plausibility (to assess candidates for the degree courses); comprehension, and knowledge.

TABLE III. Composition of the external advisor group for expert validation

Tests	Group members
CCCT	- Two lecturers from the University of Girona (one of whom was a coordinator for the communicative competence admission tests in Catalonia)
CLOM	- Two schoolteachers from primary education centres who were experts in logic and mathematical training - Two lecturers from the University of Barcelona (one of whom was a coordinator for the logic and mathematical admission tests in Catalonia) - A lecturer from the University of Girona
VP and GI	- Five lecturers from the University of the Balearic Islands - One lecturer from the Pompeu Fabra University

Source: Compiled by authors

Validation by secondary education teachers

Ten teachers, counsellors and heads from different schools involved in the pilot test took part in this validation process. The assessors were asked to score (from 0 to 10) the need to run admission tests, returning an average score of 9.4.

The validation process comprised assessing the clarity, suitability, quantity and quality of the tests on a four-level Likert scale (see Table IV), as well as the efficacy, number of questions and length of the test (see Table V).

TABLE IV. Score for the instructions and information to answer the questions in the cognitive tests

	Mean (4=excellent; 1=poor)
	CCCT	CLOM
Clarity	3.6	3.7
Suitability	3.4	3.5
Quantity	3.3	3.2
Quality	3.4	3.6

Source: Compiled by authors

TABLE V. Score for the overall design of the cognitive tests

	Mean (4=excellent; 1=poor)
	CCCT	CLOM
Efficacy in providing useful information to assess the suitability of future students	3.4	3.9
Number of questions	3.3	3.3
Test length	3.5	3.4

Source: Compiled by authors

They also assessed the instructions, information and design of the tests quantitively (with average scores between 3.2 and 3.9). In all but two instances, the top score was four and the lowest three, meaning agreement was high amongst the assessors.

The assessors made several suggestions to improve the test content. The most significant are highlighted below:

Suggested improvement for both tests:

– Include skill scoreboards for SEN / SLD / ADHD students for dyslexia or dyscalculia in order to assess how this may affect their teaching future.

Suggested improvements for CCCT:

– Include a question where they have to write a text with a register more suited to the pupils they will teach

– Include some critical thinking activities that assess the psychological profile of the students

– More clearly separate between Spanish and Catalan to avoid confusion

– Change the font size and line spacing. Perhaps also clearly separate questions in Catalan and in Spanish.

Suggested improvements for CLOM:

– Make them argue in different resolution strategies or in an activity

– Add more logic tests and remove a mathematical exercise

– Take into account the methods: knowing how a result is attained.

This process enabled content validity for the tests to be established through expert agreement (Pedrosa, Suárez-Álvarez and García-Cueto, 2013).

Validation by students

Students participating in the pilot test (n=107) were asked to fill in an anonymous questionnaire about the difficulty and duration of the tests, the clarity of the information and the number of questions. The aspects that scored highest amongst students were (on a 10-point Likert scale): clarity of information and instructions in the GI (9.5); the CCCT (9.1); the number of questions in the GI (9.5), and the length of the test (9.1 for the CCCT and 9.3 for the CLOM and GI). The aspect that scored lowest was the level of difficulty: CLOM (7.7) and CCCT (7.8) (see Table VI). Most students gave scores between seven and ten, representing a fairly high level of consensus.

TABLE VI. Scores for the tests given by students

Test features	Average (10 = very suitable; 1 = not at all suitable)
	CCCT (n=107)	CLOM (n=107)	GI (n=16)
Level of difficulty	7.8	7.7	8.9
Clarity of information and instructions to help answer the questions	9.1	8.8	9.5
Number of questions	8.6	8.8	9.5
Length	9.1	9.3	9.3

Source: Compiled by authors

Pilot test

The pilot trial was performed with the sample of 107 students who took part in the validation (from the second year of baccalaureate and upper level VET) from 14 centres (nine public schools and five charter schools). In light of the impossibility to run a random sample in the strictest sense, it was decided to do an ad-hoc sample with representation from different types of centres. The students at these centres who showed interest in registering on one of the two GIP degree programmes were invited to sign up for the pilot test and, if they passed the first phase, to take part in the mixed selection phase.

Descriptive statistics of central tendency (mean, mode and median), absolute dispersion (standard deviation) and relative dispersion (coefficient of variation) were used to analyse the results from each test. The Pearson correlation coefficient was used with a view to determining the possible link between different analysed variables. The SPSS v.24 statistical analysis program was used to process and analyse the data.

FIGURE III Histogram for result frequency in the cognitive tests

Source: Compiled by authors

The bivariate analysis shows a low correlation between the CCCT and CLOM tests, with a Pearson correlation coefficient of 0.313 (p=0.001). This makes sense since the tests measure different attributes.

Nearly half of the candidates (49.6%) failed to pass one of the tests in the first phase in line with the set criteria; 50.4% moved on to the next phase. The screening effect of this phase is clear (Klassen and Kim, 2017), ensuring that candidates who pass it have a high level of cognitive attributes.

With regard to the mixed tests, it should be noted that from the 55 students who agreed to take part, 25 passed Phase 1. A representative group of 16 was selected from this set (sex, selected course—early childhood or primary—, home centre and marks from phase one) who had given their consent to take part in this phase. Students were split into two equal groups for the VP and GI.

TABLE VII. Descriptive statistics results for students who did the mixed tests

Descriptive statistics	VP Score	GI Score
Mean	7.61	7.05
Median	7.22	6.94
Mode	8.88	10
Standard deviation	1.55	2.09
Coefficient of variation	0.20	0.30
n	16	16

Source: Compiled by authors

All candidates passed the VP, whilst four students failed the GI. After applying percentages (30% VP and 70% GI), only one candidate failed the test globally by not attaining the necessary mean, with low scores in most cognitive skill indicators but, especially, communicative and interpersonal skills which were not detectable in the cognitive phase.

External validation

In order to determine the relationship between the students’ results and other assessment tests performed at their home centre, a similarity indicator was used for the cognitive tests based on an analysis of the information provided about the participating students from their tutors. This information showed to what extent the test results matched the usual results for said students during the year at their home education centre. It was noted that coincidence was high for both the CCCT (79.4%) and CLOM tests (75.7%) (see Table VIII). After calculating the central tendency and variability measures, it was observed that the differences in means were very low, meaning the indices of dispersion were very small. In this sense, it was concluded that the score given by the judges was convergent.

TABLE VIII. Coincidence level for the cognitive test results with the usual results for students at their home centre

Coincidence level	CCCT		CLOM
	Frequency	Percentage	Frequency	Percentage
No coincidence	5	4.7%	3	2.8%
Low coincidence	17	15.9%	22	20.6%
Fairly similar	53	49.5%	50	46.7%
Total coincidence	32	29.9%	31	29%
Missing	0	0%	1*	0.9%
Total	107	100%	107	100%

* One student was not taking mathematics during this school year.

Source: Compiled by authors

Based on all the information collected in the validation, WG-2 reformulated and improved9 the AT proposal, which was approved by the UIB Governing Council in July 2019 and received a positive report on modifying course syllabi from the National Agency for Quality Assessment and Accreditation (ANECA) in April 202010.

Phase 3 Post-design (2019-2021)

Within the framework of planning and designing supplementary measures based on critical contributions, Workshop-4 was run with 39 specialists (university lecturers and current schoolteachers) with a view to designing measures to attract the best prepared and most motivated students.

The main measures adopted by consensus included:

■ Offering pre-candidates the chance to visit innovative educational centres

■ Running a promotion and awareness campaign aimed at families and students to reappraise the teaching profession (press, education debates, etc.)

■ Organising guidance sessions in primary and secondary education on the teaching profession, where students from the later years of the degree would take part

■ Enhancing ongoing training for secondary education teachers with a skill focus

■ Initiating the transformation process for the university entrance examinations in the Balearics (PBAU) so that they included a more skill-based approach

■ Undertaking longitudinal monitoring of students who joined the degree programmes through the admission test in order to analyse their progress throughout their training and career.

Discussion on the results, conclusions and outlook

Beyond developing admission tests to be used for teacher training at a specific university, the results from the case study in this article propose a template (see Figure IV) to broach the topic from a participatory perspective—a methodology not used to date for this type of work based on current literature on the topic.

In terms of the tests produced, it should be stated that the case study confirms there is no single focus to approach the candidate selection process for pre-service teacher training (ASPA, 2015). Based on the evidence, establishing a universal template to identify and assess the necessary student qualities for this training is an extremely complicated task (Darmody and Smyth, 2016). It should be approached by considering the specificities of each cultural and educational context. In any event, a series of general guidelines has emerged from the results obtained. The most prominent is perhaps the finding that based on an analysis of eligibility, the Grade Point Average (GPA) as sole criterion is an unsuitable selection system. The data underscore that suitable selection processes should be based on both cognitive and non-cognitive candidate abilities. In this regard, it should be highlighted that the consensus-based options for attributes to be considered in the design process (phase 2) are in line with current evidence (Casey and Childs, 2007; Klassen and Kim, 2017; Klassen, Durksen, Kim, Patterson, Rowett, Jane, Warwick and Wolpert, 2017; Klassen, Durksen, Hashmi, Kim, Longden, Metsäpelto, Poikkeus and Györi, 2018). In turn, the low correlation between the CCCT and CLOM test results underscores this appropriateness.11

Including the written profile in the tests (VP) is in line with current proposals, given that it is the second most common strategy in ATs (Casey and Childs, 2011). With regard to the group interview (GI), it should be stated that it is a widely used strategy in other countries as part of the ATs for admission onto teacher training courses (Casey and Childs, 2007; Klassen, Dolan and Afzal 2015). Although it is a complex process, it has shown itself as having a better predictive capacity than cognitive criteria alone (Shechtman, 1992); it may also be deemed a fair selection system for applicants (Bye and Sandal, 2016), as well as being particularly useful for detecting students with unsuitable profiles (suitability score) for education courses (Leshem, 2012) and for ranking students (setting value).12 Nonetheless, experiments have demonstrated the difficulty of this type of test and the importance of using scorecards and assessor training, since candidates may skew their responses to increase their chances of success (Johnson and Saboe, 2011).

Attracting the best candidates to the teaching profession is a major challenge and, therefore, an important topic for educational research (see, Morris, Gorard, Kokotsaki and Abdi, 2020). The results from Workshop-4 on ‘defining supplementary measures for admission tests’ are in the same vein as the results from work by Klassen, Gragner and Bardach (2020). They are also consistent with education policies from the UK Department for Education that proposes candidates undertake testing days at teaching centres within the framework of the Discover Teaching programme (Department for Education, 2019). These steps, as well as those agreed in Workshop-4, enable candidates to perform realistic job previews (RJP) based on person-environment and person-vocation theories (Klassen et al., 2020).

The research team is aware that the designed prototype and its validation are an initial contribution that will need to be supplemented by using more robust instruments and statistical tests. This process is planned to take place with the currently ongoing design for the longitudinal assessment.

In terms of the template, it should be highlighted that just as using participatory and group approaches in university teaching has significance for wider and better learning and student motivation (Calvo-Sastre, 2020; Martínez-Dominguez, Arandia-Loroño, Alonso-Olea and Castilla Prieto, 2011), their adoption in improvement processes for training or management programmes is fundamentally supported by involving stakeholders as key elements in organisational success (Wilcox, 1994; Involvement and Participation Association, 2014; Karasti, 2014; Abu-Amsha et al., 2019). In this context, selecting a participatory approach to producing ATs involves consulting, mediating, negotiating and seeking consensus in decision-making (Booker and Goldman, 2016; Engelbertink et al., 2020)—activities that were closely followed in the different phases of the process presented here. Nevertheless, one should ask whether these activities could have been investigated even further. Since a global evaluation of the process has not yet been performed (a limitation that the authors are aware of), a clear answer cannot be provided. Thus, issues such as to what extent the process widened involvement and commitment from lecturers on the GIP degree courses or how using ATs could have contributed to improving the training programme remain to be answered through evidence that will be looked at in future works.

FIGURE IV. Admission test design based on a participatory process13

Source: Compiled by authors

Finally, it should be noted that when writing this article, the longitudinal skill assessment study is currently being designed, based on contributions from current experiences (Bieri and Schuler, 2018) and expert validation. This study has enabled data to be obtained that aids in drawing inferences to improve the selection process.

Acknowledgements

This article is one of the outcomes of the project: ‘Producing Student Selection Tests for Admissions onto Pre-service Early Childhood and Primary Education Teacher Training’, funded as part of the Call for Research and Innovation Projects aimed at Pre-service Teacher Training (PRI_FIP) from the Department of Education and Universities of the Regional Government of the Balearic Islands. (2017-2021, Code D1703). The Education and Citizenship Research Group at the Institute for Research and Innovation in Education (IRIE) undertook this research.

References

Abu-Amsha, O., Gordon, R., Benton, R., Vasalou, M., and Webster, B. (2019). Reflection on a Participatory Design Process with Refugees. Journal on Education in Emergencies, 5(1), 156-176. https://doi.org/10.33682/tm40-9q50

Alexander, P.A. (2020). Methodological Guidance paper: The Art and Science of Quality Systematic Reviews. Review of Educational Research, 90(1), 6-23.

ASPA. (2015). Initial Teacher Education: Selection Criteria for Teacher Education Candidates. Retrieved from https://docplayer.net/18252962-Initial-teacher-education-selection-criteria-for-teacher-education-candidates.html

Bachman, L. (1990). Fundamental Considerations in Language Testing. Oxford: Oxford University Press.

Bayona, J.A., and Heredia, O. (2012). El concepto de equipo en la investigación sobre efectividad en equipos de trabajo. Estudios Gerenciales, 28(123), 121-132.

Bieri, C., and Schuler, P. (2018). Predictive validity of a competence-based admission test-mentors’ assessment of student teachers’ occupational aptitude. Assessment and Evaluation in Higher Education, 43(4), 640-651. https://doi.org/10.1080/02602938.2017.1390545

Booker, A. and Goldman, S. (2016). Participatory Design Research as a Practice for Systemic Repair: Doing Hand-in-Hand Math Research with Families. Cognition and Instruction, 34(3), 222-235. https://doi.org/10.1080/07370008.2016.1179535

Bustamante, A.M., Brendel, N., Degbelo, A. and Kray, C. (2018). Participatory Design and Participatory Research: An HCI Case Study with Young Forced Migrants. ACM Transactions on Computer-Human Interaction, 25(1), 24.

Bye, H.H., and Sandal, G.M. (2016). Applicant Personality and Procedural Justice Perceptions of Group Selection Interviews. Journal of Business and Psychology, 31, 569-582. https://doi.org/10.1007/s10869-015-9430-9

Callejo-Gallego, M.J., and Viedma-Rojas, A. (2016). La investigación-acción participativa. In M. García-Ferrando, F.R. Alvira-Martín, L.E. Alonso-Benito and M. Escobar-Mercado (Eds.), El análisis de la realidad social: métodos y técnicas de investigación (pp. 295-328). Madrid: Alianza.

Calvo-Sastre, A. (2020). Teaching social education with and through groups. Social Work with Groups, 43(3), 227-240. https://doi.org/10.1080/01609513.2018.1593918

Calvo-Sastre, A., Sureda-Negre, J., and Oliver-Trobat, M.F. (2018). Consultation of teacher trainers as a strategy for improving teacher training. The case of the University of the Balearic Islands. International Conference Teacher Education and Educational Research in the Mediterranean, Malta.

Casey, C., and Childs, R. (2007). Teacher Education Program Admission Criteria and What Beginning Teachers Need to know to be Successful Teachers. Canadian Journal of Educational Administration and Policy, 67, 24.

Casey, C., and Childs, R. (2011). Teacher Education Admission Criteria as Measure of Preparedness for Teaching. Canadian Journal of Education, 34(2), 3-20.

Childs, R.A. and Ferguson, A.K. (2015). Changes In, To, and Through the Initial Teacher Education Program Admission Process. In L. Thomas and M. Hirschkorn (Eds.), Change and Progress in Canadian Teacher Education. Canadian Association for Teacher Education, pp. 420-440.

Codina L. (2018). Revisiones bibliográficas sistematizadas: procedimientos generales y Framework para ciencias humanas y sociales. Barcelona: Universitat Pompeu Fabra, Departamento de Comunicación, Máster Universitario en Comunicación Social. Retrieved from http://hdl.handle.net/10230/34497

Conferencia de Decanos/as de Educación. (2017). Documento de la Conferencia Nacional de Decanos/as y Directores/as de Educación sobre la formación y el acceso a la profesión docente. Retrieved from http://www.conferenciadecanoseducacion.es/wp-content/uploads/2018/02/documento-conferencia-decanos-desarrollo.pdf

Corcoran, E. (2000). Goforth and publish-Software. Forbes, 166(9), 170-172.

Corcoran, R.P. and O’Flaherty, J. (2018). Factors that predict pre-service teachers’ teaching performance. Journal of Education for Teaching: International Research and Pedagogy, 44(2), 175-193. https://doi.org/10.1080/02607476.2018.1433463

Darmody, M. and Smyth, E. (2016). Entry to Programmes of Initial Teacher Education. T. E. a. S. R. Institute. Retrieved from https://www.teachingcouncil.ie/en/Publications/Teacher-Education/Documents/Entry-Requirements.pdf

Department for Education (2019). Teacher recruitment and retention strategy. Available from:

https://assets.publishing.service.gov.uk/government/uploads/system/uploads/attachment_data/file/786856/DFE_Teacher_Retention_Strategy_Report.pdf

Diamond, I.R., Grant, R.C., Feldman, B.M., Pencharz, P.B., Ling, S.C., Moore, A.M. and Wales, P.W. (2014). Defining consensus: A systematic review recommends methodologic criteria for reporting of Delphi studies. Journal of Clinical Epidemiology, 67(4), 401–409. https://doi.org/10.1016/j.jclinepi.2013.12.002

Díez-Gutierrez, E.J. (2020). Otra investigación educativa posible: Investigación-acción participativa dialógica e inclusiva. Revista de Educación de la Universidad de Málaga, 1(1), 115-128. https://doi.org/10.24310/mgnmar.v1i1.7154

Ebel, R. L. and Frisbie, D. A. (1991). Essentials of Educational Measurement. New Jersey: Prentice Hall

Egido-Gálvez, I. (2020). El acceso a la formación docente inicial en Europa: políticas e investigación. Revista Española de Educación Comparada (35), 197-211.

Engelbertink, M.M.J., Kelders, S.M., Woudt-Mittendorff, K.M. and Westerhof, G.J. (2020). Participatory design of persuasive technology in a blended learning course: A qualitative study. Education and Information Technologies, 25, 4115-4138. https://doi.org/10.1007/s10639-020-10147-x

Eurydice. (2018). Teaching careers in Europe: access, progression and support. Publications Office of the European Union. https://doi.org/10.2797/917260

Foro de Sevilla. (2014). El profesorado y su formación como actores necesarios y comprometidos con la educación pública. Retrieved from https://www.aulalibre.es/IMG/pdf_documento_sobre_el_profesoradio_Foro_de_Sevilla_200314_1_.pdf

Generalitat de Catalunya (2021). Prova d’aptitud personal (PAP) per a l’accés als graus en educació infantil i primària. Retrieved from http://universitats.gencat.cat/ca/altres_pagines/educacio_inf_i_prim/

González, J. and Jornet, J.M. (2009). Evaluación criterial: determinación de estándares de interpretación (EE) para pruebas de rendimiento educativo. ESE. Estudios sobre educación, 16, 103-123.

Grupo Palma. (2018). La Universidad y la formación inicial del profesorado. Retrieved from http://www.grupopalmaedu.org/documento_grupo_palma.pdf

Holden, M. and Kitchen, J. (2017). Choosing How We Choose: A Review of Teacher Education Admissions Literature. Teaching and Learning, 11(1), 16-31.

Involvement and Participation Association. (2014). Employee Engagement in Further Education. Association of Colleges Limited. Retrieved from https://www.ipa-involve.com/Handlers/Download.ashx?IDMF=3e76d3fa-dc66-4537-b719-2ae1dd734e6a

Janssen, F.J.J.M., Könings, K.D. and van Merriënboer, J.J.G. (2017). Participatory educational design: How to improve mutual learning and the quality and usability of the design? European Journal of Education Research, Development and Policy, 52(3), 268-279. https://doi.org/10.1111/ejed.12229

Johnson, R.E. and Saboe, K.N. (2011). Measuring Implicit Traits in Organizational Research: Development of an Indirect Measure of Employee Implicit Self-Concept. Organizational Research Methods, 14(3), 530-547. https://doi.org/10.1177/1094428110363617

Karasti, H. (2014). Infrastructuring in participatory design PDC’14. Proceedings on the 13th Participatory Design Conference: Research Papers, Windhoek. https://dl.acm.org/doi/10.1145/2661435.2661450

Katz, S., and Frish, Y. (2016). A new admission system model for teacher colleges. Quality Assurance in Education, 24(4), 463-489. https://doi.org/10.1108/QAE-01-2014-0003

Klassen, R., Dolan, R. and Afzal, F. (2015). Selection for Teacher Education in the UK and the Republic of Ireland. In Proposal for Innovation ECER 2015, Budapest. Retrieved from https://eera-ecer.de/ecer-programmes/conference/20/contribution/35327/

Klassen, R., Gragner, H. and Bardach, L. (2020). Attracting Prospective STEM Teachers Using Realistic Job Previews: A Mixed Methods Study. Teacher Selection Project Working Paper. Retrieved from https://psyarxiv.com/r6efk/

Klassen, R.M., Durksen, T.L., Hashmi, W.A., Kim, L.E., Longden, K., Metsäpelto, R.L., Poikkeus, A.M. and Györi, J.G. (2018). National context and teacher characteristics: Exploring the critical non-cognitive attributes of novice teachers in four countries. Teaching and Teacher Education, 72, 64-74. https://doi.org/10.1016/j.tate.2018.03.001

Klassen, R.M., Durksen, T.L., Kim, L.E., Patterson, F., Rowett, E., Jane, W., Warwick, P. and Wolpert, M.A. (2017). Developing a Proof-of-Concept Selection Test for Entry into Primary Teacher Education Programs. International Journal of Assessment Tools in Education, 4(2), 96-114. https://doi.org/10.21449/ijate.275772

Klassen, R.M., and Kim, L.E. (2017). Assessing Critical Attributes of Prospective Teachers: Implications for Selection into Initial Teacher Education Program. In D. W. Putwain and K. Smart (Eds.), British Journal of Educational Psychology Monograph Series II: Psychological Aspects of Education, 5-22. Wiley.

Könings, K.D., Bovill, C. and Woolner, P. (2017). Towards an interdisciplinary model of practice for participatory building design in education. European Journal of Education Research, Development and Policy, 52(3), 306-317.

Lee, B., Leong, B., and Chan, G. (2015). A reflection on designing participatory design workshop - case study of elderly product development workshop with multidisciplinary collaboration. DesignEd Asia Conference 2015, Hong Kong. Retrieved from http://ira.lib.polyu.edu.hk/handle/10397/81448

Leshem, S. (2012). The Group Interview Experience as a Tool for Admission to Teacher Education. Education Research International, 8. https://doi.org/10.1155/2012/876764

López, P (2009). Construcción y validación de una prueba para medir conocimientos matemáticos. Horizontes Pedagógicos, 11(1), 29-37.

Manso, J. (2019). La formación inicial del profesorado en España. Madrid: Ministerio de Educación y Formación Profesional.

Marina, J.A., Pellicer, C. and Manso, J. (2015). Libro blanco de la profesión docente y su entorno escolar. Retrieved from http://educalab.es/documents/10180/38496/LIBRO-BLANCO_ProfesionDocente_JAM_v11.pdf/e4e1d927-6a61-4897-bca0-ada011dca331

Martínez-Domínguez, I., Arandia-Loroño, M., Alonso-Olea, M.J. and Castilla Prieto, L.d. (2011). Trabajar con metodologías participativas en la formación universitaria, todo un desafío. Revista Investigación en la Escuela (75), 101-113.

Martínez-Martin, M., Prats-Gil, E., and Marín-Blanco, A. (2015). La millora de la formació inicial de mestres: el Programa de Millora i Innovació en la Formació de Mestres, MIF. Revista Catalana de Pedagogia, 9, 24-42.

Menter, I., Hulme, M., Elliot, D., and Lewin, J. (2010). Literature Review on Teacher Education in the 21st Century. Other. The Scottish Government, Edinburgh, UK. Retrieved from http://www.scotland.gov.uk/Publications/2010/09/24144019/0

Mikitovics, A. and Crehan, K.D. (2002). Pre-Professional Skills Test Scores as College of Education Admission Criteria. The Journal of Educational Research, 95(4), 215-223. https://doi.org/10.1080/00220670209596594

Ministerio de Educación y Formación Profesional. (2018). Informe de síntesis de las aportaciones al foro Educar para el Siglo XXI. Desafíos y propuestas para la profesión docente. Retrieved from http://www.ub.edu/obipd/wp-content/uploads/2019/04/INFORME-DE-S%C3%8DNTESIS-DE-LAS-APORTACIONES-AL-FORO-Educar-para-el-siglo-XXI-Desaf%C3%ADos-y-propuestas-para-la-profesi%C3%B3n-docente.pdf

Nickelsen, N.C.M. and Bal, R. (2021). Workshop as Tool for Developing Collaborative Practice across Professional Social Worlds in Telemonitoring. International Journal of Environmental Research and Public Health, 18(181), 15. https://doi.org/10.3390/ijerph18010181

Oliver-Trobat, M.F., Sureda-Negre, J. and Calvo-Sastre, A. (2018). Improvement Programme for Initial Teacher Training in Bachelor’s Degrees in Early Childhood, Primary, and Secondary Education at the University of the Balearic Islands. International Conference Teacher Education and Educational Research in the Mediterranean, Malta.

Parker, S. (2018). Literature Review on Teacher Education Entry Requirements. School of Education, University of Glasgow. Retrieved from http://www.gtcs.org.uk/web/FILES/research/GTCS-Literature-Review-on-Teacher-Education-Entry-Requirements.pdf

Pedrosa, I., Suárez-Álvarez, J., and García-Cueto, E. (2013). Evidencias sobre la Validez de Contenido: Avances Teóricos y Métodos para su Estimación. Acción Psicológica, 10(2), 3-18. http://dx.doi.org/10.5944/ap.10.2.11820Pérez-Granados, L. (2015). Análisis de las pruebas de acceso a la formación de docentes en España y Finlandia: conocimientos o competencias. Revista Complutense de Educación, 26(3).

Pérez-Granados, L. (2018). Using specific test to access for initial teacher training. Journal for Educators, Teachers and Trainers, 9(1), 12.

PM_FIP. (2017). Documento estratégico y de propuestas de intervención del Programa de Mejora de la Formación Inicial de los Estudios de Maestro y del Profesorado de Secundaria de la UIB (PM_FIP). Palma: Universitat de les Illes Balears. Retrieved from http://pmfip.uibvirtual.es/wp-content/uploads/sites/41/2019/10/def_pm_fip.pdf

REDE (2018). La profesión docente a debate. Retrieved from https://www.dialogorede.es/wp-content/uploads/2019/03/MdD_documentobase_debatepublico.pdf

See, B. H., Morris, R., Gorard, S., Kokotsaki, D. and Abdi, S. (2020). Teacher Recruitment and Retention: A Critical Review of International Evidence of Most Promising Interventions. Education Sciences, 10(262), 45. https://doi.org/10.3390/edusci10100262

Shechtman, Z. (1992). A group assessment procedure as a predictor of on-the-job performance of teachers. Journal of Applied Psychology, 77(3), 383-387. https://doi.org/10.1037/0021-9010.77.3.383

Simonsen, J. and Robertson, T. (2013). Routledge International Handbook of Participatory Design. Routledge.

Sureda-Negre, J. and Oliver-Trobat, M.F. (Dir.) (2015). La formació inicial del professorat d’Educació Infantil i Primària a les Illes Balears. Estat de la qüestió i propostes per a la millora. Palma: Universitat de les Illes Balears. Retrieved from https://diari.uib.cat/digitalAssets/329/329375_1-dossier.pdf

Sureda-Negre, J., Oliver-Trobat, M.F. and Comas-Forgas, R. (2016). Medidas para la mejora de la formación inicial de los maestros según el profesorado de un departamento de Pedagogía. Bordón, 68(2), 155-168. https://doi.org/10.13042/Bordon.2016.68210

Sureda-Negre, J., Oliver-Trobat, M. and Comas-Forgas, R. (2020). Improving the Training of Primary School Teachers: Agreements and Disagreements Between Active Teachers and University Professors. Educational Practice and Theory, 42(2), 77-104. https://doi.org/10@font-face

The Australian Secondary Principals’ Association. (2015). Initial Teacher Education: Selection Criteria for Teacher Education Candidates. The Australian Secondary Principals’ Association on behalf of the peak national principal associations.

UIB (2017a). Documento estratégico y de propuestas de intervención del Programa de Mejora de la Formación Inicial de los Estudios de Maestro y del Profesorado de Secundaria de la UIB (PM_FIP). Palma: Universitat de les Illes Balears. Retrieved from http://pmfip.uibvirtual.es/wp-content/uploads/sites/41/2019/10/def_pm_fip.pdf

UIB (2017b). Resolución del Rectorado de día 12 de septiembre de 2017 por la que se hace pública la composición de la Comisión de la UIB para la mejora de la formación inicial de los estudios de maestro y de profesorado de secundaria. Retrieved from https://seu.uib.cat/fou/acord/12458/

Universidad de Deusto (2021). Proceso de ingreso grado. Retrieved from https://www.deusto.es/cs/Satellite/deusto/es/nuevos-estudiantes-de-grado/proceso-de-ingreso

Wilcox, D. (1994). Guide to Effective Participation. Delta Press.

Wright, V. (2015). Is ATAR Useful for Predicting the Success of Australian Students in Initial Teacher Education? Australian Journal of Teacher Education, 40(9), 16. https://doi.org/10.14221/ajte.2015v40n9.1

Contact address: Miquel F. Oliver-Trobat. Universidad de las Islas Baleares, Facultad de Educación. Departamento de Pedagogía Aplicada y Psicología de la Educación. E-mail: m.oliver@uib.es

---