Spanish validation of the Vocational Engagement Instrument in Vocational Training

Validación española del Instrumento de Vinculación Estudiantil en Formación Profesional

https://doi.org/10.4438/1988-592X-RE-2024-406-644

Inmaculada Sureda-García

https://orcid.org/0000-0002-5024-1799

Universitat de les Illes Balears, Institute of Research and Innovation in Education o Institut de Recerca I Innovació

Rafael Jiménez

https://orcid.org/0000-0002-1181-3221

Universitat de les Illes Balears, Instituto de Investigación Sanitaria Illes Balears (IdISBa)

Albert Sesé

https://orcid.org/0000-0003-3771-1749

Universitat de les Illes Balears, Instituto de Investigación Sanitaria Illes Balears (IdISBa)

Francesca Salvà-Mut

https://orcid.org/0000-0001-8517-0014

Universitat de les Illes Balears, Institute of Research and Innovation in Education o Institut de Recerca I Innovació

Abstract

The concept of student engagement plays a crucial role in understanding the risk factors leading to dropout, particularly among vulnerable students. The literature has divided school engagement into three subtypes: academic, cognitive, and emotional (Fredericks et al., 2004). This model proves highly relevant in contributing to empirical studies that assess academic engagement among vocational education students, as this educational level faces a significant dropout rate. In this context, this article presents evidence of the validity of scores from the Vocational Engagement Instrument (VEI), a new psychometric tool designed to measure engagement in the context of Vocational Education and Training (VET) students. The study was conducted using stratified cluster sampling of 4,522 students (64.6% males) from three Spanish regions: Mallorca (n = 1,511, 33.4%), Barcelona (n = 1,038, 23%), and Valencia (n = 1,973, 43.6%). Participants were enrolled in the first year of basic VET (n = 1,370, 30.3%) or intermediate VET (n = 3,152, 69.7%). A model comparison strategy included different latent structures (unifactorial, pentafactorial, bifactor) through Confirmatory Factor Analysis (CFA) and Exploratory Structural Equation Models (ESEM), using the Weighted Least Squares Mean and Variance adjusted (WLSMV) estimation method. The bifactor model achieved the best overall fit (chi-square/df=4.64, RMSEA=.031, CFI=.970, TLI=.970, SRMR=.039). Furthermore, measurement invariance was verified considering the VET level (basic/intermediate) (RMSEA=.028 and CFI=.910) and self-reported sex (male/female) (RMSEA=.029, CFI=.913). These promising results regarding the latent structure of the VEI may lead to the identification of differential patterns of student engagement. Additionally, they could facilitate the development and implementation of intervention programs focused on minimizing school dropout and enhancing academic performance.

Keywords: student engagement, vocational education training, test validation, psychometric networks, latent models.

Resumen

El concepto de vinculación escolar desempeña un papel crucial en la comprensión de los factores de riesgo que conducen al abandono escolar, especialmente entre un alumnado vulnerable. La literatura ha desglosado la vinculación escolar en tres subtipos: aspectos académicos, cognitivos y emocionales (Fredericks et al., 2004). Este modelo se revela de suma importancia en la contribución a estudios empíricos que evalúan la vinculación académica entre estudiantes de formación profesional, dado que este nivel educativo enfrenta un índice significativo de abandono. En este contexto, este artículo presenta pruebas de validez de las puntuaciones del Instrumento de Vinculación Estudiantil (IVE), una nueva herramienta psicométrica diseñada para medir la Vinculación del Estudiante en el ámbito de la Formación Profesional (FP). El estudio se llevó a cabo utilizando una muestra estratificada por conglomerados compuesta por 4522 estudiantes (64.6% hombres) pertenecientes a tres regiones españolas: Mallorca (n = 1511, 33.4%), Barcelona (n = 1038, 23%), y Valencia (n = 1973, 43.6%). Los participantes estaban matriculados en el primer año de FP básica (n = 1370, 30.3%) o FP de grado medio (n = 3152, 69.7%). Se aplicó una estrategia de comparación de modelos que incorporó diversas estructuras latentes, como unifactorial, pentafactorial y bifactorial, mediante Análisis Factorial Confirmatorio (CFA) y modelos de Ecuaciones Estructurales Exploratorias (ESEM), utilizando el método de estimación WLSMV. El modelo bifactorial demostró el mejor ajuste (ji-cuadrado/gl=4.64, RMSEA=.031, CFI=.970, TLI=.970, SRMR=.039). Además, se verificó la invariancia de medias considerando el nivel de FP (básico/grado medio) (RMSEA=.028 y CFI=.910) y el sexo autoinformado (masculino/femenino) (RMSEA=.029, CFI=.913). Estos resultados prometedores sobre la estructura latente del IVE pueden conducir a la identificación de patrones diferenciales de vinculación estudiantil. Asimismo, podrían facilitar el desarrollo e implementación de programas de intervención centrados en la minimización del abandono escolar y en la mejora del rendimiento académico.

Palabras clave: vinculación estudiantil, formación profesional, validación de test, redes psicométricas, modelos latentes.

Introduction

The literature on early leaving from education has focused on the study of risk factors, interactions between these factors, and the processes that lead to dropout (Soler et al., 2021). In line with this perspective, Reschly & Christenson (2012), through their intervention program “Check and Connect”, (designed to promote student engagement through relationship building, problem solving, and persistence for marginalized students), establish a model that aims to relate context, student engagement, and students’ academic results. This model is of special relevance for the aims of the present study with VET students and represents a relevant contribution due to a scarcity of information and empirical studies on this issue and stage of education in Spain (Echeverría & Martínez, 2017). In the Spanish education system, vocational secondary education is arranged in two levels: basic VET (henceforth BVET) and intermediate VET (hereafter IVET). BVET is aimed at students who have not completed compulsory secondary education. Age of entry is 15 years, and it has a duration of two academic years. The literature about the specificities of these students explain the social and education vulnerability as personal characteristic of this group, showing the high levels of dropout in this training between students, priority for basic VET students (Olmos et al., 2020). In IVET, students have the qualification of GESO [Compulsory Secondary Education Certificate] or equivalent training; nonetheless, entry into these studies is undervalued when compared to generalized upper secondary education studies (Spanish baccalaureate). These students come from diverse forms of admission, including non-linear educational pathways (those who have experienced interruptions in their secondary or postsecondary academic studies, or the labor market) (Cerdà et al., 2022).

Despite the lack of specific studies, within this theoretical framework, the concept of student engagement plays a fundamental role in understanding the processes of dropout in VET and the prevention of said dropout in other academic studies (OECD, 2023). The construct student engagement refers to students being involved, committed, or attached to the academic and social activities in their school (Wang et al., 2019).The concept of engagement has emerged as a way to understand, and improve, outcomes for students at risk of school failure.

Factors of the Vocational Engagement Instrument (VEI)

The literature concluded that student engagement was composed of three subtypes: behavioral-academic (e.g, positive conduct, effort, participation), cognitive (e.g. self-regulation, learning goals, investment in learning), and emotional or affective (e.g. belonging, and relationships with teachers, peers and family) (Fredericks et al., 2004). Based on the theoretical work of Reschly & Christenson (2012), Fredricks et al. (2004) and social cognitive career theory that reinforces the study of cognitive variables as self-efficacy and outcome expectations (Lent et al., 1994), we have proposed five factors of engagement in the VEI. The factors are the following: school duties and discipline, goal and expectations, social integration (classmates), teacher’s support, and family’s support.

First, school duties and discipline (SD) explain the behavioral engagement across all the literature (Estell & Perdue, 2013; Fredericks et al., 2004). School duties and discipline is defined as some students focus on doing homework academic for assessment behaviors such as persistence, effort, attention of the student. Also, this factor contemplates positive conduct, such as following the rules, adhering to classroom norms, and the absence of disruptive behaviors such as skipping school or getting in trouble.

A second factor identifies goals and expectations (GE) of the students implicating cognitive engagement. The concept of outcome expectations involves the anticipated consequences of a course of action. Vocational outcome expectation could be considered as the imagined consequences of performing academic and career behaviors that would be useful to subsequent career options and decisions (Betz & Voyten, 1997).

The third factor explains social integration (classmates, SI). The type of relationship and social support received by the peer group conditions students’ short- and long-term student engagement. A good relationship with peers provides emotional support, improves self-esteem and identity development, and has important consequences for subjects’ social, emotional, and cognitive well-being (Sureda et al., 2021). The fourth factor determines teacher’s support (TS) as a important key in emotional engagement because teachers are considered to be a proximal, and therefore crucial, influence on student engagement (Niittylahti et al., 2019; Quin et al., 2018). Finally, the fifth factor contemplates the family’s support (FS). Students who have the support and perceived commitment of their families show more interest and are more involved in academic activities, hence preventing school disengagement or risk for dropout (Sureda et al., 2021; Wang et al., 2019).

Relevant Instruments for the VEI

The new instrument addressed the multi-components conception of engagement, aimed at a practical intervention for improving student engagement, and targeted a profile of students in vocational training. In this case, the three most relevant instruments by the aforementioned criteria and the design and construction of our evaluation measure are the Student Engagement Instrument (SEI) (Appleton et al., 2006), the Trousse d’évaluation des décroheurs potentiels (TEDP) scale [Potential abandonment evaluation kit] (Janosz et al., 2007), and the Vocational Outcome Expectations Scale (VOE; McWhirter et al., 2000).

The Student Engagement Instrument (SEI), used in the abovementioned project “Check & Connect” (Appleton & Christenson, 2004; Appleton et al., 2006), operationalizes emotional and cognitive engagement, used 35 items. This instrument addresses different contexts of interaction (school, family, and community), and students’ results (bonding, success, and dropout in order to evaluate behavior engagement) in a joint, related way. This instrument is based on variables that are highly modifiable from school intervention, enabling educational actions aimed at preventing and reducing school dropout (Reschly & Christenson, 2012).

This instrument used an initial sample of 1,931 ninth-grade students from an ethnically diverse, majority low-income, urban school district. Further studies on SEI used a sample of students from grades 6 to 12. Regarding psychometric properties, criterion validity evidence was obtained with significant correlations with academic variables (reading and math achievement, number of failed subjects, and grade point average). The tool also obtained adequate evidence of internal consistency of its subscales ranging from .72 to .92. This adequate psychometric behavior was also complemented with appropriate test-retest reliability, ranging from .60-.62. (Christenson et al., 2012).

Among the limitations of the SEI, it is pointed out that in some items it is a little complicated to establish whether its contents are mutually exclusive and, hence, assert their assignment to a single subtype of engagement (Mameli & Passini, 2017). However, the SEI must be completed with academic data from the educational administration or other instruments, which enable us to find out about behavior engagement for the purpose of evaluating multidimensional engagement. Further, the SEI was designed to cater for a general secondary education population and not, specifically, a vocational secondary education population.

Another relevant tool is the Trousse d’évaluation des décroheurs potentiels (TEDP) [Potential abandonment evaluation kit] (Janosz et al., 2007), aimed at educational practice, and designed to predict school dropout in different academic studies.The TEDP is made up of 54 items, used originally in Quebec, and makes it possible to identify typologies of students, aged between 12 and 18 years, who run the risk of dropping out of school before graduating in secondary education. Responses are ordered according to 4 Likert type options: 1, untrue; 2, somewhat untrue; 3, somewhat true; 4, true; although the content of some questions might modify the format of the response and offer other options, or an open answer.

The validation study of the TEDP instrument provides adequate evidence of decision validity, with ROC curve analysis, and appropriate area under the curve values of between .70 and .85. As for stability over time of the measure, adequate values of around .70 are obtained. The validation was carried out in a sample of 35,000 secondary students, aged between 12 and 18 years, in 79 schools evaluated as part of an intervention strategy (Janosz et al., 2007). The TEDP indicates that although dropout risk factors are numerous and have multiple origins (individual, family, socio-economic factors, etc.), it is not necessary to measure them all in order to obtain a reliable evaluation (Janosz et al., 2007). On the other hand, it makes it possible to postulate that not all early school leavers have the same vulnerabilities or strengths, since their school or social experience may be different. Among its limitations, it measures school engagement according to the instrument l’Adaptation Sociale et Personnelle pour les Adolescents Québécois (MASPAQ) by LeBlanc (1996), focusing more in behavior factors of engagement. Among other limitations of this instrument, the dropout risk index varies depending on the studies being taken and the characteristics of students (age, sex, training); hence there is, a priori, no dropout risk index.

The third instrument of interest for the study of a profile of vocational education and training students is the Vocational Outcome Expectations Scale (VOE) (McWhirter et al., 2000). Specifically, vocational outcome expectations could be considered as the imagined consequences of performing academic and career behaviors that would be useful to subsequent career options and decisions in their secondary or postsecondary academic studies. (Betz & Voyten, 1997; Işik (2013)). The VOE is a 12 item-scale that measures respondents’ level of positive expectations regarding the outcomes of their career choice. Ratings are made on 4 point scale with anchors ranging from 1 “strongly disagree” to 4 “strongly agree”. The range of possible scores varies from a minimum score of 12 to a maximum score of 48, with higher scores reflecting more positive outcome expectations. The concurrent validity of the scale was supported by a positive correlation (r = .54) with another outcome expectation scale (Fouad & Smith, 1996). McWhirter et al. (2000) showed that test-retest reliability of the scale was .59, and that Cronbach’s Alpha internal consistency coefficient was .83. Işik (2013) determined that test-retest reliability of the Turkish version was .79 and Cronbach’s Alpha internal consistency coefficient was .87, indicating adequate reliability. The limitations of this scale, in terms of our aims, are that it does not use a multicomponents measure of student engagement. Based on the strengths of the instruments reviewed, the aim was to validate the scores of a new instrument that will enable student engagement to be evaluated in a VET population.

Method

Participants

The sample consisted of 4,522 Spanish students (64.6% males), from Mallorca (n = 1,511, 33.4%), Barcelona (n = 1,038, 23%), and Valencia (n = 1,973, 43.6%), all enrolled in the first year of BVET (n = 1,370, 30.3%) or IVET (n = 3,152, 69.7%) (Table 1). Stratified cluster sampling was used. The strata were formed from 17 professional families according to the type of center and location. This initial population was restricted to students aged 14 to 19 years in order to ensure the evolutionary homogeneity of the sample. In the case of BVET students, they represented a mean age of 16.06 years (SD = 0.77) and in the case of IVET, students represented a mean age of 17.34 years (SD = 0.97). The sample as a whole represented a mean age of 16.88 years (SD = 1.09).

Instrument development

The new instrument consideres the importance of evaluating factors about the emotional, cognitive, and behavioral dimension of engagement (as social integration with classmates, teacher’s support, family’s support, professional goal, and shool duties and discipline) by the SEI (Appleton & Christenson, 2004) and the TEDP (Janosz et al., 2007), incorporating the evaluation of expectations created towards studies as a significant variable in the vocational studies from the VOE instrument (McWhirter et al., 2000) (Table II).

The VEI had initially 52 items with four Likert response options (as strongly disagree, disagree, agree, and strongly agree) that operationalize a five dimensional latent structure. The item contents and their descriptive statistics are presented in a table in the Annex.

Content validity

The instrument was assessed by a panel of 16 experts (9 women) with experience in vocational, secondary, and university studies for content validation (six teachers of the vocational education training, five teachers of secondary education and five teachers at Spanish Universities). By Poli and Hungler (2000), two levels of analysis of the items were established based on their level of relevance and pertinence using values from 1 (not pertinent or not relevant) to 4 (very relevant or pertinent). We asked the experts possible suggestions about questions not pertinent o not relevant in this context or population. The experts could do a proposition of change in the redaction or the elimination of the items. Finally, doing presential group work with the experts, we concreted the suggestions, valuations and agreeing. The statistical means of agreement were 3.65 for relevance and 3.68 for pertinence. The lower mean value was 3 with Aiken’s V=.67 and 90% CI Aiken’s V=[.55;.77], and 4 the highest one with Aiken’s V=1 and 90% CI Aiken’s V=[.95;1]. Both the relevance and pertinence of the items with CI Aiken’s V and the changes in the redaction or elimination of the items are provided in the following file retrieved from the Zenodo online academic repository: https://doi.org/10.5281/zenodo.10257332.

Face validity

The questionnaire was administered to a pilot sample of 172 VET students (24.4% in BVET and 75.6% in IVET, 65.1% males, M = 18.4 years). The respondents were asked to fill out the questionnaire either written. Afterwards, the respondents were encouraged to give comments on the formulation of the questions, and if the questionnaire corresponded to their understanding of student engagement. Based on ordinal Cronbach’s alpha (Elosua & Zumbo, 2008), some items from the original were reformulated based on the commentaries provided by the participants in the pilot study, improving the translation of the items with no loss of content validity. Overall, the items were understood as intended and were possible to answer with the response alternatives offered. Ordinal Alpha coefficient values were calculated in the pilot study, and all factors obtained acceptable internal consistency with values ranging from .76 to .81.

Procedure

The study was approved by the Research Ethics Committee of the University of the Balearic Islands (registration nº 5488), which made it possible to create an encrypted file of personal data managed by the university and research group. After obtaining the collaboration of the schools, informational meetings were conducted with the principals and, then, informed consent was obtained from the families of younger students. Students were informed that their participation would be anonymous, confidential, and voluntary. The average time required to complete the questionnaire was 30 minutes. The survey was conducted over a period of two to three months after the beginning of the school year, before the first academic results, preventing academic assessment from interfering with the answers, and dropout of some students later. The instrument was administered inside a regular class without introducing distracting factors.

Data analysis

Univariate descriptive data analysis was implemented to explore the statistical behavior of the 52 items, to evaluate data quality, and to verify fulfilment of statistical assumptions (univariate and multivariate normality). No imputation methods for missing values were implemented, using a pairwise deletion. The occurrence of missing values in items’ responses was virtually negligible, with their percentage presence fluctuating between 0% and 1.2%. Preliminarily, the 52 items of the questionnaire’s initial version were subjected to a latent structure analysis using Psychometric Networks (PN). An EGA (Exploratory Graph Analysis) analysis (Golino & Epskamp, 2017) was used using the EGAnet package (Hudson, 2020) of the R program. Specifically, the “glasso” and “TMFG” (Triangulated Maximally Filtered Graph) methods were used to estimate the latent network (Massara et al., 2017). These methods allow detecting the existence of latent factors from groupings of items in an interrelated network structure. The “walktrap” and “louvain” estimation algorithms were implemented. A parametric bootstrap simulation (1000 samples) was carried out to evaluate the stability of the obtained models. Using this analytical strategy, a double objective was intended: to check if the NP captures the hypothesized theoretical five-factor structure, and to detect and remove items with poor psychometric functioning in the latent structure.

Next, this refined version implementing PN was used to carry out a model comparison strategy including 5 hypothesized competing models: one-factor, five-factor, and bi-factor models under CFA estimation; and for discarding the existence of high items cross-loadings, Exploratory Structural Equation Modeling (ESEM) was applied both to the five-factor and the bi-factor models. The conceptual diagram of all models is depicted in Figure I.

All proposed models were estimated using the Weighted Least Squares with Mean and Variance adjustment (WLSMV) with robust standard errors since the Likert scale of the items had less than five response categories. The WLSMV method employs the polychoric correlation matrix when conducting confirmatory factor analysis with categorical variables. This approach is particularly suitable for situations where working with data that deviates from the normality assumption involves variables of an ordinal or nominal nature (Brauer et al., 2023; Martínez-Abad & Rodríguez-Conde, 2017; Muthén et al., 1997). Once the best fitted model was obtained, further analyses of multi-group invariance were implemented across two variables: VET level of studies (Basic/Intermediate) and self-reported sex (Male/Female). A measurement invariance procedure (Brown, 2014) including five invariance configurations ranging from the lowest to highest level of equality constraints on the model parameters to be estimated across different subsamples was used. The first step was configural invariance, which examines whether the data from the two subsamples are represented by the same factor structure. The second step, weak/metric invariance, where factor loadings are constrained to be equivalent across subgroups, was estimated. The third step, strong/scalar invariance, in which all intercepts are constrained to be equal in the two subsamples, was completed. The fourth step checked the strict invariance, where measurement error variance is constrained to be equal across compared groups. And finally, the fifth step, where the equality of latent factor means is constrained, was also estimated.

The following indices were used to assess models overall goodness-of-fit: (a) chi-square statistic (χ²), (b) the chi-square to degrees of freedom ratio (χ²/df), (c) RMSEA and its 90% confidence interval (CI), with a p-value for RMSEA < .05, (d) CFI, (e) TLI, and (f) SRMR. The cut-off values proposed by Byrne (2016), Hair et al. (2008), Hu & Bentler (1999), Kline (2005), and Steiger (2007) were used to determine an adequate fit: non-significant chi-square values; χ²/df ≤ 5; RMSEA ≤ .05; CFI and TLI ≥ .95; and SRMR ≤ .08.

As for the adequate analysis of bifactor models, focused on the strength of the general factor and potential unidimensionality, different specific indices were estimated. The Hierarchic Omega (ω_H) (squared) represents total variance explained can be estimated for the general and specific factors; when the value for the general factor is ≥ .70, it can be partially indicative of unidimensionality, and when the value for the specific factors is ≥ .30, these can be considered significant. The H index allows evaluating how well a latent variable is represented by a set of items, with a minimum value of adequacy equal to .70, and it can be calculated for both the general factor and the specific ones.

The PUC estimates the proportion of uncontaminated correlations by multidimensionality, the ECV-SS is the Explained Common Variance of a specific factor with respect to itself, and the ECV-SG is the ECV of a specific factor with respect to the general factor. It is recommended to interpret conjointly the PUC and ECV values taking into account that the unidimensionality of a model can be found when obtaining ω_H ≥ .70, PUC > .70, and ECV > .60 (Constantinou & Fonagy, 2019; Reise et al., 2013). Regarding the items’ behavior, the ECV-i index indicates what percentage of the true variance of each item is explained by the general factor, expecting values ≥ .80 to conclude on a significant influence of the general factor (Stucky & Edelen, 2015); lower ECV-i values could indicate higher loadings onto specific factors. An adequate analysis of a bi-factor model requires that all these indices be interpreted together to verify if the general factor is enough to explain the variance of the items, and therefore, the specific factors are irrelevant, or on the contrary, specific factors can be considered significant for the construct measurement.

As regards invariance analysis, a ∆χ² and ∆CFI tests were performed to compare the fit from less restricted to totally restricted constrains (equality of parameters across groups). Given that the chi-square test is really restrictive in order not to be significant (biased by the sample size) as indicative of a good fit, a ∆CFI between models less than or equal to .01 was also considered as statistically non-significant (Cheung & Rensvold, 2002). The CFA models were estimated using lavaan (Rosseel, 2012) and semTools (Jorgensen et al., 2021) packages from R program (R Core Team, 2021), and ESEM models with the MPlus 7.4 software program (Muthén & Muthén, 2017).

Results

Psychometric Networks models applied to the 52 items of the VEI questionnaire showed a clear five-factor latent structure using the TMFG method (100% of replications) and the glasso method found a six-factor model in 81.4% of replications and a five-factor model in the remaining 18.6%. In both solutions, the five hypothesized factors for operationalizing the construct can be clearly identified, but there are discrepancies regarding the detection of a sixth factor. Considering its composition (5 items), the sixth factor is a split from the Family’s support factor. The content of these items, for example, “My parents expect me to continue my studies for as long as possible”, or “I’d upset my parents if I left school/center”, tries to measure the reaction of parents to certain behaviors or their expectations, but perceived by the children. This indirect way of measuring Family’s support does not seem to obtain good psychometric behavior.

Also, two items from the Goals and expectations factor: “When I do a school activity, I try to grasp what I’m doing” and “I compare myself with my classmates to see if I’m learning at the right pace”, as well as an item of the factor School duties and discipline: “I like going to school/center”, presented less stability in their factor loadings within the latent structure through bootstrap resampling, especially with the glasso method. For this reason, these 8 items were removed from the initial version of the questionnaire. The refined version of the questionnaire with 44 items is shown in the Annex of this paper.

The following step was to carry out the SEM models comparison strategy to assess the best fitting latent structure of the questionnaire. The five hypothesised latent models (Figure 1), CFA and ESEM, were estimated using WLSMV method. Table III shows all the overall goodness of fit index for all estimated models. According to the results, the worst fit corresponded to the one-factor CFA model (χ²=17637.83, df=902, p<.001, χ²/df=19.55, RMSEA=.071, 90% CI RMSEA=[.070;.071], CFI=.81, TLI=.80, and SRMR=.095), in which none of the values of the fit indices meet the cutoff values for an adequate fit. And the best model was the Bi-factor 5F CFA model (χ²=3983.24, df=858, p<.001, χ²/df=4.64, RMSEA=.031, 90% CI RMSEA=[.030;.032], CFI=.97, TLI=.97, and SRMR=.039), in which χ²/df is lower than 5, RMSEA is clearly below .05, CFI and TLI are above .95, and SRMR is below .08; although chi-square is statistically significant, all indices indicates a good fit.

Including ESEM models in the analytical strategy responds to the need to rule out the existence of high factor cross-loadings between factors. ESEM models can be considered a hybrid between exploratory and confirmatory factor analysis since it allows establishing a priori relationships such as CFAs, but at the same time, it does not constrain the factor cross-loadings to zero as EFAs. Results obtained for the two estimated ESEM models, five-factor and bi-factor 5F, showed worse fit values than those obtained by CFA models (excepting one-factor CFA model) and obtained statistically significant differences using the chi-square tests for models comparison. After observing the factor matrices of the estimated ESEM models, no factor cross-loadings with high values were detected since all of them were less than .15.

Once an adequate fit for the bi-factor 5F CFA model had been obtained, an invariance study was carried out considering two variables: the level of VET (Basic/Intermediate) and the self-reported sex (Male/Female). Table IV shows the overall goodness-of-fit indices and comparison tests for all constrained invariance levels and both variables.

The complete invariance study obtained good fit results at all levels of constriction (Configural, Weak, Strong, Strict, and Means) for VET level and self-reported sex. Although the ∆χ² tests were only non-significant between Weak and Configural levels, the ∆CFI remained below .01 for all invariance levels considering VET studies and self-reported sex. The RMSEA showed an adequate fit with values of .028 and .029 for VET levels and self-reported sex, respectively.

Before presenting the results of the analytical fit of the parameters of the bi-factor model with 1 general factor and 5 specific uncorrelated factors, it must be taken into account that the general assumption of this kind of model is that the general factor explains a greater amount of items variance than specific factors. However, if only the global fit indices values are considered, it is possible to conclude that the bi-factor is the best model when in fact it is not. This possibility of the appearance of false positives is increased if it is taken into account that traditional fit indices (RMSEA, CFI, TLI, etc.) tend to favor bi-factor models (Gignac, 2016). Specific indices for bi-factor models (Omega H, H index, PUC, and ECVs) are needed to be interpreted for ensuring the plausibility of a true bi-factor structure with relevant specific factors, against the unidimensionality of a general factor. In this study, Omega H was .72, the H index was .88, the PUC was .80, and the ECV was .40 for the engagement general factor; these values indicate a strong general factor but unidimensionality can not be ensured. The Omega H values for the five specific factors were all above .30 and H index values were all above .70, except for the first specific factor School duties and discipline (H=.61). These results are indicating that the five specific factors can be considered significant in the bi-factor model (Table V).

In regards to the ECV-i value, only 14% of the items (6 of 44) presented a value equal to or greater than .80, so its variability was explained almost entirely by the factor general. These results reaffirm the significant weight of the specific factors in contributing to the items explained variance together with the general factor. The factors with the highest specific weight according to the average value of the ECV-i value of the items that compose them were, in this order, Social integration (.20), Family’s support (.25), and Teacher’s support (.39 ); while Goals and expectations (.63) and School duties and discipline (.60) presented a lower specific weight compared to the general factor.

Finally, all standardized factor loadings both of general and specific factors were statistically significant (p<.01), ranging from .24 to .56 for the general factor (M=.37) and from .06 to .68 (M=.44) for specific factors. No items have been detected that simultaneously presented low factorial loads in the general factor and the specific factors, so no item has been removed from the latent structure, also to preserve the content validity of the instrument.

Discussion

The measure of student engagement may be highly relevant in preventing school problems and avoiding school failure and, therefore, school dropout (Wang et al., 2011). Unfortunately, the literature shows a limited number of instruments for evaluating student engagement (Cedefop, 2016b; Christenson et al., 2012; Wang et al., 2019), which entail some limitations in terms of the domain of the measure and the target population. Further, in the Spanish education context, there are no studies aimed at measuring student engagement in the field of vocational secondary education (Echeverría & Martínez, 2017).

The present study provides an instrument with reliable and valid scores of the student engagement construct in a VET student population, both basic and intermediate. Results showed five uncorrelated factors for operationalizing the construct engagement in this population: school duties and discipline, goal and expectations, social integration (classmates), teacher’s support, and family’s support. These findings explain that the construct shows multiples components loads on a general factor, student engagement. This general factor reflects what is shared among the items and represents the students’ differences in the target dimension.

In this case, the study suggests that scores can be created to differentiate students with specific engagement patterns (Cedefop, 2016a). This result reinforces the idea that student engagement does not always entail the same degree of vulnerability or detachment in all students. In this line, the study of the factors implicated in the process is essential. School context, experiences lived about oneself, school activity, teachers, peers, and parents, all condition student engagement, hence the needs to measure different engagement factors.

Besides, according to the results, five items of the family’s support does not obtain good psychometric behavior. These items consider an indirect way of measuring family’s support, explaining parent’s reaction to certain behaviors or their expectations, without contemplating what the family does when the student need help or personal attention. Also, this result could explain that the weight of these family variables is not the same for BVET students or IVET students, in accordance with the evolutionary stage (Alonso, 2014). Thus, this variable would be more important among younger students, specifically BVET students (Elffers, 2013), and not so important among IVET students, who are usually over the theoretical entry age (16 years).

Meanwhile, items of factor two, goals and expectations as “I compare myself with my classmates to see if I am learning at the right pace” or “When I do a school activity, I try to grasp what I am doing” might not correspond to the instructions given by the teacher, mainly, in BVET and IVET studies, justifying their bad psychometric behavior. According to the literature (Fix et al., 2019), VET teachers prioritize instructions to assess intraindividual progress and supply feedback and social support to students depending on their results.

The shortage of empirical studies on engagement and vocational secondary education students in Spain makes the contribution of this study valuable in itself within the psycho-educational and social context. Besides, the VEI instrument with adequate evidence of reliability and validity is of great use to develop a simple instrument that will enable low levels of student engagement to be detected and also prevent the school dropout that takes place in the aforementioned student population. Further, the VEI instrument can favor preventive psycho-educational interventions and individualized school adaptations, thereby improving educational guidance on these issues (Pilcher et al., 2020).

On the other hand, future research ought to involve longitudinal studies that will enable the results to be expanded. In addition, administration of the VEI instrument could be completed with other scales addressing the school context (school organization, methodology of the center, etc.) completing the evaluation of student engagement and prevention of dropout.

Funding

The data for this article originates from the R&D project “Itinerarios de éxito y abandono en la formacion profesional de nivel I y II” (EDU2013-42854-R), funded by the Ministry of Economy and Competitiveness, the State Research Agency, and the European Regional Development Fund (Government of Spain).

Bibliograhpic references

Alonso, C. (2014). Familia, escuela y clase social: sobre los efectos perversos de la implicación familiar. [Family, school and social class: on the perverse effects of family involvement]. Revista de Asociación de Sociología de la Educación, 7(2), 395-409.

Appleton, J. J., & Christenson, S. L. (2004). Scale description and references for the Student Engagement Instrument. Unpublished manuscript.

Appleton, J. J., Christenson, S. L., Kim, D., & Reschly, A. L. (2006). Measuring cognitive and psychological engagement: Validation of the Student Engagement Instrument. Journal of School Psychology, 44(5), 427-445. https://doi.org/10.1016/j.jsp.2006.04.002

Betz, N., & Voyten, K. K. (1997). Efficacy and outcome expectation influence career exploration. Career Development Quarterly, 46(2), 197-189. https://doi.org/10.1002/j.2161-0045.1997.tb01004.x

Brauer, K., Ranger, J., & Ziegler, M. (2023). Confirmatory factor analyses in psychological test adaptation and development. A nontechnical discussion of the WLSMV estimator. Psychological Test Adaptation and Development, 4, 4-12. https://doi.org/10.1027/2698-1866/a000034

Brown, T. A. (2014). Confirmatory factor analysis for applied research. Guilford Publications.

Byrne, B. M. (2016). Structural equation modeling with AMOS: Basic concepts, applications, and programming. Routledge.

Cedefop (2016a). Leaving education early: Putting vocational education and training centre stage. Volume I: Investigating causes and extent. Publications Office. Cedefop research paper, 57. https://doi.org/10.2801/893397

Cedefop (2016b). Leaving education early: Putting vocational education and training centre stage. Volume II: Evaluating policy impact. Publications Office. Cedefop research paper, 58. http://dx.doi.org/10.2801/967263

Cerdà-Navarro, A., Quintana-Murci, E., & Salvà-Mut, F. (2022). Reasons for dropping out of intermediate vocational education and training in Spain: the influence of sociodemographic characteristics and academic background. Journal of Vocational Education & Training, 1-25. https://doi.org/10.1080/13636820.2022.2049625

Cheung, G. W., & Rensvold, R. B. (2002). Evaluating goodness-of-fit indexes for testing measurement invariance. Structural Equation Modeling, 9(2), 233-255. https://doi.org/10.1207/S15328007SEM0902_5

Christenson, S. L., Reschly, A. L., & Wylie, C. (2012). Handbook of research on student engagement. Springer. Retrieved from https://www.researchgate.net/profile/Azkananda_Widiasani/publication/310773130_Handbook_of_Student_Engagement/links/5836a0dd08aed45931c772b7.pdf

Constantinou, M., & Fonagy, P. (2019). Evaluating Bifactor Models of Psychopathology Using Model-Based Reliability Indices. PsyArXiv. https://doi.org/10.31234/osf.io/6tf7j

Echeverría, B., & Martínez, P. (2017). Diagnóstico de la investigación sobre la Formación Profesional Inicial en España (2005-2017). [Diagnosis of research on Initial Vocational Training in Spain (2005-2017)]. Fundación Bankia. Retrieved from https://www.fundacionbertelsmann.org/wp-content/uploads/2021/10/2019_Publicacion_Diagnostico_Investigacion_FPI_digital.pdf

Elffers, L. (2013). Staying on track: Behavioral engagement of at-risk and non-at-risk students in post-secondary vocational education. European Journal of Psychology of Education, 28(2), 545-562. https://doi.org/10.1007/s10212-012-0128-3

Elosua, P., & Zumbo, B. (2008). Coeficientes de fiabilidad para escalas de respuesta categórica ordenada. Psicothema, 20(4), 896-901. Retrieved from https://reunido.uniovi.es/index.php/PST/article/view/8747

Estell, D. B., & Perdue, N. H. (2013). Social support and behavioral and affective school engagement:The effects of peers, parents, and teachers. Psychology in the Schools, 50, 325-339. https://doi.org/10.1002/pits.21681

Fix, G. M., Ritzen, H. T. M., Pieters, J. M., & Kuiper, W. A. J. M. (2019). Effective curricula for at-risk students in vocational education: a study of teachers’ practice. Empirical Research in Vocational Education and Training, 11, 1-17. https://doi.org/10.1186/s40461-018-0076-5

Fouad, N. A., & Smith, P. L. (1996). A test of a social cognitive model for middle school students: Math and science. Journal of Counseling Psychology, 43(3), 338-346. https://doi.org/10.1037/0022-0167.43.3.338

Fredericks, J. A., Blumenfeld, P. C., & Paris, A. H. (2004). School engagement: Potential of the concept, state of the evidence. Review of Educational Research, 74, 59-109. https://doi.org/10.3102/00346543074001059

Gignac, G. E. (2016). The higher-order model imposes a proportionality constraint: That is why the bifactor model tends to fit better. Intelligence, 55, 57-68. https://doi.org/10.1016/j.intell.2016.01.006

Golino, H. F., & Epskamp, S. (2017). Exploratory graph analysis: A new approach for estimating the number of dimensions in psychological research. PLOS ONE, 12(6), e0174035. https://doi.org/10.1371/journal.pone.0174035

Hair, J. F., Anderson, R. E., Tatham, R. L., & Black, W. C. (2008). Multivariate Analysis (5.ª Ed.). Prentice Hall.

Hu, L., & Bentler, P. M. (1999). Cutoff criteria for fit indexes in covariance structure analysis: Conventional criteria versus new alternatives. Structural Equation Modeling: A Multidisciplinary Journal, 6(1), 1-55. https://doi.org/10.1080/10705519909540118

Hudson, G. (2020). EGAnet: Exploratory Graph Analysis–A framework for estimating the number of dimensions in multivariate data using network psychometrics. R package version 0.9.5.

Işik, E. (2013). Perceived Social Support and Locus of Control as the predictors of Vocational Outcome Expectations. Educational Sciences: Theory & Practice, 13(3), 1426-1430. Retrieved from https://files.eric.ed.gov/fulltext/EJ1017701.pdf

Janosz, M., Archambault, I., Lacroix, M., & Lévesque, J. (2007). Trousse d’évaluation des décrocheurs potentiels (TEDP): Manuel d’utilisation. Montréal : Groupe de recherche sur les environnements scolaires. Université de Montréal.

Jorgensen, T. D., Pornprasertmanit, S., Schoemann, A. M., & Rosseel, Y. (2021). semTools: Useful tools for structural equation modeling. R package version 0.5-5. Retrieved from https://CRAN.R-project.org/package=semTools

Kline, R. B. (2005). Methodology in the social sciences. Principles and practice of structural equation modeling. Guilford Press.

Le Blanc, M. (1996). MASPAQ, mesures de l’adaptation sociale et personnelle pour les adolescents québécois: manuel. Montréal, École de psycho-éducation, Groupe de recherche sur l’inadaptation psycho-sociale chez l’enfant, Université de Montréal.

Lent, R. W., Brown, S. D., & Hackett, G. (1994). Toward a unifying social cognitive theory of career and academic interest, choice, and performance. Journal of Vocational Behavior, 45, 79-122. https://doi.org/10.1006/jvbe.1994.1027

Mameli C., & Passini S. (2017). Measuring four-dimensional engagement in school: A validation of the student engagement scale and of the agentic engagement scale. TPM - Testing, Psychometrics, Methodology in Applied Psychology, 24(4), 527-541. https://doi.org/10.4473/TPM24.4.4

Martínez-Abad, F., & Rodríguez-Conde, M. J. (2017). Behaviour of product-moment and tetrachoric-polychoric correlations in ordinal scales: a simulation study. RELIEVE, 23(2), 1-19. http://dx.doi.org/10.7203/relieve.23.2.9476

Massara, G. P., Di Matteo, T., & Aste, T. (2017). Network filtering for big data: Triangulated maximally filtered graph. Journal of Complex Networks, 5(2), 161-178. https://doi.org/10.1093/comnet/cnw015

McWhirter, E. H., Crothers, M., & Rasheed, S. (2000). The effects of high school career education on social-cognitive variables. Journal of Counseling Psychology, 47(3), 330-341. https://doi.org/10.1037//0022-0167.47.3.330

Muthén, B., du Toit, S. H. C., & Spisic, D. (1997). Robust inference using weighted least squares and quadratic estimating equations in latent variable modeling with categorical and continuous outcomes. Technical report. Retrieved from https://www.statmodel.com/download/Article_075.pdf

Muthén, L. K., & Muthén, B. O. (2017). Mplus: Statistical Analysis with Latent Variables: User’s Guide (Version 8). Muthén & Muthén.

Niittylahti, S., Annala, J., & Mäkinen, M. (2019). Student engagement at the beginning of vocational studies. Nordic Journal of Vocational Education & Training, 9(1), 21-42. https://doi.org/10.3384/njvet.2242-458X.199121

OECD (2023). Propuestas para un plan de acción para reducir el abandono escolar temprano en España. OECD Publishing. https://doi.org/10.1787/9bc3285d-es

Olmos Rueda, P., Mas Torello, O., & Salva Mut, F. (2020). Educational disengagement profiles: a multidimensional contribution within basic vocational education and training. Revista de Educación, 389, 67-91. https://doi.org/10.4438/1988-592X-RE-2020-389-455

Pilcher, S.; Torii, K., & Andow, Jo. (2020). Evidence scan of educational interventions for children and young people disengaged from education. Mitschell Institute.

Poli, D., & Hungler, B. (2000). Investigación científica de la Salud. [Scientific Health Research]. McGraw-Hill.

Quin, D., Heerde, J. A., & Toumbourou, J. W. (2018). Teacher support within an ecological model of adolescent development: Predictors of school engagement. Journal of School Psychology, 69, 1-15. https://doi.org/10.1016/j.jsp.2018.04.003

R Core Team (2021). R: A language and environment for statistical computing. R Foundation for Statistical Computing, Vienna, Austria. Retrieved from https://www.R-project.org/

Reschly, A. L., & Christenson, S. L. (2012). Jingle, Jangle, and Conceptual Haziness: Evolution and Future Directions of the Engagement Construct. In S.L. Christentson, A. Reschly, & C. Wylie (Ed.) Handbook of Research on Student Engagement (pp 3-20). Springer.

Reise, S.P. Scheines, R., Widaman, K.F., & Haviland, M.G. (2013). Multidimensionality and structural coefficient bias in structural equation modeling: A bifactor perspective. Educational and Psychological Measurement, 73(1), 5-26. https://doi.org/10.1177/0013164412449831

Rosseel, Y. (2012). “lavaan”: An R Package for Structural Equation Modeling. Journal of Statistical Software, 48(2), 1-36. https://doi.org/10.18637/jss.v048.i02

Soler, A., Martínez Pastor, J. I., López-Messeguer, R., Valdés, M. T., Sancho Gargallo, M. A., Morillo, B., & de Cendra, L. (2021). Mapa del abandono educativo temprano en España. Informe General. Fundación Europea Educación y Sociedad.

Steiger, J. H. (2007). Understanding the limitations of global fit assessment in structural equation modeling. Personality and Individual Differences, 42(5), 893-898. https://doi.org/10.1016/j.paid.2006.09.017

Stucky, B. D., & Edelen, M. O. (2015). Using heierarchical IRT models to create unidimensional measures from multidimensional data. In S. P. Reise & D. A. Revicki (Eds.), Handbook of item response theory modeling: Applications to typical performance assessment, (pp. 183-206). New York: Routledge.

Sureda, I., Jiménez, R., Álvarez, O., & Quintana, E. (2021). Emotional and Behavioural Engagement among spanish Student in vocational education training. Sustainability, 13(7), 3882. https://doi.org/10.3390/su13073882

Wang, M. T., Fredricks, J., Ye, F., Hofkens, T., & Linn, J. S. (2019). Conceptualization and assessment of adolescents’ engagement and disengagement in school: A Multidimensional School Engagement Scale. European Journal Psychological Assessment, 35(4), 592-606. https://doi.org/10.1027/1015-5759/a000431

Wang, M. T., Willett, J. B., Eccles, J. S. (2011). The assessment of school engagement: examining dimensionality and measurement invariance by gender and race/ethnicity. Journal of School Psychology, 49(4), 465-480. https://doi.org/10.1016/j.jsp.2011.04.001

Annex

Item contents and their descriptive statistics by factors of the VEI bi-factor 5F latent model. Descriptive statistics: [N valid, M, SD, skewness (g1), kurtosis (g2)]