Noncognitive factors related to academic performance

Factores no cognitivos relacionados con el rendimiento académico

https://doi.org/10.4438/1988-592X-RE-2022-398-556

Ingrid Garcia Pinzón

https://orcid.org/0000-0002-0694-2360

Centro Nacional de Evaluación para la Educación Superior, CENEVAL, México.

Margarita Olivera Aguilar

https://orcid.org/0000-0001-6160-1254

Educational Testing Service, Center for Research on Validity, Fairness & Equity in Learning and Assessment,
Estados Unidos.

Abstract

There are few holistic models looking at the interrelations between noncognitive variables and their effects on academic outcomes. The objectives of this study were to examine a holistic model that seeks to predict academic performance based on the relationships between noncognitive variables and academic performance, and to examine its stability in different educational grades.

Data from two samples of Mexican students (19,826 students seeking admission to high school; 85,040 seeking admission to college) were analyzed to examine relationships between academic behaviors, perseverance, assertiveness, cooperation, internal locus of control, external locus of control and test scores and high school and college admission test scores. A series of structural equation models were conducted to examine scale dimensionality, relationships between variables, and measurement invariance.

The models showed good fit to the data and the effects were stable across the two samples. Cooperation and perseverance had direct effects on academic behaviors and indirect effects on test scores. Internal locus of control was a strong predictor of cooperation, perseverance, and assertiveness, and showed a positive indirect effect on test scores, but a negative effect on academic behaviors. External locus of control and assertiveness were weak predictors of test scores.

The importance of examining the effects of noncognitive variables in a holistic framework is highlighted, as it allows identifying variables with weak effects as well as unexpected results.

Key words: noncognitive variables, academic performance, cooperation, perseverance, assertiveness, academic behaviors.

Resumen

Existen pocos modelos holísticos sobre relaciones entre variables no cognitivas y sus efectos en los resultados académicos. Los objetivos del estudio fueron examinar un modelo holístico que busca predecir el rendimiento académico basado en las interrelaciones entre variables no cognitivas, y examinar su estabilidad en diferentes grados educativos.

Se analizaron datos de dos muestras de estudiantes mexicanos (19,826 estudiantes buscando el ingreso a la educación media superior; 85,040 buscando el ingreso a la universidad), para examinar relaciones entre conductas académicas, perseverancia, asertividad, cooperación, locus de control interno, locus de control externo y los resultados a exámenes de admisión a la educación media superior y superior. Se realizaron una serie de modelos de ecuaciones estructurales para examinar la dimensionalidad de las escalas, las relaciones entre variables, y la invarianza de medición entre las muestras.

Los modelos tuvieron un buen ajuste a los datos y los efectos fueron estables a través de las dos muestras. La cooperación y la perseverancia tuvieron efectos directos en las conductas académicas y efectos indirectos en los puntajes de los exámenes. El locus de control interno fue un fuerte predictor de la cooperación, perseverancia y asertividad, y mostró un efecto indirecto positivo en los puntajes de las pruebas, pero un efecto negativo en los comportamientos académicos. El locus de control externo y la asertividad fueron predictores débiles de los puntajes en los exámenes.

Se destaca la importancia de examinar los efectos de las variables no cognitivas en un marco holístico que permita identificar variables con efectos débiles así cómo resultados inesperados.

Palabras clave: variables no cognitivas, rendimiento académico, cooperación, perseverancia, asertividad, conductas académicas.

Introduction

Noncognitive skills entail an extensive array of characteristics such as personality, motivation, attitudes, curiosity, and study skills (Heckman & Kautz, 2012; Kyllonen, et al., 2014). The growing interest in the study of noncognitive variables is related to an increase in empirical evidence regarding their ability to predict academic outcomes (Duckworth & Yeager, 2015; Gamazo & Martínez-Abad, 2020; Heckman, et al., 2014; Pitsia, et al., 2017; Resino et al., 2019), and their malleability through interventions (Durlak, et al., 2011; Kautz, et al., 2014).

Various meta-analyses have examined the association of noncognitive skills with academic outcomes (Richardson, et al., 2012; Robbins, et al., 2004), but studies have rarely focused on the interactions between more than two variables or on the mechanisms through which they influence each other and influence academic outcomes. In order to have a holistic understanding of the mechanisms through which noncognitive variables influence academic performance, it is necessary to propose and evaluate models concerning the relationships between these variables. Although several models have been proposed to explain academic performance (e.g., Bean, 1980; Ryan & Deci, 2019; Tinto, 1993), they have not focused on the interrelations between noncognitive variables. An important exception is the theoretical model of Farrington, et al. (2012), which states hypotheses about the mechanisms through which noncognitive variables may influence academic performance.

Few studies have provided empirical support to the model proposed by Farrington and collaborators (for an exception see Farruggia et al., 2016). The main objective of this study was to evaluate the model in a sample of Mexican students who took the high school (EXANI-I) and college (EXANI-II) Admission Tests elaborated by the Centro Nacional de Evaluación para la Educación Superior (Ceneval). A secondary objective was to examine the stability of the model between the two samples.

Conceptual Framework

In this study, we focus on the model of Farrington et al., (2012) that classifies noncognitive skills in five categories and states hypotheses of the relationships between them and with academic performance. Next, we describe the five categories of noncognitive variables proposed by Farrington et al., (2012) and the relationships expected between them1 (Figure 1).

FIGURE 1. Section of the holistic model proposed by Farrington et al. (2012)

Note: Learning strategies and their effects are shown in grey since they were not assessed in this study due to the lack of items aligned with this construct.

Academic Behaviors

They are defined as the activities related to school work, such as attending classes, doing homework, etc. The model holds that this is the only category that has a direct effect on academic performance (Allensworth & Easton, 2007; Cooper, et al., 2006).

Academic Perseverance

It has a direct effect on academic behaviors and includes constructs such as grit, tenacity, delayed gratification, self-discipline, and self-control. Perseverance is what distinguishes those students who demonstrate academic behaviors necessary to pass a class and engage in those behaviors over long periods of time (Duckworth, et al., 2007).

Academic Mindsets

It is a set of attitudes or beliefs about oneself with regards to academic work. It includes variables such as sense of belonging, the belief that ability can increase through effort, the belief about having self-control over the likelihood of being successful in an assignment, and the perception about the usefulness and value of an assignment (Bandura, 1986; Dweck & Leggett, 1988; Eccles et al., 1983; Ryan & Deci, 2000). The model suggests that academic mindsets have direct relationships with academic behaviors, perseverance, learning strategies, and social skills.

Learning Strategies

They are the processes and tactics that are used to remember events, and the action plans and thoughts used for monitoring learning. They include study skills, metacognition, and self-regulated learning (Flavell, 1979; Pintrich & De Groot, 1990; Zimmerman & Schunk, 1989). In the model, there is a direct relationship of learning strategies with perseverance and academic behaviors, as well as an interaction between academic behaviors and learning strategies.

Social Skills

They are defined as the behaviors that improve social interactions among peers or among students and teachers. They include constructs such as cooperation, assertiveness, and empathy. There are studies that indicate positive direct relationships between social skills and academic outcomes (Malecki & Elliot, 2002), and indirect relationships through academic behaviors (Wentzel, 1991).

Purpose of the Study

Despite the importance of the model proposed by Farrington, et al. (2012), which lies in proposing hypotheses regarding the interrelations among noncognitive variables, only one article examining such relationships was found. Farruggia et al., (2016) examined a modified version of the model that included perseverance, academic mindset, and learning strategies as predictors of academic performance and retention. The study by Farrugia et al., conducted with 1,603 college students, found a strong effect of academic mindsets and a moderate effect of perseverance on academic performance. Academic performance was the only predictor with a strong effect on retention. A limitation of the study is that the effect of academic behaviors was not studied and, according to the model of Farrington et al., (2012), it is the only category that has a direct effect on academic performance. The main purpose of the present study was to examine a section of the model proposed by Farrington et al. (2012); this model does not include learning strategies due to the lack of items conceptually aligned with this construct.

The model by Farrington et al., assumes that the relationships between noncognitive variables and academic outcomes are stable across various education levels. However, there is ample evidence showing that noncognitive skills are developed at different stages in life (Kautz et al., 2014; Wigfield, et al., 2006), and previous studies indicate that the relationship between these variables and academic outcomes change over time (Gore, 2006; Poropat, 2009). It is expected that the relationships put forward in the model change at different school grade levels. Hence, the second purpose of this research was to examine stability of the scales and the model proposed in two samples of distinct education grades.

Method

Sample

Data were obtained from two large-scale assessments used in Mexico in 2015 for admission to high school, EXANI-I, and college, EXANI-II, developed by Ceneval. The total sample size in EXANI-I was 19,826, with an average age of 16.28 years, 47% men from 284 institutions, 78.5% of which were public middle schools. The sample of EXANI-II comprised 85,040 people, with an average age of 19.26, 50.4% of whom were men from 1,214 institutions, of which 70.6% were public schools.

Instruments

The noncognitive items were administered in a pilot study in 2013 and 2014 as part of the background questionnaires, also developed by Ceneval, that test-takers need to fill in when registering for EXANI-I and EXANI-II. Statistical analyses were conducted to identify and correct problematic items. The corrected scales were administered in 2015.

Academic behaviors

In this study they were defined as activities related to school work that demonstrate students’ engagement with school (Hart, et al., 2011), and were measured with four items about the frequency with which examinees studied for exams, handed in homework on time, participated in class, and were prepared for class. The items (for example, “I spend time studying outside of school”) were responded with a Likert scale with four response options (never o almost never; sometimes, frequently, always or almost always).

Perseverance of effort

This scale was adapted from the study by Duckworth & Quinn (2009), who defined perseverance as the persistence and passion to accomplish long-term goals. The Likert scale consisted of four items with statements like “I finish whatever I begin,” with four response options (not at all like me, somewhat like me, like me, just like me).

Academic mindsets

This construct was measured with two scales: internal locus of control with three items, and external locus of control with four items. Internal locus of control was defined as the beliefs that examinees have concerning the control of their lives through self-motivation or self-determination, while external locus of control assessed the degree to which people attribute the events in their lives to luck, fate, other people or external factors (Ryan & Connell, 1989; Visdómine-Lozano & Luciano, 2006). Students showed their agreement with statements such as “The fact that I do well or badly at school totally depends on me” for internal locus of control and “My grades at school are due to how lucky I am” for external locus of control. The same response options of the perseverance scale were used.

Social skills

Social skills were measured with two scales: assertiveness with three items and cooperation with five items. Assertiveness was defined as the direct expression of feelings and was measured using some items of the study by Peneva & Mavrodiev (2013) (for example, “I communicate my opinions even when they are different from the group’s”). Cooperation was defined as effective relationships to achieve group objectives by means of an exchange of knowledge and skills (Harris & Harris, 1996). The cooperation scale was based on the items included in the study by Pfaff & Huddleston (2003) (for example, “I make suggestions to improve the performance of the team”). The same response options for the academic behaviors were employed.

Academic performance

We used the students’ scores in EXANI-I, with 92 items, and EXANI-II, with 112 items, as dependent variables. These aptitude tests examine generic competences in the areas of mathematical thinking, analytical thinking, language structure, and reading comprehension. In this study, we solely used the general score.

Procedure

Design

The study applies a secondary analysis of two large-scale assessments, thus implementing a non-experimental cross-sectional cohort design.

Statistical analysis

All analyses were conducted in Mplus version 7.2 (Muthén & Muthén, 1998-2012) using WLSMV as the estimator given the categorical nature of the items. We cross-validated the results to avoid drawing conclusions highlighting the characteristics of the sample used. Each sample was divided randomly in a test sample and a validation sample of approximately the same size. The test sample of EXANI-I (subsequently called EXANI-I-T) and the validation sample (EXANI-I-V) each comprised 9,913 examinees. The test sample of EXANI-II (EXANI-II-T) comprised 42,625 examinees, while the validation sample (EXANI-II-V) comprised 42,415 examinees. The statistical models were first examined in the test samples and the final model was examined in the validation samples to evaluate the replicability of results.

As a preliminary step, scale dimensionality was examined to provide evidence of construct validity. Through a confirmatory factor analysis (CFA) for categorical data, a model of six correlated factors was examined: academic behaviors, perseverance of effort, internal locus of control, external locus of control, assertiveness, and cooperation. Items with standardized factor loadings higher than 0.4 were selected. The overall model fit was assessed through a chi-squared test of model fit. As this test is sensitive to large sample sizes, more emphasis was given to fit indices such as the comparative fit index (CFI) and the root mean square error of approximation (RMSEA). CFI values higher than 0.90 and RMSEA values lower than 0.08 were indicative of adequate fit, while CFI values higher than 0.95 and RMSEA values lower than 0.05 were indicative of good model fit (West, et al., 2012).

Measurement invariance

Measurement invariance was conducted to examine the extent to which the items had equivalent psychometric properties between the two samples. A series of multi-group CFA models were compared, where each model had an increasing number of constraints (Vandenberg & Lance, 2000). First, configural invariance was assessed by allowing all item parameters to be freely estimated in each group. Next, metric invariance was examined by constraining the factor loadings to equality between the two samples. In a third model, strong measurement invariance was examined by constraining the thresholds or intercepts to equality. Finally, strict measurement invariance was assessed by adding equality constraints in the unique variances. These nested models were compared through the chi-square difference test, but due to its sensitivity to large sample sizes, the change in CFI was also considered, where changes of 0.01 or lower were considered evidence in favor of models with more constraints (Cheung & Rensvold, 2002). When one of the models showed lack of fit, partial invariance models were examined by allowing some parameters to be freely estimated. Modification indices (MI) were examined to determine which item parameters needed to be freely estimated across groups.

Relationships between latent variables

Structural equation modeling was used to examine the hypotheses stated in the model by Farrington et al., (2012). Using the CFA results as the basis for the structural equation models, we added paths between the latent variables according to Figure 1.

To examine the stability of the model across samples, we compared the fit of two models. In the unconstrained model, the regression (i.e., path) coefficients were freely estimated in each sample, while in the constrained model, all regression coefficients were constrained to equality between the two samples. The two models were compared using the chi-square test difference for categorical data, where a significant chi-square difference was interpreted as evidence that at least one of the coefficients was different between the samples. Considering the sensitivity of the chi-square to large sample sizes, the RMSEA and the CFI were also examined.

Once the stability of the model was tested, the mediation effects indicated in the model of Farrington et al., (2012; Table 1) were examined in the final model. Mediated effects were calculated as the product of the regression coefficients (MacKinnon, et al., 2002) and assessed using 95% confidence intervals obtained from 200 bootstrap samples (MacKinnon, et al., 2004).

TABLE 1. Indirect effects examined in the model

Indirect Effect	Independent variable	Mediator	Dependent variable
1	Perseverance	Academic behaviors	Test scores
2	Assertiveness	Academic behaviors	Test scores
3	Cooperation	Academic behaviors	Test scores
4	Internal locus of control	Academic behaviors	Test scores
5	External locus of control	Academic behaviors	Test scores
6	Internal locus of control	Perseverance- Academic behaviors	Test scores
7	Internal locus of control	Assertiveness- Academic behaviors	Test scores
8	Internal locus of control	Cooperation - Academic behaviors	Test scores
9	External locus of control	Perseverance- Academic behaviors	Test scores
10	External locus of control	Assertiveness- Academic behaviors	Test scores
11	External locus of control	Cooperation - Academic behaviors	Test scores

Results

Construct validity

The 6-factor model showed adequate fit to the data in both samples (RMSEA < 0.08 and CFI > 0.90); an item in the perseverance of effort scale showed standardized loadings of 0.226 and 0.227 in the EXANI-I-T and EXANI-II-T samples, respectively. This item was deleted in both samples and the CFA models were reexamined. The new model showed an adequate fit to the data according to the RMSEA and CFI, and all the factor loadings had values above 0.5 (Table 2).

TABLE 2. Results for the final model of six correlated factors in each sample

	EXANI-I-T		EXANI-II-T
Items	Standardized factor loadings	Non-standardized factor loadings	Standardized factor loadings	Non-standardized factor loadings
Academic behaviors
I bring to class all the materials I need	0.92	1.00	0.69	1.00
I participate in class	0.77	0.84	0.69	1.00
I spend time studying outside of school	0.82	0.90	0.66	0.96
I hand in homework on time	0.98	1.07	0.77	1.11
Reliability (omega)	0.80		0.67
Perseverance
I finish whatever I begin	0.80	1.00	0.75	1.00
I am a person that strives	0.84	1.06	0.81	1.07
I am a hard worker	0.87	1.08	0.82	1.09
Reliability (omega)	0.79		0.75
Cooperation
I participate in the planning of activities	0.86	1.00	0.83	1.00
I collaborate in the development of strategies to accomplish work goals	0.90	1.04	0.85	1.03
I intervene to try to solve disagreements among team members	0.76	0.88	0.71	0.86
I make suggestions to improve the performance of the team	0.85	0.99	0.80	0.97
I do the tasks that I have to do	0.82	0.95	0.84	1.01
Reliability (omega)	0.76		0.73
Assertiveness
I defend my rights	0.82	1.00	0.78	1.00
I communicate my opinions although they are different from the group’s	0.86	1.05	0.82	1.04
I defend myself when someone accused me of something I didn’t do	0.76	0.93	0.71	0.91
Reliability (omega)	0.75		0.71
Internal locus of control
If I try enough, I’ll be successful at school	0.90	1.00	0.85	1.00
The fact that I do well or badly at school totally depends on me	0.72	0.81	0.64	0.75
If I set my mind to it, I do better at school	0.76	0.85	0.71	0.84
Reliability (omega)	0.69		0.62
External locus of control
My grades at school are due to how lucky I am	0.89	1.00	0.87	1.00
If my family supported me more, I’d do better at school	0.54	0.60	0.53	0.61
My grades at school are due to things I can’t change	0.77	0.86	0.75	0.87
If I had other teachers, I’d do better at school	0.50	0.55	0.47	0.55
Reliability (omega)	0.54		0.53
Model fit	χ2(194)= 6079.35, p<.001 RMSEA=.055, CFI=.970		χ2(194)= 22800.30, p<.001 RMSEA=.053, CFI=.952

Note: EXANI-I-T test sample of high school applicants; EXANI-II-T test sample of college applicants.

The correlations between academic behaviors, perseverance of effort, assertiveness, cooperation, and internal locus of control were positive and high, with correlation values between 0.50 and 0.83 (Table 3), while the correlations between external locus of control and the rest of latent variables were negative. In general, these correlations and the fit of the CFA models provide evidence that the scales measure related but independent constructs.

TABLE 3. Correlations between latent variables in the EXANI-I test sample (lower triangular matrix) and in the EXANI-II test sample (upper triangular matrix)

	Academic behaviors	Perseverance	Cooperation	Assertiveness	Internal locus of control	External locus of control
Academic behaviors	1	.50	.54	.38	.45	-.26
Perseverance	.72	1	.62	.56	.71	-.22
Cooperation	.83	.59	1	.62	.61	-.30
Assertiveness	.56	.59	.55	1	.56	-.22
Internal locus of control	.63	.75	.58	.59	1	-.23
External locus of control	-.24	-.15	-.21	-.20	-.14	1

Note: All values were significant p < .001.

Measurement invariance

The difference in the chi-square test of model fit in the configural, metric and strong invariance models was significant in most comparisons due to the large sample sizes. Nonetheless, the change in CFI was smaller than 0.01 in each comparison, suggesting that the scales were invariant. Given the contradictory information between the chi-square test of model fit and the change in the CFI, we examined the MIs to identify possible noninvariant items. In each scale, the number of parameters freely estimated was minimized to avoid overemphasizing nuisances of the assessed samples.

The invariance analyses for assertiveness and external locus of control did not reveal noninvariant parameters according to the change in CFI and MIs. In these scales, strict measurement invariance was found (Table 4).

TABLE 4. Fit for invariance models

Model	χ2 (df)	RMSEA	CFI	Δχ2 (Δdf)	Δχ2 p	ΔCFI
Academic behaviors
Configural 1	3585.99 (5)	0.166	0.976
Configural 2	195.98 (3)	0.050	0.999
Metric	367.38 (6)	0.048	0.998	194.52 (3)	0	0.001
Strong	1250.24 (13)	0.061	0.992	878.33 (7)	0	0.006
Partial strong	542.52 (12)	0.041	0.996	216.99 (6)	0	0.002
Strict strong	1048.325 (14)	0.053	0.993	492.17 (2)	0	0.003
Perseverance
Configural	66.74 (1)	0.050	0.999
Metric	79.94 (3)	0.031	0.999	11.54 (2)	0.003	0.003
Strong	464.42 (8)	0.047	0.996	392.43 (5)	0	0.001
Partial strong	232.33 (7)	0.035	0.998	158.50 (4)	0	0.001
Partial strict	322.58 (8)	0.039	0.997	99.82 (1)	0	0
Cooperation
Configural 1	4485.82 (11)	0.125	0.980
Configural 2	418.77 (9)	0.039	0.998
Metric	324.52 (13)	0.042	0.999	3.94 (4)	0.41	-0.001
Strong	1074.35 (21)	0.029	0.995	718.81 (8)	0	0.004
Partial strong	527.93 (20)	0.038	0.998	220.04 (7)	0	0.001
Partial strict	995.489 (23)	0.036	0.996	467.43 (3)	0	0.002
Assertiveness
Configural	4.56 (1)	0.012	1
Metric	6.63 (3)	0.007	1	1.55 (2)	.46	0
Strong	95.36 (8)	0.021	0.999	90.51 (5)	0	0.001
Strict	243.45 (10)	0.030	0.997	150.34 (2)	0	0.002
Internal locus of control
Configural	24.48 (1)	0.030	1
Metric	35.68 (3)	0.021	0.999	12.72 (2)	0	0.001
Strong	94.87 (8)	0.017	0.998	62.60 (5)	0	0.001
Strict	315.94 (10)	0.034	0.994	196.53 (2)	0	0.004
Partial strict	184.18 (9)	0.027	0.997	78.57 (1)	0	0.001
External locus of control
Configural	905.19 (5)	0.083	0.979
Metric	813.55 (8)	0.062	0.981	2.74 (3)	0.43	-0.002
Strong	818.54 (15)	0.046	0.981	86.02 (7)	0	0
Strict	825.95 (18)	0.042	0.981	67.66 (3)	0	0

In the case of perseverance of effort, although the change in CFI from the metric to the strong invariance model was below 0.01, the MI for the threshold of the item “I am a hard worker” suggested the presence of noninvariance. The threshold for this item was freely estimated and the final model consisted of partial strict factorial invariance.

In the case of internal locus of control, the MIs showed a noninvariant unique variance in the item “The fact that I do well or badly at school totally depends on me.” Therefore, that parameter was freely estimated in each group and the final model consisted of partial strict factorial invariance.

The configural model for academic behaviors showed poor model fit according to RMSEA = 0.166. The MIs showed that a correlation between the items “I participate in class” and “I spend time studying outside of school” was necessary in the EXANI-II sample. We decided to include the correlation since, in most cases, in order to participate in class students must prepare outside of school time. Likewise, the configural model for cooperation showed poor model fit according to the RMSEA value of 0.125. The MIs showed that a correlation between the items “I make suggestions to improve the performance of the team” and “I intervene to try to solve disagreements among team members” was necessary. We decided to include the correlation as in both cases cooperation involves interposing in the team’s dynamics to make positive changes. After including the correlations in academic behaviors and cooperation in both samples, the configural models showed good fit to the data. In both scales, the MIs in the strong measurement invariance model indicated the presence of a noninvariant threshold in one of the items, which was freely estimated in each sample. The final models for academic behaviors and cooperation consisted of partial strict measurement invariance.

Relationships between latent variables

The unconstrained model showed a significant chi-square test of model fit χ2 (501) = 36,098.36, p <.001, but adequate fit according to the RMSEA = 0.052 and CFI = 0.947. The model with equality constraints in the path coefficients showed a significant chi-square χ2 (513) = 31,863, p <.001, but good fit according to RMSEA = 0.048 and CFI = 0.953. Due to the large sample sizes examined, the comparison between the two models revealed a significant value in the chi-square difference test Δχ2 (12) = 546.09, p <.001, but the change in CFI showed that the constrained model had better fit to data. Therefore, the constrained model was selected as the final model. This model was examined in the validation samples and showed a similar model fit than the test sample, χ2 (513) = 29,643.38, p <.001, RMSEA = 0.047, and CFI = .957.

The final regression coefficients of the model in Figure 1 are shown in Table 5. Due to the equality constraints, the unstandardized regression coefficients are identical in the samples of EXANI-I and EXANI-II, so only one value is shown. The comparison of unstandardized coefficients in the test and validation samples indicates similar regression coefficients, providing evidence in favor of the generalization of results.

The standardized results indicate that cooperation was the strongest predictor of academic behaviors, followed by perseverance of effort (Table 5)2. External locus of control was a negative predictor of academic behaviors, perseverance of effort, cooperation, and assertiveness. Internal locus of control was a negative predictor of academic behaviors, but as significant positive predictor of the rest of the noncognitive variables.

TABLE 5. Regression coefficients of the final model

Effect	Non-standardized values		Standardized values
	Test	Validation	EXANI-I-T	EXANI-II-T	EXANI-I-V	EXANI-II-V
DV: Test scores IV:
Academic behaviors	4.04 (0.08)	4.24 (0.09)	0.45 (0.01)	0.25 (0.00)	0.44 (0.01)	0.25 (0.00)
DV: Academic behaviors IV:
Perseverance	0.32 (0.01)	0.31 (0.01)	0.28 (0.01)	0.42 (0.02)	0.27 (0.01)	0.40 (0.02)
Internal locus of control	-0.17 (0.02)	-0.14 (0.02)	-0.13 (0.02)	-0.20 (0.03)	-0.11 (0.02)	-0.17 (0.03)
External locus of control	-0.01 (0.00)	-0.01 (0.00)	-0.04 (0.00)	-0.06 (0.01)	-0.05 (0.00)	-0.08 (0.01)
Cooperation	0.47 (0.01)	0.46 (0.01)	0.42 (0.01)	0.68 (0.01)	0.43 (0.01)	0.66 (0.01)
Assertiveness	0.10 (0.01)	0.08 (0.01)	0.07 (0.01)	0.11 (0.01)	0.06 (0.01)	0.09 (0.01)
DV: Perseverance IV:
Internal locus of control	0.90 (0.01)	0.89 (0.01)	0.79 (0.01)	0.81 (0.00)	0.79 (0.01)	0.82 (0.00)
External locus of control	-0.01 (0.00)	-0.03 (0.00)	-0.06 (0.01)	-0.07 (0.01)	-0.06 (0.01)	-0.06 (0.01)
DV: Cooperation IV:
Internal locus of control	0.83 (0.01)	0.83 (0.01)	0.70 (0.01)	0.68 (0.01)	0.72 (0.01)	0.70 (0.00)
External locus of control	-0.03 (0.00)	-0.03 (0.00)	-0.16 (0.01)	-0.16 (0.01)	-0.13 (0.01)	-0.13 (0.01)
DV: Assertiveness IV:
Internal locus of control	0.70 (0.01)	0.68 (0.01)	0.70 (0.01)	0.69 (0.00)	0.70 (0.01)	0.70 (0.00)
External locus of control	-0.02 (0.00)	-0.02 (0.00)	-0.12 (0.01)	-0.13 (0.01)	-0.11 (0.01)	-0.11 (0.01)

Note: Standard errors are shown in parentheses. DV: dependent variable; IV: independent variable; EXANI-I-T: high school admission exam-test sample; EXANI-II-T: college admission exam-test sample; EXANI-I-V: high school admission exam-validation sample; EXANI-II-V: college admission exam-validation sample.

Mediation effects

None of the 95% confidence intervals of the mediation effects included zero, indicating that all effects were statistically significant (Table 6). Unstandardized mediation effects in the test and validation samples are very similar to each other, which suggests that the results are stable throughout the samples.

TABLE 6. Mediation effects

Indirect effect	Unstandardized effect	95% confidence interval	Standardized effect
			EXANI-I	EXANI-II
Perseverance – Academic behaviors – Test scores	1.28	1.18, 1.38	0.13	0.10
	(1.28)	(1.15, 1.40)	(0.12)	(0.10)
Assertiveness– Academic behaviors – Test scores	0.39	0.32, 0.46	0.03	0.03
	(0.33)	(0.25, 0.40)	(0.03)	(0.02)
Cooperation – Academic behaviors – Test scores	1.89	1.80, 1.99	0.19	0.17
	(1.94)	(1.85, 2.03)	(0.19)	(0.17)
ILC – Academic behaviors – Test scores	-0.69	-0.87, -0.53	-0.06	-0.05
	(-0.58)	(-0.75, -0.41)	(-.05)	(-0.04)
ILC – Perseverance – Academic behaviors – Test scores	1.15	(1.06, 1.24)	0.10	0.08
	(1.13)	(1.02, 1.23)	(0.10)	(0.08)
ILC – Assertiveness – Academic behaviors – Test scores	0.27	0.22, 0.32	0.02	0.02
ILC – Cooperation – Academic behaviors – Test scores	(0.22)	(0.17, 0.28)	(0.02)	(0.02)
	1.58	1.50, 1.66	0.14	0.11
	(1.60)	(1.51, 1.68)	(0.14)	(0.12)
ELC – Academic behaviors – Test scores	-0.03	-0.04, -0.02	-0.02	-0.01
	(-0.05)	(-0.06, -.004)	(-0.02)	(-0.02)
ELC – Perseverance – Academic behaviors – Test scores	-0.01	-0.02, -0.01	-0.01	-0.01
	(-0.02)	(-0.02, -0.01)	(-0.01)	(-0.01)
ELC – Assertiveness– Academic behaviors – Test scores	-0.01	-0.01, -0.005	-0.004	-0.004
	(-0.01)	(-0.01, -0.005)	(-0.003)	(-0.003)
ELC – Cooperation – Academic behaviors – Test scores	-0.05	-0.05, -0.04	-0.03	-0.03
	(-0.05)	(-0.06, -0.05)	(-0.03)	(-0.02)

Note: ILC: internal locus of control; ELC: external locus of control. The validation samples results are shown in parenthesis. Unstandardized path coefficients are the same in test and validation samples, hence, only one value is shown. Considering that the latent variables have different variances, standardized coefficients differ in each sample.

The standardized effects indicate that the strongest indirect effects were those of academic behaviors as the mediator of the relationship between cooperation and academic performance; academic behaviors as the mediator of the relationship between perseverance of effort and academic performance; and cooperation and academic behaviors as mediators of the relationship between internal locus of control and academic performance.

Internal locus of control had a negative indirect effect on test scores when the relationship was mediated only by academic behaviors. Nevertheless, when the relationship also was mediated by perseverance of effort, assertiveness, or cooperation, internal locus of control showed a positive indirect effect on the test scores.

Discussion

The purposes of this study were to assess a section of the model proposed by Farrington et al., (2012) and examine the stability of the model across two education levels. The structural equation modeling results provide evidence of construct validity, of the stability of scales and the overall model through education levels and provide information regarding the strongest predictors of academic performance.

The results indicate that cooperation and perseverance were the strongest predictors of academic behaviors, with indirect effects on test scores. Various studies have demonstrated that perseverance of effort has a strong association with academic performance (Duckworth, et al., 2007; Eskreis-Winkler, et al., 2014), while the benefits of cooperation have been reported in previous studies (Davidson & Major, 2014; Dingel, et al., 2013; Malecki & Elliot, 2002).

The final model revealed interesting effects of internal locus of control, which was a strong predictor of cooperation, perseverance, and assertiveness, and showed a positive indirect effect on test scores. In contrast with the findings of previous studies (Agnew, et al., 1993), internal locus of control had a negative direct effect on academic behaviors. A possible explanation is that if a greater sense of responsibility is not matched by ability and the appropriate circumstances to achieve academic goals, it can result in higher stress which may be negatively related to academic behaviors. Although this hypothesis must be tested, previous studies have shown negative consequences of internal locus of control (Avtgis, 1998; Dweck, 1986; Whitley, 1998).

We found that external locus of control had a negative direct effect on all noncognitive variables, but its highest effects were on cooperation and assertiveness. Consistent with previous studies (Coleman et al., 1966; Grimes, 1997), its indirect effect on test scores, though statistically significant, was close to zero and, therefore, of limited practical importance.

We also found that the relationship between assertiveness and academic behaviors was weaker than the relationships of perseverance, internal locus of control, and cooperation with academic behaviors. A possible explanation is that assertiveness may have a curvilinear relationship with academic outcomes, as has been reported in the context of leadership (Ames, 2009). In addition, most of the studies examining the relationship between assertiveness and academic success have been conducted with preschool students (Montroy, et al., 2014) and further research is needed with high school and college student samples.

Conclusions

This study is one of the first attempts to empirically test the model proposed by Farrington et al., (2012) and as such, it provides an important contribution to the understanding of the interrelations between noncognitive variables and their effects on academic performance. The results underline the importance of examining the effects of noncognitive variables under a more holistic approach. By doing so, we were able to identify unexpected results, for example, the negative relationship between internal locus of control and academic behaviors, and the small statistical effects of external locus of control and assertiveness on test scores. These findings would have been more difficult to observe if the variables were examined in isolation.

Furthermore, this study provides evidence about the stability of the results in two samples of distinct education levels. In addition to theoretical implications, these results also have practical implications as educational institutions could use this information to design interventions with the reassurance that variables related to academic success of students about to enter high-school are the same variables related to academic success of students interested in pursuing a college degree.

Nonetheless, as any investigation, this study has limitations that must be considered in future studies. The noncognitive scales were part of the background questionnaires of admission tests, and therefore there were constraints concerning the number of items that could be included in each scale. Although the scales were reviewed by content experts to ensure the items reflected the construct definitions, the limited number of items included made it difficult to fully capture all the aspects of the categories proposed by Farrington et al., (2012). The small number of items also affected scale reliability, which may cause attenuation in the estimated parameters. Future studies must consider the balance between measuring scales with a high number of items while at the same time assessing as many noncognitive variables as possible, allowing for a more holistic understanding of the interactions among them.

Finally, an important limitation is that this is a correlational study where conclusions on causality cannot be made. Although the holistic model proposes causal relationships between the variables, this study cannot support such conclusions. Future longitudinal experimental studies must be conducted to examine if the variables have a causal effect as suggested in the model.

References

Allensworth, E., & Easton, J.Q. (2007) What matters for staying on-track and graduating in Chicago Public Schools. Chicago: College of Chicago Consortium on Chicago School Research. https://files.eric.ed.gov/fulltext/ED498350.pdf

Agnew, N. C., Slate, J. R., Jones, C. H., & Agnew, D. M. (1993). Academic behaviors as a function of academic achievement, locus of control & motivational orientation. NACTA Journal, 24-27. https://nactateachers.org/attachments/article/802/Agnew_NACTA_Journal_June_1993-4.pdf

Ames, D. (2009). Pushing up to a point: Assertiveness and effectiveness in leadership and interpersonal dynamics. Research in Organizational Behavior, 29, 111-133. https://doi.org/10.1016/j.riob.2009.06.010

Avtgis, T.A. (1998). Locus of control and persuasion, social influence, and conformity: A meta-analytic review. Psychological Reports, 83, 899-903. https://doi.org/10.2466/pr0.1998.83.3.899

Bandura, A. (1986) Social foundations of thought and action: A social cognitive theory. Eglewood Cliffs, NJ: Prentice Hall.

Bean, J. P. (1980). Dropouts and turnover: The synthesis and test of a causal model of student attrition. Research in College, 12, 155–187. https://doi:10.1007/BF00976194

Cheung, G. W., & Rensvold, R. B. (2002). Evaluating goodness-of-fit indexes for testing measurement invariance. Structural Equation Modeling, 9, 233-255. https://doi.org/10.1207/S15328007SEM0902_5

Coleman, J. S., Campbell, E. Q., Hobson, C. J., McPartland, J., Mood, A. M., Weinfeld, F. D., & York, R. L. (1966). Equality of educational opportunity. Washington, DC: U.S. Government Printing Office. https://files.eric.ed.gov/fulltext/ED012275.pdf

Cooper, H., Robinson, J.C., & Patall, E.A. (2006) Does homework improve academic achievement? A synthesis of research, 1987-2003. Review of Educational Research, 76, 1-62. https://doi.org/10.3102/00346543076001001

Davidson, N., & Major, C. H. (2014). Boundary crossings: Cooperative learning, collaborative learning, and problem-based learning. Journal on Excellence in College Teaching, 25, 7-55. http://www.sun.ac.za/english/learning-teaching/ctl/Documents/Davidson%202014%20BoundaryCrossings.pdf

Dingel, M. J., Wei, W., & Huq, A. (2013). Cooperative learning and peer evaluation: The effect of free riders on team performance and the relationship between course performance and peer evaluation. Journal of the Scholarship of Teaching and Learning, 13, 45-56. https://scholarworks.iu.edu/journals/index.php/josotl/article/view/2133/3068

Duckworth, A. L., Peterson, C., Matthews, M. D., & Kelly, D. R. (2007). Grit: Perseverance and passion for long-term goals. Journal of Personality and Social Psychology, 92, 1087-1101. https://doi.org/10.1037/0022-3514.92.6.1087

Duckworth, A. L., & Quinn, P. D. (2009). Development and validation of the Short Grit Scale (GRIT–S). Journal of Personality Assessment, 91, 166-174. https://doi.org/10.1080/00223890802634290

Duckworth, A. L., & Yeager, D. S. (2015). Measurement matters: Assessing personal qualities other than cognitive ability for educational purposes. Educational Researcher, 44, 237-251. https://doi.org/10.3102/0013189X15584327

Durlak, J.A., Weissberg, R.P., Dymnicki, A.B., Taylor, R.D., & Schellinger, K.B. (2011). The impact of enhancing students’ social and emotional learning: A meta-analysis of school-based universal interventions. Child Development, 82, 405-432. https://doi.org/10.1111/j.1467-8624.2010.01564.x

Dweck, C.S. (1986). Motivational processes affecting learning. American Psychologist, 41, 1040-1048. https://doi.org/10.1037/0003-066X.41.10.1040

Dweck, C.S., & Leggett, E.L. (1988) A social-cognitive approach to motivation and personality. Psychological Review, 95, 256-273. https://doi.org/10.1037/0033-295X.95.2.256

Eccles J.S., Adler, T.F., Futterman, R., Goff, S.B., Kaczala, C.M., Meece, J.L., and Midgley, C. (1983) Expectancies, values, and academic behaviors. In J.T. Spence (Ed.), Achievement and achievement motivation (pp. 75-146). San Francisco: W.H. Freeman.

Eskreis-Winkler, L., Duckworth, A. L., Shulman, E. P., & Beal, S. (2014). The grit effect: Predicting retention in the military, the workplace, school and marriage. Frontiers in Psychology, 5. https://doi.org/10.3389/fpsyg.2014.00036

Farrington, C. A., Roderick, M., Allensworth, E., Nagaoka, J., Keyes, T. S., Johnson, D. W., & Beechum, N. O. (2012). Teaching adolescents to become learners: The role of noncognitive factors in shaping school performance. A critical literature review. Chicago: College of Chicago Consortium on Chicago School Research. https://consortium.uchicago.edu/sites/default/files/2018-10/Noncognitive%20Report_0.pdf

Farruggia, S. P., Han, C., Watson, L., Moss, T. P., & Bottoms, B. L. (2016). Noncognitive factors and college student success. Journal of College Student Retention: Research, Theory, & Practice, 20. 308-327. https://doi.org/10.1177/1521025116666539

Flavell, J.H. (1979) Metacognition and cognitive monitoring: A new area of cognitive developmental inquiry. American Psychologist, 34, 906-911. https://doi.org/10.1037/0003-066X.34.10.906

Gamazo, A., & Martínez-Abad, F. (2020). An exploration of factors linked to academic performance in PISA 2018 through data mining techniques. Frontiers in Psychology, 11. https://doi.org/10.3389/fpsyg.2020.575167

Gore Jr, P. A. (2006). Academic self-efficacy as a predictor of college outcomes: Two incremental validity studies. Journal of Career Assessment, 14, 92-115. https://doi.org/10.1177/1069072705281367

Grimes, S. K. (1997). Underprepared college students: Characteristics, persistence, and academic success. Community College Journal of Research and Practice, 21, 47-56. https://doi.org/10.1080/1066892970210105

Harris, P. R., & Harris, K. G. (1996). Managing effectively through teams. Team Performance Management: An International Journal, 2, 23-36. https://doi.org/10.1108/13527599610126247

Hart, S. R., Stewart, K., & Jimerson, S. R. (2011). The Student Engagement in Schools Questionnaire (SESQ) and the Teacher Engagement Report Form-New (TERF-N): Examining the preliminary evidence. Contemporary School Psychology: Formerly “The California School Psychologist,” 15, 67-79. https://link.springer.com/article/10.1007/BF03340964

Heckman, J. J., Humphries, J. E., & Kautz, T. (Eds.). (2014). The myth of achievement tests: The GED and the role of character in American life. College of Chicago Press.

Heckman, J.J., & Kautz, T. (2012). Hard evidence on soft skills. Labour Economics, 19, 451-464. https://doi.org/10.1016/j.labeco.2012.05.014

Kautz, T., Heckman, J. J., Diris, R., ter Weel, B., & Borghans, L. (2014). Fostering and measuring skills: Improving cognitive and noncognitive skills to promote lifetime success (OECD Education Working Paper #1110). Paris, France: OECD Publishing. https://www.nber.org/papers/w20749

Kyllonen, P.C., Lipnevich, A.A., Burrus, J., & Roberts, R.D. (2014). Personality, motivation, and college readiness: A prospectus for assessment and development. ETS Research Report Series, 2014, 1-48. https://doi.org/10.1002/ets2.12004

MacKinnon, D. P., Lockwood C. M., Hoffman, J. M., West, S. G., & Sheets, V. (2002). A comparison of methods to test mediation and other intervening variable effects. Psychological Methods, 7, 83-104. https://doi.org/10.1037/1082-989X.7.1.83

MacKinnon, D. P., Lockwood C. M., & Williams, J. (2004). Confidence limits for the indirect effect: Distribution of the product and resampling methods. Multivariate Behavioral Research, 39, 99-128. https://doi.org/10.1207/s15327906mbr3901_4

Malecki, C. K., & Elliot, S. N. (2002). Children’s social behaviors as predictors of academic achievement: A longitudinal analysis. School Psychology Quarterly, 17, 1-23. https://doi.org/10.1521/scpq.17.1.1.19902

Montroy, J. J., Bowles, R. P., Skibbe, L. E., & Foster, T. D. (2014). Social skills and problem behaviors as mediators of the relationship between behavioral self-regulation and academic achievement. Early Childhood Research Quarterly, 29, 298-309. https://doi.org/10.1016/j.ecresq.2014.03.002

Muthén, L.K. & Muthén, B.O. (1998-2012). Mplus User’s Guide. Seventh Edition. Los Angeles, CA: Muthén & Muthén. https://www.statmodel.com/download/usersguide/Mplus%20user%20guide%20Ver_7_r3_web.pdf

Peneva, I., & Mavrodiev, S. (2013). A historical approach to assertiveness. Psychological Thought, 6, 3-26. https://doi.org/10.23668/psycharchives.1912

Pfaff, E., & Huddleston, P. (2003). Does it matter if I hate teamwork? What impacts student attitudes toward teamwork. Journal of Marketing Education, 25, 37-45. https://doi.org/10.1177/0273475302250571

Pintrich, P.R., & De Groot, E. (1990) Motivational and self-regulated learning components of classroom academic performance. Journal of Educational Psychology, 82, 33-40. https://doi.org/10.1037/0022-0663.82.1.33

Pitsia, V., Biggart, A., & Karakolidis, A. (2017). The role of students’ self-beliefs, motivation and attitudes in predicting mathematics achievementA multilevel analysis of the Programme for International Student Assessment data. Learning and Individual Differences, 55, 163-173. Scopus. https://doi.org/10.1016/j.lindif.2017.03.014

Poropat, A. E. (2009). A meta-analysis of the five-factor model of personality and academic performance. Psychological Bulletin, 135, 322–338. https://doi.org/10.1037/a0014996

Resino, D. A., Amores, I. A. C., & Muñoz, I. A. (2019). La repetición de curso a debate: Un estudio empírico a partir de PISA 2015. Educación XX1, 22. https://doi.org/10.5944/educxx1.22479

Richardson, M., Abraham, C., & Bond, R. (2012). Psychological correlates of college students’ academic performance: A systematic review and meta-analysis. Psychological Bulletin, 138, 353-387. https://doi.org/10.1037/a0026838

Robbins, S.B., Lauver, K., Le, H., Davis, D., & Langley, R. (2004). Do psychosocial and study skill factors predict college outcomes? A meta-analysis. Psychological Bulletin, 130, 261-288. https://doi.org/10.1037/0033-2909.130.2.261

Ryan, R. M., & Connell, J. P. (1989). Perceived locus of causality and internalization: Examining reasons for acting in two domains. Journal of Personality and Social Psychology, 57, 749-761. https://doi.org/10.1037/0022-3514.57.5.749

Ryan, R.M., & Deci, E.L. (2000) Self-determination theory and the facilitation of intrinsic motivation, social development, and well-being. American Psychologist, 55, 68-78. https://doi.org/10.1037/0003-066X.55.1.68

Tinto, V. (1993). Leaving college: Rethinking the causes and cures of student attrition (2nd ed.). Chicago, IL: College of Chicago.

Vandenberg, R. J., & Lance, C. E. (2000). A review and synthesis of the measurement invariance literature: Suggestions, practices, and recommendations for organizational research. Organizational Research Methods, 3, 4-70. https://doi.org/10.1177/109442810031002

Visdómine-Lozano, J. C., & Luciano, C. (2006). Locus de control y autorregulación conductual: revisiones conceptual y experimental. International Journal of Clinical and Health Psychology, 6, 729-751. https://www.redalyc.org/pdf/337/33760313.pdf

Wentzel, K.R. (1991). Social competence at school: Relationships between social responsibility and academic achievement. Review of Educational Research, 61, 1-24. https://doi.org/10.3102/00346543061001001

West, S. G., Taylor, A. B., & Wu, W. (2012). Model fit and model selection in structural equation modeling. In R. H. Hoyle (Ed.), Handbook of structural equation modeling (pp. 380-392). New York, NY: Guilford Press. Whitley, Jr., B. E. (1998). Factors associated with cheating among college students: A review. Research in College, 39, 235-274. https://doi.org/10.1023/A:1018724900565

Whitley, Jr., B. E. (1998). Factors associated with cheating among college students: A review. Research in College, 39, 235-274. https://doi.org/10.1023/A:1018724900565

Wigfield, A., Eccles, J. S., Schiefele, U., Roeser, R., & Davis-Kean, P. (2006). Development of achievement motivation. In W. Damon (Series Ed.) & N. Eisenberg (Volume Ed.), Handbook of child psychology (6th ed., Vol. 3, pp. 933–1002). New York, NY: John Wiley.

Zimmerman, B.J., and Schunk, D.H. (Eds.). (1989) Self-regulated learning and academic achievement: Theory, research, and practice. New York: Springer Verlag.

Contact address: Ingrid Gracia Pinzón, Centro Nacional de Evaluación para la Educación Superior, CENEVAL. Dirección de Investigación, Calidad Técnica e Innovación Académica. Subdirección de Investigación e Innovación. Departamento de Investigación. Av. Camino al Desierto de los Leones 37, Col. San Ángel, Alcaldía Álvaro Obregón, C.P. 01000, Ciudad de México. E-mail: ingrid.garcia@ceneval.edu.mx

---