A total sample size of 184 was available to the researchers, of which 5 cases were missing NCLEX-RN outcome data due to not having taken the examination at the time of data collection or having taken the examination out of state. This left a final sample size of 179 for the purposes of this analysis.
First Exit Exam scores yielded a group mean of 832.69 (SD = 116.07); scores ranged from 473 to 1251. Final Exit Exam scores yielded a group mean of 917.27 (SD = 76.12); scores ranged from 648 to 1251. Final Exit Exam scores include the final scores from students who took the examination up to 5 times to achieve a score of 850 on the examination. Overall, the mean number of examination attempts necessary to achieve a score of 850 was 2.13 (SD = 1.16), with 38.6% (n = 69) needing to take the examination three or more times to achieve a required score of 850 to graduate.
On the basis of their final Exit Exam scores, students scoring greater than 850 were predicted to pass the NCLEX-RN. In this sample, 167 students were predicted to pass the NCLEX-RN and 12 were expected to fail (i.e., these students scored lower than 850 but were allowed to graduate after the faculty voted to rescind the progression policy). Of the 167 students expected to pass, 22 failed the NCLEX-RN. Of the 12 expected to fail, 10 passed the NCLEX-RN, meaning only 2 of those expected to fail actually failed the NCLEX-RN. On the basis of the percentage of students who score in HESI Exit Exam Categories A through C (i.e., students scoring 850 or greater) who pass the NCLEX-RN, which ranges from 94.1% to 98.3% according to Nibert, Young, and Adamson (2002), one can assume an approximate 94% NCLEX-RN pass rate for the school, considering only 2 of the 12 students predicted to fail actually failed. For the total sample, the NCLEX-RN pass rate was 86.6% (n = 155). The actual NCLEX-RN pass rate was considerably lower than the expected pass rate for the school. To examine the HESI Exit Exam scores of students who passed versus those who failed the NCLEX-RN, both first Exit Exam scores and final Exit Exam scores were examined using crosstabulation. Results are presented in Table 1. To assess for differences in HESI Exit Exam scores with NCLEX-RN outcomes as the grouping variable, one-way ANOVAs were calculated and are also presented in Table 1.
Table 1: HESI Exit Exam Score Descriptive Statistics by NCLEX-RN Outcomes and Anova Comparison
Research Question 1: What Is the Relationship Between Students’ First Exit Exam Scores and NCLEX-RN Outcomes Versus Students’ Final Exit Exam Scores and NCLEX-RN Outcomes? To answer this question, pointbiserial correlation coefficients were calculated with first Exit Exam scores and then final Exit Exam scores as the continuous variables and NCLEX-RN outcomes as the dichotomous variables. Results revealed a statistically significant relationship between first Exit Exam scores rpb = −0.275, p ≤ 0.005) and NCLEX-RN outcomes. There was no statistically significant relationship between final Exit Exam scores and NCLEX-RN outcomes (rpb = 0.026, p = 0.733). It appears that when students were allowed to retake the Exit Exam multiple times to achieve the minimum 850 required to graduate, the relationship between Exit Exam scores and NCLEX-RN outcomes nearly disappears. Allowing students indefinite attempts to pass the Exit Exam apparently introduces error in the form of spurious Exit Exam scores into the relationship, which causes the relationship to decrease in strength.
Research Question 2: Do HESI Exit Exam Scores Statistically Significantly Predict NCLEX-RN Outcomes, and What Is the Accuracy of the Classification from this Logistic Model? To examine whether HESI Exit Exam scores can significantly predict NCLEX-RN outcomes, binary logistic regression analysis was conducted. Two separate analyses were conducted; the first used students’ first Exit Exam scores; the second used students’ final Exit Exam scores. Each analysis was run separately using an ENTER method; because only one variable was used as a predictor in each analysis, the model fit statistics compare the model with a predictor to a model without a predictor to assess whether the model with a predictor predicts statistically significantly better than does a model with a constant only. Results of the logistic regression analysis are presented in Table 2.
Table 2: Logistic Regression of HESI Exit Exam Scores Predicting NCLEX-RN Outcomes
According to Tabachnick and Fidell (2001), logistic regression models should be evaluated on two fronts: First, how well the model fits the data, and second, how well the predictor classifies individual cases. There was good model fit for first Exit Exam scores, as evidenced by a significant Wald statistics (12.230, p ≤ 0.005) and omnibus analysis of chi-square ( = 14.299, p ≤ 0.005); this indicates that first Exit Exam scores can distinguish between who will pass and who will fail the NCLEX-RN in a better-than-chance way. Using the range provided by the Cox and Snell R2 and Nagelkerke R2 as estimates of the proportion of the variance in NCLEX-RN outcomes accounted for by first Exit Exam scores, 7.7% to 14.1% of the variance in NCLEX-RN outcomes can be accounted by first Exit Exam scores. The model performed poorly in predicting NCLEX-RN failure, with none of the NCLEX-RN failures being accurately classified. The weakness in this prediction can be further seen in the OR of only 0.992 for the model; this OR is near 1 and therefore shows little change in likelihood of NCLEX-RN failure for a one-unit change in first Exit Exam score.
To visually examine how well first Exit Exam scores predict NCLEX-RN outcomes, a Receiver Operating Characteristics (ROC) curve was calculated (Figure 1). Spurlock and Hanks (2004) used this method of showing the tradeoff between sensitivity and specificity. According to Norusis (2005), once the curve is calculated, the area under the curve (AUC) is known as the c statistic. Children’s Mercy Hospitals & Clinics (Simon, 2003) provides this guide for interpreting c values: 0.50 to 0.75 = fair, 0.75 to 0.92 = good, 0.92 to 0.97 = very good, and 0.97 to 1.00 = excellent. The c value for this model was 0.739 (p = 0.000), which would classify first Exit Exam scores as “fair” predictors of NCLEX-RN outcomes, the poorest class for predictors.
Figure 1: Receiver Operating Characteristics (ROC) Curve for First HESI Exit Exam Scores. Area Under the Curve (AUC) Is c = 0.739, p ≤ 0.005. a Perfect AUC = 1.0, Indicating 100% Sensitivity and 100% Specificity. the Center Diagonal Reference Line Above Indicates an AUC of 0.5, or Chance.
Model fit for final Exit Exam scores was not impressive nor statistically significant as indicated by an insignificant Wald statistic and omnibus analysis chi-square ( = 2.751, p = 0.097). When the range provided by the Cox and Snell R2 and Nagelkerke R2 was used, only 1.5% to 2.8% of the variance in NCLEX-RN outcomes can be accounted by final Exit Exam scores. These results are not unexpected in light of the poor relationship reported earlier between final Exit Exam scores and NCLEX-RN outcomes and the insignificant F for differences in final Exit Exam mean scores for those who failed versus those who passed the NCLEX-RN. Classification using this model was poor as well, with no NCLEX-RN failures being accurately classified. The 95% confidence interval for ORs in this model includes an OR of 1, which further indicates the insignificance of final Exit Exam scores in predicting NCLEX-RN outcomes. An ROC curve was also calculated for this model (Figure 2). Results from calculating the curve reveal, again, that final Exit Exam scores are insignificant and poor predictors of NCLEX-RN outcomes.
Figure 2: Receiver Operating Characteristics (ROC) Curve for Final HESI Exit Exam Scores. Area Under the Curve (AUC) Is c = 0.591, p = 0.152.
Research Question 3: What Cutoff Scores for the HESI Exit Exam Yield the Most Accurate Classification of Students as Predicted to Fail and Predicted to Pass? The logistic regression model can be used to calculate predicted probabilities of an event (in this case, NCLEX-RN failure), but to easily and more accurately demonstrate how setting different HESI Exit Exam scores can affect the overall ability of the Exit Exam to classify students as either predicted to pass or fail the NCLEX-RN, we used the model presented by Spurlock and Hanks (2004). This model, which is useful for evaluating how well a test can classify or diagnose a case (NCLEX-RN pass or fail), was applied to various cutoff scores. As Spurlock and Hanks demonstrated, only students who achieve a score of 900 or above are classified as predicted to pass by the HESI Exit Exam. Students achieving scores lower than 900 fall into other descriptive categories, further discussed under research question four. At the institution from which data for this study were collected, for example, students had to achieve a score of 850 to progress to graduation. So in this case, the prediction to pass was set at an Exit Exam score of 850 and above. Students who could not achieve an 850 or more on the Exit Exam, even after multiple attempts, would have technically, according to this institution’s progression policy, been prohibited from graduation (although that has not happened).
There are several predictive test characteristics that are important to nursing faculty. These test characteristics are summarized in Table 3, which also provides a clinical-educational comparison (for a more detailed explanation, see Spurlock and Hanks, 2004). These values are calculated from inserting actual student performance data into a 2×2 contingency table (Table 4). By changing the cutoff scores for when students are predicted to pass and, therefore, when they are predicted to fail, the overall performance of the various cutoff scores can be assessed. As Spurlock and Hanks noted (2004), the most important characteristics for nurse educators to be concerned with are the sensitivity and the positive predictive value (PPV). These values tell nurse educators what it is they need to know—how well the test predicts NCLEX-RN failure, both retrospectively (sensitivity) and prospectively (PPV).
Table 4: Contingency Table Used to Figure Predictive Test Parameters
Table 3: Predictive Test Characteristics of Importance to Nurse Educators with Clinical-Educational Comparison
Students from this sample were classified according to HESI Exit Exam cutoff score as either predicted to pass or predicted to fail. Only the first Exit Exam scores were used because they were the only scores found to be worthwhile in using as predictors. The starting cutoff score was 900 and the last score was 550; so for the 900 cutoff, students scoring less than 900 were predicted to fail and students scoring 900 or above were predicted to pass. This procedure was performed in 25-point increments to the lowest cutoff score. Results of each of the test characteristics calculated are provided in Table 5.
Table 5: Performance of Different Cutoff Scores for Predictions to Pass or Fail on First Exit Exam Scores
An evaluation of the data in Table 5 reveal that the best cutoff score for the students in this sample was 650. A HESI Exit Exam score of 650 on the first attempt yields the best classification of students: PPV and overall accuracy are the highest on the table, and the OR increases greatly to 7.89, meaning that students predicted to fail (those scoring lower than 650) are 7.89-fold more likely to fail than those scoring above 650. Also, the overall accuracy of the test is highest as well, at 87%. A point of caution is due here regarding the calculation of positive predictive values. As Smith, Winkler, and Fryback (2000) noted, calculating PPVs in populations with low prevalence can be troublesome. Essentially, when the prevalence of the problem (failing the NCLEX-RN) is low (e.g., approximately 10%), the PPV of a test will tend to be low as well. One can use, as Smith et al. (2000) did, Bayesian methods to try to account for pre-prediction prevalence of the problem, but this would seem too wieldy for nurse educators in diverse academic settings. A key point to remember is this: Because for most schools NCLEX-RN failures are relatively low in prevalence, predicting them with a diagnostic or predictive test can be challenging, as has been demonstrated here. Difficulty in predicting NCLEX-RN failure has been reported by Spurlock and Hanks (2004) and Seldomridge and DiBartolo (2004), and further research will no doubt bolster this claim.
Research Question 4: Do Descriptors of HESI Exit Exam Score Categories Actually Reflect the Real Probability of a Student Failing the NCLEX-RN? To answer this question, we go back to the binary logistic regression model for first Exit Exam scores. One of the benefits of the logistic regression is that the logistic regression equation constructed from the logistic regression model can be used to predict the probability of, in this study, NCLEX-RN failure (Cohen, Cohen, West, & Aiken, 2002; Norusis, 2005; Tabachnick & Fidell, 2001). According to Norusis (2005), the logistic regression model applicable here is:
are coefficients estimated from the data, X
is the value of the independent variable, and e
is the base of the natural logarithms, which is approximately 2.718. In this analysis, the regression model applied would be this:
Using this equation, predicted probabilities for NCLEX-RN failure were calculated for each of the scoring categories reported by HESI. Those values, along with the assigned descriptors of risk and definition of categories as reported by HESI (2002) are reported in Table 6. As can be seen in Table 6, the descriptors for Category A and B seem correct; those students scoring very highly on their first Exit Exam do in fact have little chance of failing the NCLEX-RN, based on the Exit Exam as a sole predictor (p ≤ 0.03 to 0.05). Category C, which indicates an average probability of passing, is described incorrectly for this sample; according to the National Council of State Boards of Nursing (2005a, 2005b), the national pass rate for the NCLEX-RN in 2004 was 85.3% and in 2005, the national pass rate was 89.2%. Therefore, the average failure rate is actually much higher, at approximately p = 0.11 to 0.15., not the 0.05 to 0.08 that was found in this study. Looking at the lowest scoring categories, Categories G and H, HESI describes students in these categories as at “grave risk of failing” and “poor performance expected” (Evolve Reach, Powered by HESI, 2008, p. 2). The actual predicted probabilities of NCLEX-RN failure for this group is estimated to range from 0.22 to 0.29 (the lower the score in category H, the greater the risk), certainly not eliciting any thoughts of grave danger—increased risk, perhaps, but not grave danger.
Table 6: HESI Exit Exam Categories, Descriptors of Risk, and Calculated Probabilities of Failure for Each Category