The importance of academic support services, including academic advising services, has been established, in part, by its designation as a key element in the American Association of Colleges of Nursing’s Standards for Accreditation (2009). In addition to this designation, research and scholarly commentary indicate that effective academic advisors play a role in student development (Light, 2001; Pizzolato, 2008; Reinarz & Ehrlich, 2002). Good advising is tied also to students’ academic success (Bahr, 2008; Campbell & Nutt, 2008; Museus & Ravello, 2010) and may influence decisions about and attitudes toward lifelong learning (Hunter & White, 2004; Smith & Allen, 2006; Stickle, 1982). Finally, effective academic advising has been associated with student satisfaction, recruitment, and retention (Elliott & Healy, 2001; Freeman, 2008; Peterson, Wagner, & Lamb, 2001). Therefore, it is important, to understand what constitutes good academic advising and to have access to tools that enable students to evaluate the quality of advising.
Advancements in academic advising research are, for the most part, the aegis of education and pedagogy. Nursing’s contribution to the study of academic advising scholarship has been minimal. The purpose of this study was to develop and pilot test the Faculty Advisor Evaluation Questionnaire (FAEQ).
Five to nine questions were developed to measure each characteristic (Table 1). Items were revised multiple times to ensure that they met criteria for structurally sound and appropriately worded queries. According to Dillman (2007), survey questions should be written in a way that potential respondents would interpret them in the same manner, and they should be willing and able to answer each question accurately. Other principles for writing survey questions include, but are not limited to, avoiding bias, using simple words, being precise, and avoiding hypothetical and double-barreled questions.
Questions were formatted in a six-point Likert-type scale (completely agree, generally agree, neither agree nor disagree, generally disagree, completely disagree, and not applicable) and 12 demographic items were developed. The demographic data included, but were not limited to, questions about the student’s major area of study, type of nursing program, gender, ethnicity, status of advisor (faculty or nonfaculty), and frequency of advising meetings. Finally, sets of instructions for completing the FAEQ and demographics were constructed.
Validity and pilot reliability of the FAEQ were conducted in two phases at different universities (referred to here as university A and university B) due to a change in employment by the principal investigator (PI). Institutional review board approval was obtained at university A for validity testing and at university B for the reliability pilot study.
Two nursing students confirmed face validity. Face validity is the extent to which the study participant feels that the questionnaire measures what it is intended to measure (Nunnally & Bernstein, 1994). Although face validity cannot be quantified and is often confused with content validity, it provides helpful evidence when used in conjunction with other forms of evidence (Lynn, 1986; Nunnally & Bernstein, 1994; Rattray & Jones, 2007). Face validity was deemed valuable to the development of the FAEQ because both student and faculty perceptions of effective academic advising were used in its development.
Content validity is concerned with the adequacy of questions to measure the concept of interest—in this case, the characteristics of an effective academic advisor. Content validity may be established through literature review, personal reflection, or analytical critique (Higgins & Straub, 2006). Seven nurse experts confirmed content validity and revisions were made to the FAEQ based on their evaluations. For the purpose of this study, analytical critique was used, and the experts were chosen based on their experience as faculty advisors and their willingness to serve. All seven panel members were female; one was Black, six were White; their experience in nursing ranged from 16 to 25+ years; four members had been academic advisors for 16 to 25 years and two members had 2 to 5 years and one member had less than 2 years of experience in advising. Five panel members were masters’ prepared, and two had doctorates in nursing. The members were solicited from public and private universities in Florida, Minnesota, Pennsylvania, Mississippi, California, Iowa, and Wisconsin.
The panel members were asked to link each question to an objective or characteristic and evaluate each using a four-point Likert-type scale (1 = totally irrelevant, 2 = somewhat relevant, 3 = quite relevant, 4 = very relevant). Members were provided with instructions that included the study objectives and definition of major concepts. For example, objective one, ‘To determine the student’s perception of the advisor’s knowledge of program and university requirements,” reflects the first major concept or characteristic: knowledge. Knowledge was defined as being familiar with facts and a range of information, including, but not limited to, knowledge of students and their development. In addition, panel members were asked to evaluate the adequacy and redundancy of the questionnaire as a whole in a series of yes or no questions.
Packets containing an invitation to participate, the questionnaire, study purpose, objectives, and major concepts, as well as a demographic data collection sheet for the validity experts, were mailed to the participants with a postage-paid return envelope. A reminder letter was mailed to participants several weeks later.
The assessment of the analytical critique or judgment-quantification phase in establishing content validity included the calculation of the index of content validity (CVI) and the evaluation of qualitative responses to questions about the instrument’s adequacy and redundancy. The CVI is the proportion of items or questions that receive a rating of 3 = quite relevant or 4 = very relevant (Lynn, 1986). The CVI range of the 60 questions was 0.43 to 1.00. Twenty-nine questions had a CVI of 1.00. The CVI of the remaining 31 questions ranged from 0.43 to 0.86.
According to Davis (1992), new instruments should achieve an 80% or higher agreement among reviewers. To achieve this significance level and reduce the number of items in the FAEQ, 27 questions rated below 0.85 were considered for elimination. Four of 27 were either reworded or retained based on their conceptual or theoretical significance. For example, the item, “My advisor helps me with my career goals,” was retained despite a CVI of 0.57 because academic advising research and scholarly commentary consistently cite the considerable role academic advisors play in helping students to develop career goals (Beasley-Fielstein, 1986; Hunter & White, 2004; Kelley & Lynch, 1991; O’Banion, 1972; Pizzolato, 2008; Shultz, Colton, & Colton, 2001; Walsh, 1979; Yarbrough, 2002).
Ten questions were added based on the evaluation of qualitative responses from the content judges to meet the requirement of adequacy. “I believe that my advisor is committed to promoting my success,” and “My advisor demonstrates the behavior of a good mentor” are examples of questions suggested by the panel members. Several additions and revisions were also made to the demographic data collection sheet based on the content judges’ qualitative responses. “Do you have an individual assigned to you as an academic advisor?” and “Is English your second language?” are examples of demographic questions added or revised based on feedback from panel members. Based on the aforementioned revisions, the FAEQ was reduced from 63 to 50 items.
Reliability Pilot Study
The next step in the development of the FAEQ was to conduct a pilot study to discover and solve problems before implementation of the full study. The questionnaire format was recreated for Survey Monkey ( http://www.surveymonkey.com/). Students at a medium-sized public university in the southern United States received a brief invitation to participate in the research and a link to the questionnaire in the university’s student e-news. After accessing the site, students were able to read the formal invitation to participate. It included an explanation of the study, estimated time to complete, inclusion criteria, and PI and institutional review board contact information. The inclusion criteria were enrollment full time or part time in a 4-year college or university; age 18 years or older; ability to read, write, and understand English; and willingness to complete the questionnaire. To estimate completion time, two students were timed while taking a 50-item hard copy form of the questionnaire, and their times were averaged. Completion of the questionnaire signified consent. The instructions and demographic data section also preceded the questionnaire.
Six hundred thirty-three students responded to the invitation, and the results were evaluated for internal consistency reliability (Cronbach’s alpha) and factor analysis. Internal consistency reliability refers to the consistency with which a questionnaire measures the concept of interest—in this case, the quality of academic advising (Waltz, Strickland, & Lenz, 1991). Cronbach’s alpha should exceed 0.70 for a developing questionnaire and 0.80 for an established questionnaire (Rattray & Jones, 2007). The initial overall reliability of the FAEQ was 0.81, indicating that the questionnaire consistently measures the concept.
In general, a 10:1 ratio (10 respondents for each question) is necessary for factor analysis (Hair, Black, Babin, Anderson, & Tatham, 2006); therefore, the number of responses in this pilot study was sufficient for factor analysis.
A principle components factor analysis with oblimin rotation (chosen because questions were related to each other) was conducted to explore the relationship of variables. Principle components factor analysis enables the identification of underlying subscales and allows the researcher to identify and remove redundant or unnecessary items. The Kaiser-Meyer-Olkin measure, a statistic associated with factor analysis, is used to test sampling adequacy and indicates that sufficient numbers of completed questionnaires are available to substantiate further analysis. The Kaiser-Meyer-Olkin for the FAEQ was 0.963. According to Kaiser (1974), values >0.90 are considered “marvelous.” The Bartlett’s test of sphericity (χ2 = 8583.51, df = 406, p = 0.000) indicated significant correlations among the items and provided support for the factorability of the data.
Two criteria were used to determine how many factors should be extracted. First, the Kaiser-Guttman eigenvalue greater-than-one-rule indicated that three factors should be extracted. The second criterion was the scree plot, which suggested a four-factor solution. This four-factor solution accounted for 81% of the variability. Items were then assessed based on their individual factor loadings (>0.40) and theoretical significance. Based on this evaluation, the 50-item questionnaire was reduced to 30 items. Table 2 displays the factor loadings for the individual questions of the yet unnamed factors in the 30-item questionnaire. Cronbach’s alpha values were 0.974, 0.965, 0.885, and 0.927 for factors 1 through 4, respectively, indicating homogeneity among items.
Table 2: Four-Factor Structure of Measurements for Questionnaire Items
The purpose of this study was to develop and pilot test the FAEQ. The questionnaire was developed based on qualitative nursing research that identified faculty and student perceptions’ of effective academic advisors. The psychometric properties of the FAEQ were explored using face and content validity, internal consistency reliability, and principle components factor analysis. The panel of experts agreed that the questionnaire items were relevant (CVI = 0.43 to 1.00). The questionnaire was reduced from 63 to 50 items. The CVI of the 50-item questionnaire was not calculated due to the addition of unjudged questions suggested by content experts.
Internal consistency reliability of the questionnaire was 0.81 based on a pilot study conducted at a medium-sized university using undergraduate students across disciplines. Four factors were extracted with Cronbach’s alphas ranging from 0.885 to 0.974. Pilot study data provide tentative evidence that the 30-item FAEQ should elicit valid and reliable results, adequately measuring the quality of academic advising.
The questions retained in the 30-item solution correspond to the initial 10 characteristics of the effective academic advisor (Harrison, 2009a, 2009b). Knowledge and availability questions appear stronger, whereas only one question related to accountability was retained (Table 1). Accountability was unique to the faculty study (Harrison, 2009a) and was not found in the extant advising research or scholarly commentary.