Journal of Nursing Education

The articles prior to January 2012 are part of the back file collection and are not available with a current paid subscription. To access the article, you may purchase it or purchase the complete back file collection here

Development of an Instrument to Measure Effectiveness of Clinical Instructors

Mary M Reeve, EdD, RN

Abstract

ABSTRACT

Evaluation of clinical instructors by their students is an important issue in nursing education. This study describes the outcome of the development of a method by which nursing students can evaluate their clinical instructors with a reliable and valid instrument. The instrument was developed using characteristics from the literature, which were rated in importance by students, faculty, and new graduates. The ratings are then subjected to statistical analysis using frequency, means, alpha coefficient, and item-to-total correlation and expert assessment to provide reliability and validity for the instrument and to aid in reducing the original list of 50 characteristics to a more usable number of 27. The instrument is pilot tested once with students rating clinical faculty. Results from the pilot are provided.

Abstract

ABSTRACT

Evaluation of clinical instructors by their students is an important issue in nursing education. This study describes the outcome of the development of a method by which nursing students can evaluate their clinical instructors with a reliable and valid instrument. The instrument was developed using characteristics from the literature, which were rated in importance by students, faculty, and new graduates. The ratings are then subjected to statistical analysis using frequency, means, alpha coefficient, and item-to-total correlation and expert assessment to provide reliability and validity for the instrument and to aid in reducing the original list of 50 characteristics to a more usable number of 27. The instrument is pilot tested once with students rating clinical faculty. Results from the pilot are provided.

Introduction

A large percentage of student contact time for most nursing faculty is spent in clinical settings. It is important that there be a method by which students can evaluate the effectiveness of their clinical instructors. Evaluation of faculty is important in higher education for accountability as well as faculty growth (Centra, 1980; Miller, 1987). Many universities have reliable and valid instruments for faculty evaluations in classrooms but do not have established instruments that are appropriate for use in clinical settings.

This paper focuses on the development of an instrument to measure the effectiveness of clinical instructors at a large state university in northern California. Gien (1991) provides guidelines that can be used in selecting or developing an instrument. It should be (a) multidimensional, measuring as many desirable behaviors as possible, (b) reliable, (c) stable with a correlation between end-of-term and retrospective ratings, (d) valid against other indicators of faculty effectiveness, (e) useful for faculty feedback, and (f) evaluate the instructor and not the course in the space of one page maximum.

Five characteristics have been regularly and consistently attributed to excellent college teachers according to Sherman, Armistead, Fowler, Barksdale, and Reif (1987). These five characteristics are "enthusiasm, clarity, preparation/organization, stimulating, and love of knowledge" (p. 67). Centra (1980) summarized additional characteristics of effective teachers to include communication skills, favorable attitudes toward students, knowledge of the subject, organization of subject matter, enthusiasm, fairness, flexibility, encouragement of students to think for themselves, and being an interesting lecturer.

In clinical nursing there are few studies that define and evaluate teaching effectiveness in the clinical area (Stafford & Graves, 1978, p. 495). Zimmerman and Waltman (1986) compiled a categorization of overall characteristics for clinical instructors including such items as being available to students, professional competence, interpersonal relationship, teaching practice, personal characteristics, and evaluation practices. Five general categories of teaching ability, nursing competence, evaluation, interpersonal relationships, and personality were identified by Knox and Morgan (1987) and replicated by Nehring (1990).

In summary, the basic characteristics in the literature that are appropriate for clinical instructors to possess were identified. The instructor should have a professional manner and the ability to function as a role model, the ability to provide appropriate evaluation and feedback, to be a resource person and be able to assist students to meet objectives without taking over, to have good interpersonal relationships and communication skills, to display respect for the student, and to have appropriate teaching practices (Bonapart, 1977; Brown, 1981; Brown & Hayes, 1979; Griffith & Bakanauskas, 1983; Knox & Morgan, 1985; Meleca, Schimpfhauser, Witteman, & Sachs, 1981; Van Ort, 1983).

Table

TABLE 1Characteristics: 50 Original and 27* Chosen by Participants

TABLE 1

Characteristics: 50 Original and 27* Chosen by Participants

Methodology

Phase I Initial 50 Questions

Specific characteristics of teachers in classroom and nonclassroom settings such as clinical were identified using a thorough literature review (Bonapart, 1977; Brown, 1981; Brown & Hayes, 1979; Griffith & Bakanauskas, 1983; Knox & Morgan, 1985; Meleca et al., 1981; Van Ort, 1983). These characteristics or categories were used to develop an evaluation instrument.

For the next step, the faculty members in the Department of Nursing in a large California State University were given the list of characteristics of an effective clinical instructor and were asked to review them, to comment on appropriateness, and to add to the list as desired. Based on the literature review and the faculty input, a list of items to match characteristics for a clinical instructor was compiled. To avoid a halo effect and to increase reliability, four "lie" or negatively worded characteristics were interspersed within the list. This was done to assess that the raters were reading critically and not marking all one answer, which produces a halo effect. The halo effect means a tendency for a factor in a unit of study to influence the relevant feature in a favorable or unfavorable direction. Typically, a strong initial positive or negative impression may influence rating on all subsequent observations (Table 1).

Six education experts from the university (four nurses and two non-nurses from the instructional resources division of the university) were independently asked to rate the characteristics. This step helped increase the content validity of the findings.

The 50 characteristics were matched to verify that they covered the categories of effective clinical instructors from the literature. Some of the questions fit in more than one category. The characteristics, as found in the literature, were grouped into categories and questions were written to cover each. The categories and the related questions are summarized in Table 2.

Phase II First Rating of Instrument

Questionnaires were distributed to the faculty (n = 42) and undergraduate students including advanced placement RNs (n = 420), and mailed to two classes of recent graduates (n = 142). Faculty, students, and recent graduates then rated on a five-point Likert scale (1 = least desirable, 5 = most desirable) as to how important each characteristic was for a clinical instructor to possess. When the completed computer answer sheets were returned, a set of 11 subfiles was created, one for each of the semester levels, the faculty, and the recent graduates. Thus, subgroups could be compared in addition to seeing the results of the total responses.

Phase III Statistical Analysis

Statistical analysis (frequency, alpha coefficient, means, standard deviations, item-to-total correlation) using SPSS was completed to obtain reliability data. This was done to reduce the number of desirable characteristics from the original 50. Reduction criteria were based on retaining items with a mean of 4.0 or higher, standard deviation of .79, and a frequency of 160 responses or higher in the 4 (considerable importance) to 5 (most importance) range. In addition, at least 50% of the expert faculty rated the characteristic as important. An alpha rating of at least .85 and item-to-total correlation close to the range of .30 to .70 were also used (Hinshaw & Atwood, 1982; Ritter, 1975). A factor analysis was also completed.

Table

TABLE 2Characteristics From Literature and Related Item Number

TABLE 2

Characteristics From Literature and Related Item Number

Phase IV Second Rating of Instrument

The Instrument was reduced to 27 questions using the established criteria. It was then resubmitted to the available faculty and students for a second rating.

Phase V Pilot Evaluation of Faculty with Instrument

All clinical faculty during the spring semester were asked to be rated by their current students using the 27-item characteristic tool. The data were then analyzed to assess that the instrument did provide discrimination when comparing the results of each questionnaire against the results of the others.

Results

Results of Phase II First Rating of Instrument

In the first rating of the 50 characteristics, the population responding was 324 (77%) of the undergraduates, 24 (57%) of the faculty, and 45 (33%) of the new graduates. The alpha was .93, with at least 160 (50%) of the raters and four of the experts rating the characteristic in the range labeled "considerable importance*' to "most importance."

Table

TABLE 3Means, Standard Deviations, Frequency of Responses and Number of Expert Responses for the Most Positive and Most Negative Responses

TABLE 3

Means, Standard Deviations, Frequency of Responses and Number of Expert Responses for the Most Positive and Most Negative Responses

Table

TABLE 4Quartimax Rotated Factor

TABLE 4

Quartimax Rotated Factor

Results of Phase IV Second Rating of Instrument

Based on these results, the number of characteristics was reduced to 27 items. In the second rating of the 27 characteristics, 297 (64%) of the undergraduates and 24 (57%) of the faculty responded. The alpha was .94 with 21 of 27 fitting in the range of .30 to .70 item-to-total correlation (Table 3).

Using the results from the reduced number of characteristics as rated by faculty, students, and new graduates, factor analysis was completed with the rotated Quartimax showing the cleanest loading. The first factor (Eigenvalue= 11.11) accounted for 84% of variance. None of the remaining three factors was statistically significant. The single factor accounting for 84% of variance was classified by the researcher as an "effective clinical teacher." See Table 4 for the outcome of the factor load.

Results of Phase V Pilot Use of the Instrument

The 27-item characteristics instrument on teacher effectiveness in the clinical setting was administered to clinical groups at the end of a spring semester. The population available consisted of 34 faculty and 319 students. AU clinical faculty were invited to participate but only 18 instructors accepted the invitation. The population finally completing the evaluation of teachers was 205 (64%) students and 18 (43%) faculty including three who taught and were thus rated in two different clinical levels. The rating scale ranged from 1 (far below average) to 5 (excellent). Although an evident halo effect was seen in the results of the faculty evaluation, it occurred at different levels of students (i.e., juniors versus sophomores) and with different faculty.

Table

TABLE 5Examples of Individual Ratings of Pilot Testing of Clinical Evaluation of Teacher Effectiveness

TABLE 5

Examples of Individual Ratings of Pilot Testing of Clinical Evaluation of Teacher Effectiveness

Three teachers were rated at different clinical levels (e.g., sophomores and juniors). All three faculty received high ratings in the 4 (above average) and 5 (excellent) range. The remaining faculty were rated by only one clinical group and received ratings from 1 (far below average) to 5 (excellent) with a predominance in 3 (average) to 5 (excellent) range. It appears that one student rated Faculty "K* in the 1 (far below average) or 2 (below average) range for all 27 characteristics. These ratings represent a negative halo effect with the small number rating (7) causing the very skewed negative results for that individual faculty member (Table 5).

Demographic information for the respondents was as follows: the student population ranged in age from 20 to 60 with a mean of 26, had nursing background experience from one semester to 20 years (for some of the RNs), were primarily female with 25 males involved, included over 30 who were working on a second bachelor's degree and two who had master's degrees in other fields. The new graduates were a similar makeup with between eight and 18 months of professional experience in the field of nursing in a variety of settings. The faculty had a minimum of a master's degree (n = 20) with 10 in the process of obtaining doctorates and 12 with earned doctorates. Teaching experience ranged from one to 27 years in a wide variety of specialities.

Discussion

Instrument Development

As previously discussed, the items were based on the characteristics from the literature and faculty input. Items relating to interpersonal relationships scored high (i.e., honesty, being sensitive to student needs, demonstrating respect for the student, interacting well with students, and respecting confidentiality). The literature indicated that a sense of humor was highly regarded but this study refuted that finding with only a third of the responses for that item being positive.

The ability of the faculty to evaluate students was not always viewed strongly by students. Using appropriate evaluation criteria and clarifying course objectives did not rank high; however, providing appropriate feedback did. The literature also indicated that faculty should have the ability to use and relate to the conceptual framework of the program and that this should be included in the evaluation instrument. However, the two questions relating to the conceptual framework of the program did not receive support from students or faculty in either rating.

Different levels of students stressed different characteristics as being important. For example, the younger, less experienced individuals felt it was very important for the faculty to be available and to assist them. The seniors, advanced placement RNs, and new graduates downplayed this characteristic. All levels felt it was important for the faculty to function as a role model.

Pilot Testing of Evaluation Tool

It is important to remember that the small number of students in the clinical groups (often 10 or less) could lead to skewed results from evaluation instruments. With 10 or fewer ratings, results should be interpreted with caution as one overrater or one underrater can change the totals more easily, as in the case of Faculty "K."

As indicated, the instrument was used to evaluate faculty at the end of the spring semester. The totals for each of the 27 characteristics on which the faculty were rated provided discrimination (Table 5). Nine of the 27 characteristics were rated in the 1 (far below average) category for at least one faculty and each of the other categories of 2 (below average), 3 (average), 4 (above average), and 5 (excellent) were used (Table 6).

One of the possible reasons many of the faculty chose not to participate in the pilot study was because the researcher was able to see the results. Traditionally, the official evaluation results from the university are known only to the faculty member and the chair of the department. The teachers who were rated volunteered to be so rated and are primarily the faculty members who generally receive high ratings and thus did not mind making their ratings known to others. The results would probably show more discrimination if all faculty were rated.

Summary

This study provided the School of Nursing of this university with a reliable and valid instrument by which students could evaluate faculty in the clinical settings. It was not normed, as only one rating was acquired in the pilot testing. As with any new tool, for the first one to two years that it is in use, it should be used for formative evaluation and personal growth only. There will not be adequate norms available to use it for summative, personnel decisions such as retention and promotion. When adequate numbers are available, the norming will have to be done using department ratings only, as this is an instrument specific to the School of Nursing. Once norms are established, faculty would strive to be able to report results in the "average" to "above average" to "excellent" range for inclusion in their personnel action file.

As Gien (1991) wrote, the instrument could be used mid-term as well as end of term to provide a correlation. It must be valid against other indicators of faculty effectiveness so the results would only be one factor in evaluation of faculty. The results could and should be used to provide feedback to faculty to assist them in improving teaching methods. Finally, as based on literature review categories, this instrument does evaluate the faculty and not the course.

Table

TABLE 6Combined Totals of Responses of Students' Evaluations of Clinical Instructors (205 raters)

TABLE 6

Combined Totals of Responses of Students' Evaluations of Clinical Instructors (205 raters)

References

  • Bonapart, T.H. (1977). Characteristics of the ideal clinical teacher. Paper presented at the meeting of the National League for Nursing, New York, NY.
  • Brown, D., & Hayes, E. (1979). Evaluation tools: Student's assessment of faculty. Nursing Outlook, 27, 778-781.
  • Brown, S. (1981). Faculty and student perceptions of effective clinical teacher. Journal of Nursing Education, 20, 4-15.
  • Centra, J. (1980). Determining faculty effectiveness. San Francisco: Jossey-Base.
  • Gien, L.T. (1991) Evaluation of faculty teaching effectiveness: Toward accountability in education. Journal of Nursing Education, 30, 92-94.
  • Griffith, J., & Bakanauskas, A. (1983). Student-instructor relationships in nursing education. Journal of Nursing Education, 22, 104-107.
  • Hinshaw, A.A., & Atwood, J.A (1982). Patient satisfaction instrument: Precision by instrumentation. Nursing Research, 31, 170-175.
  • Knox, J., & Morgan, J. (1985). Important clinical teacher behaviors as perceived by university nursing faculty, students, and graduates. Journal of Advanced Nursing, 10, 25-30.
  • Knox, J., & Morgan, J. (1987). Characteristics of "best' and 'worst' clinical teachers as perceived by university nursing faculty and students. Journal of Advanced Nursing, 12, 331-337.
  • Meleca, B, Schimpfhauser, F., Witteman, J., & Sachs, L. (1981). Clinical instruction in nursing: A national survey. Journal of Nursing Education, 20, 32-39.
  • Miller, R. (1987). Evaluating faculty for promotion and tenure. San Francisco: Jossey-Bass.
  • Nehring, V. (1990). Nursing clinical teacher effectiveness inventory: A replication study of the characteristics of "best' and "worst' clinical teachers as perceived by nursing faculty and students. Journal of Advanced Nursing, 15, 934-940.
  • Ritter, N. (1975). Development of an instrument to measure patient satisfaction with nurses and nursing care in primary care settings. Nursing Research, 24, 45-52.
  • Sherman, T., Armistead, L.P., Fowler, E, Barksdale, M. A, & Reif) G. (1987). The quest for excellence in university teaching. Journal of Higher Education, 48, 65-84.
  • Stafford, L., & Graves, C.C. (1978). Some problems in evaluating teaching effectiveness. Nursing Outlook, 26, 494-497.
  • Van Ort, S. (1983). Developing a system for documenting teaching effectiveness. Journal of Nursing Education, 22, 324-328.
  • Zimmerman, L., & Waltman, L. (1986). Effective clinical behaviors of faculty: A review of the literature. Nurse Educator, 11, 31-34.

TABLE 1

Characteristics: 50 Original and 27* Chosen by Participants

TABLE 2

Characteristics From Literature and Related Item Number

TABLE 3

Means, Standard Deviations, Frequency of Responses and Number of Expert Responses for the Most Positive and Most Negative Responses

TABLE 4

Quartimax Rotated Factor

TABLE 5

Examples of Individual Ratings of Pilot Testing of Clinical Evaluation of Teacher Effectiveness

TABLE 6

Combined Totals of Responses of Students' Evaluations of Clinical Instructors (205 raters)

10.3928/0148-4834-19940101-05

Sign up to receive

Journal E-contents