In recent years, the assessment of educational outcomes has received attention from a variety of constituencies. In 1988, policies of the regional accreditation associations for higher education were changed to reflect the emphasis on outcomes assessment by requiring all institutions to develop and implement educational outcomes assessment plans. These accreditation criteria apply to schools of nursing that are located in colleges and universities. In addition, baccalaureate nursing programs are accredited by the National League for Nursing (NLN), using criteria developed and approved by the Council of Baccalaureate and Higher Degree Programs. The NLN is the nationally recognized agency for the specialized accreditation of nursing programs.
The NLN Accreditation Program is founded on the belief that specialized accreditation provides for the maintenance and enhancement of educational quality, provides a basic assurance of program improvement, and contributes to the improvement of nursing practice (NLN, 1992, p. v).
In 1989, the nurse representatives of the Council of Baccalaureate and Higher Degree Nursing Programs of the NLN revised accreditation criteria to include outcome measures. One of these measures is "Required Outcome Criterion 1: Critical Thinking- This outcome reflects students' skills in reasoning, analysis, research, or decision making relevant to the discipline of nursing (NLN, 1992, p. 26). For the self-study report, which is a required component of the accreditation process, the nursing program must document the following:
A. Give the nursing unit's definition of critical thinking appropriate to each nursing program.
B. Provide a rationale and assessment of methods or processes used to evaluate or measure critical thinking.
C. Report critical thinking outcome data and its use in the development, maintenance, and revision of program/s (NLN, 1992, p. 26).
The newly revised criteria were optional for baccalaureate nursing programs scheduled for an accreditation site visit in the fall of 1992 and the spring of 1993. Nursing programs were required to use the revised criteria beginning with accreditation visits scheduled for the fall semester of 1993, and thereafter. With the implementation of the revised NLN criteria, it is timely to examine and describe the prevailing practice of baccalaureate nursing programs with regard to Required Outcome Criterion 1: Critical Thinking.
EVALUATION OF OUTCOMES
There is no universally accepted, all-inclusive framework or set of criteria by which to describe or evaluate an outcome measure of critical thinking. Terenzini and Pascarella (1991) made several recommendations for future assessment studies in higher education that included: differentiation of changes that occur during college versus those that occur due to college; estimation of the magnitude of those changes; examination of when those changes occur, the effects of student characteristics on changes; and the increased use of qualitative in addition to quantitative methods. Lumsden and Knight (1991) recommend the use of multiple measures and locally or faculty developed instruments. Use of individualized, locally developed instruments can enhance the institution's ability to measure outcomes based on its mission and objectives in a meaningful way (El-Khawas, 1992; Moore, 1986). Smith and Weith (1985), Lenning (1988), Lumsden and Knight (1991), and Wilcox and Ebbs (1992) support the concept of assessing value added concepts, such as critical thinking, in concert with academic achievement.
Paul (1993) outlines the recommendations for the assessment of higher order thinking from the Center of Critical Thinking:
1. Inclusion of the elements of thought to be assessed: purpose; question or problem; frame of reference; empirical and conceptual dimensions of reasoning; assumptions; implications and consequences; and inferences and conclusions.
2. Use of intellectual standards, such as accurate, clear, significant, fair, realistic, precise, and so forth.
3. Variety of assessment strategies, such as essays, multiple-choice, and multiple-rating items.
4. Delineation of affective traits, such as fair-mindedness, intellectual integrity, and willingness to suspend judgment.
Paul (1993) notes that use of commercially available, standardized tests presents a twofold problem. They are not based on a comprehensive model, and fail to take into account the recent scholarship on critical thinking.
A survey of nurse educators demonstrated consensus regarding the inclusion of both cognitive abilities and affective qualities in definitions of critical thinking, and raised the question of the need to identify the complexity of patient care situations, or the need to identify context when evaluating critical thinking abilities of nursing students (Videbeck, 1995).
The literature provides support for professional registered nurses to use critical thinking in practice, and therefore defines a need for nurse educators to teach critical thinking skills to nursing students. However, there is no general consensus on the meaning of critical thinking or its relationship, if any, to clinical judgment or decision-making. Nursing is still seeking answers to those questions.
The purpose of this study was to describe prevailing practice in baccalaureate schools of nursing with respect to NLN Required Outcome 1: Critical Thinking. The published list of NLN-accredited baccalaureate schools of nursing was used to identify the nursing programs that were scheduled for an accreditation site visit during the 1992 to 1993 and 1993 to 1994 academic years. Invitations and a follow-up postcard reminder were sent to the Deans and Directors of 124 schools of nursing in 42 states asking them to participate by sending copies of the accreditation self-study report pages that pertained to NLN Required Outcome 1: Critical Thinking. Receipt of the requested materials was considered the consent to participate. Ninety-five programs, or 77%, responded to the letter or postcard. Thirty-one programs indicated that the previous NLN accreditation criteria had been used for the self-study which was an option for programs being visited during the 1992 to 1993 academic year. Nine programs responded that they had used the newly approved criteria, but declined to participate in the study. Fifty-five programs in 30 states submitted the materials requested. The 55 nursing programs in the sample represented the population of 124 nursing programs with respect to college or university affiliation, i.e., religious, public, or private, and the number of enrolled nursing students as demonstrated by a Chi-square test for homogeneity, or "goodness-of-fit."
A framework of criteria to describe prevailing evaluation practices of critical thinking was constructed from recommendations in the higher education literature, the Center for Critical Thinking in Sonoma, California (Paul, 1993), and a survey of nurse educator experts (Videbeck, 1995). This framework provided the initial structure for the content analysis of the data. The framework included:
Definition of critical thinking
Primarily cognitive, or cognitive and affective
Specified to discipline of nursing
Source/author referenced or readily recognized
Critical thinking outcome
Outcome contained in program, curriculum or course objectives
Methods of measurement/evaluation
Single or multiple types of measurement/evaluation
Type(8) of measurement/evaluation used
Timing of measurement
Rationale and assessment of methods/processes used
Use of outcome data for program evaluation
Additional criteria were developed as they emerged from the process of content analysis and are described in the next section.
Analysis of the definitions of critical thinking showed that 43 definitions specified both cognitive abilities and affective qualities, while 12 were primarily cognitive in nature. Twenty-six of the definitions were specific to nursing, that is, they identified nursing activities or stated "relevant to nursing" or "in discipline or practice of professional nursing." Fifty of the definitions referenced adaptation or adoption of a particular author's critical thinking definition, or the source was readily identifiable (Table 1). Twenty-eight of the definitions originated in the nursing literature and twenty-two from authors writing about critical thinking not related to nursing specifically.
The definition of critical thinking in one program was operationalized to establish outcomes to be evaluated. Nineteen programs identified critical thinking outcomes as selected program and/or course objectives. Of these 19 programs, 8 used the term critical thinking in the objectiveis), and 11 selected objectives which were viewed as related to critical thinking, such as problem-solving and decision-making.
Initial categories or types of measurements used in the analysis were: standardized tests (widely or nationally recognized); locally developed instruments; and coursespecific measures, with subdivisions of clinical performance, written tests or examinations, and class or clinical written assignments. During data analysis, the category of capstone projects emerged. These were written and/or oral presentations during the last two semesters of the nursing major designed to demonstrate the cumulative knowledge and abilities of students integrated from the program of study rather than a specific course. Eighteen of the programs used one category or type of measurement and 37 used multiple (two or more) categories of measurement. The measurement categories and timing of measurement are presented in concert due to the relationship between the two factors.
Twenty-nine programs used 44 standardized tests to evaluate students' critical thinking abilities. For four programs, standardized tests were the only category of measurement. Thirteen of the twenty-nine programs used more than one standardized test. Twelve of the 44 tests were administered upon entry and exit to the college/university or the nursing major, yielding a gain or change score. Twenty-six of the tests were administered at the end of the major, two were administered after graduation (NCLEX-RN licensure examination), and four were the NCLEX-RN summary from McGraw-Hill, placed in this category since it is a summary of a standardized test, though not an actual test per se. The frequency with which specific tests were used is shown in Table 2. Programs using the ACT COMP and Collegiate Assessment of Academic Proficiency tests indicated they did so as part of their institutional outcomes assessment plan.
Twenty-two programs used a locally developed instrument with 17 of those instruments being a graduate/alumni and/or employer post-graduation survey. These surveys requested respondents to rate on a Likerttype scale the satisfaction with or the degree to which graduates were prepared to use critical thinking, problem-solving, or decision-making abilities. Five programs used other locally developed instruments that were designed to measure critical thinking abilities. They are as follows:
Source of Definition
1. Faculty developed clinical judgment tool used at the end of each clinical course, and compared for change.
2. Faculty developed Nursing Critical Thinking Appraisal, multiple-choice test administered at end of the major.
3. Critical Thinking Survey, student self-report Likerttype scale administered upon entry into and completion of nursing major.
4. Clinical case study administered upon entry into and completion of nursing major, compared for change/growth.
5. College-based outcome assessment, developed at the college/university and administered prior to graduation for all students at the institution.
Course-specific measures were divided into three subcategories: written tests or examinations; clinical performance; and class or clinical written assignments. Thirtynine programs utilized course specific measures to assess critical thinking. Seventeen of the programs used coursespecific measures in concert with other categories of measurement, 12 used this type of measure exclusively. Twelve programs used all three subcategories of measures, 19 used 2 of the 3 subcategories, and 8 programa used one subcategory of measurement, all of which were class and clinical written assignments. Viewed from another perspective, 38 of the 55 programs used the subcategory of class and clinical written assignments, 24 programs used course tests and examinations, and 20 programs used clinical performance measures, 5 of which were observed clinical performance and 15 of which were case study presentations or clinical conference discussions. Written nursing care plans or case studies were the most frequently used written assignments. Other examples of written assignments were journals or logs, process papers or recordings, managemenVchange papers, teaching projects, small group projects, critique of research literature, and research projects. The timing of the evaluative measures occurred within the courses in the nursing major. Information about the types of items on course tests or examinations was not included in the data sets. Also, in the majority of cases, it was not possible to distinguish whether written class or clinical assignments were quantitative or qualitative measures.
Frequency of Standardized Test Use
Sixteen programs used from one to four capstone projects to evaluate students' critical thinking abilities. Two programs used this category as the sole type of measure. In all cases, the projects were completed during the last one or two semesters of the major, and were not specific to any one course in the nursing major. The 16 programs used 25 capstone projects in their evaluation of critical thinking. The topics and frequency of their use were: research, 7; management/leadership, 5; community, 4; complex topic or issue, 3; and one each of portfolio of paired projects, problem-solving, decision-making, values clarification, interactive video clinical simulation, and a manuscript to submit for publication.
The rationale cited for the selection of the evaluation methods or processes was universal. All 55 programs indicated that they selected their methods and processes of evaluation due to the relationship between the activities being evaluated and the expectations of the professional nurse in practice. Though not all programs explained this relationship by enumerating the specific professional nursing activities to which they referred, 36 programs listed multiple activities. The most frequently cited activities were: nursing process, 33; research, 24; problem-solving, 18; decision-making, 16; and clinical judgments/decisions, 9. Other activities cited 6 or fewer times were: management, ethical decisions, thinking/reasoning skills, analysis and synthesis, and affective components, such as values clarification and creativity.
All programs indicated that the results of evaluation activities would/have been reviewed to determine if courses, teaching strategies, or student learning experiences needed revision to improve students' critical thinking abilities. Not all programs specified the method by which changes would be made. The 38 programs that did indicate specific methods referred to the systematic evaluation plan for the program, review by the faculty as a whole, specific faculty committees, or teaching faculty within a given course.
Programs evidenced the assessment of the cognitive abilities of critical thinking in concert with academic achievement in many of the evaluation methods and processes. All course-specific measures were in concert with academic achievement, as were the NLN-generated and Mosby AssessTest in the standardized tests category. The 18 locally developed instruments which included graduate, alumni, and/or employer surveys and the college-wide tool for all students may have a strong or no relationship with academic achievement, it was not possible to determine any relationship from the data provided. The other four faculty-developed instruments were related to nursing, and therefore could be assumed to be assessments made in concert with academic achievement.
While forty-three programs included both affective qualities and cognitive abilities in their definition of critical thinking, some of these programs used measures generally viewed as primarily cognitive in nature. Most of the definitions provided by programs were global in scope, with only one program providing an operational definition of critical tliinking. Some of the programs included program/curriculum or course objectives to "operationalize" critical thinking, though in some cases, critical thinking was not directly mentioned, but inferred from objectives related to problem-solving, decision-making or use of the nursing process. It was not clear if these activities were seen as synonymous with critical thinking or if they were viewed as a foundation for critical thinking. Use of the nursing process, problem-solving, and decision-making were identified more frequently than clinical judgments when describing activities in nursing practice that required critical thinking, and therefore provided the rationale for their selection as evaluation activities. However, it is possible that some programs would assume inclusion of clinical judgments under nursing process activities without using the term clinical judgments. It was not possible to determine the existence of this assumption in the data provided by the self-study report pages.
The designation of written clinical assignments as evaluation measures for critical thinking was made more frequently than the observation of actual clinical performance. This may be due to the difficulty of a more precise meaning for the performance of clinical judgments in actual clinical or practice situations, or the assumption that a written nursing care plan can adequately provide evidence of clinical judgments.
The sample size of 55 programs and the use of a relatively new criterion where little or no history exists prevent generalization from this study. Also, with one primary investigator, the possibility of bias exists.
There is no well-established framework for the evaluation of a construct such as critical thinking. The framework used in the content analysis methods employed in this study was created for this investigation, and therefore has been used by no one else, or in any other context.
The amount of information in the self-study report sections on critical thinking varied among programs, as did the specificity of the information. This is a limitation of information presented in narrative form. Therefore, more detailed data that exceeded the required content was available from some programs, but not from others.
As programs described their evaluation methods and processes, some programs made very clear distinctions between teaching strategies designed to promote critical thinking or formative evaluation activities versus summative evaluation, or those methods designated as measuring/assessing the outcome of critical thinking. Some of the data could have been open to misinterpretation as the investigator made decisions about distinctions among the various activities described.
The prevailing practice of baccalaureate programs of nursing with respect to the evaluation of critical thinking raises several areas for consideration by nurse educators.
Standardized, paper-and-pencil tests are often selected as an evaluation measure since normative data are available, and reliability and validity have been established. However, none of the available instruments are specific to nursing, and there is no consistent relationship between scores on this type of test and clinical judgment or decision-making (Pless & Clayton, 1993). The use of a preand post-test format with standardized tests to obtain a "gain" score to infer a change in students' critical thinking abilities, must also be considered carefully. For example, instruments designed to measure a particular attribute are static, and there is a high correlation across two points in time if the measure is valid and reliable. Therefore, to pre- and post-test with the same instrument, and use the difference as a change score either yields few or no measurable results of change, or calls into question the reliability and validity of the instrument if the change score is significant (Hanson, 1988).
The use of locally or faculty developed instruments to assess student outcomes is strongly recommended (El-Khawas, 1992; Lumsden & Knight, 1991). However, careful consideration is required if the only instrument of this type that is used is the graduate/alumni/employer survey. This survey provides only self-reported data which alone are insufficient to lead to program revision or improvement.
Course-specific measures, such as clinical performance criteria or written assignments, have the advantage of being very specific to nursing practice expectations. But one semester can be too short a time to detect much change in such a complex construct as critical thinking. Consideration must be given to allow adequate time for students' critical thinking abilities to develop and for faculty to evaluate these changes. One evaluation method that addresses the need for time is the keystone or capstone project that requires students to use knowledge and abilities from many courses, occurs toward the completion of the major, and is not specifically contained in one particular course. Another mechanism to address this consideration is careful planning and coordination of knowledge and experiences to facilitate critical thinking across the curriculum in most or all courses in the nursing major.
Critical thinking as an outcome in nursing education is currently being evaluated using a variety of methods and processes. This is due to the individual differences among programs, different conceptions of critical thinking and how it is evidenced in nursing practice, the relative "newness" of evaluating critical thinking as an outcome measure, and the lack of a model for evaluation of a complex construct such as critical thinking. A model could provide needed direction for the evaluation of critical thinking, for teaching critical thinking, as well as the framework for research. Research is needed to answer many questions, such as, Is there is a relationship between critical thinking and clinical judgment? or nursing process? or problem-solving? What level of critical thinking is appropriate in the undergraduate curriculum? What are the most effective ways of fostering students' critical thinking abilities? If it is valid to assume that nurses need critical thinking skills to take an active, leadership role in health care today, then nursing needs to begin to answer these and many other questions.
- Bandman, EX., & Bandman, B. (1995). Critical thinking in nursing (2nd ed.). Norwalk, CT: Appleton & Lange.
- Bowers, B., & McCarthy, D. (1993). Developing analytic thinking skills in early undergraduate education. Journal of Nursing Education, 32(3), 107-114.
- Brookfield, S.D. (1987). Developing critical thinkers: Challenging adults to explore alternative ways of thinking and acting. San Francisco, CA Jossey-Bass.
- El-Khawas, E. (1992). Campus Trends, 1992. Higher Education Panel Report, Number 82. Washington, D.C.: American Council on Education.
- Ennis, R.H. (1985). A logical basis for measuring critical thinking skills. Educational Leadership, 43(10), 44-48.
- Hanson, G.R. (1988). Critical issues in the assessment of value added in education. In T.W. Banta (Ed.), Implementing outcomes assessment: Promises and perils (pp. 53-67). New Directions for Institutional Research, Number 59. San Francisco, CA: Jossey-Bass.
- Kurfiss, J. (1988). Critical thinking: Theory, research, practice, and possibilities. ASHE-ERIC Higher Education Report No. 2, Washington, DC: Association for the Study of Higher Education.
- Lenning, O.T. (1988). Use of noncognitive measures in assessment. In T.W. Banta (Ed.). Implementing outcomes assessmentPromises and perils (pp. 41-51). New Directions for Institutional Research, Number 59. San Francisco, CA: Jossey-Bass.
- Lumsden, D., & Knight, M.E. (1991). Getting started in outcomes assessment: Setting objectives, selecting instruments, utilizing findings. Assessment update: Progress, trends and practices in higher education, 3(5), 10-11.
- Matthews, CA, & Gaul, A.L. (1979). Nursing diagnosis from the perspective of concept attainment. Advances in Nursing Science, 2(1), 17-26.
- Miller, M.A, & Malcolm, NS. (1990). Critical thinking in the nursing curriculum. Nursing & Health Care, 11(2), 67-73.
- Moore, K.M. (1986). Assessment of institutional effectiveness. In J. Losak (Ed.), Applying institutional research in decision making (pp. 49-60). New Directions for Community Colleges, Number 56. San Francisco, CA: Jossey-Bass.
- National League for Nursing. (1992). Criteria and guidelines for the evaluation of baccalaureate and higher degree programs of nursing education. New York, NY: Author.
- Paul, R. W. (1993). Critical thinking: How to prepare students for a rapidly changing world. Santa Rosa, CA: Foundation for Critical Thinking.
- Pless, B.S., & Clayton, GM. (1993). Clarifying the concept of critical thinking. Journal of Nursing Education, 32(9), 425-428.
- Smith, TB., & Weith, R.A. (1985). Value-added: The student affairs professional as a promoter of intellectual development. NASPA Journal, 23(2), 19-24.
- Terenzini, P.T., & Pascarella, E.T. (1991). Twenty years of research on college students: Lessons for future research. Research in Higher Education, 32(1), 83-92.
- Videbeck, S.L. (1995). Critical thinking as an outcome measure in baccalaureate schools of nursing. Dissertation Abstracts International, 9531798.
- Watson, G, & Glaser, E. (1964). Critical thinking appraisal manual. New York, NY: Harcourt, Brace & World.
- Wilcox, J.R., & Ebbs, S.L. (1992). Promoting an ethical campus culture: The values audit. NASPA Journal, 29(4), 253-260.
- Wilkinson, J.M. (1991). Nursing process in action: A critical thinking approach. Redwood City, CA: Addison- Wesley.
Source of Definition
Frequency of Standardized Test Use