Nursing faculties generally agree that students who know how to think make better clinical judgments than those who have merely memorized facts. Tb meet the challenges of clinical practice, graduates of nursing programs must be capable of applying nursing concepts, learned throughout a nursing curriculum, to the infinite variety of clinical situations that will happen during one's nursing career. Though most nursing faculties acknowledge their responsibility to develop thinking individuals, measuring students' ability to think critically is an ongoing challenge.
Recently, the accrediting body of the National League for Nursing (NLN), the National League for Nursing Accrediting Commission (NLNAC), noted that one criterion for the assessment of educational effectiveness in nursing was "a written plan for the systematic evaluation of all components in the nursing program based on outcomes" (NLNAC, 1997, p. 2). Further, the NLNAC described evaluation of critical thinking as a required outcome that should be evaluated. In contrast, the Commission on Collegiate Nursing Education (CCNE, 1997) did not specifically refer to the measurement of critical thinking in nursing education, but stated that "expected results are the measurable indicators of a program . . . student outcomes may be demonstrated as attained competencies, knowledge, skills, and attitudes" (p. 14). With or without the specific mention of critical thinking, it is clear that accrediting bodies are concerned with measurement of student outcomes demonstrated by the ability to think critically when making clinical judgments.
Much has been written about the definition of critical thinking as well as methods that promote critical thinking. In 1987, a committee of the National Academy of Education strongly recommended the development of subject-specific higher-order thinking tests (Alexander, James, and Glaser, 1987). Within the academic community, considerable debate exists about the nature of critical thinking, the transferability of critical-thinking skills from one setting or situation to another, and the best methods to assess critical-thinking skills. Some describe critical-thinking proficiency as a generalized skill, with applicability to a variety of situations and contexts, and further assert that critical thinking can be tested with general-content, critical-thinking instruments (Ennis, 1993; Facione, Facione, and Sanchez, 1994; Paul, 1987, 1990). Others contend that critical-thinking mastery is improved if developed and assessed within the context of a discipline (Blatz, 1989; McPeck, 1981, 1990).
McPeck asserted that critical thinkers evaluate information in light of background knowledge, context, and reflective skepticism (McPeck, 1981, 1990) and postulated that it is impossible and incoherent to attempt to teach critical thinking in isolation from the skills being taught to students (McPeck, 1981). Further, McPeck noted that "truly suggestive, and therefore useful, thinking skills tend to he limited to specific domains or narrower areas of application" (McPeck 199Oa, p. 12) and that "critical thinking is not a content-free 'general ability", nor is it a set of 'specific skills™ (McPeck, 199Ob, p. 27). McPeck postulated that the development of increasingly sophisticated use of language to express the tenets and principles of the discipline constitutes critical knowledge and represents critical thought (McPeck, 199Ob, p. 40).
Some nurse authors and educators (Alfaro-LeFevre, 1995; Brigham, 1993; Cascio, Campbell, Sandor, Rains, and Clark, 1995; Doona, 1995; Miller and Babcock, 1996; Miller and Malcolm, 1990) have described the context of critical thinking within the discipline of nursing. AlfaroLeFevre (1995) asserted that "a key point to realize is that critical thinking is contextual . . . these skills require job-specific knowledge, and must be mastered within the context" (p. 35). Bandman and Bandman (1995) described the universality of critical thinking and their view of critical thinking as both subject-specific and general.
There are as many definitions of the nature of critical thinking as there are nursing authors writing on the subject. Pacione et al. (1994) remarked that "defined as purposive, self-regulatory judgment, critical thinking is a construct that greatly overlaps the conceptual boundaries of the process nurses call clinical judgment" p. 349). Young (1998) stated that "in our hearts, we know what critical thinking is ... an individual who is able to respond to problems by using the nursing process effectively is a critical thinker" (p. 153). In contrast, Morin (1998) responded that "critical thinking is more than following a problem-solving method ... I believe that the nursing process format makes thinking out-of-the-box (what I consider creative thinking) more difficult" (p. 154). Numerous studies have attempted to correlate the results of general critical thinking tests with those tests that assess decision-making within the context of nursing (Brooks and Shepherd, 1990; Hölzerner and McLaughlin, 1988; Pardue, 1987). In all cases, no correlation, or a weakly positive correlation, was found between general tests of critical thinking and tests of decision- making, prioritization of care, or argument analysis in nursing.
Many nurse educators have written about teaching methods that promote critical thinking. Inventive strategies and suggestions for enhancing critical-thinking skills in nursing students have been developed and described by several nurse writers. Case (1994) suggested that to promote critical-thinking skills in learners, active dialogue between the instructor and the student was vital. Doona (1995) suggested that nursing education could expand the critical-thinking abilities of students by encouraging reflective thinking through such activities as writing of journals and using group discussion to explore alternatives and arrive at conclusions. Free (1997) used a critical-thinking game called What If? What Else? What Then? to encourage students to formulate alternatives to clinical or ethical decisions. Reynolds (1994) described a teaching tool called a "patho-flow diagram" (p. 333), designed to assist nursing students in connecting clinical events or decisions with information obtained in the classroom.
There is considerable agreement among nursing researchers that critical thinking is a vital component of successful nursing practice (Birx, 1993; Brigham, 1993; Jones and Brown, 1993; Pond, Bradshaw, and Turner, 1991; Miller and Malcolm, 1990; Pardue, 1987; Rubenfield and Scheffer, 1995; Tiessen, 1987; Woods, 1993). However, nurse researchers (Saarman, Freitas, Rapps, and Riegel, 1992; Hickman, 1993) have noted with concern the lack of discipline-specific assessment mechanisms to evaluate critical-thinking competency in nursing students. Hickman (1993) stated that:
There is not a strong research base supporting a relationship between nursing curricula and critical thinking. It may be that this is due to the lack of an appropriate instrument to measure critical thinking in nursing (p. 46).
Indeed, a relative paucity of data exists regarding productive ways to measure critical-thinking abilities of nursing students or to evaluate the effectiveness of curricular elements in fostering this important skill. Therefore, an approach to writing test items designed to evaluate students' critical-thinking abilities within the discipline of nursing was developed.
Based on the position that measurement of critical thinking is discipline-specific, the evaluation of nursing students for measurement of critical thinking was examined. Since there is a propensity for schools of nursing to employ the multiple-choice test item format for student evaluation, it was determined worthwhile to investigate a method of ensuring that such test items promote and assess student's ability to think critically.
Using the model described by Morrison, Smith, and Britt (1996), the process of developing critical thinking multiple-choice test items is described. Although difficult to construct, closed-end multiple-choice test items are easy to administer, and with the assistance of scoring and item-analysis computer software programs, they are especially easy to analyze. An item analysis, which includes item discrimination and test reliability, can provide definitive data regarding test items' effectiveness in discriminating between those who know the content and those who do not. Within the context of nursing, knowing implies the ability to apply concepts to clinically-oriented situations.
When properly written, multiple-choice test items can be highly discriminating and can therefore test a student's ability to apply concepts to clinically-oriented situations. Additionally, multiple-choice questions lend themselves to machine grading that can expedite the scoring of exams and reporting of grades.
WRITING TEST ITEMS
To begin the process of writing test items that measure students' ability to think critically within the discipline of nursing, one must first have knowledge of current clinical practice. Morrison, et al. (1996) reported that:
There is no substitute for clinical knowledge; it is the foundation upon which good critical thinking test items are developed" (p. 15).
Sound clinical knowledge takes years to develop and perfect, whereas the basics of curriculum design and testitem writing can be imparted within the context of faculty inservice education programs. Schools of nursing, as well as their students, are therefore well served when excellent clinicians are hired and retained as nursing faculty and when deans and directors take responsibility for developing such clinical experts into capable faculty members with the skills necessary to develop criticalthinking test items. Likewise, experienced faculty, responsible for teaching clinical courses must maintain and update their knowledge and expertise in the clinical areas they are teaching if they are to write effective, practice-oriented test items.
Morrison, et al. (1996) described four criteria for critical-thinking multiple-choice test items:
* Include rationale for each test item.
* Write questions at the application or above cognitive level.
* Require multilogical thinking to answer questions.
* Require a high level of discrimination to choose from among plausible alternatives (p. 21).
The main purpose of testing is evaluation. However, testing can also serve as a learning tool. There is no doubt that test items influence students' learning, and generally, faculties intend that such learning should be positive. Providing rationales for each test item helps to make testing a positive learning experience, while at the same time serves the evaluative purpose for which the exam was designed. In keeping with concepts of critical thinking, students should be encouraged to analyze how they think by describing why they chose a certain answer. Such an examination of one's thinking process is one of the definitions of critical thinking described by Paul (1993):
Thinking about your thinking in order to make your thinking better: more clear, more accurate, or more defensible (p. 462).
Test-item rationales should contain information as to why a choice is correct as well as why distracters are incorrect. Carefully written rationales can provide teaching and may help to increase students' learning while perhaps decreasing the time faculty spend reviewing the content contained in the rationale. Developing such rationales may seem like an awesome task, especially when faculties are experiencing normal time constraints related to conducting class, supervising students in clinical settings, counseling students, and developing exams. However, in the long run, developing rationales for each test item can save faculty time in explaining test questions as well as decrease the time spent by students arguing with faculty about test items.
Students should be permitted to view test items and rationales for test items after the test analysis process has been completed. However, test reviews are usually ineffective in promoting critical thinking when they are conducted for the entire class with one faculty member attempting to answer questions and perhaps defend test items. Such reviews can serve to reinforce argumentative behavior, which can in turn result in hostile feelings between faculty and students. At no time should the class, as a whole, be permitted to argue with a faculty member regarding test items. Policies and procedures to formally protest a test item's correctness should be developed along with test review procedures that ensure security and promote student learning.
An effective method for conducting a test review is to tape pages of the test to desktops or to the wall of a classroom and allow a limited number of students at a time to view the test items and rationales. To maintain security, the test review should be supervised by faculty and conducted in silence with no books or paper permitted in the room where the review is conducted. Such a review enables students to learn from the testing process, while at the same time maintaining security and providing order to the review process. Additionally, if class time is a factor, the review can be conducted before or after class.
Computer programs, such as ParSYSTEM®, a test item banking and analysis program distributed by Scantron®, generate reports that are helpful in conducting a test review. Test items are stored in item banks along with rationales, and a report of the rationales contained for each test item can be printed. The system also provides a printout of students' individual test scores and identifies the correct answer for each test item and the answer that was chosen by the student. Such computer-generated reports can be helpful with producing and reporting students' scores as well as with the test review process. Though more laborious, scores and test reviews of rationales can also be developed manually. Whether computer generated or manually generated, the test review process is best conducted in silence so students have the opportunity to read the content of the questions and the rationale for each item. Such a review process encourages the student to think critically. By reviewing each test item's rationale, the students are analyzing how they reached a certain conclusion and are "thinking about their thinking." Therefore, according to Paul (1993), the student is thinking critically.
Cognition at the Application Level and Above
Critical-thinking test items should focus on application of concepts. Memorizing implies "habitual thinking," described by Bloom (1956) as knowledge, the lowest level of cognition. Comprehension, Bloom's second level of cognition, is essentially a restatement of memorized facts. Critical-thinking test items should be written at higher cognitive levels than both "habitual thinking" and "comprehension." Based on Bloom's Taxonomy (1956), criticalthinking test items should be written at the application level and above.
Verbs Associated with Categories of Cognition for Bloom's Taxonomy of Educational Objectives
Table 1 describes verbs for each of Bloom's cognitive levels. Though these verbs might be helpful when writing test items, it must be remembered that only the faculty responsible for presentation of the content being tested can authentically categorize a test item's cognitive level. If a test item is reviewed prior to administrating the test containing the item, then no matter what verb is used, the reviewed test item automatically becomes a mere regurgitation of facts or a knowledge-based test item. It is important, therefore, for faculty to determine the cognitive level of a test item in relation to presentation of the content.
Paul (1993) defines multilogical thinking and multilogical problems as one's ability to view problems from different perspectives. Morrison et al. (1996) narrowed Paul's definition and incorporated Bloom's description of cognitive levels to create a definition of multilogical thinking that can be used for the purpose of writing critical thinking test items:
Thinking that requires knowledge of more than one fact to logically and systematically apply concepts to a clinical problem (p. 28).
An easy way to determine if a question meets the criterion of requiring multilogical thinking to answer is to ask if the student must know more than one concept, construct, or fact to answer the test item. If the answer to a test item can be found on one page of a text, it is not a critical-thinking test item. In such a case, the item would be knowledge-based because it would only require memorizing that portion of the text, and it would therefore not meet the criterion for multilogical thinking. To challenge students to think critically, test items must require the ability to relate and apply concepts to clinically-oriented situations. Such multilogical test items promote critical thinking as well as measure the students' ability to think critically within the discipline of nursing.
An example of a test-item stem that would require multilogical thinking is:
Which assessment finding is most important in determining nursing care for a client with chronic obstructive pulmonary disease (COPD)?
To answer this question, the student would have to know about COPD as well as the nursing care involved in caring for such a client. This question is different from one that might simply ask:
Which assessment finding would the nurse expect a client with COPD to exhibit?
Such a question would only require knowledge of the signs and symptoms of COPD.
High Level of Discrimination
To promote critical thinking, multiple-choice test items should require a high level of discriminating judgment to answer. The student should be required to discriminate from among plausible alternatives. In other words, all choices provided should be relatively possible, but one choice should be better than the others. Questions that require calculation or ask what is the best, most important, first, highest priority, and so forth, require a high level of discrimination to answer, and therefore promote critical thinking. For example:
In responding to a client who describes himself as being able to read minds because he is the Pope, it is most important for the nurse to demonstrate which type of behavior?
This test-item stem could be a highly discriminating item if all choices are plausible. The student would have to know how to deal with delusional clients and would have to pick the behavior that is most important for the nurse to demonstrate based on concepts of therapeutic communication. Table 2 provides samples of highly discriminating test-item stems, which can assist in writing critical thinking test items. However, faculties know that the most difficult part of developing a good multiplechoice test item is writing the distracters. Clinical practice settings are the best resources for such distracters. Students' nursing care plans, as well as nursing rounds conducted by students, can provide a plethora of distracters. Gathering these incorrect statements or practtices and filing them under subject topics may help to make writing distracters a little easier.
Highly Discriminating Test-Item Stems
Highly Discriminating Test-Item Stems
Table 3 describes two test items covering the same content and compares the ability of each test item to promote critical thinking within the discipline of nursing.
Faculties are concerned about measurement of critical thinking for a variety of reasons. First and foremost is the desire to prepare capable practitioners of nursing who can use the language of nursing to apply facts and concepts to clinical problems. Such ability requires critical thinking. More concretely, faculties are also concerned about meeting accreditation requirements of the NLNAC (1997). For those who subscribe to the position that mastery of critical thinking must be assessed within the context of a discipline, then the measurement of nursing students' ability to think critically must be conducted within the discipline of nursing. Faculties are hired to teach nursing because of their education and expertise within the field of nursing. Therefore, measuring students' ability to think critically within the discipline of nursing is within the purview of nursing evaluation. In contrast, general content critical-thinking instruments do not measure what faculties are hired to teach. Rather, such instruments test individuals' general critical-thinking ability. The degree to which a student possesses the abilities tested with general-content critical-thinking instruments is likely to be related to skills that were developed prior to entering a school of nursing. Such scores have nothing to do with the nursing education that students paid for when enrolling in a college or university. It is nursing faculties' responsibility to promote critical thinking within the discipline of nursing. More specifically, it is faculties' responsibility to facilitate students' self-sufficiency and autonomy and their ability to make judgments regarding the practice of nursing. It follows, therefore, that critical thinking cannot be measured pre and post entrance into a nursing program, rather it must be measured systematically, as an ongoing process in relation to the nursing content being tested. As nursing knowledge increases, so should students' ability to use critical-thinking skills with an increasing variety of nursing problems. Therefore, it behooves nursing faculty to develop critical-thinking evaluation tools within the discipline of nursing. Writing critical-thinking multiple-choice test items using the four criteria described by Morrison et al. (1996) is one means of measuring students' critical-thinking skills. When students view rationales for test items and "think about their thinking" (Paul, 1993), they are developing critical-thinking skills. Test items that are written at Bloom's (1956) application and above cognitive level require students to use the language of nursing, to apply concepts related to nursing, and to make judgments related to nursing problems. To make such applications, the student must have the ability to think critically. Critical-thinking skills are also required to answer multilogical thinking test items since the student must have knowledge of more than one fact to logically and systematically apply concepts to a clinical problem. Test items that are highly discriminating, which require calculation or ask such questions as "What is the best, first, most important, etc." require students to discern from among plausible alternatives, thereby requiring them to think critically. Implementing these four criteria into test item writing practices helps to ensure that critical-thinking test items are being written. It follows, therefore, that test items that require critical thinking to answer, measure not only the students' nursing knowledge, but also measure the students' criticalthinking abilities within the discipline of nursing.
Comparison of Two Test Items in Ability to Promote Critical Thinking
- Alexander, L., James, H.T., & Glaser, H. (1987). The nation's report card: Improving the assessment of student achievement. Cambridge, MA: National Academy of Education.
- Alfaro-LeFevre, R. (1995). Critical thinking in nursing: A practical approach. Philadelphia: W.B. Saunders.
- Bandman, E.L., & Sandman, B. (1995). Critical thinking in nursing. Norwalk, CN: Appleton and Lange.
- Birx, E. (1993). Critical thinking and theory-based practice. Holistic Nursing Practice, 7(2), 21-27.
- Blatz, C.V. (1989). Contextualism and critical thinking: Programmatic investigations. Educational Theory, 35(2), 107119.
- Bloom, B.S. (1956). Taxonomy of educational objectives: The classification of educational goals. Handbook 1: Cognitive domain. New York, NY: McKay.
- Brigham, C. (1993). Nursing education and critical thinking: Interplay of content and thinking. Holistic Nursing Practice, 7(1), 48-54.
- Brooks, K.L., & Shepherd, J.M. (1990). The relationship between clinical decision-making skills in nursing and general critical thinking abilities of senior nursing students in four types of nursing programs. Journal of Nursing Education, 29, 391399.
- Cascio, R.S., Campbell, D-, Sandor, M.K., Rains, A.P., & Clark, M. C. (1995). Enhancing critical thinking skills. Nurse Educator, 20(3), 38-43.
- Case, B. (1994). Walking around the elephant: A critical thinking stategy for decision making. Journal of Continuing Education in Nursing, 25, 101-109.
- Commission on Collegiate Nursing Education. (1997). Standards for accreditation of baccalaureate and graduate nursing education programs. Washington: CCNE Publications.
- Doona, M. E. (1995). Nurses' judgment as they care for persons who exhibit impaired judgment: A phenomenological study. Journal of Professional Nursing, 11, 98-109.
- Ennis, R.H. (1993). Critical thinking assessment. Theory Into Practice, 32, 180-186.
- Facione, N.C., Facione, P.A., & Sánchez, CA. (1994). Critical thinking as a measure of competent clinical judgment: The development of the California critical thinking dispositions inventory. Journal of Nursing Education, 33, 345-349.
- Free, K. (1997). Teaching tools: What If? What Else? What Then? A critical thinking game. Nurse Educator, 22(5), 9-12.
- Hansen, J. (1997). Quality multiple-choice questions: Item-writing guidelines and an analysis of auditing test banks. Journal of Education for Business, 73(2), 94.
- Hickman, J.S. (1993). A critical assessment of critical thinking in nursing education. Holistic Nursing Practice, 7,36-47.
- Holzerner, W., & McLaughlin, F. (1988). Concurrent validity of clinical simulations. Western Journal of Nursing Research, 10,78-83.
- Jones, SA., & Brown, L.N. (1993). Alternative views on defining critical thinking through the nursing process. Holistic Nursing Practice, 7, 71-76.
- McPeck, J.E. (1981). Critical thinking and education. New York: St. Martin's Press.
- McPeck, J.E. (1990). Critical thinking and subject specificity: A reply to Ennie. Educational Researcher, 4(2), 10-12.
- McPeck, J.E. (1990). Teaching critical thinking. New York: Routledge.
- Miller, M.A., & Babcock, D.E. (1996). Critical thinking applied to nursing. St. Louis: Mosby.
- Miller, M.A., & Malcolm, N.S. (1990). Critical thinking in the nursing curriculum. Nursing and Health Care, 11(1), 67-73.
- Morin, K. (1998). Response to the importance of critical thinking. (Letter to the editor). 37(4), 154.
- Morrison, S., Smith, P., & Britt, R. (1996). Critical thinking and test item writing. Houston: Health Education Systems, Inc.
- National League for Nursing Accrediting Commission. (1997). Interpretive guidelines for standards and criteria: 1997. (Online). Available: www.NLN.org.
- Pardue, S.F. (1987). Decision-making skills and critical thinking ability among associate degree, diploma, baccalaureate, and master'e-prepared nurses. Journal of Nursing Education, 26, 354-361.
- Paul, R. W. (1987). Dialogical thinking: Critical thought essential to the acquisition of rational knowledge and passions. In J. Baron & R. Sternberg (Eds), Teaching thinking skills: Theory and practice. New York: W.H. Freeman.
- Paul, R. W. (1990). Critical thinking, Rohnert Park, CA: Center for Critical Thinking and Moral Critique.
- Paul, R. W. (1993). Critical thinking: What every person needs to survive in a rapidly changing world (rev. 3rd ed.). Santa Rosa, CA: The Foundation for Critical Thinking.
- Pond, E.F., Bradshaw, M.J., & Turner, S.L. (1991). Teaching strategies for critical thinking. Nurse Educator, 16(3), 18-22.
- Reynolds, A. (1994). Patho-flow diagraming: A strategy for critical thinking and clinical decision making. Journal of Nursing Education, 33, 333-336.
- Rubenfield, M.G., & Scheffer, B.K. (1995). Critical thinking in nursing: An interactive approach. Philadelphia: J.B. Lippincott.
- Saarman, L., Prietas, L., Rapps, J., & Riegel, B. (1992). The relationship of education to critical thinking ability and values among nurses: Socialization into professional nursing. Journal of Professional Nursing, 8(5), 26-34.
- Tiessen, J.B. (1987). Critical thinking and selected correlates among baccalaureate nursing students. Journal of Professional Nursing, 3, 118-123.
- Woods, J.H. (1993). Affective learning: One door to critical thinking. Holistic Nursing Practice, 7, 64-70.
- Young, D. (1998). The importance of critical thinking. (Letter to the editor). Journal of Nursing Education, 37(4), 153.
Verbs Associated with Categories of Cognition for Bloom's Taxonomy of Educational Objectives
Highly Discriminating Test-Item Stems
Highly Discriminating Test-Item Stems
Comparison of Two Test Items in Ability to Promote Critical Thinking