The Journal of Continuing Education in Nursing

The articles prior to January 2012 are part of the back file collection and are not available with a current paid subscription. To access the article, you may purchase it or purchase the complete back file collection here

Criterion Referenced Measurement for One Continuing Education Offering: Pre- and Postoperative Visits by Operating Room Nurses

Julie Kneedler

Abstract

In the last decade greater emphasis has been put on the measurement of achievement. Perhaps the advent of behavioral objectives has provided the impetus. None the less, achievement measures are now becoming a separate entity from aptitude measures. Achievement measures are concerned with characteristics and properties of present performance with emphasis on the content. Aptitude measurement in contrast is the demonstrated relationship between present performance and future attainment of knowledge and skill. The term which has been given to the measurement of a student's performance in relation to objectives of specified content is criterion referenced measurement. Criterion refers to a set standard or behavior. This standard does not have to be an end of course standard or behavior, as levels of performances can be measured at any point of instruction.

In order to provide a clear understanding of the two types of measures frequently utilized, I will refer to Glaseas statement about what each one does:

Criterion referenced measures indicate the content of the behavioral repertory, and the correspondence between what an individual does and the underlying continuum of achievement.

Norm referenced measures tell that one student is more or less proficient than another, but do not tell how proficient either of them is with respect to the subject matter tasks involved.1

Because instructional specialists have started to distinguish between these two approaches to measurement many questions have been asked in terms of the framework, construction, and revision of criterion referenced measurement

Before these questions can be answered it is important to explore how criterion referenced measurement is used. Criterion referenced measurement is used to determine an individual's status in relation to specific determined criterion. This cou Id be reflected as a continuum of knowledge which ranges from no proficiency to perfect performance. Each individual's achievement level then falls at some point on the continuum. In order to measure the achievement level the instructor must first develop objectives which he intends to be met at the end of the course. The end result will then be a measurement of the learner's performance based on achievement of the original objectives.

We can readily see then that criterion referenced measurement is invaluable to evaluate the learner's performance based on objectives and to evaluate instructional material. We are most interested in what the individual can do, not how he stands in comparison to others. An example could be the driver's license test, where an individual must display certain driving skills to pass the examination without any consideration of how well others perform on the test.

The scope of criterion referenced measures include establishment of instructional behavioral objectives, teaching content which is valid in terms of the objectives, and finally developing a test which is called criterion referenced test The tests used are devised to make decisions about individuals and instructional programs. The test is a method of determining whether a learner has mastered the objective or criterion skill.

In terms of the instructional program one could design a measure which reflected a set of objectives supposed to be achieved. The measure is then administered to appropriate learners after one segment of instruction has been completed. Methods for evaluating the instructional programs are needed and are valuable for the educator.

THE LEARNER

The nurse participants at continuing education offerings are adults; thus, the concepts of adult education must be taken into account when planning, implementing, and evaluating any offering. Characteristics of an adult learner which the nurse usually displays are initiative, self-direction, a spirit of inquiry and the ability to think critically. These must be used to advantage when developing the continuing education offering…

In the last decade greater emphasis has been put on the measurement of achievement. Perhaps the advent of behavioral objectives has provided the impetus. None the less, achievement measures are now becoming a separate entity from aptitude measures. Achievement measures are concerned with characteristics and properties of present performance with emphasis on the content. Aptitude measurement in contrast is the demonstrated relationship between present performance and future attainment of knowledge and skill. The term which has been given to the measurement of a student's performance in relation to objectives of specified content is criterion referenced measurement. Criterion refers to a set standard or behavior. This standard does not have to be an end of course standard or behavior, as levels of performances can be measured at any point of instruction.

In order to provide a clear understanding of the two types of measures frequently utilized, I will refer to Glaseas statement about what each one does:

Criterion referenced measures indicate the content of the behavioral repertory, and the correspondence between what an individual does and the underlying continuum of achievement.

Norm referenced measures tell that one student is more or less proficient than another, but do not tell how proficient either of them is with respect to the subject matter tasks involved.1

Because instructional specialists have started to distinguish between these two approaches to measurement many questions have been asked in terms of the framework, construction, and revision of criterion referenced measurement

Before these questions can be answered it is important to explore how criterion referenced measurement is used. Criterion referenced measurement is used to determine an individual's status in relation to specific determined criterion. This cou Id be reflected as a continuum of knowledge which ranges from no proficiency to perfect performance. Each individual's achievement level then falls at some point on the continuum. In order to measure the achievement level the instructor must first develop objectives which he intends to be met at the end of the course. The end result will then be a measurement of the learner's performance based on achievement of the original objectives.

We can readily see then that criterion referenced measurement is invaluable to evaluate the learner's performance based on objectives and to evaluate instructional material. We are most interested in what the individual can do, not how he stands in comparison to others. An example could be the driver's license test, where an individual must display certain driving skills to pass the examination without any consideration of how well others perform on the test.

The scope of criterion referenced measures include establishment of instructional behavioral objectives, teaching content which is valid in terms of the objectives, and finally developing a test which is called criterion referenced test The tests used are devised to make decisions about individuals and instructional programs. The test is a method of determining whether a learner has mastered the objective or criterion skill.

In terms of the instructional program one could design a measure which reflected a set of objectives supposed to be achieved. The measure is then administered to appropriate learners after one segment of instruction has been completed. Methods for evaluating the instructional programs are needed and are valuable for the educator.

THE LEARNER

The nurse participants at continuing education offerings are adults; thus, the concepts of adult education must be taken into account when planning, implementing, and evaluating any offering. Characteristics of an adult learner which the nurse usually displays are initiative, self-direction, a spirit of inquiry and the ability to think critically. These must be used to advantage when developing the continuing education offering or program.

Because the adult is a different kind of learner compared to the child, there are some characteristics which are important to understand before attempting to utilize criterion referenced measurement for continuing education offerings. The nurse learner is continuing her learning. She has already obtained the basic skills and preparation for her profession and now has the desire to know more, to continue learning, or you might say to incease her level of competency. Therefore, she has more experience, is able to abstract, synthesize, and has insight. Learning needs are more immediate and concrete and if those needs are not met she will leave or stop attending the educational offering. She has a desire to be included in the learning situation and should actively participate in determining learning needs. When thinking in terms of utilizing criterion referenced measures remember the learner will take the responsibility for learning because she controls the amount of knowledge acquired, and to what extent learning takes place. Be sure to involve the learner by allowing a sharing of experiences, exchanging ideas, group interaction, and role-playing. Free and open communication makes possible the exchange of ideas and makes learning more viable and relevant.

BASE OF CRITERION REFERENCED MEASUREMENT

Before measuring the achievement level or evaluating the learner or instructional program it is essential that the entire planning and development of the instructional module be done using a criterion referenced base. Popham 2 who is one of the forerunners in criterion referenced measurement outlines steps which I will refer to as components which make up the whole of criterion referenced measurement.

Subject Matter

First, delimit the subject matter. Decide on the overall topic for the continuing education offering. An example which we will follow through will be preparing a twoday seminar on pre- and postoperative visits. The topic then is preoperative and postoperative visits. It will be assumed that this topic has been chosen because it is a high-need choice. Operating room nurses are needing and requesting a seminar on this topic, because they want to expand their knowledge and want to provide quality care to all patients experiencing surgery.

Major Categories

The second step then is to select major categories which will be included. In the pre- and postoperative visit seminar, it must be decided which categories will make up the content. The major categories then might be·. (1) structuring of pre- and postoperative visits; (2) acquiring skills in interviewing; (3) developing selfawareness; (4) assisting the terminally ill patient and family; (5) recognizing sexuality needs of patients; and (6) managing preoperative anxiety.

It may be necessary to make subcategories in order to further delimit content to facilitate the next component To continue with our example, subcategories under the major category structuring of pre- and postoperative visits would be: (1) objectives and goals; (2) identify pros and cons; (3) implementing a program; and (4) modifying resistance. (See Figure 1 for complete subcategories.)

General Objectives

The third step is to decide on some general objectives in order of importance in terms of skills and knowledge which constitute a subject matter skeleton. Remember that the general objectives will be the base for selecting the next level of objectives. An example of general objectives for the pre- and postoperative visit seminar might be: (1) to expand the nurse's ability to effectively interview patients within a short-term time period during pre- and postoperative visits; and (2) to enhance the nurse's insight into her own behavior in her efforts to communicate with others. (See Figure 1 for complete general objectives.)

Specific Objectives

The fourth step is to develop specific objectives. They should be clearly defined objectives which are written as performance objectives. The two characteristics of performance objectives are that they contain an action statement using a verb that describes observable behavioral output and incorporate a description of the conditions and materials that the learner must use to perform the action. Specific objectives must be clearly defined and may be divided into three domains: cognitive, affective, and psyche-motor. The cognitive domain includes objectives that are specific to thinking, knowing, and problem-solving. The affective domain includes those objectives dealing with attitudes, values, interests, and appreciation. The psychomotor objectives are those having to do with manual and motor skills. An example of a specific objective for the preand postoperative visit seminar in the cognitive domain is: to discuss a method by which a pre- and postoperative visiting program by operating room nurses could be implemented in the hospital of current employment. In the affective domain an objective might be: to appreciate listening to a patient's verbal expressions as a means of sampling his phenomenal field. I p the psychomotor domain an objective might be: to use principles and techniques of interviewing through involvement in a simulated interviewing situation. (See Figure 1 for complete specific objectives.)

Table

FIGURE 1: CONTINUING EDUCATION OFFERING OUTLINE FOR USE IN CRITERION REFERENCED MEASUREMENT

FIGURE 1: CONTINUING EDUCATION OFFERING OUTLINE FOR USE IN CRITERION REFERENCED MEASUREMENT

Table

FIGURE 1: CONTINUING EDUCATION OFFERING OUTLINE FOR USE IN CRITERION REFEHENCEO MEASUREMENT

FIGURE 1: CONTINUING EDUCATION OFFERING OUTLINE FOR USE IN CRITERION REFEHENCEO MEASUREMENT

Table

FIGURE 1: CONTINUING EDUCATION OFFERING OUTLINE FOR USE IN CRITERION REFERENCED MEASUREMENT

FIGURE 1: CONTINUING EDUCATION OFFERING OUTLINE FOR USE IN CRITERION REFERENCED MEASUREMENT

After having progressed this far in criterion referenced measurement it is advisable to seek some advice or consultation of other experts or educators who can look critically at what has been developed and provide input. Some changes in content or general objectives will be to the advantage of the program and should be done before progressing any further.

Constructing the Test

The criterion referenced test is constructed to yield measurements that can be interpreted in terms of specific performance standards. When writing specific objectives they may be divided into three domains, cognitive, affective and psychomotor. In order to measure performance representative samples of each of the three domains may be essential. It may be well to remember at this point that the criterion referenced test is constructed to give information about the individual's performance relative to a specified domain of tasks.

One problem inherent in constructing criterion referenced tests is how to build a test of reasonable length in order to determine if the learner can perform. Other problems in terms of test construction are in relation to variability, validity, reliability and item construction. (These problems will be expanded in the section on issues.)

Principles of criterion reference testing. Literature is not readily available in the area of criterion reference test development; however, Gronlund has outlined the following principles to provide a framework for direction. Some of the following principles were referred to earlier in the guidelines in relation to subject matter, categories, and objectives.

Criterion referenced testing requires:

- clearly defined and delimited domain of learning tasks.

- instructional objectives clearly defined in behavioral terms.

- standards of performance be clearly specified.

- student performance be adequately sampled within each area of performance.

- test items be selected on the basis of how well they reflect the behavior specified in the instructional objectives.

- scoring and reporting systems that adequately describes student performance on clearly defined learning tasks.3

Scoring and reporting systems rerer to the "percentage-correct" score. In criterion-referenced testing 90 percent is widely used by educators. This report indicates the leve) oí proficiency at which the learner performs. We cannot use percenti le ranks or standard scores as they apply only to norm referenced tests and criterion referenced tests are designed to describe a learner's performance in absolute terms - terms which are meaningful without reference to the performance of other learners.

Criterion reference test uses. When using criterion referenced tests for continuing education offerings such as the pre- and postoperative visit seminar it is possible to design the tests to serve two purposes. Measurements can be designed for use during the instructional process to stimulate, guide, and evaluate learning and to appraise the ongoing effectiveness of the instruction. The term for this type test is formative.

Another use is the summative test used at the end of the seminar to determine what learners have gained. Pretests to measure prerequisite knowledge can be used if they are developed according to principles of criterion referenced tests. Diagnostic tests can also be used to determine difficulties during instruction in order to take corrective action.

If using the pretest, formative test, summative test or diagnostic test, be sure the terms are not used interchangeably with criterion referenced tests.

Masfery and deve/opmenia/ levels of learning. Two levels of learning must be considered when developing the criterion referenced test. The easiest level to construct tests is at the mastery level of learning. Mastery level learning refers to measuring the mastery of minimal essentials. An example would be learning to administer an intravenous medication. What are the minimal essential knowledge and skills that must be mastered before a nurse can administer the intravenous medication? In relation to any continuing education offering consideration should be given to Gronlund's guide:

1. What are the minimum knowledge and skills that must be a prerequisite to further learning in the same area?

2- What basic skills are prerequisite to learning in another area?

3. What minimum skills are needed for safe performance of some activity?24

Criterion referenced testing is most useful at the mastery level of learning because the outcomes are relatively simple, domain of behavior is limited, learning is sequential in nature, and "percentage score" correct provides a meaningful report. This will be the easiest to accomplish with continuing education offerings such as the pre- and postoperative visit seminar.

The second level ot learning is the developmental level. This level is more complex and does not allow the educator the simplicity and management characteristic of the mastery level of learning. Developmental learning is defined as that learning which takes place beyond the minimum essentials. The learning outcomes are complex, domain of learning tasks unlimited, and learning is not sequential. The developmental level will be of great importance when measuring performance of the learner and effectiveness of the continuing education offering. In this level the learner is encouraged to strive for the maximum level of achievement and excellence. Criterion referenced test construction may be impossible for some learning outcomes, however norm referenced tests can be constructed using criterion referenced treatment. Gronlund illustrates some developmental learning outcomes:

1. Understands concepts and principles,

2. Applies concepts and principles to new situations.

3. Uses a scientific approach in solving problems.

4. Demonstrates reasoning ability.5

Many of the mentioned objectives take a lifetime to grasp and the level at which the learner accomplishes the objective is very hard to determine.

Selecting test items. When selecting the items for testing it is important to make sure a representative sampling of instructional objectives is measured. It would be beneficial to predetermine approximately how many items should be used for each objective and also the total number of items for the test before starting to construct the test. Remember a representative sample is all that is needed; however, keep in mind that each learner is going to be measured according to the performance objectives. Educators are familiar with test types and also how to actually construct items. For a brief review there are two major test types: supply where a short answer or essay is used, and the selection type used in multiple choice, true-false, and matching. Some authorities prefer the multiple choice and short answer for criterion referenced testing. This does not mean others cannot be used and it would seem that for many continuing education offerings a short essay would be most appropriate. When writing test questions, it will depend on how the specific instructional objective is stated as to the type of test to use. An objective for the pre- and postoperative visit seminar might be to list six objectives for doing preand postoperative visits. This objective would require a supply type question. The selection type question would be derived from an objective such as: "to identify ways that nurses and hospital personnel abandon the dying or terminally ill patient"

A sample of test questions for the objectives previously mentioned would be:

SHORT ANSWER ITEMS

Specific Learning Outcome: List six objectives for doing pre- and postoperative visits.

1. List four objectives for doing preoperative visits.

2. List two objectives for doing postoperative visits.

MULTIPLE CHOICE ITEMS

Specific Learning Outcome: Identify ways that nurses and hospital personnel abandon the dying or terminally ill patient

Which of the following verbal responses to a dying patient's statement, "I don't think I'll ever get well," constitute abandonment?

1. Why are you saying that?

2. Tell me why you are feeling this way?

3. I'm not certain what you are trying to tell me.

4. Getting well is often a matter of having faith.

5. Are you saying that you believe you are going to die?

Some rules which should be followed when writing objective test items have been identified by Gronlund:

1. The test item must be written so that it reflects the specific behavior described in the learning outcome.

2. The test item must be written clear and definite so the learner knows exactly what the task is.

3. The test item must contain only material relevant to the problem,

4. The test item must be free from irrelevant factors that would interfere with a correct response.

5. The test item must not include clues to right answers.

6. The test item must not provide a clue to other items on the test.

7. The test item must be written at a proper level in order to determine the maximum level of achievement.

8. The test item must be constructed so that experts would also agree on the correct answer.

9. The test item must be written in a positive form.

10. The test must include enough questions to adequately determine a measure of the learning outcomes.6

After test items are written they are then arranged into a test form. In criterion referenced test all questions that test the same objective should be grouped together in sequence from the easiest to the most difficult.

Reporting in criterion referenced testing. When the test is completely constructed and has been administered, the next step will be to score the results. Criterion referenced marking and reporting involves judging the learner's performance in terms of the objectives. Therefore, the use of percentile ranking is inappropriate for learners but a percentage correct score can be used to provide meaningful data. The object is to determine whether or not the learner was able to perform and also to evaluate the instructional material. Grades are not assigned, academic credit is not given, but emphasis is on the level of achievement for each individual. In continuing education offerings such as the pre- and postoperative visit seminar the learner receives a certificate stating they attended the seminar and are given a specific number of contact hours or receive a specific number of CEUs. Neither of these have anything to do with growth, changes in behavior, or in any way measure the amount of learning the participant may have as a result of attending the continuing education offering.

ISSUES FOR CONSIDERATION

For educators who have not been using criterion referenced measurement there will be a question of how test construction differs from norm referenced tests. Because controversies and differences of opinion exist» it is necessary to briefly discuss some of the issues.

One of the first issues comes from the fact that test constructors are repeatedly told test items must be analyzed and they must have validity and reliability. This concept is noi applicable to criterion referenced tests. In fact some authorities say it is injurious to the proper development and use of criterion referenced lests. This is because validity, reliability, and item analysis are all based on the variability among scores. In criterion referenced measurement the critical aspect in terms of validity is as it relates to content. Test items must reflect content With criterion referenced tests the chief rule is to make sure the item is an accurate reflection of the criterion behavior. The item writers must make the items represent behaviors delimited and pay close attention to defining domain of relevant test responses and the specific situation when required.

Another idea which has been reported is that all of the test iterns do not have to be given to all of the learners. Therefore, a representative sampling can be done by using different tests which seem most appropriate for use in continuing education offerings.

Another issue is that of conflicting views in relation to behavioral objectives. As one author so appropriately stated, "In certain influential circles, anyone who confesses to reservations about the use of behaviorally stated objectives for curriculum planning runs the risk of being labeled as the type of individual who would attack the virtues of motherhood."7 Atkin does concede to some benefits of behavioral objectives particularly that of the researcher who needs to determine some reliable measures of educational output. Also in relation to long-range planning and decision-making about curriculums and changes which evolve.

He discusses limitations and cautions in their use.

1. We assume as educators that we know or can easily identify the educational objectives we hope to attain and therefore predict what the educational outcomes will be. He contends that we are making progress towards thousands of goals and we not only can articulate them but don't even recognize some that may be very important.

2. By identifying behavioral objectives it may be that over a period of time the program or curriculum will tend to emphasize only those in terms of being important. Therefore, other important outcomes which were or are not identified get dropped and instruction then focuses only on those that have been easily identifiable.

3. Creativity and innovation are thwarted and frustrated by the demands of behavioral objectives. Styles of teaching and intellectual inquiry are hampered. Building programs or curriculums around behavioral objectives limits exploration and possible productive tangents.

4. Because it is impractical to pursue all goals thoroughly and because it is important to relate the educational goals to the situation at hand, educational potential is lost. Subjects which are taught that have rich meaning need to be emphasized repeatedly and in various ways. Many times concepts come up at unplanned times and it is detrimental to learning not to capitalize on îhe opportune moment to teach the concept.

5. Behavioral analysts assume that for an objective to be worthwhile we must be able to measure progress. Goals are derived from individual's needs and philosophies not from how we are going to measure them. Therefore it may be that behavioral goals are worthless and are articulated from a vast majority of goals because they are easily stated.9

Popham, who is one of the forerunners in criterion referenced measurement, also lists ten arguments against behavioral objectives and goes on to refute each of them. Many are the same which have been identified. Briefly, three of the refutations are:

1. Explicit objectives make it far easier for educators to attend important instructional outcomes.

2. Instructional opportunities which unexpectedly occur "should always be justified in terms of its contribution to the learner's attainment of worthwhile objectives."

3. Instructors that are not writing objectives in behavioral terms should begin now. Just because they aren't doing it. doesn't make it right.

Other problems and issues exist in constructing criterion referenced tests and although there are no solutions to many of these it is still of value to continue to explore and to develop tests which can be analyzed and improved in the future.

SUMMARY

The evaluation or measurement in continuing education is the weakest part of most programs. Evaluation is significant and the performance of the learner and the effectiveness of the program must be determined. To be able to make decisions about educational offerings, some questions need to be answered such as: Are the objectives being attained and to what extent? Are the objectives appropriate? Are the teaching methodologies effective? and Are the learner's needs being met?

Criterion referenced measurement is one method which can be utilized to evaluate the learner's achievement of the objectives of continuing education programs and also an invaluable mechanism whereby program effectiveness can be evaluated.

Using criterion referenced measurement for continuing education offerings will not be a simple task; however, it can be effective.

A challenge awaits educators who explore the principles and concepts inherent in criterion referenced measurement The more common pitfalls will be avoided if it is recognized that criterion referenced measurement needs more theory and research to guide the educator. Therefore, caution must be exercised as we proceed to use criterion referenced measurement in continuing education programs.

REFERENCES

  • 1. Glaser R: Instructional technology and the measurement of learning outcomes: some questions. Amer Psychoío 18:519-521, 1963.
  • 2. Popham WJ: Selecting objectives and generating test items for objective-based tests. In Problems in Criterion Referenced Measurement. CSE Monograph Series in Evaluation. No. 3, Los Angeles. Center for the Study of Evaluation. 1974. pp 22-24.
  • 3. Gronlund NE; Preparing Criterion Referenced Tests for Classroom Instruction. New York, The Macmillan Company, 1973. pp 3-5.
  • 4. Ibid, ? T.
  • 5. Ibid, p 17.
  • 6. Ibid, pp 34-37.
  • 7. Worthen BR, Sanders JR: Educational Evaluation: Theory and Practice. Worthington, Ohio. Charles A. Jones Publishing Company. 1973, p 232.
  • 8. Ibid, pp 232-239.
  • 9. fold, pp 239-246.

FIGURE 1: CONTINUING EDUCATION OFFERING OUTLINE FOR USE IN CRITERION REFERENCED MEASUREMENT

FIGURE 1: CONTINUING EDUCATION OFFERING OUTLINE FOR USE IN CRITERION REFEHENCEO MEASUREMENT

FIGURE 1: CONTINUING EDUCATION OFFERING OUTLINE FOR USE IN CRITERION REFERENCED MEASUREMENT

10.3928/0022-0124-19760301-08

Sign up to receive

Journal E-contents