Journal of Nursing Education

A Multidimensional Approach to Evaluation

Patricia R Dean, RN, MSN; Tonya A Edwards, RN, MSN

Abstract

Each terminal competency is awarded a letter grade by the jury based upon the following criteria:

A - demonstrates extensive breadth and depth of knowledge and application of theory and principles.

B - demonstrates knowledge and applies theory and priniciples. Utilizes appropriate resources to complement knowledge and application base.

C - demonstrates basic knowledge and application of theory and principles.

The jury examines the students only on subject content and skill areas consistent with the statement of attainment. The final grades are submitted in writing to the student's mentor, the Dean and the student's file. The mentor has the responsibility of submitting grades to the Registrar's Office of the University. If at any time during the process the student does not satisfactorily complete the evaluation method or fails to meet scheduled events without prior notification, the process is aborted and the student must apply for readmission to the terminal jury.

The student may appeal a grade a warded by a jury through the normal grievance procedures already existing within the conventional University structure.

Time Allotment: A total of 18V* hours are required to complete the jury process. On the average, these hours are spread overa two week period. Occasionally, a student will request less time or more time to complete the process, based on work schedules or personal goals. Because of the flexibility built into the program, the student's request generally is granted.

Each jury team has a minimum of four faculty members. A minimum of 9Vt hours per faculty member is necessary to process one student. The total hours for student and faculty are based upon the student's successful completion of the process the first time. When a student aborts the process, reapplication for evaluation is required, increasing the student and faculty hours.

Due to the number of hours a faculty member puts into this process, a strong commitment to the multidimensional approach is necessary. However, a minimum of four faculty members could carry the load of evaluation by this method in addition to other teaching responsibilities in a traditional generic program.

Nineteen students were in the group of registered nurse students assessed for this report. Seven students successfully completed the jury process on their first attempt. Eleven students successfully completed the jury process their second attempt. One student withdrew from the program following her third attempt through the jury process and, therefore, did not complete the process.

In the Table a breakdown of unsuccessful completions in each component is identified. The table identifies the written obstetrical examination as the major area of difficulty. Personal interviews reflected a weakness in this area of the program which correlates with the results identified upon examination. These results have been communicated to the program director and alterations in the program are being made.

The evaluation methods which the students and jurors identified as the better methods of certifying mastery of the terminal competencies were:

1. oral grand rounds.

2. clinical practicum.

3. simulation experiences.

The battery of written examinations was identified as a fair method, but not an accurate method of evaluating synthesis, integration and mastery. The recommendations from students and jurors for change in the process were:

1. to place the examination at the end or the Registered Nurse Program and to eliminate them from the jury process.

2. to utilize the examinations in the Registered Nurse Program as an indicator of readiness for the jury process.

3. to totally eliminate the examinations because students have been tested in the Registered Nurse Program on the specific content in the examination and do not need retesting.

The average time for completion of…

A Curriculum of Attainments Program began at Florida State University in 1972 when Senate Bill 455 of the Florida Legislature required the Division of Universities to "revise the requirements -for the granting of baccalaureate degrees and to grant degrees after three years' academic work, unless there were accreditation conflicts."1 Professor David Riesmann of Harvard University requested Florida State University's participation in a research project designed for implementation and evaluation of a Curriculum of Attainments Program. In 1974, funds were provided for programs in Marine Biology, Nursing, and Urban and Regional Planning at Florida State University.

The principal concept of the Curriculum of Attainments Program was that students could demonstrate competencies, without regard to methods used or the length of time taken to meet the attainments. This method provided a unique component of separating instruction from evaluation in the educational process.2

The pilot project in nursing dealt specifically with 14 generic nursing students in Florida State University's baccalaureate program. The results of the Curriculum of Attainments Program demonstrated that these students could attain competency in nursing without regard to the teaching methods used or specified time assigned to attain a particular competency. This process did not necessarily reduce the length of time students remained in the program, but it permitted students to learn at their own rate. The Nursing faculty determined, based upon the pilot project, that this program offered a curriculum design which provided a unique opportunity for registered nurses to obtain a baccalaureate degree while meeting their family and job responsibilities. Therefore, in 1975 this curriculum was established as an alternative to the traditional generic baccalaureate curriculum and designated the Registered Nurse Program.

Integral to the Curriculum of Attainments Program was the method of assessing student competency. Competency was evaluated by a process of oral, written and practical examination by a panel of faculty members and professionals. This multidimensional evaluation is termed the Jury Evaluation. The jury process was retained as the means of measuring competency when the curriculum was adopted for the Registered Nurse Program. Every registered nurse student must complete successfully the jury process before the baccalaureate degree is granted. This paper discusses the jury process as utilized in the Registered Nurse Program curriculum.

Basic Concept of Jury Evaluation

Each jury is composed of faculty members and practicing professionals who are specialists in their disciplines. The jury assesses each student's attainment in the program's terminal competencies. Terminal competencies are established by the total faculty and the measurement of these competencies by the jury. The Registered Nurse Program mentors do not evaluate the program's students on a summative basis. The major responsibilities of the jury are:

1. Certification of minimal level of mastery in each terminal competency, and

2. Assignment of grades for each student in each terminal competency.

The jury examines a student's capacity to integrate broad areas of knowledge and skills, which require nontraditional assessment techniques such as oral examinations, videotaped performances, live demonstrations, anecdotal records and portfolios. They primarily evaluate a student's behavior in high-order cognitive skill areas and in complex problem-solving situations, such as analysis and synthesis.1

Definition of Terms

Tutor: A tutor is a specialist in the discipline who complements the mentor role by helping students master basic intellectual content and fundamental skills of the discipline.1

Mentor: The mentor advises students on the acquisition and use of learning resources in order to help them reach mastery. The mentor designs instructional materials which enable students to acquire knowledge and skills. Mentors teach as little as possible- Their primary role is to help a student integrate knowledge and skills gained from a variety of educational resources, and to assess a student's progress toward attainment of competencies. A mentor does not participate in the jury process.

jury: The jury certifies that the student has met the competencies of the program, and awards grades for each of the terminal competencies. The jury is also responsible for certifying that scholastic standards were valid and adequate, and they determine the criteria for evaluation of the terminal competencies.1

The Jury Process

Upon formally entering the Registered Nurse Program, the terminal jury process is explained to the students by two methods. An oral explanation along with a packet of written materials is presented to the student by the chairman and several members of the existing jury. The written packet contains a statement of the purpose of the jury, instructions to the student for application and procedure of the process, grading guidelines, terminal competencies, application form and schedule of jury events. The student is then asked to see a 15 minute videotape showing the highlights of the jury process and several methods used by the jury for evaluation.

U pon completion of the Regis tered N urse Program, the student makes a formal written application to the jury. This form is signed by student and mentor certifying that all learning packets have been completed with a grade of satisfactory and she/he has met all graduation requirements for the University from which the degree is sought.

Within five days of the receipt of the application, the jury provides the student with a written schedule of events required to complete the jury process. This scheduling of events may or may not be worked out with the student, depending upon the jury member's responsibilities in the generic program and the student's work schedule or personal goals. The student is assigned toa team of four jurors who represent the four major areasof nursing - medical/surgical, maternal-child health, psychmental health, and community health. All communication and written assignments between student and team is processed through the team captain.

At the time of application to the jury process, the student's mentor provides the jury with a portfolio and written summary of the student's work in the program and biographical data of the student's work experiences. In addition to the portfolio, the mentor provides an evaluation of the student's personal and professional development.

The terminal jury evaluation is in four major categories: (1) written evaluations, (2) simulated clinical, (3) clinical practicum with grand rounds, and (4) a personal interview. The failure by the student of any of the first three components halts the process and the student is sent back to the mentor for additional study or experience.

Written Evaluation: The written part of the jury process consistsof essay, short answer, and objective questions. Each of the questions are developed around the terminal competencies but do not evaluate all of the competencies. Out of the 20 identified competencies, 16 are evaluated on the written examinations. The written examinations generally evaluate theory and application of theory to clinical practice. A total of nine hours is required for the written part of the process, three hours for the essay and two hours each for the examinations evaluating maternal-child health, psych-mental health, and community health content.

One jury member is required to monitor all four written exams, but it does not require the same jury member to monitor all four exams. Successful completion of all written examinations is 80% or above. Upon successfully passing the written examinations, the student progresses to the next method of evaluation. If the student does not successfully complete the written examinations, the process is halted and the student returns to the mentor for further instruction in the area(s) in which additional work is needed. The jury team provides the mentor with a written summary of the student's needs.

Simulation: The student participates in ä one-and-a-half-hour simulated clinical problem related to the expanding family. This simulated clinical covers components of growth and development, community health practices, communication skills, psychosocial needs, prenatal, labor, delivery, postnatal and neonatal care occurring at any point on the health-illness continuum. Sixteen of the 20 competencies are evaluated by this method. Two jurors must be present for observation and interaction with the student. The student is given written detailed guidelines for participation in the simulation and is allowed to ask questions for clarification at any time during the simulation. This component is also audiotaped and retained by the jury.

Clinical Practicum - Grand Rounds: Utilizing the nursing process, the student will provide care to a selected client or family and make a comprehensive formal presentation of the nursing care experiences. The jury team selects the client for which the student will provide care using the following criteria:

1. a client that needs some physical care;

2. a client who is alert and can communicate in some form;

3. a client selected from an agency with which the School of Nursing has an established contract.

The student is given the clinical assignment by one of the jurors. She is then required to do an assessment prior to the implementation of care. Within 24 hours of receiving the clinical assignment, the student will render comprehensive nursing care over a two to four hour period. The jurors must be present during the time care is rendered for evaluation of practice and interaction with the student. Within 24 hours of completion of the care rendered, the student submits to the team captain written materials consisting of the nursing history, priority of needs, care plan, and annotated list of all resources utilized.

Following the submission of written material, the student conducts a grand rounds session which is three hours in length. The student makes a 30 minute presentation to the jury about the client which includes the development of the nursing care plan in which she/he demonstrates evidence of integration of the terminal competencies. The student is permitted to use notes but is not allowed to read from a written manuscript. Once the student has made the presentation, the jurors begin orally questioning the student, evaluating her/his theory base, cognitive skills and problem-solving abilities. All questions are related to the 20 terminal competencies. For example, a question might be, "Using crisis theory, at which level would you place your client and on what basis would you initiate intervention?" A minimum of three jurors must be present. The grand rounds presentation is taped and retained by the jury for further reference. Upon satisfactory completion of the grand rounds component, the jury process proceeds to the personal interview.

Personal Interview: Following the grand rounds component a break is taken, and then a personal interview is conducted. The interview has several purposes. First, it gives the jury the opportunity to clarify any areas about which the jurors desire additional information prior to assessing the student's grand rounds performance. Second, it provides the jury the opportunity to orally advise the student of strengths or undeveloped potential revealed by the jury process. Lastly, if the student has completed the process successfully, the student's perception of the curriculum and jury process is solicited.

The jurors ask the students to speak to each component of the jury process identifying strengths and weaknesses, resources utilized during the jury process and to what extent, and the sequence and time allotment for the components of the process. This data was gathered by the jury for evaluation of the jury process in order to make modifications.

The student is then asked to speak to aspects of her/his program of study identifying strengths and weaknesses of the program. Aspects of the program which the jurors ask the student to speak to are as follows:

1. The student's adjustment into the program.

2. The student's likes and dislikes of a self-paced program.

3. The content areas of the program.

4. The utilization of previous work experiences.

5. The student's exposure to faculty other than mentor.

6. The utilization of seminar methods to promote socialization and interaction.

7. The knowledge that has been gained as a result of baccalaureate education.

Grading: Each terminal competency is assigned a letter grade of A, B, C, or U by the evaluating jury team. All components of the jury process are utilized when determining the grade for each competency. Of the 20 terminal competencies, 11 of them are evaluated by all of the components of the process. The remaining nine are evaluated by at least two of the components. Utilizing at least two of the components for evaluation of the competencies, students are evaluated both by oral and written methods. This provides for a margin of error, as some students do not do well on written examinations but can express themselves orally and vice-versa.

Table

TABLEBREAKDOWN OF UNSUCCESSFUL COMPLETIONS IN EACH COMPONNT N - 12

TABLE

BREAKDOWN OF UNSUCCESSFUL COMPLETIONS IN EACH COMPONNT N - 12

Each terminal competency is awarded a letter grade by the jury based upon the following criteria:

A - demonstrates extensive breadth and depth of knowledge and application of theory and principles.

B - demonstrates knowledge and applies theory and priniciples. Utilizes appropriate resources to complement knowledge and application base.

C - demonstrates basic knowledge and application of theory and principles.

The jury examines the students only on subject content and skill areas consistent with the statement of attainment. The final grades are submitted in writing to the student's mentor, the Dean and the student's file. The mentor has the responsibility of submitting grades to the Registrar's Office of the University. If at any time during the process the student does not satisfactorily complete the evaluation method or fails to meet scheduled events without prior notification, the process is aborted and the student must apply for readmission to the terminal jury.

The student may appeal a grade a warded by a jury through the normal grievance procedures already existing within the conventional University structure.

Time Allotment: A total of 18V* hours are required to complete the jury process. On the average, these hours are spread overa two week period. Occasionally, a student will request less time or more time to complete the process, based on work schedules or personal goals. Because of the flexibility built into the program, the student's request generally is granted.

Each jury team has a minimum of four faculty members. A minimum of 9Vt hours per faculty member is necessary to process one student. The total hours for student and faculty are based upon the student's successful completion of the process the first time. When a student aborts the process, reapplication for evaluation is required, increasing the student and faculty hours.

Due to the number of hours a faculty member puts into this process, a strong commitment to the multidimensional approach is necessary. However, a minimum of four faculty members could carry the load of evaluation by this method in addition to other teaching responsibilities in a traditional generic program.

Nineteen students were in the group of registered nurse students assessed for this report. Seven students successfully completed the jury process on their first attempt. Eleven students successfully completed the jury process their second attempt. One student withdrew from the program following her third attempt through the jury process and, therefore, did not complete the process.

In the Table a breakdown of unsuccessful completions in each component is identified. The table identifies the written obstetrical examination as the major area of difficulty. Personal interviews reflected a weakness in this area of the program which correlates with the results identified upon examination. These results have been communicated to the program director and alterations in the program are being made.

The evaluation methods which the students and jurors identified as the better methods of certifying mastery of the terminal competencies were:

1. oral grand rounds.

2. clinical practicum.

3. simulation experiences.

The battery of written examinations was identified as a fair method, but not an accurate method of evaluating synthesis, integration and mastery. The recommendations from students and jurors for change in the process were:

1. to place the examination at the end or the Registered Nurse Program and to eliminate them from the jury process.

2. to utilize the examinations in the Registered Nurse Program as an indicator of readiness for the jury process.

3. to totally eliminate the examinations because students have been tested in the Registered Nurse Program on the specific content in the examination and do not need retesting.

The average time for completion of the process has been 12 days. This time varies depending upon the student's goals and the faculty's availability.

Although the evaluation process was somewhat lengthy, the jurors did determine that in a baccalaureate program, a multidimension method of evaluation is as reliable as the traditional methods of written examinations in assessing a registered nurse's integrated cognitive abilities and affective skills.

Presently the jury members are in the process of obtaining a grant to further study this method of evaluation. The jury's goal is to reduce the time factor for completion of the process in order to develop a mobile program and terminal evaluation method for registered nurses throughout the state of Florida. This mobile program will afford a better opportunity for registered nurses wishing to obtain a baccalaurate degree in nursing to do so without having to leave their job and/or families for an extended period of time.

With additional educational backgrounds, better quality nursing care will be provided to the consumer.

References

  • 1. Peterson GW: Curriculum of Attainments, Final Report. Center for Educational Design, Florida State University, August 31, 1976, pp 4, 8, 10, 61, 62.
  • 2. Registered Nurse Program, Terminal Jury Packet, Florida State University, 1976.

TABLE

BREAKDOWN OF UNSUCCESSFUL COMPLETIONS IN EACH COMPONNT N - 12

10.3928/0148-4834-19820201-04

Sign up to receive

Journal E-contents