During the past decade, nursing faculty across the United States have increased access to their programs by incorporating new technologies and developing distance education opportunities. These programs have included methods such as interactive television and online or Web-based technologies. Questions have been raised about the quality of learning and learner outcomes in these programs. This article describes the evaluation of an online graduate pharmacology course and compares the results with those of a face-to-face course taught simultaneously.
Evaluation of all aspects of online courses, from student learning to effectiveness of the course, instructor, and overall program, is important (Palloff & Pratt, 1999). Examinations, papers, and student course evaluations are common methods of evaluating student learning and faculty performance. In online learning, asynchronous discussions, focus groups, and written feedback about the online learning environment can be rich sources of evaluation data.
In their study, Ryan, Carlton, and Ali (1999) found common student and faculty perceptions of both positive and negative attributes of online education. Students appreciated the time flexibility, convenience, and decreased travel associated with online courses, and the authors reported high-quality asynchronous discussions, increased networking possibilities, and myriad resources available on the Internet. Some negative experiences included establishing and maintaining Internet connectivity and the need for technical support. Some students reported that online courses stimulated self-learning and helped them examine content in greater depth, while others believed online courses create a sense of isolation.
The “no significant difference” phenomenon has surfaced as educators in many fields compare learning outcomes between Web-based and traditional classroom instruction. Merisotis and Phipps (1999) described limitations in the research to date. Most of the studies they reviewed concluded that distance education courses compare favorably with classroom courses and have high student satisfaction levels. However, the studies often did not control for extraneous variables, did not use random assignment to type of course, used instruments for which validity and reliability were questionable, and did not control for students’ and faculty members’ feelings and attitudes. Randomly assigning students to courses is difficult and leaves faculty to use quasi-experimental methods when comparing course delivery methods.
Yucha and Princen (2000) compared 14 students in an online module and 163 students in a face-to-face graduate pathophysiology module (one credit of a total of three) taught by the same instructor. Students in the face-to-face module had access to the Web content, which could be used as a supplement to or in lieu of class attendance. Student learning was evaluated by multiple-choice examination. No difference was found between the two groups (F = 1.524, NS, probability not reported).
Woo and Kimmick (2000) compared 53 graduate nursing students in a face-to-face research course with 44 students in a Web-based course. The course materials and instructor were the same. Student learning was evaluated using identical midterm and final examinations. A limitation of this study was that online students could attend classroom sessions at any time. All but 12 students (27%) attended at least one class, and nearly three quarters of the students (n = 55, 73%) attended three or four class sessions. Use of online course resources by students in the face-to-face course was not reported.
No significant differences were found in examination scores or overall student satisfaction. However, students in the online course scored significantly higher (p = .04) on “stimulation of learning,” compared to students in the face-to-face course, using a single question on the student satisfaction tool. The authors did not state the question used for this part of the evaluation.
Leasure, Davis, and Thievon (2000) conducted a similar study of 66 baccalaureate students in a nursing research course. The students chose the teaching method. Forty-eight students chose to participate in the face-to-face course and 18 chose the online course. The same instructor taught both courses, using the same course materials and examinations. No significant difference was found between the two groups on the examinations (t = −.96; p = .343).
Online nursing courses have been found to be similar to face-to-face courses in measures of student learning outcomes and student satisfaction. Students are able to complete courses that are primarily Web-based and meet learning objectives in a way similar to students in more traditional courses. Technical problems and adequate communication between faculty and students and among students remain unresolved issues.
Development of the Web-Based Course
In 1996, the state legislature provided funding for a system of state colleges and universities to develop the Web-based graduate pharmacology course. Nurse educators and Web instructional designers collaborated to develop the course using WebCT as the course management system. The course was developed to meet the competencies in the model pharmacology/pharmacotherapeutics curriculum guidelines (U.S. Department of Health and Human Services, 1998) for graduate family nurse practitioner programs. The course was selected by WebCT as an “Exemplary WebCT Course for 2000” by meeting criteria of academic rigor and content robustness (Palloff & Pratt, 1999).
Students from many nurse practitioner specialties and clinical nurse specialist areas, and nurse-midwifery students complete this pharmacology course. It includes study of drug therapy; principles of altered pharmaco-dynamics relative to age, race, and ethnic background; and the cost-benefit aspects of pharmacological interventions. This 3-credit, Web-based course was developed in a modular format. All modules contain links to Web resources to keep the information current. Three foundational modules include clinical decision making, case management principles, and legal, ethical. and role issues. These are followed by 15 modules based on body systems. The foundational modules serve as a basis for the critical thinking, discussion, and decision making that will be used throughout the course. The course materials include a pharmacology text, a small drug handbook, and a case study workbook.
Learning activities for the foundational modules include a visit to a pharmacy to analyze costs of over-the-counter medications, followed by a Web-based discussion. Students write a prescriptive practice agreement in accordance with the law, rules, and regulations of their home states. In the body systems modules, learning activities are case based and include a defined group of questions with each case study (Schwinghammer, 2000). These questions are posted on the discussion bulletin board and serve as the focus for discussion and the clinical decision-making process.
In the spring 2000 semester, 18 students enrolled in the Web-based course. Computer and e-mail requirements, course access codes, the syllabus, and a WebCT tutorial were mailed to students 2 weeks prior to the on-campus orientation. Students were encouraged to access the course in advance. The model for Web-based courses at our university includes a minimum of two face-to-face sessions each semester. Three face-to-face sessions (at weeks 1, 7, and 15) were held for this course.
Students were divided into small groups for asynchronous discussion of cases related to weekly content. The self-assigned group leader (a rotating role) initiated the weekly discussions, and each student was required to contribute. The instructor read the discussions and provided feedback to stimulate depth and completeness of the discussions. Students submitted individual prescriptions to the instructor for a portion of the cases discussed.
A 3-credit, face-to-face course with 52 enrolled students was taught during the same 15-week semester, combining 2½ hours of lecture with a ½ hour of case discussion every week. Lectures were presented by the course instructor, guest nursing faculty, and university pharmacy faculty. There was no Web-based component. The foundational topics were taught at the beginning of the course, and a specific body system was taught each week. Learning activities were similar to those in the Web-based course, except that students in the face-to-face course wrote a paper instead of participating in an online discussion after the pharmacy visit. Each student had the opportunity to practice writing prescriptions. Students were provided with a case prior to the class session and met briefly in small groups during class time. The entire class then discussed the case, with facilitation by the instructor. Students discussed aspects related to their specialty area, as well as cultural and ethical issues and cost. Students in the face-to-face course used the same textbooks, were taught by the same instructor, and took the same written examinations as students in the online course.
Methods for Course Evaluation
There are many methods to evaluate learning. According to Lockee, Moore, and Burton (2001), the distinction between research and evaluation can blur because similar methods are used, but the intentions differ. Research involves testing theories and constructs to inform practice, while evaluation seeks to determine whether a program was successfully developed and implemented. To assess the effectiveness of a Web-based course, investigators can answer relevant questions through evaluation techniques such as focus groups, weekly student assessments, and knowledge testing. Our goal was to evaluate the Web-based course and add to the body of research examining learning outcomes using new technologies.
Evaluation of this course included several measurements. A focus group was conducted with students who took the course at another college prior to its first offering at our university. Another focus group was conducted with university students following the first offering of this course. We constructed a set of questions, beginning with broad questions related to the students’ experience with the course and moving toward more detailed questions about the specific course components. Prior to conducting the first focus group, we drafted the questions and then revised them based on the review and critique of two individuals in the university’s digital media center with experience conducting focus groups. During the focus group, one author asked the questions and the other took notes about the discussion. The students consented to audiotaping the sessions, and the audiotapes were transcribed and reviewed. The same questions were used with both focus groups.
At the beginning of both the face-to-face and Web-based courses, the students were asked to provide several pieces of information, including their age, type of basic nursing program (i.e., associate degree, diploma, or baccalaureate), years of nursing practice, and whether they took a formal pharmacology course in that nursing program. The instructor explained that the information would be used to evaluate the course and that no students would be individually identified.
The objective measures of student outcomes for the courses were instructor-designed midterm and final examinations. The same examinations were administered to students in both courses. These examinations could have a maximum score of 100%. Midterm and final examination scores were compared between student groups.
Students in the Web-based course also completed an evaluation tool specifically developed for that course. They were asked the degree to which they thought they had met each of 10 course objectives on a 5-point Likert scale (from 1 = “not very much” to 5 = “a great deal”). Students also rated their use of Web sites linked to the course, the textbooks, the amount of professional knowledge gained, and their ability to communicate effectively with the instructor and other students in the course, using a similar Likert scale. After the completion of the course, we sought and received Institutional Review Board permission to use these data for the evaluation process.
The research questions were:
- Are there differences in characteristics between students in a Web-based course and those in a face-to face course (e.g., age, years of nursing practice prior to graduate school, type of basic nursing education, whether students had taken a previous formal pharmacology course)?
- Is there a difference in objective learning measures, such as midterm and final examinations, between students in a Web-based course and those in a face-to face course?
- What was the experience of students in the online course?
Data were first analyzed descriptively based on the variable type, including the final Web-based course evaluation. Age, years of nursing practice, and examination scores were compared between the two groups using the two-tailed t test. Previous pharmacology course and type of basic nursing education were examined for association with course choice using the chi-square test of association. All results were considered significant at the p < .05 level.
First Focus Group
Students who participated in the Web-based pharmacology course at another college were pleased with the course overall. Suggested modifications included reducing the number of and prioritizing the most important Web links used for the course; reorganizing and clarifying parts of the syllabus; and considering other textbooks for a future course. Changes implemented prior to the first offering of the course at the university included providing a face-to-face orientation to the course, which included an introduction to prescription writing and orientation to WebCT and assigning a small drug handbook as a supplement to the textbook.
Students’ mean age in the face-to-face course was 34.4 (SD = 8.8) and 36.9 (SD = 6.9) in the Web-based course. The difference was not statistically significant (t = −1.23, df = 59, p = .23). Years of nursing practice prior to course registration was similar (t = −1.28, df = 63, p = .21). The average years of nursing practice was 9.6 years (SD = 7.3) for students in the face-to-face course and 12.2 years (SD = 7.8) for those in the Web-based course.
No significant associations were found between course method choice (face-to-face or Web-based) and previous pharmacology course or type of basic nursing program. Among students in the face-to-face course, 75% had taken a previous pharmacology course, while 61% of students in the Web-based course had taken a previous pharmacology course (χ2 = 1.12, df = 1, p = .29). In the face-to-face course, 81% of students had completed a baccalaureate degree, 17% had completed an associate’s degree, and 2% had completed a diploma program. In the Web-based course, 72% of students had completed a baccalaureate degree, 22% had completed an associate’s degree, and 6% had completed a diploma program (χ2 = .88, df = 1, p = .67).
No significant differences were found between the mean examination scores of students enrolled in the two courses. Average midterm examination scores were 75.5 (SD = 6.7) for students in the face-to-face course and 75.8 (SD = 6.5) for students in the Web-based course (t = −.195, df = 68, p = .85). Average final examination scores were 77.8 (SD = 8.4) for students in the face-to-face course and 75.3 (SD = 8.7) for students in the Web-based course (t = 1.09, df = 68, p = .28).
Web-Based Course Evaluation
For the 10 questions related to meeting each course objective, the majority of students in the Web-based course selected 4 or 5 (scale: 1 = not at all to 5 = a great deal). Responses at the 4 or 5 level for meeting each course objective ranged from 50% (for one objective) to 100% (for one objective). Only 25% of students ranked their use of the linked Web sites as 4 or 5; the majority of students would not recommend the same textbooks; and 50% of students rated communication with both students and faculty as 4 or 5. Another 37.5% to 50% of students selected the neutral rating of 3 for the questions related to faculty and student communication.
Students also responded to seven open-ended questions. Eight of 16 students responded yes to a question about interest in the school offering of more online courses. One student replied no, and the remainder made comments such as requesting more face-to-face time and suggesting that Web-based learning may be better suited to certain courses.
Students liked the case studies and prescription-writing activities, and several asked for less-complex cases. When asked if they would recommend the course to other students, 10 replied yes, 3 said no, and the others made conditional responses, depending on future changes to the course.
Second Focus Group
The students’ comments about the course and the systems approach to pharmacology were generally favorable. Students suggested changes in the order of topics and requested that the systems taught coincide with the content taught in the primary care course offered the same semester. Students again made comments about the complexity of the case studies and some of the technical difficulties they experienced, including problems logging onto the course site. One student stated that the course was no different than any other in that the students felt overwhelmed at first and then figured out how to study and complete the work. One student commented that writing responses in asynchronous discussions takes more time than answering questions in class and participating in discussions. Specific recommendations were made to increase the clarity of the syllabus and assignments, and to reduce the writing required each week.
No significant differences or associations were found when students who chose the online course were compared with those who chose the face-to-face course for age, type of basic nursing preparation, years of nursing experience, and whether the student had taken an undergraduate pharmacology course. Because the majority of students had taken an undergraduate pharmacology course and completed a baccalaureate degree for their basic nursing education, there may not have been enough power to observe an association, if one existed.
There were also no significant differences in the midterm and final examination scores between the two groups of students. Students seemed to perform equally well on these two objective measures of their learning. There were no dropouts from either the Web-based or face-to-face courses. However, students are required to take this course to progress in their graduate program.
The majority of students in the online course felt they had met the course objectives. Half of the students rated interaction with other students in the high range, but 13% of the students indicated a low quality of interaction with other students. Half of the students also rated interaction with faculty in the high range. However, no students chose the low ratings for interaction with faculty. This indicates a need to continue to improve communication, particularly among students, so a higher percentage of students feel this aspect of the course is more positive.
Our results were similar to other studies in finding no differences in student outcome measures (Leasure et al., 2000; Merisotis & Phipps, 1999; Woo & Kimmick, 2000; Yucha & Princen, 2000). Students self-selected the teaching method, and we were able to evaluate several student characteristics. We found the students in the two courses to be similar in age, years of nursing practice, and previous nursing education.
This study was limited in the same ways as similar projects. Students self-selected the teaching method they preferred. We did not believe it was prudent to randomly assign students to a course method because our goal in developing online courses is to increase student access and flexibility by offering more choices.
Changes have been made to this graduate pharmacology course based on student feedback. These changes include adding more examinations, moving from administering only midterm and final examination to administering four examinations during the semester. In addition, students write more prescriptions throughout the semester for each case study. Because the course is required by many specialty areas in our graduate program, it continues to be offered by both online and face-to-face teaching methods.
As the research evaluating Web-based courses continues, it is important to begin to move beyond comparisons of face-to-face and Web-based courses and evaluate online programs including other aspects of student achievement. Billings, Connors, and Skiba (2001) reported the use of benchmarking as a means of evaluating quality in Web-based nursing courses. The variables used for the benchmarking project were derived from a model that was developed from nursing and higher education concepts to examine the effect of the use of technology on student outcomes (Billings, 2000). Further development of a framework that is useful across nursing programs and common tools to evaluate technology in nursing education should help develop the research in this area.
No differences were found between the two groups in learning outcomes, as measured by instructor-developed examinations. Overall, students were satisfied with their experience in the online course. Evaluation of online courses should continue, with particular attention to interaction among faculty and students and overall student satisfaction.
- Billings, D.M. (2000). A framework for assessing outcomes and practices in Web-based courses in nursing. Journal of Nursing Education, 39, 60–66.
- Billings, D.M., Connors, H.R. & Skiba, D.J. (2001). Benchmarking best practices in web-based nursing courses. Advances in Nursing Science, 23(3), 41–52. doi:10.1097/00012272-200103000-00005 [CrossRef]
- Leasure, A.R., Davis, L. & Thievon, S.L. (2000). Comparison of student outcomes and preferences in a traditional vs. World Wide Web-based baccalaureate nursing research course. Journal of Nursing Education, 39, 149–154.
- Lockee, B., Moore, M. & Burton, J. (2001). Old concerns with new distance education research. Education Quarterly, 2, 60–62.
- Merisotis, J.P. & Phipps, R.A. (1999). What’s the difference? Outcomes of distance vs. traditional classroom-based learning. Change, 31(3), 12–17. doi:10.1080/00091389909602685 [CrossRef]
- Palloff, R. & Pratt, K. (1999). Building learning communities in cyberspace. San Francisco: Jossey-Bass.
- Ryan, M., Carlton, K.H. & Ali, N.S. (1999). Evaluation of traditional classroom teaching methods versus course delivery via the World Wide Web. Journal of Nursing Education, 38, 272–277.
- Schwinghammer, T.L. (2000). Pharmacotherapy casebook. Stamford, CT: Appleton & Lange.
- U.S. Department of Health and Human Services. (1998). Curriculum guidelines and regulatory criteria for family nurse practitioners seeking prescriptive authority to manage pharmacotherapeutics in primary care. Rockville, MD: Health Resources and Services Administration.
- Woo, M.A. & Kimmick, J.V. (2000). Comparison of internet versus lecture instructional methods for teaching nursing research. Journal of Professional Nursing, 16, 132–139. doi:10.1053/PN.2000.5919 [CrossRef]
- Yucha, C. & Princen, T. (2000). Insights learned from teaching pathophysiology on the World Wide Web. Journal of Nursing Education, 39, 68–72.