Nursing schools are offering Web-based courses and programs to meet needs for access and convenience for students seeking basic and advanced degrees. Additionally, Web technology is being used to create learning communities and support learner-centered strategies for students in on-campus courses ("Webenhanced courses"). In spite of the increasing use of the internet to deliver courses and enhance learning, little is known about the outcomes, what teaching and learning practices contribute to positive outcomes, what supports need to be in place for students and faculty, or how Web technology and its learning tools contribute to teaching and learning. Answers to questions about outcomes, teaching and learning practices, quality, and academic standards are being sought by accrediting bodies, commissions of higher education, academic institutions, schools of nursing, employers, as well as the students and faculty who are the pioneers in this new way of teaching and learning. The purpose of this article is to propose a framework for guiding the evaluation/assessment of nursing courses and programs offered on the World Wide Web.
As with other forms of distance education (Billings & Bachmeier, 1994; Cobb & Billings, in press), teaching and learning in Web-based courses involve a dynamic interaction of the technology, the teaching-learning practices, faculty and student development, and the outcomes enabled by the technology (Billings, 1997; Ehrmann, 1994). An understanding of the complexity of teaching and learning in Web-based courses can be facilitated by a framework that specifies the variables and their relationships. Several national groups are currently leading efforts to develop common frameworks, instruments, and benchmarks that will guide assessment of quality, cost, and outcomes of technology-enabled courses and programs (Ehrmann, 1994; Ehrmann & Zuniga, 1997; Harrison, Seeman, Behnm, Saba, Molise, & Williams, 1986; Johnstone & Krauth, 1996). The need for a consistent, theory-driven approach to educational program assessment/evaluation has been urged by nurse educators as well (Cobb & Billings, in press; Ingersoll, 1996; Ingersoll & Sauter, 1998).
The framework proposed here has been adapted from models developed to study the impact of the use of technology in higher education (Chickering & Ehrmann, 1996; Ehrmann 1995; Ehrmann & Zuniga, 1997; Johnstone & Krauth, 1996, Harrison, et al.). A review of nursing literature about distance education (DE) and the emerging literature from pilot projects and classroom studies of the uses of Web-based and Web-enhanced courses guided the development of this framework for nursing.
The framework has 5 major concepts (Figure) with their associated operational variables (Table). Although not all concepts or variables may be of interest or relevant to every study, the framework is intended to provide a context for relating a variety of likely variables.
The framework begins with outcomes that are enabled by Web-based courses (full Web courses, Webenhanced courses). The outcomes are influenced by educational practices. Effective teaching and learning in DE courses and programs are dependent on faculty and student development and orientation for role changes, as well as technical support. Thus, the third and fourth components of the model are faculty support and learner support. Successful Web courses require reliable Internet connections and appropriate use of learning tools (e.g., discussion boards, collaborative learning tools, email). The final component of the model, then, is the use of the technology. Each of the variables are further described as follows.
Figure. Framework for assessing outcomes and practices in Web-based courses in nursing.
Outcomes identified in the model include those course and program outcomes typically associated with undergraduate and graduate nursing curricula. Other outcomes that are particularly enabled by Web-based learning technologies are also included.
Learning. There have been innumerable studies of learning outcomes in distance education in nursing. Generally these studies have revealed no significant differences in learning outcomes when comparing technology mediated courses with classroom courses, regardless of the type of technology (print, audio, television or Internet) (Billings & Bachmeier, 1994; Bachman, & Panzarine, 1998; Rosenblund, Damask-Bembenek, Hugie, & Matsumura, 1999; Ryan, Carlton, & Ali, 1999). Critical thinking (Ryan, Carlton, & Ali, 1999; Todd, 1998) and creativity (Niederhauser, Bigley, Hale, & Harper, 1999) have been particularly highlighted as learning outcomes of Web-based courses.
Recruitment, Retention, Graduation, Employment, and Certification. As with other types of DE courses, Webbased courses can be an important recruitment strategy for schools of nursing. Additionally, the use of the Internet makes DE an attractive, cost-effective option for attracting learners world wide.
Access. Using DE to provide access to educational programs has been the primary reason for nursing schools to offer distance education programs. The need for access often exists because of geographic distances, to provide workforce development in rural areas, and for nurses returning for additional degrees who are employed and/or have family responsibilities.
Convenience. Because of the asynchronous learning tools used in Web-based courses, students can participate in these courses any time and any place, thus making course participation convenient. Cragg (1994) found that registered nurses in a post-RN baccalaureate program found "time-shifting," the ability to participate in learning activities at the learners' convenience to be a major advantage of a computer-mediated conference course. Connors, Smith, DeCock, & Langer (1996) note that many nurses want to further their education, and asynchronous Internet courses provide that convenience. Cobb & Mueller (1998) also found convenience and the ability to work at the student's own pace to be major benefits for enrolling in Web-based courses.
Assessing Outcomes and Practices in Web-Based Courses: Variables and Definitions
Connectedness. Although Web-based courses and collaborative learning tools such as email and discussion forums enable students and faculty to be connected to each other for frequent and meaningful interaction, many studies have revealed that students in Web courses feel isolated (Cobb & Mueller, 1998; Cragg, 1994; Landis & Wainwright, 1996). Ryan, Carlton, and Ali (1999) reported that students felt disconnected from the faculty, and Anderson (1995) reported that some of the students in her course had strong preferences for face-to-face meetings with faculty. Landis and Wainwright (1996) noted that off-campus students had greater needs for connectedness and that computer conferencing software may have contributed to meeting this need.
Preparation for Real-World Work. Nursing is a practice profession that requires transfer of learning to clinical practice, and because of the ease of access to real world settings and situations, Internet technologies have the capabilities of preparing nurses for work in the knowledge age. For example, simulations using data bases, clinical information systems, or spreadsheets (Gravely & Fullerton, 1998; Wambach, et al., 1999) can prepare nurses for the realities of practice. Facilitating client conferences on the Web, developing teaching care plans (Colling & Rogers, 1999), and assessing real and virtual clients are additional examples.
Computer Tool Proficiency (computer literacy). One of the reasons faculty integrate experiences with Web-based learning technologies into courses is to provide an opportunity for students to acquire computer literacy skills and proficiency with knowledge tools such as search engines, conducting online searches, using bibliographic databases, and accessing large databases for research. Most studies of Web-based learning report its beneficial effects on improving basic computer competency (Anderson, 1995; Bachman & Panzarine, 1998; Clark, 1998; Colling & Rogers, 1999; Kirkpatrick & Brown, 1999; Landis & Wainwright, 1996). Cragg (1994) found that students overcame their fear of computers and had improved selfesteem. McGonigle and Mastrian (1998), reported that students' feelings of being overwhelmed and frustrated with computers decreased after taking an Internet-based course. Bachman and Panzarine (1998) reported that computer skills improved and transferred to use in other courses, and Niederhauser, Bigley, Hale, and Harper (1999) found students had increased comfort with using the Internet one year after the course.
Internet-based technologies also have great power to enhance knowledge skills such as critical thinking, clini· cal decision-making, and use of data sets. Ribbons (1998), studying the development of higher order skills of metacognition, found that an instructional database served as a cognitive template for clinical decision-making. Other knowledge skills that can be developed in Web courses are online team building and collaboration.
Professional Practice Socialization. Nursing is a clinical profession with values, roles, and professional practices that are developed through collaboration with faculty, peers, experts, and mentors. Clinical practice experiences are essential to professional role socialization and development and require contact with role models. A variety of clinical practice models have been used to provide clinical experience and practice socialization in Web courses and programs (Block et al., 1999; Carlton, Ryan & Siktberg, 1998; Connors et al., 1996; Wambach et al., 1999). Some programs employ on-site faculty, others use faculty from the home campus who travel to outreach sites; preceptors and "limited-cohort'' programs are other strategies to assure clinical practice socialization.
Socialization can also occur through the use of peer groups that emerge from the Web course. Cragg (1994) found that students in Internet courses form strong bonds in online courses. Bachman and Panzarine (1998) also found that students in Internet courses communicated with their classmates and formed support groups that facilitated professional growth.
Acquisition of professional values is also an important program outcome, and these values can be developed in Web courses. Niederhauser, Bigley, Hale, and Harper (1999) noted improved abilities of students to be selfdirected, and pointed out that skills in accessing information developed in the course should promote the likelihood of lifelong learning.
Satisfaction. Student satisfaction with Web-based courses and learning experiences is critical to the ongoing success of Web-based courses and programs. When compared to similar educational experiences in the on-campus classroom, students are generally as satisfied with their Web course and indicate they would take a Webbased course again (Block et al., 1999).
Although students may be satisfied with the courses, they may be dissatisfied with the course delivery. Niederhauser, Bigley, Hale, and Harper, (1999) report that students were satisfied with the course experiences offered on the WWW, but were frustrated with the technology. Other studies (Cragg, 1994; Cobb & Mueller, 1998) have found similar mixed levels of student satisfaction; students were satisfied with the course, access, and convenience, but frustrated with the technology, and distressed by the lack of connectedness to their faculty and classmates.
Positive learning outcomes are best achieved by using teaching-learning practices adapted to the use of the technology. Seven principles of good practices in education have been identified, which, when used consistently, result in student learning and satisfaction (Chickering and Gamson, 1987). These principles, explained in the following paragraphs, include active learning, time on task, collaboration with peers, interaction with faculty, rich and rapid feedback, high expectations, and respect for diversity. These principles also are enabled by distance education technologies (Chickering and Ehrmann, 1997).
Active Learning. Students learn best when they are engaged and interact with the content and concepts of the course and assume responsibility for their own learning. McGonigle and Mastrian (1998) established goals in their RN-BSN transition course to promote active participation. When interactive activities such as scavenger hunts that required students to engage in the learning process were used, students' enthusiasm was high and written work revealed attainment of learning outcomes beyond course expectations. Case studies, databases, clinical problems, and simulations are other methods faculty can use to promote active learning (Niederhauser, Bigley, Hale & Harper, 1999; Ribbons, 1998; Ryan, Carlton, & Ali, 1999; Wambach, et al., 1999). In addition, richer discussions enabled by Web course learning tools can encourage more active participation in learning.
Time on Task. Achieving course learning outcomes is related to the time spent actively engaged in course activities. Web-based courses and learning experiences must be designed to ensure sufficient time for learning, while avoiding barriers that cause students to spend undue time learning how to use the technology, accessing the computer, logging on to the Internet, or reading a myriad of responses to questions or bulletin boards. Bachman and Panzarine (1998) found that students in their Web-enhanced course spent significantly more time using the computer for email, searching assignments, and accessing chat groups than those in a comparison group. Kirkpatrick, Brown, and Atkins (1998) found that students in Web-courses tended to go beyond the task assignment.
Although Web-based learning tools can promote active learning, critical thinking, and reflection, posting written comments and reading postings from faculty and classmates can be time-consuming. Niederhauser, Bigley, Hale & Harper (1999) reported that students spent five to ten hours per week on course related activities, but Ryan, Carlton, and Ali (1999) found no significant differences in the amount of time to complete assignments between classroom and Web courses.
Feedback. Feedback (from self, classmates, faculty, and experts) about the process and outcome of learning promotes progress towards attaining learning outcomes. One of the primary advantages of Web-based courses is the opportunity for rich and rapid feedback. Todd (1998) found that it was helpful to be able to give feedback to correct students' misunderstandings of course content. She also found that students in the course who had learning disabilitities made significant contributions to the discussion in a way that they had not in the classroom and thus she was able to give them additional feedback. Immediate access to faculty and the opportunity to answer questions or give feedback to drafts of work is another advantage of the Web (Anderson, 1995).
In Web courses, feedback is also important to overcome isolation and to reassure students that they are completing assignments according to course and faculty expectations. Ryan, Carlton, and Ali (1999) found that students preferred the immediacy of the verbal and nonverbal feedback they received in the classroom. In addition, students in these Web courses were unsure about how their responses were being graded and if they were progressing satisfactorily.
Student-Faculty Interaction. Learning is a social activity based on interaction with the faculty member both inside and outside of class and includes interaction about course content and processes, as well as personal and professional goals. Anderson (1995) found that Web-courses enabled discussion between students and faculty that might not have occurred in a traditional classroom, and Landis and Wainwright (1996) found that, with this interaction, there was less isolation and more of a sense of connectedness to the faculty for the students who were at outreach sites, than for students who attended class on campus.
Interaction and Collaboration Among Peers. Web courses, and the collaborative learning tools used to support these courses, have the potential to create learning communities that promote interaction among peers, collaborative group work, networking, and information sharing. Connors, Smith, DeCock, and Langer (1996) found increased personal interaction and sharing among students in a nurse practitioner program. Anderson (1995) noted a sense of community for students who were geographically dispersed, and Halstead, Hayes, Reising, and Billings (1995) reported increased collegiality when students from several campuses had an opportunity to have social connections mediated through computer conferencing software. Theile, Allen, and Stuckey (1999) reported that, because of the way the course used the Web, students were more likely to collaborate with their peers, and Ryan, Carlton, and Ali (1999) found that students believed the interaction in the Web modules enabled them to learn from each other and to be well prepared for class participation. Landis and Wainwright (1996) noted that students at outreach sites had greater need for and attached higher importance to interaction with classmates.
Respect for Diversity. Students come to all classes from diverse cultures with diverse ways of learning, and with varying learning abilities, styles, and interests. Web courses, through well-designed teaching and learning practices, have the capability of creating options for learning, eliciting and promoting respect for diverse opinions, and bringing individuals and groups with varying cultural backgrounds into the course. For example, Kirkpatrick, Brown, and Atkins (1998) and Kirkpatrick & Brown (1999) established an international exchange that highlighted international awareness and promoted cultural competence.
High Expectations. When faculty and students have high expectations for what can be accomplished, positive results are achieved. High expectations for quality courses, learning experiences, and outcomes from Web courses can be established in Web-based courses. Niederhauser, Bigley, Hale, and Harper (1999), reported the students in their Web courses were empowered and learned at their highest potential.
When Internet technology is used in nursing courses and programs, longstanding practices and traditions in teaching and learning are challenged. Unlike other forms of distance education (video, audio) that primarily replicate the faculty-centered classroom, teaching in Webbased courses requires a learner-centered approach to course and curriculum design, which may require assisting faculty with instructional design for the course, orientation to the use of technology, considerations for the workload involved, and rewards and recognition for serving as pioneers during paradigm shifts. Additionally, technical and legal issues of copyright, intellectual property, and scholarship must be addressed (AAUP, 1999).
Faculty Development and Assistance for Developing and Teaching Web Courses. Developing and teaching Web courses require changes in teaching practices as well as faculty support through planned faculty development. New theories and philosophies of education suggest that learning outcomes are improved when there is a shift from teacher-centered to learner-centered pedagogies (Barr & Tagg, 1995; Bonk & Cunnigham, 1998; Skiba, 1997), and that faculty benefit from working with a team of pedagogical and technical consultants to design Web courses. Wambach et al. (1999), Carlton, Ryan, and Siktberg (1998), and Rosenlund and colleagues (1999) described the faculty development process and the multidisciplinary team of experts needed to assist faculty design Web courses: instructional designers, librarians, Web programmers, computer support personnel, and clerical assistants.
Orientation to Technology. In addition to course redesign, faculty support includes orientation to the hardware and software used to offer the course. As several authors (Milstead & Nelson, 1998; Verhey 1999) have pointed out, faculty must have computer skills before integrating them into the curriculum, and time must be available for faculty to acquire these skills (AAUP, 1999).
Ongoing Technical Support. Support for technology must be available during course development, implementation, and revision. Changing software and hardware and using updated versions of tools all require support (Milstead & Nelson, 1998).
Workload Recognition. Faculty workload increases in DE courses because of the time needed for orientation to the technology, the time needed to develop new teaching materials, and the time required to plan learning activities and evaluation strategies. Course development can take several semesters (AAUP, 1999; Milstead & Nelson, 1998; Ryan, Carlton, & Ali, 1999; Wambach et al., 1999). McGonigal and Mastrian (1998) describe the steps that are involved in developing a Web learning experience and advise adequate planning time. Additional time, at least initially, also may be required to teach the course. Todd (1998) and Rosenlund and colleagues (1999) notes there is increased time spent when giving individual feedback to large numbers of students.
Faculty Rewards and Recognition. Faculty who teach Web courses should have recognition for their work from administrators, colleagues, and peers, and formal plans for these activities should be in place (AAUP, 1999). Promotion, tenure, and merit raises are other ways faculty excellence in teaching can be rewarded.
Web-based courses place additional demands on students as well. Students must be self-directed and self-starting (Ryan, Carlton, & Ah, 1999). However, developing these attributes is likely to happen earlier and more easily when students are informed about the requirements of Webbased courses, are oriented to the technology, have ongoing technology support, have access to learning resources, and can use online student services such as those provided by the registrar, bursar, advising, records, bookstores, and academic counseling (Mueller & Billings, in press).
Information. Web-based courses are substantially different from on-campus courses and students should have information about the course, that it will be offered on the Internet, how to assess their readiness for Web-based learning, the amount of time required for the course, the hardware and software requirements, and the computer literacy expectations (Milstead & Nelson, 1998; Ryan, Carlton & Ali, 1999). For example, Gravely and Fullerton (1998) provided students with a tutorial three weeks before the course began; Milstead and Nelson (1998) allowed sufficient start-up time to solve any problems. Todd (1998) recommends giving detailed information about computer assignments and expectations in the syllabus. Cobb and Mueller (1998) found that even though students had been informed that a course would be offered on the Internet in the course catalog, a follow-up letter was helpful in alerting students to the resources for acquiring computer skills.
Orientation to the Use of the Technology. Students must be oriented to the technical components of using the technology before the course begins. This can be accomplished by using student handbooks; posting orientation information on the Internet; conducting orientation sessions online, on-campus, or at the outreach site prior to the use of the technology; or using the technology itself during the first class session. Students are initially uncomfortable with class participation when using new technology and benefit from time spent on orientation to the technology. Cragg (1994) found that face-to-face orientation for the technology was the best approach. Niederhauser, Bigley, Hale, and Harper (1999) required an on-campus short course before the Web assignments in the course were given; and Halstead, Hayes, Reising, and Billings (1995) used a campus orientation preceded by a newsletter and followed by having a student consultant to assist students learn the conferencing software. Orientation to the use of the technology also includes specifics about using course software, collaborative work tools, and to "netiquette" related to being a member of the online learning community.
Students who are successful in Web courses are usually self-directed, independent learners, able to network with classmates and faculty, and able to seek assistance, request feedback, and collaborate with others to enhance their learning and that of others. Therefore, orientation should include information about these student responsibilities as well.
Ongoing Technology Support. Students need ongoing technical help as they use hardware/software and Internet service providers, log into courses, and post information to the course. Technical help is particularly critical during the first few weeks of the course while students are learning the use of the hardware and software (Rosenlund & colleagues, 1999; Todd, 1998). Cragg (1994) notes that students need technical help to prevent frustration, and it should be available 24 hours a day, 7 days a week (Milstead and Nelson, 1998).
Learning Resources. Students in Web courses often do not have easy access to libraries and learning labs-the basic learning resources for on-campus support. Learner support in Web courses, therefore, must include provisions for assuring that course assignments can be accomplished with resources that can be obtained from the Internet, local libraries, health care agencies, or from instructor-developed course resource packets.
Student Services. Services must be available for students who are at a distance from the originating site of the educational offering. Ideally, for example, academic advising, access to the bookstore (online and mail order), registration, bursar, and financial aid services should be available to students without coming to campus. Not all colleges and universities with schools of nursing offering DE courses have well-developed systems, although most are now using the Internet or telephone to provide immediate access to necessary student services or are providing access through outreach sites (Block, 1999; Milstead & Nelson, 1998).
Of particular concern for students who are taking Web courses is the need for personal and study support systems. Because of the isolating effects of studying alone, some students need additional help and, therefore, services often offered on-campus such as study groups, writing centers, and tutoring sessions, may not be available in a Web environment.
Use of Technology
The use of the Internet not only makes education accessible and convenient, but is changing the way faculty and students teach, learn, develop, and disseminate knowledge. The infrastructure supporting Web courses must be accessible and reliable, and faculty and students must use appropriate hardware and course management software to support course goals and learning activities.
Infrastructure. There is abundant anecdotal evidence in the literature about students' inability to access the Internet, system "crashes," local service providers losing connections during storms, and resultant anxiety for students. Having accessible and reliable infrastructure is imperative (Connors, Smith, & DeCock, 1996). Additionally, Theile, Allen, and Stuckey (1999) found that lack of access at home is a barrier for students in Web courses.
Use of the Technology. Successful teaching and learning depends not only on reliable hardware and Internet connections, but also on course management software, and related learning tools such as email, discussion, forum, chat, testing, and cognitive templates. These tools must support teaching, learning, and evaluation, and be used appropriately (Carlton, Ryan, & Siktberg, 1998). Cobb (1998) and Cobb and Mueller (1998) found that students were overwhelmed by the large number of messages on the bulletin board, some of which were not relevant to the content and learning activities of the course. Also, when course management software is cumbersome or used inappropriately, students can become side-tracked by surfing and engaging in noncourse dialogue, and thus increasing frustration with the course.
On the other hand, Web-based courses can promote productive use of study time by providing access to student services and learning resources, saving time that might otherwise be spent standing in line to register for classes or checking out library materials. Time management can also be enhanced by ease of contact with faculty and classmates. Gravely and Fullerton (1998) reported that students were more productive in an Internet-based course because they received answers to their questions in a timely manner. Ryan, Carlton, and Ali (1999) indicate that students in their courses appreciated saving time by not having to commute to campus.
Teaching and learning in a Web-based environment is a complex and multivariate process. A framework for identifying the components and their relationships can serve to guide assessment/evaluation activities. Nursing faculty teaching in Web courses and other stakeholders are asking questions about the relationships between teaching/learning practices and outcomes, faculty and student support and teaching/learning practices and outcomes, and the overall utility of the use of Web technology. These questions will be answered as nursing faculty use theory-driven frameworks to guide inquiry that will provide evidence for determining the best practices for teaching and learning in Web-based courses.
- American Association of University Professors. (1999). Special Committee on Distance Education and Intellectual Property Issues [On-line]. Available: http://www.AAUP.org/spcintro.htm.
- Anderson, D.G. (1995). Electronic education: E-mail links students and faculty. Nurse Educator, 20(4), 8-11.
- Bachman, J.A., & Panzarine, S. (1998). Enabling student nurses to use the information superhighway. Journal of Nursing Education, 37(4), 155-161.
- Barr, R, & Tagg, J. (1995). From teaching to learning- a new paradigm for undergraduate education. Change, (November J December), 13-25.
- Billings, D.M. (1997, January). Current status and future directions of distance education in nursing. Paper presented to Division of Nursing. Alexandria, VA.
- Billings, D.M., & Bachmeier, B. (1994). Teaching and learning at a distance: A review of the literature. In L.R. Allen (Ed.), Review of research in nursing education (pp 1-32). New York: National League for Nursing.
- Block, D., Josten, L.E., Lia-Hoagber, B., Kerr, M., Smith, M.J., Lewis, MX., & Hutton, S.J. (1999). Fulfilling regional needs for specialty nurses through limited cohort graduate education. Nursing Outlook, 47(1), 23-29.
- Bonk, C.J., & Cunningham, D.J. (1998). Searching for learnercentered, constructivist, and socioculturel components of collaborative educational learning tools. In GJ. Bonk & K.S. King (Eds.), Electronic collaborators (pp. 25-50). Mahwah, NJ: Lawrence Erlbaum Associates.
- Byrne Colling, K., & Rodgers, A.E. (1999). Nursing students surf the Web: Resources for patient teaching. Journal of Nursing Education, 38(6), 1-3.
- Chickering, A. W., & Ehrmann, S. (1996). Implementing the seven principles: Technology as lever [On-line]. Available: http://www.tltgroup.org/ehrmann.htm.
- Chickering, A. W., & Gamson, Z.F. (1987). Seven principles for good practice in undergraduate education. AAHE Bulletin, 39(7), 3-6.
- Clark, D.J. (1998). Incorporating an Internet Web site into an existing nursing class. Computers in Nursing, 16(4), 219-222.
- Cobb, K, & Yago, K. (1997). Evaluation by students enrolled in nursing courses offered via the World Wide Web: Using a flashlight in the dark. Paper presented at the Distributed Education Conference.
- Cobb, K., & Billings, D. (in press). Assessing distance education in nursing. In J. Novotny (Ed.), Learning together apart -distance learning in nursing. New York: Springer Publishing.
- Cobb, K.L., & Mueller, C. (1998). Evaluation of graduate nursing students in a virtual classroom. Unpublished manuscript.
- Connors, H.R., Smith, C, DeCock, T.S., & Langer, B. (1996). Kansas nurses surf Web for master's degrees. Reflections, 2nd Quarter.
- Cragg, CE. (1994). Distance learning through computer conferences. Nurse Educator, 19(2), 10-14.
- Erhmann, S. (1995). Asking the right questions: What does research tell us about technology and higher learning? Change, 27(2), 20-27.
- Ehrmann, SC, & Zuniga, R.E. (1997). The flashlight evaluation handbook. Washington, DC: Corporation for Public Broadcasting.
- Graveley, E., & FuUerton, JT. (1998). Incorporating electronicbased and computer-based strategies: Graduate nursing courses in administration. Journal of Nursing Education, 37(4) 186-188.
- Halstead, J, Hayes, R., Reising, D., & Billings, D. (1995). Nursing student information network: Fostering collegial communications using a computer conference. Computers in Nursing, 13(2), 55-59.
- Harrison, PJ, Seeman, B.B., Behnm, R., Saba, F, Molise, G., & Williams, M.D. (1991). Development of a distance education assessment instrument. Educational Technology Research and Development, 39(4), 65-77.
- Hodson-Carlton, KH., Ryan, M.E., & Siktberg, L.L. (1998). Designing courses for the internet. Nurse Educator, 23(3), 4550.
- Ingersoll, GL. (1996). Evaluation research. Nursing Administration Quarterly, 20(4), 28-40.
- Ingersoll, G.L., & Sauter, M. (1998). Integrating accreditation criteria into educational program evaluation. Nursing and Health Care Perspectives, 19(b), 224-229.
- Johnstone, S.M., & Kreuth, B. (1996). Some principles of good practice for the virtual university. Change, 28(2), 39-41.
- Kirkpatrick, M.K., & Brown, S. (1999). Efficacy of an international exchange via the internet. Journal of Nursing Education, 38(6), 1-4.
- Kirkpatrick, M.K., Brown, S., & Atkins, T. (1998). Electronic education: Using the internet to integrate cultural diversity and global awareness. Nurse Educator, 23(2), 15-17.
- Landis, B.J, & Wainwright, M. (1996). Electronic education: Computer conferencing; Communication for distance learners. Nurse Educator, 21(2), 9-14.
- McGonigle, D., & Mastrain, K. (1998). Learning along the way: Cyberspacial quests. Nursing Outlook, 46(2), 81-86.
- Milstead, J.A., & Nelson, R. (1998). Preparation for an online asynchronous university doctoral course: Lessons learned. Computers in Nursing, 16(5), 247-258.
- Mueller, C, & Billings, D. (1999). Distance education: Focus on the learner. In J Novotny (Ed.), Learning together apart- distance learning in nursing. New York: Springer Publishing.
- Neiderhauser, V.P., Bigley, M., Hale, J, & Harper, D. (1999). Cybercases: An innovation in internet education. Journal of Nursing Education, 38(6), 1-4.
- Ribbons, R.M. (1998). The use of computers as cognitive tools to facilitate higher order thinking skills in nurse education. Computers in Nursing, 16(4), 223-227.
- Rosenlund, C, Damask-Bembenek, B., Hugie, P., & Matsumura, G. (1999). The development of online courses for undergraduate nursing education. Nursing and Health Care Perspectives, 20(4), 194-198.
- Ryan, M., Hodson Carlton, K, & Ali, N.S. (1999). Evaluation of traditional classroom teaching methods versus course delivery via the World Wide Web. Journal of Nursing Education, 38(6), 1-6.
- Skiba, D. (1997). Transforming nursing education to celebrate learning. Nursing and Health Care Perspectives, 18(3), 124129, 148.
- Thiele, JE., Allen, C, & Stucky, M. (1999). Effects of web-based instruction on learning behaviors of undergraduate and graduate students. Nursing and Health Care Perspectives, 20(4), 199-208.
- Todd, NA. (1998). Using e-mail in an undergraduate nursing course to increase critical thinking skills. Computers in Nursing, 16(2), 115-118.
- Verhey, M. (1999). Information literacy in an undergraduate nursing curriculum: Development, implementation, and evaluation. Journal of Nursing Education, 38(6), 1-8.
- Wambach, K, Boyle, D., Hagemaster, J, Teel, C, Langner B., Fazzone, P, Connors, H., Smith, C, & Forbes, S. (1999). Beyond correspondance, video conferencing, and voice-mail: Internet-based masters courses in nursing. Journal of Nursing Education, 38(6), 1-5.
Assessing Outcomes and Practices in Web-Based Courses: Variables and Definitions