Journal of Nursing Education

Major Article 

The Who, What, and How of Evaluation Within Online Nursing Education: State of the Science

Bedelia H. Russell, MSN, RN, CPNP-PC, CNE

Abstract

The resource capacity in nursing programs has a direct impact on student admissions and number of graduates who enter the nursing workforce. Online delivery of nursing education is identified as a solution to expand nursing program capacity. As nursing programs continue to address capacity with online course delivery, it is essential that nurse educators maintain consistent evaluation practices to ensure successful and positive outcomes, compared with traditional models. Evaluation is a central component to determine program quality and mastery of learning outcomes. This article examines the state of the science around the current evaluation of educational practices, instructional strategies, and outcomes within the context of online nursing education. Thirty-six articles met the inclusion criteria. Despite substantive contributions to the state of the science, the findings reflect evaluation practices that are diffuse and superficial and serve as the basis for future recommendations and research opportunities. [J Nurs Educ. 2015;54(1):13–21.]

Ms. Russell is Interim Dean and Assistant Professor, Whitson-Hester School of Nursing, Tennessee Tech University, Cookeville, Tennessee.

The author has disclosed no potential conflicts of interest, financial or otherwise.

The author thanks Dr. Patricia Hayes for her continued support, wisdom, and encouragement.

Address correspondence to Bedelia H. Russell, MSN, RN, CPNP-PC, CNE, Interim Dean and Assistant Professor, Whitson-Hester School of Nursing, Tennessee Tech University, Robert and Gloria Bell Hall, Office 244, PO Box 5001, Cookeville, TN 38505; e-mail: bhrussell@tntech.edu.

Received: December 16, 2013
Accepted: October 14, 2014
Posted Online: December 28, 2014

Abstract

The resource capacity in nursing programs has a direct impact on student admissions and number of graduates who enter the nursing workforce. Online delivery of nursing education is identified as a solution to expand nursing program capacity. As nursing programs continue to address capacity with online course delivery, it is essential that nurse educators maintain consistent evaluation practices to ensure successful and positive outcomes, compared with traditional models. Evaluation is a central component to determine program quality and mastery of learning outcomes. This article examines the state of the science around the current evaluation of educational practices, instructional strategies, and outcomes within the context of online nursing education. Thirty-six articles met the inclusion criteria. Despite substantive contributions to the state of the science, the findings reflect evaluation practices that are diffuse and superficial and serve as the basis for future recommendations and research opportunities. [J Nurs Educ. 2015;54(1):13–21.]

Ms. Russell is Interim Dean and Assistant Professor, Whitson-Hester School of Nursing, Tennessee Tech University, Cookeville, Tennessee.

The author has disclosed no potential conflicts of interest, financial or otherwise.

The author thanks Dr. Patricia Hayes for her continued support, wisdom, and encouragement.

Address correspondence to Bedelia H. Russell, MSN, RN, CPNP-PC, CNE, Interim Dean and Assistant Professor, Whitson-Hester School of Nursing, Tennessee Tech University, Robert and Gloria Bell Hall, Office 244, PO Box 5001, Cookeville, TN 38505; e-mail: bhrussell@tntech.edu.

Received: December 16, 2013
Accepted: October 14, 2014
Posted Online: December 28, 2014

Currently, many schools of nursing find themselves at a crossroads of resource demand and capacity. The national impetus for the need for innovative and collaborative programs, accelerated entry into workforce educational programs, and diversification of the nursing workforce is an ever-present discussion within schools of nursing. Faculty and administration face an ongoing struggle about how to efficiently use resources and increase enrollment to keep up with demand, while simultaneously maintaining program outcomes and evaluation standards. Online delivery of nursing education has been identified by the Institute of Medicine ([IOM], 2003, 2009, 2011) and the American Association of Colleges of Nursing ([AACN], 2000a, 2000b, 2003) as a solution to expand program capacity and increase access to nursing education. According to the AACN (2014), in 2011 more than 400 nursing programs had partial online components. Kolowich (2010) reported that the number of fully online degree programs from 2007 to 2009 increased from 96 to 129. As nursing programs expand capacity through increased numbers of programs offered through various online modalities, a need exists to maintain rigorous evaluation for educational practices, instructional strategies, and learning outcomes to ensure program quality. The state of the science around the current evaluation of educational practices, instructional strategies, and outcomes within online nursing education is described in this article. Specifically, the questions of (a) who is evaluated (populations of interest), (b) what is evaluated (focus of evaluation), and (c) how evaluation is accomplished (method of evaluation) are explored.

Background

Higher Education Online Learning in the United States

The annual report by Allen and Seaman (2013) on the state of online learning in U.S. higher education reflected 10 years of tracking online education. That report showed an all-time high of 69.1% among chief academic leaders who indicated that online education is a critical part in their long-term strategy. In addition, the report indicated an all-time high of 32% of all higher-education students who have taken at least one online course. The proportional increase in the number of students who have taken an online course brings the total number to 6.7 million. Faculty continue to report that increased time and effort is necessary to teach online courses, compared with traditional face-to-face courses. Approximately 77% of academic leaders indicated that learning outcomes for online courses are the same or better than learning outcomes for face-to-face instruction (Allen & Seaman, 2013). Conversely, these academic leaders also cited retention of online students, employer and faculty perception of online degrees, and the need for discipline among online learners as barriers to the adoption of online learning (Allen & Seaman, 2013). Online learning encompasses a variety of course formats, including online, blended or hybrid, or Web-facilitated (Allen & Seaman, 2013). Allen and Seaman (2013) defined online courses as those with “at least 80% of the course content delivered online” (p. 7).

Online Learning in Nursing Education

Online learning is well-established in the nursing education literature, often with the terms distance learning, online education, or distance education used synonymously (Billings, Connors, & Skiba, 2001; Childs, Blenkinsopp, Hall, & Walton, 2005; Dutile, Wright, & Beauchesne, 2011; Skiba & Barton, 2006). In an editorial by Billings (2007), the concept of distance education was described as the “separation of the educator and learner in time and space” and, more broadly, as “the use of technology to support the connections that establish the learning community” (p. 121). Billings provided a brief 25-year review of distance education in nursing through the provision of historical context for adoption and evaluation emphasis, noting the early need of educators and administrators for “evidence” of distance education technology effectiveness prior to the adoption of distance education technology (p. 121). Billings (2007) marked the transition from studies that examined the buy-in of various distance education methods and technology effectiveness to studies that established no significant difference between a variety of phenomena associated with distance education. From the mid-1990s to mid-2000s, the pedagogical shift in educator thinking from teaching to learning was noted in the context of an online learning environment, with emphasis on how students learned, learning styles, and best practices for teaching and learning. The result was a focus on distance education and online learning and the development of theoretical models and best practices (Billings, 2000; Jeffries, 2000, 2005a, 2005b). In addition, professional nursing education organizations began to take notice, and subsequent white papers, position statements, and accreditation alliance statements evolved (AACN, 2000b, 2003; Council for Regional Accrediting Commissions, 2001; Western Interstate Commission on Higher Education, 1995).

As the adoption of online education by nursing programs expanded from the mid-2000s and beyond, additional studies emerged, with emphasis on the student and faculty experience in the online teaching–learning environment and the similarities among the online education environment and nurses’ daily practice encounters with technology use (Mancuso, 2009; Mancuso-Murphy, 2007 ; Simpson, 2006).

As more Web-intelligent technology is used for online learning in nursing education, synchronous and asynchronous course delivery and even virtual, simulated, or immersive experiences need to “continue to be mindful of the ongoing need to assess distance education practices and outcomes” (Billings, 2007, p. 121). Although additional studies continue to examine ideal faculty characteristics for online teaching (Ansah, Neill, & Newton, 2011; Jones & Wolf, 2010), nursing programs and educators must keep in mind the established position statements. The Alliance for Nursing Accreditation on Distance Education states that fully online or hybrid nursing programs should meet the same academic learning support standards and accreditation standards as traditional face-to-face programs ( AACN, 2003). The Alliance statement further added that student outcomes should remain consistent across all delivery formats and that programs should establish assessment and evaluation criteria to measure the outcomes. These statements are also in line with standards of regional accreditation bodies for institutions of higher education.

Purpose and Guiding Questions

Given the continued expansion of online nursing education, the author sought to examine the current state of the science of online learning and nursing education. More specifically, the primary question— What is the state of the science around current evaluation practices within online nursing education?—was established. To better explore the construct of online nursing education, secondary questions were developed. These questions were: Within the context of online nursing education, (a) who are the populations of interest being evaluated? (b) What educational practices, instructional strategies, and outcomes are being evaluated? and (c) How are educational practices, instructional strategies, and outcomes being evaluated? The context and variables of interest were defined prior to the initial literature search (Table 1).

Context and Variables of Interest Defined Prior to the Literature Search

Table 1:

Context and Variables of Interest Defined Prior to the Literature Search

Method

Search Strategies and Critique Methods

A systematic and iterative approach to search methodology was inspired by Whittemore and Knafl (2005a, 2005b). This methodology allowed for the review of both the theoretical and empirical literature across disciplines. A priori parameters for English- only and theoretical and empirical articles published between 2008 and 2013 were established. An initial computerized database search of CINAHL®, Nursing & Allied Health Collection, and Health Source®: Nursing/Academic Edition was performed using the search terms shown in Table 2 . The terms in the first column were searched independently. Next, the search term shown in the second column was added to the first term and was searched again to refine the search results. A third search was conducted, using the first terms, the second term, and each of the terms individually listed in the third column to best reduce the yield of the search results to a more relevant list of articles. After removal of duplicates, these searches yielded 519 articles as potential sources of data.

Search Terms for the Literature Search

Table 2:

Search Terms for the Literature Search

A full abstract review of the identified articles assisted in the development of inclusion and exclusion criteria around populations of interest. With the goal to include prelicensure and graduate nursing students at the postsecondary level of education, articles were excluded if the population emphasis was on practicing nurses, staff or professional development, or continuing education of staff nurses. The result was 156 articles identified for review.

A secondary computerized database search of PsycINFO® using the terms distance learning AND nursing education, distance education AND distance learning, online learning AND nursing education, and online education AND nursing education, with the same date and language limiters as the primary database search, yielded an additional nine potential articles. The same search criteria, with the additional limiters of higher education and postsecondary education, was applied within the ERIC database and yielded an additional 38 results. Following a full abstract review of these additional results, the final article count identified as potential data sources from all searches was 203.

A full-article review identified additional literature for exclusion. Nonsubstantive editorials, commentary, descriptions of technology use in education, or logistics for application of technology within education were eliminated due to the lack of methodological rigor, lack of theoretical or empirical basis, or lack of reference to formal evaluation. The result was 97 articles for consideration, with subsequent application of posteriori inclusion and exclusion criteria.

Inclusion and Exclusion Criteria

The remaining 97 full articles were reviewed in the context of the established guiding questions, with the population of interest being an additional exclusion criterion. The final inclusion criteria for populations of interest was baccalaureate, including RN-to-Bachelor of Science in Nursing (BSN) completion, students; graduate students at the master’s and doctoral levels; and faculty of these student populations. The student populations excluded were licensed practical nurses, associate degree nurses, and student groups that lacked specificity. Articles and studies conducted outside of the United States were eliminated as well. Because application of this exclusion criterion (non-U.S. nursing programs) eliminated studies conducted with nursing and interprofessional students, the studies conducted within the United States, which included interprofessional students, along with nursing students, were also eliminated. Additional inclusion criteria were applied to emphasized articles that (a) reflected clarity in purpose or aim, (b) referenced an evaluation method and was theoretical or empirical based, or (c) referenced principles or best practices, teaching–learning theories, specific educational practices or outcomes, specific types of instructional technology with evaluation, or a validated scale, questionnaire, or survey. The application of final inclusion and exclusion criteria yielded 32 articles for consideration. A manual search of the references within these articles resulted in the review of an additional eight articles, with only four identified as a substantive contribution. The final total number of articles identified for inclusion in the current state of the science article was 36.

Findings

Evaluation Practices Within Online Nursing Education

A final inclusion dataset of 36 articles was used to answer each of the primary and secondary questions. Within the context of online nursing education, 21 empirical studies and 15 theoretical or descriptive studies were published between 2008 and 2013 (Table 3). A delineation of empirical work, both quantitative and qualitative, and theoretical or descriptive work was distinguished to better organize the report of findings to answer the primary and secondary questions (Tables A C; available in the online version of this article).

Types of Studies, Populations of
Interest, and Evaluation Focus of Literature Search

Table 3:

Types of Studies, Populations of Interest, and Evaluation Focus of Literature Search

Qualitative Empirical Studies. Of the six qualitative empirical studies, the populations of interest included nursing faculty, instructional designers, and RN-to-BSN students, including accelerated RNs, with one of the six studies inclusive of graduate nursing students (Table A). Each qualitative study of students focused on the online learning environment and the evaluation of affective outcomes relative to the teaching–learning environment. These affective outcomes included student perception of sense of community, student socialization and learning, peer dynamics on student learning, and student emotional factors and sense of community (Gallagher-Lepak, Reilly, & Killion, 2009; Holley & Taylor, 2009; McIntyre, McDonald, & Racine, 2013; Reilly, Gallagher-Lepak, & Killion, 2012). The studies of faculty emphasized the educational practice of course redesign and transition to teaching in a different learning environment, but only one held specific reference to any type of instructional strategy ( Smith, Passmore, & Faught, 2009).

All of the qualitative empirical studies used either individual or focus group interviews. Methods of evaluation were primarily inductive analysis of transcripts through content analysis, coding, and thematic analysis (Table A). Every study noted a step of intercoder reliability, and two studies specified the use of member checks (Holley & Taylor, 2009; Reilly et al., 2012). Four of six studies referenced a theoretical underpinning, philosophical assertion, or best practice (Holley & Taylor, 2009; McIntyre et al., 2013; Reilly et al., 2012; Smith et al., 2009).

Quantitative Empirical Studies. Of the 15 empirical quantitative studies, the population of interest was inclusive of all levels of nursing education from prelicensure to graduate students (Table B), and one quantitative study focused on nursing faculty (Reneau, 2013). Educational practices, with an emphasis on the environment of online learning, were evaluated by seven studies; eight focused on active learning, with one of the total studies also incorporating a focus on peer–faculty interaction. The type of instructional strategy evaluated was clearly delineated within those studies, where the educational practice emphasis was on active learning or peer–faculty interaction (Table B). In those studies, the evaluated outcome focused on the effectiveness of the instructional strategy in relation to student satisfaction, retention, course grades, course evaluations, clinical knowledge or skills, student recall and clinical performance, and student perception on learning and satisfaction (Table B). In studies where the emphasis was on the context of the online learning environment, each examined the affective learning domain. All but one of the studies (Smith, 2010) evaluated an outcome of student perception, whereas others examined student perception of learning (Burress, Billings, Brownrigg, Skiba, & Connors, 2009; Cobb, 2009; Wells & Dellinger, 2011) and student-perceived social presence or community (Cobb, 2009; Mayne & Wu, 2011).

In all but seven studies, the format of evaluation included a researcher-developed survey or evaluation, in addition to another standardized instrument, scale, tool, or questionnaire (Carpenter, Theeke, & Smothers, 2013; Gilmore & Lyons, 2012; Guzic et al., 2012; Kardong-Edgren & Emerson, 2010; Langley & Brown, 2010; McLain, Biddle, & Cotter, 2012; Vogt, Schaffner, Ribar, & Chavez, 2010). The lack of a standardized instrument, scale, or questionnaire correlated with those studies that emphasized the educational practice of active learning or peer–faculty interaction and the evaluation of instructional strategy effectiveness as an outcome (Table B). Inferential statistics were used to report data in all but two of the quantitative studies (Gilmore & Lyons, 2012; Kardong-Edgren &Emerson, 2010). All studies reflected some degree of reported descriptive statistics, with descriptive, comparative, or correlational studies being the most predominant. In regard to theoretical framework, learning theories, or best practices, only three studies lacked such references (Gilmore & Lyons, 2012; Guzic et al., 2012; Hall, 2010). The Benchmarking Best Practices in Web-Based Courses (Billings, et al., 2001) and Mayer’s multimedia theory (Vogt et al., 2010) were the only theories or models referenced more than once (Table B).

Theoretical or Descriptive Studies. An additional 15 articles reflected a broader theoretical or descriptive basis and included the context and variables of interest, along with reference to evaluation methods (Table C). Of this research, eight exclusively evaluated graduate nursing students at the master’s and doctoral levels, one was exclusive of faculty, five were exclusive of baccalaureate students, and one had a combined population of interest that included faculty and undergraduate and graduate nursing students. The evaluated educational practices of active learning (Culley & Polyakava-Norwood, 2012; Guhde, 2010; Rounds & Rappaport, 2008) and peer–faculty interaction ( Carlson, 2011) were clearly delineated. Of the remaining studies, the emphasis on evaluation of instructional strategies, type of instructional strategy, and utilization of the online learning environment for course content or assignment delivery occurred most frequently. Of the types of instructional strategies used, Adobe Connect® and Blackboard® were referenced the most. Six articles either did not specifically discuss a type of instructional strategy evaluated or provided a general reference to Web-based or online delivery (Table C).

The outcomes evaluated in the theoretical and descriptive studies emphasized student preference, experience, satisfaction, or effectiveness related to the instructional strategy or content redesign and delivery through online education (Table C). Three studies evaluated perceived cognitive outcomes of critical thinking, clinical competence, or clinical preparation, with each study having combined multimethod teaching strategies through online delivery (Guhde, 2010; Robbins & Hoke, 2008; Rutledge, Barham, Wiles, & Benjamin, 2008). All but four studies identified the format or method of evaluation through a researcher-developed survey, formative course evaluations, summative course evaluations, or surveys (Leners, Wilson, & Sitzman, 2007; McDonald & Walters, 2009; Rounds & Rappaport, 2008; Zsohar & Smith, 2008), whereas two studies used narrative reflection, in addition to the course evaluations (Guhde, 2010; Manning & Frisby, 2011). Consistent across all findings was a general satisfaction, or perceived effectiveness, by students and the need for clarity in assignments, roles, and introduction to use of new technology. All but five studies included learning theories, models, or best practices (Table C).

Discussion

Online Nursing Education: General Impression

It is evident that nursing education embraces online teaching and learning and integrates technology use from a broad and diverse curricular perspective. The adoption of online nursing education is well- established, with mechanisms for evaluation described in the literature. However, with the research conducted thus far, it is difficult to make straight comparisons and draw definitive conclusions, as the studies are diverse, lack depth of focus, or are often fragmented due to the breadth of research emphasis.

Populations of Interest. The predominant populations of interest are graduate students at the master’s level or students in baccalaureate completion programs. Traditional prelicensure students are the second most often studied population, followed by nursing faculty. Of the entire dataset, nursing faculty was a nominal population of research interest, with representation in a mere six of 36 articles (Tables A C). Students at the doctoral level were the least studied student population. At least seven studies integrated populations of interest, and the authors found that this complicated their report of findings. An overall comparison of the populations of interest reflects inconsistent levels and depth of the population description (Caudle, Bigness, Daniels, Gillmore-Kahn, & Knestrick, 2011; Hall, 2010; Johnson, 2008; Kavanaugh et al., 2009). The broad, and sometimes nonspecific, description of populations makes it difficult to compare similar populations or delineate differences across diverse populations.

Focus of Evaluation. Similar to the populations of interest, the overall focus of evaluation practices in online nursing education is diffuse with pockets of focus at different levels of inquiry and domains of learning. In regard to what is evaluated in online nursing education, the primary emphasis in the empirical literature has been on the affective domain of learning, with student satisfaction, perception, preference, and experience with the online environment central to inquiry. Researchers are largely interested in student engagement with perception of community, connectedness, and social presence, as well as factors that best promote these attributes to influence student-perceived learning and satisfaction.

The context of the online learning environment was explored in almost half of all of the empirical studies (Tables AB). Of these, only one provided a comparison between the online learning environment and the classroom learning environment (Hart & Morgan, 2010). In that study, the researchers examined the outcome of student attitudes toward academic integrity and cheating within the different learning environments.

Another central focus of evaluation is the effectiveness of educational practices or instructional strategies within the online environment, and it is sometimes compared with traditional teaching–learning environments. In eight of the studies, the authors described instructional strategies to develop an online course, integrate concepts or programs through online delivery, or develop an online program (Kavanaugh et al., 2009; Kelley & Klopf, 2008; Kidd, Morgan, & Savery, 2012; Leners et al., 2007; Manning & Frisby, 2011; McDonald & Walters, 2009; Robbins & Hoke, 2008; Zsohar & Smith, 2008). Two studies lacked focus on delivery of online education and were about how to augment standard instruction with the use of technology (Rutledge et al., 2008; Xu, Martin, & Gribbins, 2010). In addition, teaching strategy effectiveness tends to dominate the evaluation focus within the theoretical and descriptive literature when compared with the empirical literature. Consistent with what Billings (2007) described as the “no significant difference comparison” phenomenon (p. 122), researchers continued to report similar comparative studies (Guzic et al., 2012; Wells & Dellinger, 2011), despite past recommendations for future research. Student and faculty experience with content delivery, activity integration, and overall use of technology in the delivery of online education was most prevalent in the descriptive literature.

Method of Evaluation. Used across the majority of quantitative empirical studies was a consistent evaluation method of descriptive and inferential statistical analysis based on established valid and reliable scales and questionnaires. This method of evaluation was most often coupled with a researcher-developed survey or evaluation (Table B). Of the established valid and reliable scales, the Social Presence Scale and Classroom Community Scale were the most often utilized. Mayne and Wu (2011) performed the only study that emphasized the educational practice of active learning, which included a standardized scale and questionnaire. Examples of other standardized instruments, scales, tools, or questionnaires reflected across quantitative studies were the Social Presence Scale, the Satisfaction Scale, the Organizational Climate for Caring Questionnaire, the Donald-McCabe Academic Integrity Scale–Modified, Rovai’s Self-Report Classroom Community Scale, the Social Presence Questionnaire, the Cultural Diversity Questionnaire for Nurse Educators, the Inventory for Assessing the Process of Culture Competence–Revised, Kolb’s Learning Style Inventory, and the learner–interaction tool.

Of the qualitative empirical studies, interpretive analysis was the primary method of evaluation and proved to be the most consistent method of evaluation across the entire set of articles reviewed. When compared with the empirical literature, the theoretical and descriptive literature had the most inconsistent method of evaluation. Generally, the lack of specificity by the study authors and the reliance on individual course evaluations specific to a particular study limits the generalizability or ability to replicate and expand findings to an empirical level.

Use of Theories, Models, Frameworks, and Best Practices. Similar to the populations of interest, foci of evaluation, and method of evaluation in online nursing education, the use of theories, models, frameworks, and best practices lacks continuity and reflects a broad sweep of adult learning theories. A reference to theories, models, frameworks, or best practices is not exclusive to one particular type of study. Rather, application of these theoretical underpinnings was evident across all empirical studies, both qualitative and quantitative, as well as the theoretical and descriptive literature. Within the qualitative literature, examples of either theoretical underpinning, philosophical assertion, or best practices included the framework of student socialization by Weidman et al. (as cited in Holley & Taylor, 2009) and recommendations for telephone focus groups by Krueger and Casey (as cited in Reilly et al., 2012), with the general references to theories of student socialization, hermeneutics (Gadamer & Heidegger, as cited in McIntyre et al., 2013), Bloom’s taxonomy, and community of inquiry and social presence disciplines and paradigms (Table A).

Examples of theoretical frameworks, learning theories, or best practices across quantitative studies included Benchmarking Best Practices in Web- Based Courses (Billings et al., 2001), evaluating educational uses of Web-based courses, multimedia theory, Galland’s and Drinan’s model of academic integrity institutionalization (as cited in Hart & Morgan, 2010), uses and gratification expectancy model, the constructivist theory of learning, Paivio’s dual coding theory (as cited in McLain et al., 2012), Campinha-Bacote’s conceptual model of cultural competency (as cited in Reneau, 2013), Kolb’s learning styles (as cited in Smith, 2010), Seimann’s connectivism theory (as cited in Vogt et al., 2010), and the learner–interaction model. In addition, general references to problem-based learning, social presence theory, and general, adult, experiential, and transformative learning theories were noted. Benchmarking Best Practices in Web-Based Courses (Billings et al., 2001) and Mayer’s multimedia theory (as cited in Carpenter et al., 2013; Vogt et al., 2010) were the only theories or models referenced more than once (Table B).

Specific examples of learning theories, models, or best practices across the theoretical and descriptive literature included the model of online collaboration by Palloff and Pratt (as cited in Cully & Polyakova-Norwood, 2012), the stages of critical thinking skill development by Elder and Paul (as cited in Guhde, 2010), Kolb’s experiential learning theory (as cited in Kavanaugh et al., 2009; see also Manning & Frisby, 2011), seven principles for using technology for good practice in nursing education, the constructivist approach, Lewin’s change theory (as cited in Kidd et al., 2012), benchmarks and best practices for online nursing education (Leners et al., 2007), adult learning theory, constructivist pedagogy, problem-based learning, characteristics of an exemplary online course by Hooper and Harmon (as cited in McDonald & Walters, 2009), Campinha-Bacote’s cultural competence model (as cited in Rutledge et al., 2008), and the cooperative model of nursing education (Table C).

The constructivist theory of learning and Kolb’s experiential learning theory (Kardong-Edgren & Emerson, 2010; Kavanaugh et al., 2009 ; Kidd et al., 2012; McDonald & Walters, 2009 ; Manning & Frisby, 2011; Smith, 2010) were the most referenced learning theories. This is significant for online nursing education because these theories emphasize student-centered learning. It is important to note that the theories, models, and frameworks used by the researchers of the studies evaluated largely reflected and supported the affective learning domains, with minimal emphasis on evaluation of cognitive learning domains.

Limitations

The primary limitation was the key terms used for the literature search. Although the search terms online learning, distance learning, online education, and distance education provided a large volume of articles for review, and the frequent use of the search terms of e-learning or Web-based was noted consistently in the literature prior to 2007. The author of the current literature review also acknowledges the iterative search cease date, as articles relevant to the focus of the current article that were published after the respective date were excluded from consideration.

Implications and Recommendations for Future Research

The state of the science in the current article serves as a reflection on past and current research and is a basis for future research about the evaluation practices in the context of online nursing education. The past and current research has provided substantive contributions to nursing education, and the findings can assist nurse researchers with a new focus and direction. For example, populations of interest need to be better defined in study reports, and nursing education needs research that delineates whether there are some student populations that are better served with the delivery of online nursing education. With so few studies of nursing faculty and doctoral students, it seems clear that these populations are in need of the attention of researchers. Because of the diverse use of terminology to reference online nursing education, additional emphasis and consensus for a consistent definition and utility of the definition is also warranted.

The broader implication of this state of the science article reflects the evaluation practices within online nursing education, which are diffuse and superficial. Future research in online nursing education needs to expand outcomes evaluation away from the affective learning domain. The established research in this area has demonstrated positive affective learning outcomes, but much of it is perceived learning versus actual learning. Research is now needed to examine outcomes evaluations that are focused on the cognitive learning domains, with consideration to the ability to evaluate psychomotor outcomes in the online learning environment. One recommendation is to consider the Quality and Safety Education for Nurses competencies (QSEN Institute, 2013) as well-established criteria on which to evaluate student learning and guide faculty in course redesign through integration of activities based on knowledge, skills, and attitudes.

In addition, the focus on instructional strategy effectiveness needs to transition to identification of the most effective evaluation method to best measure a particular learning outcome. For the short term, it is not realistic to conduct large-scale comparative studies of student learning outcomes in a traditional classroom versus an online classroom. However, more research on how to incorporate established models and frameworks of online education within nursing education will help to yield evidence- based course redesign, delivery, and evaluation of teaching and learning. To date, Benchmarking Best Practices in Web-Based Courses by Billings et al. (2001) is underutilized in online nursing education research, and its current utility should be revisited. Also notably absent in the nursing literature is a cross-disciplinary view of best practices in online course development and standardized course review rubrics established by other disciplines, such as education and psychology, who frequently use online education. For example, the Quality Matters program (2014) is an evidence-based peer review program that assists with quality assurance and continuous improvement for the design and delivery of online courses. The Quality Matters program is faculty centered and provides research-based rubrics that are grounded in online instructional design principles to evaluate course design.

Conclusion

Many nursing programs use additional data to evaluate learning outcomes within courses, such as the HESI® standardized examinations. Research that could relate HESI outcomes data with the effectiveness of online nursing education should be considered. On a broader scale, research that would examine nursing program outcomes, opposed to student learning outcomes, in the context of online nursing education would be of value to nurse educators and nursing education as a whole. Program outcomes used as measures of program evaluation, such as the NCLEX-RN,® alumni and graduate surveys, employer surveys, and, ultimately, nurse-sensitive indicators of patient and community care outcomes, would be important breadth and depth added to the current state of the science around evaluation practices in online nursing education.

References

  • Allen, I.E. & Seaman, J. (2013). Changing course: Ten years of tracking online education in the United States. Newburyport, MA: Sloan Consortium.
  • American Association of Colleges of Nursing. (2000a). Distance technology in nursing education: Assessing a new frontier [White paper]. Journal of Professional Nursing, 16, 116–122. doi:10.1016/S8755-7223(00)80025-3 [CrossRef]
  • American Association of Colleges of Nursing. (2000b). Tri-council for nursing: Response to the Institute of Medicine’s report To err is human: Building a safer health system. Retrieved from http://www.aacn.nche.edu/publications/position/tri-council-sept-2000
  • American Association of Colleges of Nursing. (2003). Alliance for nursing accreditation statement on distance education policies. Retrieved from http://www.aacn.nche.edu/publications/position/distance-ed-statement
  • American Association of Colleges of Nursing. (2014). Degree completion programs for registered nurses [Fact Sheet]. Retrieved from http://www.aacn.nche.edu/media-relations/fact-sheets/degree-completion-programs
  • Ansah, A.O., Neill, P. & Newton, J. (2011). Who’s on first in distance education?Online Journal of Distance Learning Administration, 14, Article 4.
  • Billings, D.M. (2000). A framework for assessing outcomes and practices in Web-based courses in nursing. Journal of Nursing Education, 39, 60–67.
  • Billings, D.M. (2007). Distance education in nursing: 25 years and going strong. Computers, Informatics, Nursing, 25, 121–123. doi:10.1097/01.NCN.0000270044.67905.4a [CrossRef]
  • Billings, D.M., Connors, H.R. & Skiba, D.J. (2001). Benchmarking best practices in Web-based nursing courses. Advances in Nursing Science, 23 (3), 41–52. doi:10.1097/00012272-200103000-00005 [CrossRef]
  • Burress, N.M., Billings, D.M., Brownrigg, V., Skiba, D.J. & Connors, H.R. (2009). Class size as related to the use of technology, educational practices, and outcomes in Web-based nursing courses. Journal of Professional Nursing, 25, 33–41. doi:10.1016/j.profnurs.2008.06.002 [CrossRef]
  • Carlson, K. (2011). Using Adobe Connect to deliver online library instruction to the RN to BSN program. Journal of Library & Information Services In Distance Learning, 5, 172–180. doi:10.1080/1533290X.2011.634979 [CrossRef]
  • Carpenter, R., Theeke, L. & Smothers, A. (2013). Enhancing course grades and evaluations using distance education technologies. Nurse Educator, 38, 114–117. doi:10.1097/NNE.0b013e31828dc2d7 [CrossRef]
  • Caudle, P., Bigness, J., Daniels, J., Gillmore-Kahn, M. & Knestrick, J. (2011). Implementing computer-based testing in distance education for advanced practice nurses: Lessons learned. Nursing Education Perspectives, 32, 328–332. doi:10.5480/1536-5026-32.5.328 [CrossRef]
  • Childs, S., Blenkinsopp, E., Hall, A. & Walton, G. (2005). Effective e-learning for health professional and students—Barriers and their solutions. A systematic review of the literature: Findings from the HEXL project. Health Information & Libraries Journal, 22, 20–32. doi:10.1111/j.1470-3327.2005.00614.x [CrossRef]
  • Cobb, S.C. (2009). Social and online learning: A current view from a research perspective. Journal of Interactive Online Learning, 8, 241–254.
  • Council for Regional Accrediting Commissions. (2001). Best practices for electronically offered degree and certificate programs. Retrieved from http://wcet.wiche.edu/wcet/docs/cigs/studentauthentication/Accrediting_BestPractices.pdf
  • Culley, J. & Polyakava-Norwood, V. (2012). Innovation center: Synchronous online role play for enhancing community, collaboration, and oral presentation proficiency. Nursing Education Perspectives, 33, 51–54. doi:10.5480/1536-5026-33.1.51 [CrossRef]
  • Dutile, C., Wright, N. & Beauchesne, M. (2011). Virtual clinical education: Going the full distance in nursing education. Newborn & Infant Nursing Reviews, 11, 43–48. doi:10.1053/j.nainr.2010.12.008 [CrossRef]
  • Gallagher-Lepak, S., Reilly, J. & Killion, C.M. (2009). Nursing student perceptions of community in online learning. Contemporary Nurse, 32, 133–146. doi:10.5172/conu.32.1-2.133 [CrossRef]
  • Gilmore, M. & Lyons, E.M. (2012). NURSING 911: An orientation program to improve retention of online RN-BSN students. Nursing Education Perspectives, 33, 45–47. doi:10.5480/1536-5026-33.1.45 [CrossRef]
  • Guhde, J. (2010). Using online exercises and patient simulation to improve students’ clinical decision-making. Nursing Education Perspectives, 31, 387–389. doi:10.1043/1536-5026-31.6.387 [CrossRef]
  • Guzic, B.L., McIlhenny, C.V., Knee, D.R., LeMoine, J.K., Wendekier, C.M., Demuth, B.R. & Bapat, A. (2012). Distance learning and clinical simulation in senior baccalaureate nursing education. Clinical Simulation in Nursing, 8, 459–467. doi:10.1016/j.ecns.2011.04.005 [CrossRef]
  • Hall, L.R. (2010). Perceptions of faculty caring: Comparison of distance and traditional graduate nursing students [Doctoral dissertation]. Available from ProQuest Dissertations and Theses database. (UMI No. AAI3404439)
  • Hart, L. & Morgan, L. (2010). Academic integrity in an online registered nurse to baccalaureate in nursing program. The Journal of Continuing Education in Nursing, 41, 498–505. doi:10.3928/00220124-20100701-03 [CrossRef]
  • Holley, K.A. & Taylor, B.J. (2009). Undergraduate student socialization and learning in an online professional curriculum. Innovative Higher Education, 33, 257–269. doi:10.1007/s10755-008-9083-y [CrossRef]
  • Institute of Medicine. (2003). Health professions education: A bridge to quality. Washington, DC: National Academies Press.
  • Institute of Medicine. (2009). Redesigning continuing education for health professionals. Washington, DC: National Academies Press.
  • Institute of Medicine. (2011). The future of nursing: Leading change, advancing health. Washington, DC: National Academies Press.
  • Jeffries, P.R. (2000). Development and test of a model for designing interactive CD-roms for teaching nursing skills. Computer Nursing, 18, 1118–1124.
  • Jeffries, P.R. (2005a). Development and testing of a hyperlearning model for design of an online critical care course. Journal of Nursing Education, 44, 366–372.
  • Jeffries, P.R. (2005b). Technology trends in nursing education: Next steps. Journal of Nursing Education, 44, 3–4.
  • Johnson, A.E. (2008). A nursing faculty’s transition to teaching online. Nursing Education Perspectives, 29, 17–22.
  • Jones, D.P. & Wolf, D.M. (2010). Shaping the future of nursing education today using distant education and technology. Association of Black Nursing Faculty Journal, 21(2), 44–47.
  • Kardong-Edgren, S. & Emerson, R. (2010). Student adoption and perception of lecture podcasts in undergraduate Bachelor of Science nursing courses. Journal of Nursing Education, 49, 398–401. doi:10.3928/01484834-20100224-04 [CrossRef]
  • Kavanaugh, K., Andreoni, V.A., Wilkiee, D.J., Burgener, S., Buschmann, M.B., Henderson, G. & Zhao, Z. (2009). Developing a blended course on dying, loss, and grief. Nurse Educator, 34, 126–131. doi:10.1097/NNE.0b013e31819fcad1 [CrossRef]
  • Kelley, F.J. & Klopf, M.I. (2008). Second-language learning in a family nurse practitioner and nurse midwifery diversity education project. Journal of the American Academy of Nurse Practitioners, 20, 479–485. doi:10.1111/j.1745-7599.2008.00353.x [CrossRef]
  • Kidd, L., Morgan, K. & Savery, J. (2012). Development of a mental health nursing simulation: Challenges and solutions. Journal of Interactive Online Learning, 11, 80–89.
  • Kolowich, S. (2010). Online cure for the nursing crisis. Inside Higher Education. Retrieved from http://www.insidehighered.com/news/2010/02/02/nursing
  • Langley, M.E. & Brown, S.T. (2010). Perceptions of the use of reflective learning journals in online graduate nursing education. Nursing Education Perspectives, 31, 12–17.
  • Leners, D.W., Wilson, V.W. & Sitzman, K.L. (2007). Twenty-first century doctoral education: Online with a focus on nursing education. Nursing Education Perspectives, 28, 332–336.
  • Mancuso, J.M. (2009). Perceptions of distance education among nursing faculty members in North America. Nursing & Health Sciences, 11, 194–205. doi:10.1111/j.1442-2018.2009.00456.x [CrossRef]
  • Mancuso-Murphy, J. (2007). Distance education in nursing: An integrated review of online nursing students’ experiences with technology-driven instruction. Journal of Nursing Education, 46, 252–260.
  • Manning, M.L. & Frisby, A.J. (2011). Multimethod teaching strategies to integrate selected QSEN competencies in a doctor of nursing practice distance education program. Nursing Outlook, 59, 166–173. doi:10.1016/j.outlook.2011.01.001 [CrossRef]
  • Mayne, L.A. & Wu, Q. (2011). Creating and measuring social presence in online graduate nursing courses. Nursing Education Perspectives, 32, 110–114. doi:10.5480/1536-5026-32.2.110 [CrossRef]
  • McDonald, L.J. & Walters, K. (2009). Collaborative teaching in a virtual environment to promote conceptual change for nurse educator master’s students. Nursing Education Perspectives, 30, 381–383.
  • McIntyre, M., McDonald, C. & Racine, L. (2013). A critical analysis of online nursing education: Balancing optimistic and cautionary perspectives. Canadian Journal of Nursing Research, 45, 36–53.
  • McLain, N.E., Biddle, C. & Cotter, J.J. (2012). Anesthesia clinical performance outcomes: Does teaching method make a difference?American Association of Nurse Anesthetists Journal, 80(4 Suppl.), S11–S16.
  • QSEN Institute. (2013). Competencies. Retrieved from http://qsen.org/competencies/
  • Quality Matters. (2014). Quality matters: A national benchmark for online course design. Retrieved from https://www.qualitymatters.org
  • Reilly, J.R., Gallagher-Lepak, S. & Killion, C. (2012). “Me and my computer”: Emotional factors in online learning. Nursing Education Perspectives, 33, 100–105. doi:10.5480/1536-5026-33.2.100 [CrossRef]
  • Reneau, M. (2013). Teaching nurses sight unseen: Comparing the cultural competency of online and on-campus BSN faculty. Journal of Transcultural Nursing, 24, 78–85. doi:10.1177/1043659612462405 [CrossRef]
  • Robbins, L.K. & Hoke, M.M. (2008). Using objective structured clinical examinations to meet clinical competence evaluation challenges with distance education students. Perspectives in Psychiatric Care, 44, 81–88. doi:10.1111/j.1744-6163.2008.00157.x [CrossRef]
  • Rounds, L.R. & Rappaport, B.A. (2008). The successful use of problem-based learning in an online nurse practitioner course. Nursing Education Perspectives, 29, 12–16.
  • Russell, T. (1999). No significant difference phenomenon. Raleigh: North Carolina State University.
  • Rutledge, C.M., Barham, P., Wiles, L. & Benjamin, R.S. (2008). Integrative simulation: A novel approach to educating culturally competent nurses. Contemporary Nurse: A Journal for the Australian Nursing Profession, 28, 119–128. doi:10.5172/conu.673.28.1-2.119 [CrossRef]
  • Simpson, R.L. (2006). See the future of distance education. Nursing Management, 37, 42–51. doi:10.1097/00006247-200602000-00012 [CrossRef]
  • Skiba, J. & Barton, A. (2006). Adapting your teaching to accommodate the next generation of learners. Online Journal of Issues in Nursing, 11, 119–128.
  • Smith, A. (2010). Learning styles of registered nurses enrolled in an online nursing program. Journal of Professional Nursing, 26, 49–53. doi:10.1016/j.profnurs.2009.04.006 [CrossRef]
  • Smith, G., Passmore, D. & Faught, T. (2009). The challenges of online nursing education. Internet & Higher Education, 12, 98–103. doi:10.1016/j.iheduc.2009.06.007 [CrossRef]
  • University of Minnesota Center for Teaching and Learning. (2013). What is active learning? Retrieved from http://www1.umn.edu/ohr/teachlearn/tutorials/active/what/
  • Vogt, M., Schaffner, B., Ribar, A. & Chavez, R. (2010). The impact of podcasting on the learning and satisfaction of undergraduate nursing students. Nurse Education in Practice, 10, 38–42. doi:10.1016/j.nepr.2009.03.006 [CrossRef]
  • Wells, M.I. & Dellinger, A.B. (2011). The effect of type of learning environment on perceived learning among graduate nursing students. Nursing Education Perspectives, 32, 406–410. doi:10.5480/1536-5026-32.6.406 [CrossRef]
  • Western Interstate Commission on Higher Education. (1995). Principles of good practice in electronically offered academic degree and certificate programs. Retrieved from http://wcet.wiche.edu/wcet/docs/publications/PrinciplesofGoodPractice1995.pdf
  • Whittemore, R. & Knafl, K. (2005a). Combining evidence in nursing research: Methods and implications. Nursing Research, 54, 56–62. doi:10.1097/00006199-200501000-00008 [CrossRef]
  • Whittemore, R. & Knafl, K. (2005b). The integrative review: Updated methodology. Journal of Advanced Nursing, 52, 546–553. doi:10.1111/j.1365-2648.2005.03621.x [CrossRef]
  • Xu, Y., Martin, M. & Gribbins, A. (2010). Learning packets in nursing education: Reviving the past. Nurse Education in Practice, 10, 164–169. doi:10.1016/j.nepr.2009.06.003 [CrossRef]
  • Zsohar, H. & Smith, J.A. (2008). Transition from the classroom to the web: Successful strategies for teaching online. Nursing Education Perspectives, 29, 23–28.
Table A: Qualitative Empirical Studies

Table A:

Qualitative Empirical Studies

Table B: Quantitative Empirical Studies

Table B:

Quantitative Empirical Studies

Table B: Quantitative Empirical Studies

Table B: Quantitative Empirical Studies

Table C: Theoretical and Descriptive Studies

Table C:

Theoretical and Descriptive Studies

Table C: Theoretical and Descriptive Studies

10.3928/01484834-20141228-02

Sign up to receive

Journal E-contents