Journal of Nursing Education

Major Article 

Evaluation of the Effects of Flipped Learning of a Nursing Informatics Course

Jina Oh, PhD, RN; Shin-Jeong Kim, PhD, RN; Sunghee Kim, PhD, RN; Rajaguru Vasuki, MSN, RN

Abstract

Background:

This study evaluated the effects of flipped learning in a nursing informatics course.

Method:

Sixty-four undergraduate students attending a flipped learning nursing informatics course at a university in South Korea participated in this study in 2013. Of these, 43 students participated at University A, and 46 students participated at University B, as a comparison group. Three levels of Kirkpatrick's evaluation model were used: level one (the students' satisfaction), level two (achievement on the course outcomes), and level three (self-perceived nursing informatics competencies).

Results:

Students of the flipped learning course reported positive effects above the middle degree of satisfaction (level one) and achieved the course outcomes (level two). In addition, self-perceived nursing informatics competencies (level three) of the flipped learning group were higher than those of the comparison group.

Conclusion:

A flipped learning nursing informatics course is an effective teaching strategy for preparing new graduate nurses in the clinical setting. [J Nurs Educ. 2017;56(8):477–483.]

Abstract

Background:

This study evaluated the effects of flipped learning in a nursing informatics course.

Method:

Sixty-four undergraduate students attending a flipped learning nursing informatics course at a university in South Korea participated in this study in 2013. Of these, 43 students participated at University A, and 46 students participated at University B, as a comparison group. Three levels of Kirkpatrick's evaluation model were used: level one (the students' satisfaction), level two (achievement on the course outcomes), and level three (self-perceived nursing informatics competencies).

Results:

Students of the flipped learning course reported positive effects above the middle degree of satisfaction (level one) and achieved the course outcomes (level two). In addition, self-perceived nursing informatics competencies (level three) of the flipped learning group were higher than those of the comparison group.

Conclusion:

A flipped learning nursing informatics course is an effective teaching strategy for preparing new graduate nurses in the clinical setting. [J Nurs Educ. 2017;56(8):477–483.]

The need to reform the pedagogical approach in nursing education for a highly technological generation has been highlighted during this era (Benner, Sutphen, Leonard, & Day, 2009). This reform has the potential to engage nursing students in ways that address the complexity of contemporary health care (Betihavas, Bridgman, Kornhaber, & Cross, 2015). Therefore, nurse educators have to design an educational experience that not only meets the needs of the contemporary health care environment, but also students' learning needs.

The flipped learning (FL) or inverted classroom has generated interest in higher education that is focused on providing student-centered learning with the idea that students retain more information by using active learning techniques, compared with sitting in a lecture-based classroom (Calimeirs & Sauer, 2015). A growing need exists for FL in nursing education, as well as other health care disciplines, and studies that examine the process and refinement of the nursing program have been warranted (Martyn, Terwijn, Kek, & Huijser, 2014). A challenge for nurse educators when creating a course design is measuring whether a change has occurred in the students in response to the instruction. Although FL in nursing education is known to be effective, studies that identify the evaluation of the process of implementing FL to assess its influence on work practices have been limited (Dowding et al., 2013). To evaluate the effects of FL on nursing informatics (NI) courses for undergraduate nursing students, it is important to apply a specific evaluation model that can improve the quality of the evaluation.

The influence of NI is evolving due to advancements in nursing practice and technology (Darvish, Bahramnezhad, Keyhanian, & MojdehNavidhamidi, 2014; Hebda & Czar, 2013). Informatics competency is defined by the Quality and Safety Education for Nurses (QSEN, 2012) project as the “use of information and technology to communicate, manage knowledge, mitigate error, and support decision making” (p. 349). Therefore, NI competencies have become an essential requirement for nurses to fulfill their professional roles (Oh, Shin, & De Gagne, 2015), and the NI courses of many nursing schools have been created with diverse learning methods.

The Kirkpatrick evaluation model (Kirkpatrick & Kirkpatrick, 2006) is a well-established standard for evaluating training programs and provides a systematic evaluation framework to explore the influence of education by health care professionals (Adamson, Kardong-Edgren, & Willhaus, 2013; Clark, Ahten, & Macy, 2013). Kirkpatrick's model (Kirkpatrick & Kirkpatrick, 2006) proposes four levels: reaction (level one), learning (level two), behavior (level three), and results (level four; Kirkpatrick & Kirkpatrick, 2006). Level one is concerned with the participants' satisfaction as a reaction to the program, level two with the participants' achievement during learning activities, level three with the participants' competency to apply what they have learned, and level four with the influence of the learning on the organization, such as the hospitals they are working in (DeSilets, 2009). According to a recent review (Rutherford-Hemming, Nye, & Coram, 2016) of the use of the Kirkpatrick model (Kirkpatrick & Kirkpatrick, 2006), 13 of 15 studies evaluated level one and level two, whereas only three studies evaluated level three; no studies have evaluated level four.

Purpose

The purpose of this study was to evaluate the effects of the FLNI course for nursing students in their second year of undergraduate nursing school and at the end of the graduation time using levels one, two, and three of the Kirkpatrick evaluation model (Kirkpatrick & Kirkpatrick, 2006). The research questions were:

  • What was the students' satisfaction with the FLNI course (level one)?
  • How different were the students' course outcomes of the FLNI course pre- and postclass (level two)?
  • How different were the students' NI competencies between the FL group and a traditional learning group (level three)?

Method

Research Design

This study comprised two parts: the first part used levels one and two of the Kirkpatrick evaluation model (Kirkpatrick & Kirkpatrick, 2006) to evaluate one group of participants using a pre- and posttest design, and the second part used level three with a nonequivalent comparison group design (Figure).

Research process based on the Kirkpatrick evaluation model (Kirkpatrick & Kirkpatrick, 2006).

Figure.

Research process based on the Kirkpatrick evaluation model (Kirkpatrick & Kirkpatrick, 2006).

Setting

The researchers developed a 15-week FLNI course for undergraduate nursing students during the fall 2013 semester and evaluated participants' satisfaction as a reaction (level one) and their course achievement (level two) during December of the same year. Using a retrospective second participant survey, the achievement of competencies among the FL group and a traditional learning group (level three) was compared in January 2016. This subsequent survey for level three was administered 2.5 years after the initial survey, and it asked participants about the effect of their achievement in NI competencies on perceived changes in their behavior during clinical practicum since the time of the NI course with FL. In this study, three of the four levels of the Kirkpatrick evaluation model (Kirkpatrick & Kirkpatrick, 2006) were used. The reason why level four could not be performed is that, in this model, 2 or 3 years need to pass after the level three evaluation or graduation (DeSilets, 2009).

Participants

Sixty-four undergraduate nursing students attended the FLNI course in the fall semester of 2013 in University A, located in Busan, South Korea. At that time, participants underwent the evaluation regarding satisfaction (level one) and course outcomes achievement (level two). Three years after experiencing the FLNI course, 43 (67.2%) of 64 students who participated in the previous study agreed to participate in the follow-up study, whereas 21 students took a leave of absence from the school (n = 3), joined the Army (n = 12), or did not want to participate in the follow-up study (n = 6). When they graduated, the participants were evaluated in terms of whether they applied knowledge, had the confidence to apply their skills, and had the attitudes of NI competencies (level three).

In contrast to the flipped class, and for level three, the nursing department of University B was recruited as a comparison group by purposive sampling. University B is approximately 20 miles from University A in the same province of South Korea, and the standardized curriculum of the bachelor degree nursing program was accredited by the Korean Accreditation Board of Nursing Education in 2015. The nursing students of the comparison group comprised 46 students at the time of graduation, and they participated in the NI course that used the traditional didactic lecture method during the fall semester of 2013, similar those on the FLNI course when they were second-year nursing students.

Regarding the homogeneity between the two groups for level three, the NI course was configured in a similar class size (n = 66), and students received a class with the same contents that was jointly developed by professors from University A (FL group) and University B (traditional learning group). In addition, the professors periodically confirmed the contents and time schedule with each other. Before the Korean RN national licensure examination in January 2017, a statistically significant difference was not found between the students' performance for the FL and traditional learning groups in a mock test conducted by the Korean Society of Nursing Science (mean score = 171.8). In addition, regarding confounding factors, they had gained experience in standardized educational hospitals with more than 200 beds, which were also accredited by the Korean Accreditation Board of Nursing Education, during the third and fourth years of the nursing program. Although some students experienced nursing practicum in the same hospitals, the students from each group were unlikely to have had an opportunity to interact during the NI performance because the practicum occurred over different periods. Therefore, the two groups were considered homogenous.

The required sample size was 33 students for each group, with a moderate effect with power (1 to β) = .80, effect size (f) = .34, and two-tailed alpha = .05 for t tests and the in-class correlation coefficient. Thus, the final sample size of 89 students for level three was acceptable in this study.

The FLNI Course

The 2013 FLNI course comprised preclass, in-class, and postclass content for 15 weeks. During the preclass, self-learning proceeded for 50 minutes via the online e-Campus Web site. All participants learned how to connect to the e-Campus Web site via personal computer or application via smartphone. They were also instructed to follow the predetermined rules and to carefully listen to the lecture, and they were permitted to take notes during the online class. Face-to-face classes were conducted once per week for 50 minutes. During the in-class time, the participants presented their opinions about the themes given to them and discussed the issues of NI. In addition, they were encouraged to ask the educators questions, and prompt answers were given. Through this process, the concepts of NI may be enhanced. During the postclass, an instructor analyzed and provided feedback about the students' questions and the semester examination results. The interaction was performed via bulletin board on the e-Campus Web site.

The traditional learning group comprised only face-to-face didactic classes for the same 15 weeks. All participants learned the basic concepts and core concepts of NI from a nurse educator during the in-class time. When they had questions, they could freely ask their educator during that time. After the in-class time, they reviewed the lessons and worked on homework assignments.

Measurements

Level One: Satisfaction. Level one measured the students' satisfaction with the FLNI course in terms of the course design, organization of materials, delivery methods, and understanding of the contents, among others. The self-reported questionnaire, which was composed of 10 items with a 5-point Likert scale ranging from 1 = absolutely disagree to 5 = strongly agree, was developed and its validity was verified with the support of the University A center for teaching and learning development.

Level Two: Course Outcomes Achievement. Data were received from students regarding whether the FLNI course actually achieved the course outcomes. The course outcomes were to (a) identify the essential concepts of informatics (knowledge), (b) understand the five core concepts of NI (knowledge), (c) navigate electronic nursing records and other clinical decision tools (skills), and (d) appreciate the necessity to seek continuous learning of information technology skills and to value nurses' involvement (attitudes). The participants underwent a pretest and a posttest. The achievement of the first and second course outcomes was evaluated using multiple-choice questions, which were based on the course outcomes and developed by this study's authors. The multiple-choice questions could be completed within the given time limit of 50 minutes, and the reliability and validity were confirmed by the center for teaching and learning development of University A. The third course outcome was evaluated with a checklist, and the fourth course outcome was evaluated with a short essay graded on a scale of 100 points.

Level Three: Self-Perceived Achievement of NI Competencies. This study examined the self-perceived NI competency levels between the FL group and traditional learning group on the NI course. Students responded to a survey that was individualized for the undergraduate program based on the QSEN project regarding informatics competencies (Cronenwett et al., 2007). The self-perceived NI competency assessment consisted of 17 detailed components that were classified as knowledge (five items), confidence to apply the skills (eight items), and attitudes (four items) that used a 4-point Likert scale ranging from 1 = not competent to 4 = very competent.

Data Analysis

The analysis was performed using SPSS® version 22. Descriptive statistics were used to summarize the participants' satisfaction with the FLNI course (level one). The paired-samples t test was used to compare the difference in the course outcomes between the pre- and postclass (level two), and a t test was used to compare the difference in NI competencies between those participating in the FLNI course and those in the traditional class (level three).

Ethical Considerations

The study was conducted in accordance with The Code of Ethics of the World Medical Association Declaration of Helsinki (World Health Organization, 2001). It was approved by University A's institutional review board for human subjects in 2013, and the approval was renewed in 2016. Informed consent was obtained from participants and the rights of participants, and the purpose and the procedures of the study were explained. No negative consequences would exist for nonparticipation, and the data would be reported as a group. All students were assured that participation was voluntary, that it did not affect academic progression, and that anonymity would be upheld. They were also informed that the questionnaire would be used at the same time. These rights were reiterated verbally before data collection.

Results

Level One: Satisfaction With the FLNI Course

The total mean score of the participants' satisfaction after taking part in the FL was 3.78 (± .35), which was a slightly more than the middle value (Table 1). The highest ranking item was “The instructor used learning activities, examples, visual aids, and practice that facilitated understanding of the course content” (3.96 ± .23), and the second-highest ranking item was “The course was intellectually challenging and stimulating” (3.88 ± .29). However, the lowest ranking item was “Feedback on assignments, examinations, and graded materials was valuable.”

The Ranking of Satisfaction with the Nursing Informatics Course With Flipped Learning (N = 64)

Table 1:

The Ranking of Satisfaction with the Nursing Informatics Course With Flipped Learning (N = 64)

Level Two: Achievement of Course Outcomes

At the end of the course semester, the mean scores of the pre- and posttest were compared between participants in the FLNI course at University A. The achievement of the course outcomes was significantly increased in the postclass, compared with the preclass. The FLNI course had four course outcomes (Table 2), including two knowledge, one skill, and one attitude outcome.

Comparison of the Course Outcomes Achievement Between the Pre- and Postclass of the Flipped Learning Group on the Nursing Informatics Course (N = 64)

Table 2:

Comparison of the Course Outcomes Achievement Between the Pre- and Postclass of the Flipped Learning Group on the Nursing Informatics Course (N = 64)

Level Three: Self-Perceived Achievement of NI Competencies

When comparing the two groups 2.5 years after participation in the FL group and the traditional learning group on the NI course, a statistically significant difference existed (t = 4.224, p < .001) in self-perceived NI competency. In addition, significant differences existed in all domains of knowledge, confidence to apply the skills, and attitudes in achieving self-perceived NI competencies between the FL group and traditional learning group (Table 3).

Achievement of Nursing Informatics Competencies Between the Flipped Learning Class and Comparison Group (N = 89)

Table 3:

Achievement of Nursing Informatics Competencies Between the Flipped Learning Class and Comparison Group (N = 89)

Discussion

In this study, the FLNI course, which was developed by the authors in 2013, was evaluated using the Kirkpatrick sequential evaluation model (Kirkpatrick & Kirkpatrick, 2006) in 2013 and 2016. Kirkpatrick and Kirkpatrick (2006) advised that the evaluation should be conducted not only at level one (reaction), but also at both levels two (learning) and three (behavior), because it is necessary to know what the participants learned from the educational activity and whether this learning could be transferred into behavioral change.

Level One: Satisfaction of Participants in the FLNI Course

An emerging learning method, FL has gained much interest lately due to its process involving prior study followed by the students' classroom involvement, the direction of which matches that of the current educational policy that emphasizes learner-centered interaction (Acelajado, 2014). Participants' satisfaction is an important first step in evaluating the success of a newly developed course. The participants' overall responses concerning the effectiveness of the FLNI course were rather positive. Some of the nursing students' reactions increased after the FL class. The top three items were related to learning activities and the instructor's materials. Therefore, participants perceived the FLNI class as intellectual, challenging, and stimulating. Although the overall evaluation of the FL method by Chen, Wang, Kinshuk, and Chen (2014) indicated that students might have resisted adopting this learning method because their old passive learning habits required less effort, the current results were inconsistent with their findings.

The online FL class was easily accessed via the Web system using computers or mobile devices. Therefore, the students experienced online learning with no restrictions on time or place. Although the students experienced a limitation regarding prompt interaction during the online class, this could be solved by a question-and-answer bulletin board. In addition, it could be discussed again in the offline class; therefore, it was not a serious problem. The educator's feedback and interaction with students are essential factors that influence the effectiveness of learning (Cant & Cooper, 2010). Cant and Cooper (2010) found that educators preferred to use interactive strategies, such as seminars, case study presentations, or debriefings, to reflect the students' progress. In this study, students' concepts of NI may have been acquired through the presentations and discussion during the extensive face-to-face class. In addition, the educator provided additional explanation and facilitated the students to introspect about the main themes. During the interactive group activities, the teacher's role is to support the students in their quest for further understanding of the lesson (Herreid & Schiller, 2013). Hantla (2014) has also proven that FL allows the educators to do more with their students during the classroom time. The FL was expected to stimulate students' interest and to secure tutorial time for individual students.

Although Chen et al. (2014) suggested a combined methodology, including a student survey, interview, and system log analysis to examine the teaching effectiveness of FL, the authors of the current study used a satisfaction questionnaire only. Thus, to evaluate the effects of the developed FL in future studies, it is suggested that a diverse assessment methodology is combined. In addition, the authors of the current study evaluated the degree of satisfaction with the FLNI course only, because the aim was not to compare the satisfaction level between the groups, but rather to know whether the FLNI course may be accepted without any resistance. Thus, it is also recommended that a comparison be attempted in the future of the satisfaction of other groups using diverse teaching–learning methods.

Level Two: Participants' Achievement of Course Outcomes

Students responded about the achievement of course outcomes and described whether the FLNI course helped them to understand the NI core concepts. The course outcomes achievement scores of the posttest, including knowledge, skills, and attitudes, were significantly increased compared with those of the pretest on the FLNI course. At the end of the course, the participants recognized the benefits of applying information and communication technology to all aspects of nursing. The students understood the core concepts of NI, and they could use electronic nursing records using electronic devices and navigate decision-making devices. They appreciated their influence on health care and the nurses' role in the informatics area, which indicates that the course outcomes were accomplished and that learning effects were meaningfully derived.

The results of this study showed that significant effects exist on the students' achievement of the course outcomes, which is consistent with Lee's and Eun's (2016) findings that students achieved the learning outcomes and gained motivation in their learning and self-efficacy, critical thinking, and communication competency through FL. The FL approach to the NI course provides an incentive for students to come to the class prepared and to assess their understanding, focusing on a higher level of cognitive activities.

Although Clark, Ahten, and Macy (2014) have insisted that positive or negative results are not an indication of whether participants will use their newly acquired knowledge in meaningful ways, the findings reaffirmed that this NI course could be both promising and challenging. Of note, reflection about the FL class may encourage students to explore their interpretive thinking experience. Without an adequate reflection experience, students remain stagnated (Manning & Frisby, 2011); however, some students may have difficulties adopting a flipped approach because of their passive learning habits (Chen et al., 2014).

Currently, as the potential for NI application is increasing, it may enhance the quality of the nursing domains outcomes (McNelis, Horton-Deutsch, & Friesth, 2012). Darvish et al. (2014) indicated that NI competencies are a prerequisite for improving patient care. Therefore, an NI course with a novel approach should be attempted to prepare and adjust students for the current rapid changes occurring in the world.

Level Three: Self-Perceived Achievement of NI Competencies

It is necessary to know whether the participants learned from the educational activities and whether this learning was transferred into behavioral change (Clark et al., 2014). Because the change is an important evaluation element of an educational intervention, nurse educators are more often interested in whether a transfer of knowledge existed from the classroom to the clinical practicum. Therefore, level three of the Kirkpatrick evaluation model (Kirkpatrick & Kirkpatrick, 2006) in nursing education examines the students' learning by measuring behavioral change demonstrated in the clinical practicum; this was measured in this study using a self-perceived NI competency questionnaire that included knowledge, confidence to apply their skills, and attitudes. The level of self-perceived achievement in each area of the NI competencies in the FLNI course was higher than that of the comparison group that used the traditional learning method.

Although a significant difference existed in the NI competencies between the two groups, all participants in level three in this study perceived that they had an upper-middle level of NI competencies. Fetter's (2009) study, which used a self-reported questionnaire, also revealed that most of the undergraduate nursing students perceived confidence in their NI competencies. Although at graduation time, the nursing students are still young and have less experience of clinical and health care informatics systems, they may feel confident because they generally are more proficient in the basic use of advanced information and communication technology devices through their education (Elder & Koehn, 2009). Therefore, specific measurement tools are needed to evaluate students' real informatics confidence in applying their skills and attitudes. In addition, varied and appropriate strategies to integrate NI competency into the nursing curriculum are required (Manning & Frisby, 2011).

Elder and Koehn (2009) stated that the actual performance and skill levels of baccalaureate nursing students were lower than the self-reported NI competency levels. If students' behavioral change does not occur after the NI class, it is critical to know whether the lack of change is a result of inadequacies of the training program or of factors other than the training (Kirkpatrick & Kirkpatrick, 2007). For the successful transfer of the confidence to apply their skills and attitudes of NI competencies into the workplace, nurse educators have to provide practical applications from the basic level of informatics and technology skills up to the integration level (Parker et al., 2016).

The theoretical and conceptual informatics knowledge, including familiarity with nursing classification, expert systems, and other core concepts of NI can be achieved effectively using FL with self-directed learning (Oh et al., 2015). Although the confidence to apply the informatics skills of undergraduate nursing students includes the use of instruments and techniques specific to informatics, nurses have to use these skills in systems analysis and project management to support safe and effective transitions in care (Yang et al., 2014). The confidence to apply their skills and attitudes in the NI competencies needs to be increased through overall education or a training period (McGonigle, Hunter, Sipes, & Hebda, 2014). In this sense, participation in an online class FLNI course may be one of the best instructional methods to familiarize students with information and communication technology.

Fetter (2009) identified the NI competencies of prelicensure nursing students, including not only knowledge of NI concepts, but also confidence in electronic document writing, Internet searching and using e-mail, and operating software programs and systems. Although nursing students have been educated about informatics and technology, graduates should be able to integrate the NI competencies in the workplace (Choi & Martinis, 2013). Therefore, nurse educators have to design NI courses and suggest educational strategies that can incorporate NI competencies into the nursing curriculum (Gassert, 2008). For students who have already been exposed to high technology, the opportunity to experience various and new teaching-learning methods, such as FL, is required, which will interest and motivate them to transfer the confidence to apply their skills and attitudes from the classroom to the actual workplace.

Limitations

A few limitations of the study need to be acknowledged. First, because the sampling was restricted to one region, the findings may lack generalizability to other practice settings and populations. Second, a contrast group is needed to compare the evaluations of levels one and two; although the authors attempted to minimize confounding factors, they may not have been excluded completely. In addition, it was a weakness that the self-perceived confidence to apply their informatics skills and attitudes toward NI were reported using a self-administered questionnaire; a gap between the perception and reality may exist. Accordingly, to ensure accurate measurement, a skills checklist needs to be used in the future. As this study focused on levels one, two, and three of the Kirkpatrick evaluation model (Kirkpatrick & Kirkpatrick, 2006), future studies should be designed to address level four, which refers to the results of the participants' ability to apply their competencies in the workplace.

Conclusion

The results of this study using the Kirkpatrick model of evaluation (Kirkpatrick & Kirkpatrick, 2006) generally included positive teaching effectiveness related to increases in the learner's satisfaction and achievement of course outcomes and NI competencies. The FL encouraged the students to be motivated and led to self-directed learning. The findings from the survey provide the opportunity to elicit a holistic understanding of the students' views and experiences. In addition, the nursing students' responses indicated that they had a better understanding of NI and that FL was an enjoyable way to learn. This program may be an ongoing process as FL continues to evolve.

References

  • Acelajado, M.J. (2014, October). Flipped teaching approach in college algebra: Cognitive and non-cognitive gains. Proceedings of KSME 2014 International Conference on Mathematical Education, 87–99.
  • Adamson, K.A., Kardong-Edgren, S.S. & Willhaus, J. (2013). An updated review of published simulation evaluation instruments. Clinical Simulation in Nursing, 9, e393–e400. http://dx.doi.org/10.1016/j.ecns.2012.09.004 doi:10.1016/j.ecns.2012.09.004 [CrossRef]
  • Benner, P., Sutphen, M., Leonard, V. & Day, L. (2009). Educating nurses: A call for radical transformation. San Francisco, CA: Jossey-Bass.
  • Betihavas, V., Bridgman, H., Kornhaber, R. & Cross, M. (2015). The evidence for ‘flipping out’: A systematic review of the flipped classroom in nursing education. Nurse Education Today, 38, 15–21. http://dx.doi.org/10.1016/j.nedt.2015.12.010 doi:10.1016/j.nedt.2015.12.010 [CrossRef]
  • Calimeirs, L. & Sauer, K.M. (2015). Flipping out about the flip: All hype or is there hope?International Review of Economics Education, 20, 13–28. http://dx.doi.org/10.1016/j.iree.2015.08.001 doi:10.1016/j.iree.2015.08.001 [CrossRef]
  • Cant, R.P. & Cooper, S.J. (2010). Simulation-based learning in nurse education: Systematic review. Journal of Advanced Nursing, 66, 3–15. http://dx.doi.org/10.1111/j.1365-2648.2009.05240.x doi:10.1111/j.1365-2648.2009.05240.x [CrossRef]
  • Chen, Y., Wang, Y., Kinshuk, & Chen, N. (2014). Is FLIP enough? Or should we use the FLIPPED model instead?Computers & Education, 79, 16–27. doi:10.1016/j.compedu.2014.07.004 [CrossRef]
  • Choi, J. & De Martinis, J.E. (2013). Nursing informatics competencies: Assessment of undergraduate and graduate nursing students. Journal of Clinical Nursing, 22, 1970–1976. Retrieved from http://onlinelibrary.wiley.com/doi/10.1111/jocn.12188/full doi:10.1111/jocn.12188 [CrossRef]
  • Clark, C.M., Ahten, S.M. & Macy, R. (2013). Using problem-based learning scenarios to prepare nursing students to address incivility. Clinical Simulation in Nursing, 9, e75–e83. http://www.dx.doi.org/10.1016/j.ecns.2011.10.003 doi:10.1016/j.ecns.2011.10.003 [CrossRef]
  • Clark, C.M., Ahten, S.M. & Macy, R. (2014). Nursing graduates' ability to address incivility: Kirkpatrick's level-3 evaluation. Clinical Simulation in Nursing, 10, 425–431. https://doi.org/10.1016/j.ecns.2014.04.005 doi:10.1016/j.ecns.2014.04.005 [CrossRef]
  • Cronenwett, L., Sherwood, G., Barnsteiner, J., Disch, J., Johnson, J. & Mitchell, P. (2007). Quality and safety education for nurses. Nurse Outlook, 55, 122–131. doi:10.1016/j.outlook.2007.02.006 [CrossRef]
  • Darvish, A., Bahramnezhad, F., Keyhanian, S. & MojdehNavidhamidi, M. (2014). The role of nursing informatics on promoting quality of health care and the need for appropriate education. Global Journal of Health Science, 6, 11–18. doi:10.5539/gjhs.v6n6p11 [CrossRef]
  • DeSilets, L.D. (2009). Connecting the dots of evaluation. The Journal of Continuing Education in Nursing, 40, 532–533. http://dx.doi.org/10.3928/00220124-20091119-09 doi:10.3928/00220124-20091119-09 [CrossRef]
  • Dowding, D.W., Currie, L.M., Borycki, E., Clamp, S., Favela, J., Fitzpatrick, G. & Dykes, P.C. (2013). International priorities for research in nursing informatics for patient care. Studies in Health Technology and Informatics, 192, 372–376. Retrieved http://ebooks.iospress.nl/publication/34021
  • Elder, B.L. & Koehn, M.L. (2009). Assessment tool for nursing computer competencies. Nursing Education Perspectives, 30, 148–152.
  • Fetter, M.S. (2009). Graduating nurses' self-evaluation of information technology competencies. Journal of Nursing Education, 48, 86–90. http://dx.doi.org/10.3928/01484834-20090201-05 doi:10.3928/01484834-20090201-05 [CrossRef]
  • Gassert, C.A. (2008). Technology and informatics competencies. Nursing Clinics of North America, 43, 507–521. http://dx.doi.org/10.1016/j.cnur.2008.06.005 doi:10.1016/j.cnur.2008.06.005 [CrossRef]
  • Hantla, B.F. (2014). The effects of flipping the classroom on specific aspects of critical thinking in a Christian college: A quasi-experimental, mixed-methods study [Doctoral dissertation]. Southeastern Baptist Theological Seminary, 1–264. Retrieved from https://eric.ed.gov/?id=ED556925
  • Hebda, T.L. & Czar, P. (2013). Handbook of informatics for nurses & healthcare professionals (5th ed.). Upper Saddle River, NJ: Prentice Hall.
  • Herreid, C.F. & Schiller, N.A. (2013). Case studies and the flipped classroom. Journal of College Science Teaching, 42(5), 62–67.
  • Lee, Y.-S. & Eun, Y. (2016). The effect of the flipped learning on self-efficacy, critical thinking disposition, and communication competence of nursing students. Journal of Korean Academy Society of Nursing Education, 22, 567–576. http://dx.doi.org/10.5977/jkasne.2016.22.4.567 doi:10.5977/jkasne.2016.22.4.567 [CrossRef]
  • Kirkpatrick, D.L. & Kirkpatrick, J.D. (2006). Evaluating training programs: The four levels (3rd ed.). San Francisco, CA: Berrett-Koehler.
  • Kirkpatrick, D.L. & Kirkpatrick, J.D. (2007). Implementing the four levels. San Francisco, CA: Berrett-Koehler.
  • Manning, M.L. & Frisby, A.J. (2011). Multimethod teaching strategies to integrate selected QSEN competencies in a doctor of nursing practice distance education program. Nursing Outlook, 59, 166–173. doi:10.1016/j.outlook.2011.01.001 [CrossRef]
  • Martyn, J., Terwijn, R., Kek, M.Y. & Huijser, H. (2014). Exploring the relationships between teaching, approaches to learning and critical thinking in a problem-based learning foundation nursing course. Nurse Education Today, 34, 829–835. http://dx.doi.org/10.1016/j.nedt.2013.04.023 doi:10.1016/j.nedt.2013.04.023 [CrossRef]
  • McGonigle, D., Hunter, K., Sipes, C. & Hebda, T. (2014). Why nurses need to understand nursing informatics. AORN, 100, 324–327. doi:10.1016/j.aorn.2014.06.012 [CrossRef]
  • McNelis, A.M., Horton-Deutsch, S. & Friesth, B.M. (2012). Improving quality and safety in graduate education using an electronic student tracking system. Archives of Psychiatric Nursing, 26, 358–363. http://dx.doi.org/10.1016/j.apnu.2012.06.006 doi:10.1016/j.apnu.2012.06.006 [CrossRef]
  • Oh, J., Shin, H.W. & De Gagne, J.C. (2015). The use of the movie ‘The Island’ for nursing informatics in nursing students. The Journal of the Korea Contents Association, 15, 330–341. http://dx.doi.org/10.5392/JKCA.2015.15.04.330 doi:10.5392/JKCA.2015.15.04.330 [CrossRef]
  • Parker, K.R., Srinivasan, S.S., Houghton, R.F., Kordzadeh, N., Bozan, K., Ottaway, T. & Davey, B. (2016). Health informatics program design and outcomes: Learning from an early offering at a mid-level university. Education and Information Technologies, 21, 1–17. http://dx.doi.org/10.1007/s10639-016-9506-9
  • Quality and Safety Education for Nurses. (2012). QSEN competencies. Retrieved from http://qsen.org/competencies/pre-licensure-ksas/#informatics
  • Rutherford-Hemming, T., Nye, C. & Coram, C. (2016). Using simulation for clinical practice hours in nurse practitioner education in the United States: A systematic review. Nurse Education Today, 37, 128–135. http://dx.doi.org/10.1016/j.nedt.2015.11.006 doi:10.1016/j.nedt.2015.11.006 [CrossRef]
  • World Health Organization. (2001). World Medical Association Declaration of Helsinki: Ethical principles for medical research involving human subjects. Bulletin of the World Health Organization, 79, 373–374.
  • Yang, L., Cui, D., Zhu, X., Zhao, Q., Xiao, N. & Shen, X. (2014). Perspectives from nurse managers on informatics competencies. The Scientific World Journal, 1–5. http://dx.doi.org/10.1155/2014/391714

The Ranking of Satisfaction with the Nursing Informatics Course With Flipped Learning (N = 64)

RankingContentM ± SD
1The instructor used learning activities, examples, visual aids, and practices that facilitated understanding of the course.3.96 ± .23
2The course was intellectually challenging and stimulating.3.88 ± .29
3The instructor's materials were well-prepared and carefully explained.3.85 ± .25
4The instructor helped students to understand the contents with explanation.3.85 ± .32
5Students were encouraged to ask questions and were promptly given meaningful answers.3.81 ± .24
6The syllabus was helpful for addressing the content and learning outcomes.3.80 ± .42
7The class kept to the time allocated according to the syllabus.3.78 ± .41
8The course was managed thoroughly, including a student's attendance record and a make-up lesson, among others.3.76 ± .38
9Course assignments, evaluation methods, and the grading procedure were fair and appropriate.3.72 ± .43
10Feedback on assignments, examinations, and graded materials was valuable.3.67 ± .48
Total3.78 ± .35

Comparison of the Course Outcomes Achievement Between the Pre- and Postclass of the Flipped Learning Group on the Nursing Informatics Course (N = 64)

DomainCourse OutcomesPretest M±SDPosttest M±SDtp
Knowledge1. Essential concepts of informatics59.88 ± 5.40382.94 ± 6.079−29.94<.001
2. Core concepts of NI2.56 ± .7014.05 ± .634−17.71<.001
Skills3. Navigation of ENRs and other tools5.88 ± .6868.62 ± .557−23.78<.001
Attitude4. Appreciation of the necessity of NI70.64 ± 9.79290.52 ± 5.078−155.65<.001

Achievement of Nursing Informatics Competencies Between the Flipped Learning Class and Comparison Group (N = 89)

CompetencyDomainFL (n= 43)M ±SDTrad. (n= 46)M ±SDtp
Knowledge16.02 ± 1.8514.76 ± 1.463.588.001
Skills24.09 ± 2.6422.48 ± 1.813.388.001
Attitude13.00 ± 1.4511.89 ± 1.044.173<.001
Total53.02 ± 5.0349.13 ± 3.594.224<.001
Authors

Dr. Oh is Professor, and Ms. Vasuki is Doctoral Candidate, Department of Nursing, Institute of Health Science, Inje University, Busan; Dr. Shin-Jeong Kim is Professor, Department of Nursing, Hallym University, Chuncheon; and Dr. Sunghee Kim is Associate Professor, Red Cross College of Nursing, Chung-Ang University, Seoul, South Korea.

The authors have disclosed no potential conflicts of interest, financial or otherwise.

This work was supported by the National Research Foundation of Korea Grant funded by the Korean Government (NRF-2013S1A5A2A01019026).

Address correspondence to Shin-Jeong Kim, PhD, RN, Professor, Department of Nursing, Hallym University, 1 Hallymdaehak-gil, Chuncheon, Gangwon-do, 200-702, South Korea; e-mail: ksj@hallym.ac.kr.

Received: February 12, 2017
Accepted: March 29, 2017

10.3928/01484834-20170712-06

Sign up to receive

Journal E-contents