Dr. Story is Assistant Professor, Dr. Butts is Associate Professor, Dr. Bishop is Assistant Professor, Ms. Green is Instructor, Ms. Johnson is Instructor, and Ms. Mattison is staff nurse, University of Southern Mississippi, School of Nursing, Hattiesburg, Mississippi.
The authors have no financial or proprietary interest in the materials presented herein.
Address correspondence to Lachel Story, PhD, RN, Assistant Professor, University of Southern Mississippi, School of Nursing, 118 College Drive #5095, Hattiesburg, MS 39406-0001; e-mail: firstname.lastname@example.org.
Program findings required to be reported to the State of Mississippi Institutions of Higher Learning and to the American Association of Colleges of Nursing’s (AACN) Commission on Collegiate Nursing Education include information commonly reported by most schools and colleges of nursing across the nation. There are numerous evaluation methods conducted in nursing programs, but a variety of summative evaluations reflect the program’s overall quality or value. If correct data are sought, summative evaluations reveal information for reporting and decision making about policy, curriculum, and funding (Davidson, 2005).
Summative evaluations are used primarily for determining the overall quality of a program and whether the program regularly meets its mission, goals, objectives, and outcomes. Certain standards, key elements, and competencies must be met for nursing programs to continue their accreditation status from their state and their professional accrediting bodies, such as the AACN. Data from summative evaluations also can be extrapolated to justify making changes for improvement in the program. Formative evaluations generally are the method used by evaluators for finding areas that need improvement, but these data do not give an overall picture of the quality of the program. The evaluations addressed in this article are summative.
Data collected from alumni and their employers include perceived satisfaction levels, program quality, competency levels, and performance of graduates. The history at one school of nursing in the southern United States revealed the program evaluation committee experienced a paucity of responses from alumni and their employers. Although responses from alumni, whether from surveys conducted 1 year or 5 years following graduation, and their employers had never been at the acceptable range of 30% to 40% during the past decade, the program evaluation committee reported record low numbers of responses in the past 2 to 3 years.
As a result, the program evaluation committee focused its attention creatively on strategies for improving response rates from alumni and employers. The traditional method of surveying with self-addressed, postage-paid envelopes had been proven as inefficient, yielding poor return rates and biased findings because of those poor returns, and consuming human and monetary resources to no avail. During the program evaluation committee’s quest, several facts came to light:
- Many of the questionnaires mailed to alumni were being returned as “undeliverable,” meaning alumni had moved and left no forwarding address with the school of nursing.
- Increased employment opportunities have resulted in more geographical movement of nurses.
- The alumni and employer surveys were long and laborious, often containing questions not needed for reporting, which culminated in participants not completing the surveys.
This article describes the innovative approaches one school of nursing used to overcome program evaluation issues.
New Surveys and Web-Based Data Collection
The program evaluation committee spent a year rebuilding its master evaluation plan process, which includes processes for data retrieval, storage, collection, reporting, and monitoring. After examining the strategies used for obtaining feedback and the type of information to include in the surveys, the program evaluation committee decided to:
- Redesign the long employer survey with the poor response rates, which resulted in the creation of two new surveys: a brief one-page Employer’s Satisfaction Survey and a Potential for Success Survey.
- Shorten and revise the 1-year alumni and employer surveys, which were the surveys with the lowest response rates and the most unnecessary items.
- Continue using the existing exit surveys for baccalaureate, master’s, and doctoral students for the time being.
In the revised surveys, the decision was made to include only questions needed for reporting to accreditation agencies by the school. The surveys for the baccalaureate nursing (BSN) program were revised first, and the surveys for the other programs currently are being reviewed.
Employer Satisfaction Survey for BSN Graduates
The program evaluation committee devised a completely new survey with only those questions needed regarding employer satisfaction. The questionnaire was limited to one page and was addressed to the human resources (HR) director rather than the chief nursing officer of hospitals and other institutions because HR departments typically maintain employee records.
This survey currently remains a paper-and-pen survey, mailed with a postage-paid return envelope, but the program evaluation committee also is considering strategies for Web-based administration. The program evaluation committee asked faculty from the senior capstone course to collect information each semester from the graduating seniors on their probable employers. A list of those employers will be maintained to target those employers for sending out future employer satisfaction surveys. Until data can be collected from employers identified by those graduates who will have been working for 1 year, the survey will be sent to those institutions in close proximity to the school of nursing’s three campuses.
The first year of this revision process has produced some bias because of limited sampling, but the program evaluation committee viewed this sampling only as a trial run for future data collections. Response rates project promise by improving from 0% last year to 17% this past semester as a result of implementing this simple change. The program evaluation committee anticipates continued improvement with logistical changes in the process.
Potential for Success Survey for New BSN Graduates
The nursing faculty is committed to providing quality education so that community and regional needs are met. The previously used survey, Preceptor Evaluation of the Student, was changed from a survey that required preceptors to grade students’ performance to a survey in which preceptors provided additional feedback about students’ readiness to perform as a registered nurse (RN).
The Potential for Success Survey completed by senior students’ preceptors is one new tool used to meet this need. The goal of this new survey is to obtain data on senior students’ ability in transitioning to the RN role and potential for successful practice as an RN after graduation. The program evaluation committee reports this information collectively to accrediting agencies in the form of the students’ potential for success as working RNs.
The students’ clinical agency preceptors complete this form each semester at the conclusion of the senior students’ role transitions preceptorship course. Some of the items removed from the discontinued and inefficient Employer Survey were revised for this new form, which clinical preceptors use to evaluate senior students completing this course. The Essentials of Baccalaureate Education for Professional Nursing Practice (AACN, 2008) was used as a guide to create the tool. This evaluation, which targets pertinent reportable information, includes areas such as competency of knowledge, skills, safe practice and performance, and management of care; appropriate use of the nursing process; and professionalism and maintenance of boundaries.
Faculty members in the role transitions preceptorship course provide the program evaluation committee with the completed surveys each semester. The most significant benefit of surveying in this manner is that return rates are 100% because the course syllabus mandates senior students have their preceptor complete and sign this paper-and-pen evaluation, and then give the evaluation to their course faculty.
Alumni Survey for BSN Graduates: A Web-Based Approach
Other long, laborious surveys with poor returns were the 1-year and 5-year alumni surveys. The program evaluation committee decided after substantial contemplation to continue only the 1-year alumni survey because of the frequent geographical relocation of nurses; the committee could not maintain current biographical data.
This survey was shortened to an inexpensive online tool. Numerous companies are available that offer programs to help build questionnaires. The most common ones were assessed, and the program evaluation committee decided on one that was inexpensive and user friendly, could be edited, and could easily be linked to a social networking program—Survey-Monkey ( http//:www.surveymonkey.com).
The Alumni Survey created by the program evaluation committee was limited to nine questions. The first six questions ask for biographical updates (some optional), type of work setting, and function in the work setting. The remaining three questions include three to seven parts that require Likert-type responses. One question on role preparation includes five parts, one question on satisfaction includes three parts, and one question on quality of the nursing program consists of seven parts. The response rates have increased dramatically, from essentially none to 52%, and the program evaluation committee is pleased with the functioning of this electronic survey tool.
Online Social Networks
Historically, the program evaluation committee had experienced dismal return rates for all mailed surveys (i.e., alumni and employer) to meet the evaluation requirements of the Institutions of Higher Learning and AACN. Attempts had been made to increase response rates by using the traditional methods (e.g., reminders). However, the numbers of returned surveys did not lend themselves to meaningful results nor warranted the cost.
While including student input, the program evaluation committee began to brainstorm for alternative approaches to increase the return rates of these surveys. One such innovative approach for reaching alumni was through the use of an online social network, Facebook. Founded in 2004, Facebook ( http://www.facebook.com/) was established to connect people worldwide, and members now number in the millions. Knowing that many students are part of the technological generation and continue to maintain their Facebook accounts after graduation, the program evaluation committee chose this online social network to reach alumni in a new way.
Facebook has social groups that share similar lifestyles and interests that can be initiated by any member. Facebook has been identified to have the potential for social research (Ellison, Steinfield, & Lampe, 2007), but the network’s use in education evaluation research has yet to be identified. To reach the BSN alumni, the program evaluation committee set up Facebook groups for each graduating class during the evaluation period and then invited students to join their respective group. On joining, the alumni were informed the group’s purpose is to provide an outlet for them to stay connected with each other and their alma mater. To the delight of the program evaluation committee, alumni quickly joined their groups.
After establishing the BSN alumni groups, a brief survey was piloted to test the system. To accomplish the pilot, the program evaluation committee generated the survey using SurveyMonkey. A Web link was created and then sent to the Facebook alumni groups. The groups were informed the survey was a pilot survey and suggestions or concerns were welcomed. Because many Facebook users log onto the Web site daily or often more frequently (Bugeja, 2006), response was generated quickly and respondent feedback was positive.
As mentioned, the revised BSN Alumni Survey was developed from a previously used survey that was daunting. Careful selection of essential items resulted in a concise survey that was administered to the alumni groups in the same manner as the pilot. Within a couple of weeks of sending the Web link to Facebook, the response rate was 52%, far exceeding previous response rates of nearly zero.
The advantages of using an online social network to collect data are numerous. The BSN alumni can have an outlet to provide feedback in a manner that is convenient to their lifestyles. The program evaluation committee benefits from having meaningful data that can be used for program improvement as well as meeting the standards of Institutions of Higher Learning and AACN. In this time of limited financial resources, Facebook provides a direct connection to alumni without incurring any cost—registration and usage are free of charge. Reminder messages also can be sent to the group with a link for immediate response at no cost.
The disadvantages of using this online social network were minimal and were far outweighed by the benefits. The returned survey sample was biased because it included only those BSN alumni who participate on the social network. However, the program evaluation committee recognized the previous results likely were biased as well because of the sparse numbers. This sample bias could be overcome by sending surveys electronically and also by mail to reach alumni not participating on Facebook.
E-mail is so popular as a method of communication that many professionals receive hundreds of e-mails per day from colleagues, staff, and friends. In light of the popularity of communicating via e-mail, the program evaluation committee analyzed ways to incorporate e-mail communication in an effective way. For the revised Employer Satisfaction Survey, for instance, the program evaluation committee questioned who within the organization would be best suited to complete the survey. After HR directors receive the cover letter and survey, they can delegate the completion of the survey to employment specialists or others within the department. To prepare HR directors for this upcoming survey, it was recommended that e-mails be sent as an additional personal touch. Receiving an e-mail would alert HR directors to the future survey.
The use of e-mail is being considered a mode of communication for master’s and doctoral alumni as well as their employers. As the number of alumni in these categories is less and these graduates tend to be more settled, maintaining e-mail addresses was not considered as daunting a task as it would be for undergraduates. During the exit interview, master’s and doctoral students will be asked to keep their e-mail current with the graduate nursing school office. Students will be informed that e-mail will be the method of communication following graduation, as it has been during the duration of their program.
Nurses often feel constrained by time to complete time-consuming surveys for institutions and programs of nursing regarding their past nursing education experience. Paper-and-pen surveys can be inefficient, do not yield sufficient response rates, and quickly are becoming outdated for alumni. The program evaluation committee at one school of nursing recognized return rates were not adequate to report to agencies and institutions requiring information for maintenance of accreditation. The movement toward the online social networking paradigm along with Web-based surveys translates to successful, reportable evaluation results for nursing programs. The lessons learned from one school of nursing’s experiences can be adapted by other nursing programs encountering similar roadblocks and frustrations associated with program evaluation. In addition, the possibilities are numerous for using some of these strategies in other areas of nursing research.
- American Association of Colleges of Nursing. (2008). The essentials of baccalaureate education for professional nursing practice. Washington, DC: Author. Retrieved from http://www.aacn.nche.edu/education/pdf/BaccEssentials08.pdf
- Bugeja, M.J. (2006). Facing the Facebook. The Chronicle of Higher Education, 52(21), p. C1.
- Davidson, E.J. (2005). Evaluation methodology basics: The nuts and bolts of sound evaluation. Thousand Oaks, CA: Sage.
- Ellison, N.B., Steinfield, C. & Lampe, C. (2007). The benefits of Facebook “friends:” Social capital and college students’ use of online social network sites. Journal of Computer-Mediated Communication, 12, 1143–1168. doi:10.1111/j.1083-6101.2007.00367.x [CrossRef]