In 2011, the Centers for Medicare & Medicaid Services approved a State Plan Amendment (under Title XIX of the Social Security Act) to provide Medicaid-reimbursable mental health Community Support Services (CSS) in supported housing agencies (U.S. Department of Health and Human Services [USDHHS], 2011). CSS are grounded in psychiatric rehabilitation values, such as self-determination, the belief that individuals with serious mental illness (SMI) have the capacity to learn and grow, and the promotion of valued social roles in normalized environments, and methods such as person-centered rehabilitation planning and goal-related skills training (Pratt, Gill, Barrett, & Roberts, 2014; USDHHS, 2011). One major aspect of implementing CSS is the need for clinically licensed staff and RNs to conduct comprehensive rehabilitation needs assessments and individualized rehabilitation plans, which are grounded in a wellness approach.
In the past, supported housing programs have not focused on hiring staff who are clinically licensed or RNs. Therefore, staff members could have experience with the agency and, due to tenure, be promoted to supervisory status without a license. With the change to CSS, agencies must incorporate staff who have clinical licenses or are RNs to deliver necessary services and supervise direct care staff. Although clinically licensed staff and RNs have extensive educational backgrounds, their education does not always address psychiatric rehabilitation methods. Because CSS is grounded in psychiatric rehabilitation, it is important to assess clinically licensed staff and RNs' learning of these practices.
There appears to be a shortage of nurses who choose to work in mental health settings or with individuals who have SMI (Thongpriwan et al., 2015). Researchers suggest some nursing students may have negative attitudes and lack of knowledge when working with individuals with SMI (Happell & Gaskin, 2013; Thongpriwan et al., 2015). Negative perceptions may be a direct result of anxiety that surrounds the stigma associated with SMI (Happell, Platania-Phung, Harris, & Bradshaw, 2014). Due to the increase in integrated care and the Affordable Care Act, now is an important time to increase awareness and education for nursing students entering the field, as this may lead to strengthening skills to work with individuals with SMI (Kirkbakk-Fjaer, Andfossen, & Hedelin, 2015; Thongpriwan et al., 2015). With an increase in training, nursing students may increase their competence to provide effective services (Happell & Gaskin, 2013) when they become RNs who may work in a supported housing setting with individuals with SMI.
Pre- and posttest development and analysis have been shown to be an effective way to measure the knowledge participants gain from attending training (Saks, Salas, & Lewis, 2014). Pre- and posttest designs have been used effectively to assess similar kinds of training among professionals and peers in the mental health workforce and in the area of work-based learning (Bachen, Brewster, Guerra-Sarabia, Merritt, & Schneider, 2015; Goncalves et al., 2013; Hemingway, Clifton, Stephenson, & Edward, 2014; Israel, Willging, & Ley, 2016; Lauritzen, Reedtz, Van Doesum, & Martinussen, 2014; White, Hemingway, & Stephenson, 2014).
The purpose of the current study was to evaluate the effectiveness of CSS training to increase participants' knowledge of psychiatric rehabilitation skills and practices. Participant knowledge was assessed at the beginning of the first training session and at the conclusion of the last training session. Researchers hypothesized that participants would increase their knowledge of psychiatric rehabilitation as indicated by an increase in correct responses from the pre- to posttest (Hypothesis 1). In addition, researchers hypothesized that participants who identified as direct care staff, were non-licensed, or had fewer years of experience would demonstrate greater improvements in their test scores than supervisory staff, licensed staff, or staff with more experience, respectively (Hypothesis 2).
Community Support Services Training
The current authors (researchers A.Z., A.R., Z.C.) conducted CSS trainings. Researchers used a work-based learning model to develop a series of eight full-day, in-person training sessions to increase participants' knowledge of psychiatric rehabilitation, as well as increase participants' skill competency with psychiatric rehabilitation methods (Clay, Barrett, Reilly, & Zazzarino, 2016). Each training was conducted once per week in three separate locations. Researchers made the learning materials, training schedules, and related resources accessible to participants via a training website that remained available to participants after completion of the curriculum. An overview of the content of the eight sessions is depicted in Table 1.
Community Support Services (CSS) Curriculum Overview
The content of the training curriculum was determined by several factors: services outlined in the CSS State Plan Amendment (USDHHS, 2011); researchers' knowledge of psychiatric rehabilitation, evidence-based practices, and other best practices in the counseling professions; and findings from agency self-assessments and onsite agency visits conducted by researchers. Agency self-assessments were tools agencies used to assess readiness to transition to CSS and identify training needs of staff. Researchers identified a common theme during the onsite agency visits: supported housing practitioners were not skilled in engaging individuals in identifying personally meaningful goals that could enhance their quality of life. Goals that are set without real input from the individual participating in services, even with the best intentions, are not likely to be motivating (Pratt et al., 2014). In addition, practitioners and consumers often reported that instead of providing intentional interventions outlined on individualized rehabilitation plans, general services such as “checking-in” and transportation to the grocery store and medical appointments were provided. The ability to engage individuals, collaboratively craft individualized rehabilitation plans (IRP), provide interventions, and document the provision of those interventions and progress toward goals requires a particular set of knowledge and skill competencies. The development of proficiencies in these psychiatric rehabilitation practice domains became the foundation of the training curriculum.
Engagement and communication skills were covered in Sessions 2 and 3 of the training series. Researchers purposefully introduced these topics early in the training series because communication is the primary vehicle for establishing a relationship and delivering meaningful services. Developing a therapeutic alliance enables practitioners to understand an individual's needs and desires, which is crucial in the development of an IRP. In addition to traditional didactic instruction, researchers created experiential learning activities so participants had an opportunity to practice the communication skills presented. In one activity, participants worked in dyads to use the techniques of active listening that are particularly important aspects of relationship building, allowing participants to see the effectiveness of practicing active listening skills while working to foster relationships with individuals participating in their services (Chase et al., 2012).
Strategies for assisting individuals in developing and maintaining activities of daily living skills and social skills were also a major part of the training curriculum. An explicit goal of CSS is to support individuals in restoring functioning that will empower them to be more independent in the community and less reliant on professional support. Bellack, Mueser, Gingerich, and Agresta (2004) articulated a structured approach to skills training that includes establishing a rationale for learning a new skill, modeling the steps of the skill, engaging individuals in skill practice role plays, providing positive and corrective feedback, and assisting individuals in finding opportunities to practice the skill in their natural environment. Researchers introduced this framework and emphasized the importance of providing goal-related skills training interventions with intentionality and consistency, as skills training is one of the six identified CSS interventions. During one exercise, practitioners created a lesson plan that included identifying the specific steps of a skill. They were then asked to demonstrate the steps used to teach that skill and use feedback received from peers and researchers to improve their delivery.
The curriculum also emphasized documentation as shown in Table 1. Supported housing practitioners must be proficient in the documentation requirements delineated by Medicaid. There was a substantial amount of time dedicated to this topic during the training with a variety of hands-on exercises to reinforce learning. One exercise had practitioners form small work groups to practice writing and identifying interventions that were measurable for rehabilitation plans. They gave and received feedback and revised their rehabilitation plans to reflect individual choice and defined the interventions to be provided.
The initial training sessions were offered only to supervisors and administrators, as it was important for individuals in those leadership positions to learn about the impending changes that a shift to CSS would require, as well as the knowledge and skills that direct care practitioners needed to acquire to provide psychiatric rehabilitation services. Supervisors needed to master the skills their supervisees would be integrating into practice, as they are ultimately responsible for determining whether direct care practitioners are using the skills effectively, giving feedback, and providing further instruction if needed. Although much of the content included in the supervisors' training mirrored that of direct care practitioners, there were a few distinctions, such as a focus on supervision strategies, managing change, and developing organizational transition plans.
Researchers delivered the training and collected pre- and posttest data at three different sites, each in a different county in New Jersey. These counties represent the northern, central, and southern regions of the state. Researchers collected personal demographic variables including: gender, age, race/ethnicity, and level of education. Participants reported general work variables including professional credentials, job title, and years of experience in the field (Table 2).
Pre- and Posttest. Pre- and posttests were developed by the three researchers who facilitated the training and are experts in the content areas. The pre- and posttests comprised the same 17 multiple choice questions, with each question having four possible answers; however, the ordering of the posttest was changed randomly to address potential testing effect. Test questions were related to learning objectives of the training such as increased knowledge of psychiatric rehabilitation methods, evidence-based practices, and documentation strategies. A total score was calculated for the pre- and posttests by summing the number of correct responses.
Researchers administered the paper and pencil pretest at the beginning of Session 1 and the posttest at the conclusion of Session 8 of the CSS training series. Researchers verbally explained the nature of data collection before administering the test questionnaire and reminded participants that participation was completely voluntary. Researchers provided each participant necessary time to complete the test and requested that participants turn the test in upon completion. Participants included their names on each pre- and posttest, so researchers could match participants' pretests with their post-tests.
Institutional Review Board approval was not necessary, as this was deemed a project evaluation.
In total, 420 participants completed both the pre- and posttests. Of all participants, 358 were direct care staff and 62 were supervisory staff (supervisory staff includes RNs and clinically licensed individuals). Chi-square analyses and t tests were conducted to assess baseline differences between direct care staff and supervisory staff. There were no significant differences at baseline related to gender or ethnicity. However, there were significant differences between direct care and supervisory staff related to age (χ2 = 21.63, p < 0.001), level of education (χ2 = 77.30, p < 0.001), licensure (χ2 = 60.05, p < 0.001), and years of experience (t = 6.89, p < 0.001). These differences were expected, considering higher level of education, licensure, and years of experience are often related to obtaining a supervisory position.
To test Hypothesis 1, a paired-samples t test was conducted to measure an increase in knowledge from before training to after training. There was a significant increase in correct answers from pretest (mean score = 10.69, SD = 2.78) to post-test (mean score = 13.47, SD = 2.36) (t= −19.96, p < 0.001). This result supports the first hypothesis that participants would increase their knowledge of psychiatric rehabilitation following the training sessions.
To test Hypothesis 2, two-way, between-subjects analysis of covariance was conducted to examine the possible differences in knowledge gained between direct care staff and supervisory staff and between licensed and non-licensed staff. There was a difference between posttest scores of direct care staff and supervisory staff such that supervisors had greater improvement in scores from pre- to posttest (F[1,389] = 4.64, p < 0.05, ηp2 = 0.01). In addition, there was a difference between posttest scores of licensed staff and non-licensed staff such that licensed staff had greater improvement in scores from pre- to posttest (F[1,389] = 4.91, p < 0.05, ηp2 = 0.01).
In another test of Hypothesis 2, a Pearson product-moment correlation was calculated to assess the relationship between number of years of experience and change scores. There was a correlation between the two variables (r = −0.12, n = 388, p = 0.02). Overall, there was a weak, negative correlation between number of years of experience and increase of knowledge as indicated by the change score, showing that as number of years of experience increase, improvement in change score decreases. Therefore, staff who are newer to the field gain knowledge at a greater rate than staff who have more years of experience in the field.
Lastly, a multiple regression analysis was used to test whether credentials, the level of staff (i.e., direct care staff or supervisory staff), or number of years working in the field significantly predicted participants' change in knowledge. Factoring in the simultaneous entry of the predictor variables, it was found that the three variables do not explain a significant amount of variance in the change in knowledge (F[3,384] = 2.40, p < 0.06, R2Adjusted = 0.011). However, the analysis shows that of all variables tested, number of years working in the field is the most predictive of an increase in knowledge (β = −0.10, t = −1.92, p = 0.05).
The purpose of the current study was to evaluate the effectiveness of CSS training to increase participants' knowledge of psychiatric rehabilitation skills and practices. Results indicate that 48-hour training curricula grounded in the work-based learning model can significantly increase participants' knowledge, as measured by improvements in pre- and posttest scores. Psychiatric rehabilitation focuses on improving the lives of individuals with SMI through recovery with an ultimate goal of living independently in the community (Pratt et al., 2014). Although most participants in the current study had some educational background, it was important to assess the increase in knowledge specific to psychiatric rehabilitation.
Throughout the process, all participants gained knowledge indicating the effectiveness of the training (Posavac, 2015). The current researchers explored differences in knowledge based on staff position (i.e., direct care staff or supervisory staff), having or not having a license, and number of years of experience. Including RNs as supervisory staff is essential when examining an increase in knowledge for CSS, as RNs are now expected to deliver services grounded in psychiatric rehabilitation (USDHHS, 2011).
When factoring all variables, years of experience appears to be the greatest predictor of participants' learning. Regardless, the results are not strong enough to provide concrete evidence that experience is a mediating factor for participant learning. Instead, there may be some connection between the three variables (i.e., years of experience, licensure, staff level) that the results do not highlight. For instance, staff with more years of experience tend to work in supervisory capacities and have more education. The study found that there is a negative relationship with years of experience in regard to change in knowledge. The less experience staff members have, the greater their change in knowledge.
Although the current study demonstrated an increase in knowledge, it did not include a control group to assess the increase in knowledge. Having a control group would provide stronger evidence that a difference in scores was directly connected to the work-based learning model of the training. Furthermore, researchers did not provide a specific timeframe for participants to complete the pre- and posttests. Instead, researchers handed out the pre- and posttests and allowed participants to complete the tests at any point throughout the first or final session, respectively. Future research should be aimed at structuring the process when giving the pre- and posttests. In addition, the study was conducted in one state with only supported housing staff; therefore, the results cannot be generalized to other states or service providers.
In addition, researchers tested participants at the end of the training sessions to assess the increase in knowledge; however, there was no follow up to determine whether participants maintained the knowledge over the long term or if the knowledge impacted the services. To strengthen the results of this study, researchers could follow up with participants at specific intervals and assess their knowledge of psychiatric rehabilitation. If participants can demonstrate continued knowledge, the researchers can substantiate their training approach, methods, and curriculum.
Future research should develop additional pre- and posttest questions to allow for greater range in scores, leading to better evaluation of knowledge. As the state supported housing agencies shift to a fee-for-service model grounded in psychiatric rehabilitation principles, it is imperative to assess and ensure that participants are gaining knowledge, skills, and practices. By doing so, supported housing providers will have the knowledge to provide more effective services to individuals with SMI.