Journal of Psychosocial Nursing and Mental Health Services

CNE Article 

Increasing Psychiatric Rehabilitation Knowledge in a Supported Housing Setting: Pre- and Posttest Analysis

Anthony Zazzarino, PhD, LPC, ACS, CPRP; Ann Reilly, MA, MSW, LSW, CPRP; Zakia Clay, DSW, LCSW, CPRP


The current study investigated the effectiveness of a state-wide Community Support Services training grounded in work-based learning to improve knowledge about psychiatric rehabilitation principles and skills. Supervisory staff and direct care staff in supported housing settings completed a 17-item, multiple choice, preand posttest after attending a 48-hour training series. Results indicated that participants gained psychiatric rehabilitation knowledge. These findings support the effectiveness of the training series grounded in work-based learning. [Journal of Psychosocial Nursing and Mental Health Services, 57(6), 39–44.]


The current study investigated the effectiveness of a state-wide Community Support Services training grounded in work-based learning to improve knowledge about psychiatric rehabilitation principles and skills. Supervisory staff and direct care staff in supported housing settings completed a 17-item, multiple choice, preand posttest after attending a 48-hour training series. Results indicated that participants gained psychiatric rehabilitation knowledge. These findings support the effectiveness of the training series grounded in work-based learning. [Journal of Psychosocial Nursing and Mental Health Services, 57(6), 39–44.]

In 2011, the Centers for Medicare & Medicaid Services approved a State Plan Amendment (under Title XIX of the Social Security Act) to provide Medicaid-reimbursable mental health Community Support Services (CSS) in supported housing agencies (U.S. Department of Health and Human Services [USDHHS], 2011). CSS are grounded in psychiatric rehabilitation values, such as self-determination, the belief that individuals with serious mental illness (SMI) have the capacity to learn and grow, and the promotion of valued social roles in normalized environments, and methods such as person-centered rehabilitation planning and goal-related skills training (Pratt, Gill, Barrett, & Roberts, 2014; USDHHS, 2011). One major aspect of implementing CSS is the need for clinically licensed staff and RNs to conduct comprehensive rehabilitation needs assessments and individualized rehabilitation plans, which are grounded in a wellness approach.

In the past, supported housing programs have not focused on hiring staff who are clinically licensed or RNs. Therefore, staff members could have experience with the agency and, due to tenure, be promoted to supervisory status without a license. With the change to CSS, agencies must incorporate staff who have clinical licenses or are RNs to deliver necessary services and supervise direct care staff. Although clinically licensed staff and RNs have extensive educational backgrounds, their education does not always address psychiatric rehabilitation methods. Because CSS is grounded in psychiatric rehabilitation, it is important to assess clinically licensed staff and RNs' learning of these practices.

There appears to be a shortage of nurses who choose to work in mental health settings or with individuals who have SMI (Thongpriwan et al., 2015). Researchers suggest some nursing students may have negative attitudes and lack of knowledge when working with individuals with SMI (Happell & Gaskin, 2013; Thongpriwan et al., 2015). Negative perceptions may be a direct result of anxiety that surrounds the stigma associated with SMI (Happell, Platania-Phung, Harris, & Bradshaw, 2014). Due to the increase in integrated care and the Affordable Care Act, now is an important time to increase awareness and education for nursing students entering the field, as this may lead to strengthening skills to work with individuals with SMI (Kirkbakk-Fjaer, Andfossen, & Hedelin, 2015; Thongpriwan et al., 2015). With an increase in training, nursing students may increase their competence to provide effective services (Happell & Gaskin, 2013) when they become RNs who may work in a supported housing setting with individuals with SMI.

Pre- and posttest development and analysis have been shown to be an effective way to measure the knowledge participants gain from attending training (Saks, Salas, & Lewis, 2014). Pre- and posttest designs have been used effectively to assess similar kinds of training among professionals and peers in the mental health workforce and in the area of work-based learning (Bachen, Brewster, Guerra-Sarabia, Merritt, & Schneider, 2015; Goncalves et al., 2013; Hemingway, Clifton, Stephenson, & Edward, 2014; Israel, Willging, & Ley, 2016; Lauritzen, Reedtz, Van Doesum, & Martinussen, 2014; White, Hemingway, & Stephenson, 2014).

The purpose of the current study was to evaluate the effectiveness of CSS training to increase participants' knowledge of psychiatric rehabilitation skills and practices. Participant knowledge was assessed at the beginning of the first training session and at the conclusion of the last training session. Researchers hypothesized that participants would increase their knowledge of psychiatric rehabilitation as indicated by an increase in correct responses from the pre- to posttest (Hypothesis 1). In addition, researchers hypothesized that participants who identified as direct care staff, were non-licensed, or had fewer years of experience would demonstrate greater improvements in their test scores than supervisory staff, licensed staff, or staff with more experience, respectively (Hypothesis 2).

Community Support Services Training

The current authors (researchers A.Z., A.R., Z.C.) conducted CSS trainings. Researchers used a work-based learning model to develop a series of eight full-day, in-person training sessions to increase participants' knowledge of psychiatric rehabilitation, as well as increase participants' skill competency with psychiatric rehabilitation methods (Clay, Barrett, Reilly, & Zazzarino, 2016). Each training was conducted once per week in three separate locations. Researchers made the learning materials, training schedules, and related resources accessible to participants via a training website that remained available to participants after completion of the curriculum. An overview of the content of the eight sessions is depicted in Table 1.

Community Support Services (CSS) Curriculum Overview

Table 1:

Community Support Services (CSS) Curriculum Overview

The content of the training curriculum was determined by several factors: services outlined in the CSS State Plan Amendment (USDHHS, 2011); researchers' knowledge of psychiatric rehabilitation, evidence-based practices, and other best practices in the counseling professions; and findings from agency self-assessments and onsite agency visits conducted by researchers. Agency self-assessments were tools agencies used to assess readiness to transition to CSS and identify training needs of staff. Researchers identified a common theme during the onsite agency visits: supported housing practitioners were not skilled in engaging individuals in identifying personally meaningful goals that could enhance their quality of life. Goals that are set without real input from the individual participating in services, even with the best intentions, are not likely to be motivating (Pratt et al., 2014). In addition, practitioners and consumers often reported that instead of providing intentional interventions outlined on individualized rehabilitation plans, general services such as “checking-in” and transportation to the grocery store and medical appointments were provided. The ability to engage individuals, collaboratively craft individualized rehabilitation plans (IRP), provide interventions, and document the provision of those interventions and progress toward goals requires a particular set of knowledge and skill competencies. The development of proficiencies in these psychiatric rehabilitation practice domains became the foundation of the training curriculum.

Engagement and communication skills were covered in Sessions 2 and 3 of the training series. Researchers purposefully introduced these topics early in the training series because communication is the primary vehicle for establishing a relationship and delivering meaningful services. Developing a therapeutic alliance enables practitioners to understand an individual's needs and desires, which is crucial in the development of an IRP. In addition to traditional didactic instruction, researchers created experiential learning activities so participants had an opportunity to practice the communication skills presented. In one activity, participants worked in dyads to use the techniques of active listening that are particularly important aspects of relationship building, allowing participants to see the effectiveness of practicing active listening skills while working to foster relationships with individuals participating in their services (Chase et al., 2012).

Strategies for assisting individuals in developing and maintaining activities of daily living skills and social skills were also a major part of the training curriculum. An explicit goal of CSS is to support individuals in restoring functioning that will empower them to be more independent in the community and less reliant on professional support. Bellack, Mueser, Gingerich, and Agresta (2004) articulated a structured approach to skills training that includes establishing a rationale for learning a new skill, modeling the steps of the skill, engaging individuals in skill practice role plays, providing positive and corrective feedback, and assisting individuals in finding opportunities to practice the skill in their natural environment. Researchers introduced this framework and emphasized the importance of providing goal-related skills training interventions with intentionality and consistency, as skills training is one of the six identified CSS interventions. During one exercise, practitioners created a lesson plan that included identifying the specific steps of a skill. They were then asked to demonstrate the steps used to teach that skill and use feedback received from peers and researchers to improve their delivery.

The curriculum also emphasized documentation as shown in Table 1. Supported housing practitioners must be proficient in the documentation requirements delineated by Medicaid. There was a substantial amount of time dedicated to this topic during the training with a variety of hands-on exercises to reinforce learning. One exercise had practitioners form small work groups to practice writing and identifying interventions that were measurable for rehabilitation plans. They gave and received feedback and revised their rehabilitation plans to reflect individual choice and defined the interventions to be provided.

The initial training sessions were offered only to supervisors and administrators, as it was important for individuals in those leadership positions to learn about the impending changes that a shift to CSS would require, as well as the knowledge and skills that direct care practitioners needed to acquire to provide psychiatric rehabilitation services. Supervisors needed to master the skills their supervisees would be integrating into practice, as they are ultimately responsible for determining whether direct care practitioners are using the skills effectively, giving feedback, and providing further instruction if needed. Although much of the content included in the supervisors' training mirrored that of direct care practitioners, there were a few distinctions, such as a focus on supervision strategies, managing change, and developing organizational transition plans.


Researchers delivered the training and collected pre- and posttest data at three different sites, each in a different county in New Jersey. These counties represent the northern, central, and southern regions of the state. Researchers collected personal demographic variables including: gender, age, race/ethnicity, and level of education. Participants reported general work variables including professional credentials, job title, and years of experience in the field (Table 2).

Participant Demographics

Table 2:

Participant Demographics


Pre- and Posttest. Pre- and posttests were developed by the three researchers who facilitated the training and are experts in the content areas. The pre- and posttests comprised the same 17 multiple choice questions, with each question having four possible answers; however, the ordering of the posttest was changed randomly to address potential testing effect. Test questions were related to learning objectives of the training such as increased knowledge of psychiatric rehabilitation methods, evidence-based practices, and documentation strategies. A total score was calculated for the pre- and posttests by summing the number of correct responses.


Researchers administered the paper and pencil pretest at the beginning of Session 1 and the posttest at the conclusion of Session 8 of the CSS training series. Researchers verbally explained the nature of data collection before administering the test questionnaire and reminded participants that participation was completely voluntary. Researchers provided each participant necessary time to complete the test and requested that participants turn the test in upon completion. Participants included their names on each pre- and posttest, so researchers could match participants' pretests with their post-tests.

Institutional Review Board approval was not necessary, as this was deemed a project evaluation.


In total, 420 participants completed both the pre- and posttests. Of all participants, 358 were direct care staff and 62 were supervisory staff (supervisory staff includes RNs and clinically licensed individuals). Chi-square analyses and t tests were conducted to assess baseline differences between direct care staff and supervisory staff. There were no significant differences at baseline related to gender or ethnicity. However, there were significant differences between direct care and supervisory staff related to age (χ2[5] = 21.63, p < 0.001), level of education (χ2[3] = 77.30, p < 0.001), licensure (χ2[1] = 60.05, p < 0.001), and years of experience (t[420] = 6.89, p < 0.001). These differences were expected, considering higher level of education, licensure, and years of experience are often related to obtaining a supervisory position.

To test Hypothesis 1, a paired-samples t test was conducted to measure an increase in knowledge from before training to after training. There was a significant increase in correct answers from pretest (mean score = 10.69, SD = 2.78) to post-test (mean score = 13.47, SD = 2.36) (t[391]= −19.96, p < 0.001). This result supports the first hypothesis that participants would increase their knowledge of psychiatric rehabilitation following the training sessions.

To test Hypothesis 2, two-way, between-subjects analysis of covariance was conducted to examine the possible differences in knowledge gained between direct care staff and supervisory staff and between licensed and non-licensed staff. There was a difference between posttest scores of direct care staff and supervisory staff such that supervisors had greater improvement in scores from pre- to posttest (F[1,389] = 4.64, p < 0.05, ηp2 = 0.01). In addition, there was a difference between posttest scores of licensed staff and non-licensed staff such that licensed staff had greater improvement in scores from pre- to posttest (F[1,389] = 4.91, p < 0.05, ηp2 = 0.01).

In another test of Hypothesis 2, a Pearson product-moment correlation was calculated to assess the relationship between number of years of experience and change scores. There was a correlation between the two variables (r = −0.12, n = 388, p = 0.02). Overall, there was a weak, negative correlation between number of years of experience and increase of knowledge as indicated by the change score, showing that as number of years of experience increase, improvement in change score decreases. Therefore, staff who are newer to the field gain knowledge at a greater rate than staff who have more years of experience in the field.

Lastly, a multiple regression analysis was used to test whether credentials, the level of staff (i.e., direct care staff or supervisory staff), or number of years working in the field significantly predicted participants' change in knowledge. Factoring in the simultaneous entry of the predictor variables, it was found that the three variables do not explain a significant amount of variance in the change in knowledge (F[3,384] = 2.40, p < 0.06, R2Adjusted = 0.011). However, the analysis shows that of all variables tested, number of years working in the field is the most predictive of an increase in knowledge (β = −0.10, t[387] = −1.92, p = 0.05).


The purpose of the current study was to evaluate the effectiveness of CSS training to increase participants' knowledge of psychiatric rehabilitation skills and practices. Results indicate that 48-hour training curricula grounded in the work-based learning model can significantly increase participants' knowledge, as measured by improvements in pre- and posttest scores. Psychiatric rehabilitation focuses on improving the lives of individuals with SMI through recovery with an ultimate goal of living independently in the community (Pratt et al., 2014). Although most participants in the current study had some educational background, it was important to assess the increase in knowledge specific to psychiatric rehabilitation.

Throughout the process, all participants gained knowledge indicating the effectiveness of the training (Posavac, 2015). The current researchers explored differences in knowledge based on staff position (i.e., direct care staff or supervisory staff), having or not having a license, and number of years of experience. Including RNs as supervisory staff is essential when examining an increase in knowledge for CSS, as RNs are now expected to deliver services grounded in psychiatric rehabilitation (USDHHS, 2011).

When factoring all variables, years of experience appears to be the greatest predictor of participants' learning. Regardless, the results are not strong enough to provide concrete evidence that experience is a mediating factor for participant learning. Instead, there may be some connection between the three variables (i.e., years of experience, licensure, staff level) that the results do not highlight. For instance, staff with more years of experience tend to work in supervisory capacities and have more education. The study found that there is a negative relationship with years of experience in regard to change in knowledge. The less experience staff members have, the greater their change in knowledge.


Although the current study demonstrated an increase in knowledge, it did not include a control group to assess the increase in knowledge. Having a control group would provide stronger evidence that a difference in scores was directly connected to the work-based learning model of the training. Furthermore, researchers did not provide a specific timeframe for participants to complete the pre- and posttests. Instead, researchers handed out the pre- and posttests and allowed participants to complete the tests at any point throughout the first or final session, respectively. Future research should be aimed at structuring the process when giving the pre- and posttests. In addition, the study was conducted in one state with only supported housing staff; therefore, the results cannot be generalized to other states or service providers.

In addition, researchers tested participants at the end of the training sessions to assess the increase in knowledge; however, there was no follow up to determine whether participants maintained the knowledge over the long term or if the knowledge impacted the services. To strengthen the results of this study, researchers could follow up with participants at specific intervals and assess their knowledge of psychiatric rehabilitation. If participants can demonstrate continued knowledge, the researchers can substantiate their training approach, methods, and curriculum.

Future research should develop additional pre- and posttest questions to allow for greater range in scores, leading to better evaluation of knowledge. As the state supported housing agencies shift to a fee-for-service model grounded in psychiatric rehabilitation principles, it is imperative to assess and ensure that participants are gaining knowledge, skills, and practices. By doing so, supported housing providers will have the knowledge to provide more effective services to individuals with SMI.

Implications for Nursing

Happell et al. (2014) highlighted the shortage of nursing students wanting to work with individuals with SMI; however, this population of individuals with SMI continues to increase and is in need of services. Although RNs have the education and knowledge of a wellness perspective, incorporating it into their mental health practice may need improvement (Doyle et al., 2017). Increasing RNs' knowledge of psychiatric rehabilitation will enhance their understanding and application of a wellness approach and help them complete the comprehensive rehabilitation needs assessment, laying the groundwork for CSS.

As supported housing agencies seek RNs to work with individuals with SMI, it is important to increase training and improve confidence and a positive perception (Happell & Gaskin, 2013). Increasing psychiatric rehabilitation knowledge may lead to an improvement in confidence and perceptions, increasing the workforce in supported housing settings. In addition, as more nurses are hired to provide CSS, specific training on psychiatric rehabilitation may also be necessary to reduce any potential anxiety of working with this population. Individuals with SMI may receive more individualized services that impact their recovery and community integration if more nurses are trained in psychiatric rehabilitation.


The purpose of the current study was to evaluate the effectiveness of CSS training to increase participants' knowledge of psychiatric rehabilitation skills and practices. With more RNs being hired to work with individuals with SMI to provide CSS, creating training programs and assessing their knowledge is important. Although this research cannot account for the long-term effects of the training, RNs increased their knowledge in psychiatric rehabilitation. Ultimately, the more knowledgeable RNs are with this practice, the better they may be at delivering effective services that can improve the quality of life and assist with community integration for individuals with SMI.


  • Bachen, C., Brewster, A., Guerra-Sarabia, R., Merritt, J. & Schneider, J. (2015). Putting social justice at the center: Assessing social justice learning outcomes as a result of classroom, community, and co-curricular experiences. Retrieved from
  • Bellack, A.S., Mueser, K.T., Gingerich, S. & Agresta, J. (2004). Social skills training for schizophrenia (2nd ed.). New York, NY: Guilford Press.
  • Chase, M., Malden, A., Lansbury, L., Hansen, J., Ambrose, A., Thomas, C. & Costall, A. (2012). ‘In sight, out of mind’: The experiences of the compliantly engaged community psychiatric out-patient. Community Mental Health Journal, 48, 574–583. doi:10.1007/s10597-011-9414-9 [CrossRef]
  • Clay, Z., Barrett, N., Reilly, A. & Zazzarino, A. (2016). Work-based learning: A training model for state wide system changes. Psychiatric Rehabilitation, 39, 371–373. doi:10.1037/prj0000228 [CrossRef]
  • Doyle, L., Ellilä, H., Jormfeldt, H., Lahti, M., Higgins, A., Keogh, B. & Kilkku, N. (2017). Preparing master-level mental health nurses to work within a wellness paradigm: Findings from the eMenthe project. International Journal of Mental Health Nursing, 27, 823–832. doi:10.1111/inm.12370 [CrossRef]
  • Goncalves, D.A., Fortes, S., Campos, M., Ballester, D., Portugal, F.B., Tófoli, L.F. & Bower, P. (2013). Evaluation of a mental health training intervention for multidisciplinary teams in primary care in Brazil: A pre-and posttest study. General Hospital Psychiatry, 35, 304–308. doi:10.1016/j.genhosppsych.2013.01.003 [CrossRef]
  • Happell, B. & Gaskin, C.J. (2013). The attitudes of undergraduate nursing students towards mental health nursing: A systematic review. Journal of Clinical Nursing, 22, 148–158. doi:10.1111/jocn.12022 [CrossRef]
  • Happell, B., Platania-Phung, C., Harris, S. & Bradshaw, J. (2014). It's the anxiety: Facilitators and inhibitors to nursing students' career interests in mental health nursing. Issues in Mental Health Nursing, 35, 50–57. doi:10.3109/01612840.2013.837123 [CrossRef]
  • Hemingway, S., Clifton, A., Stephenson, J. & Edward, K.L. (2014). Facilitating knowledge of mental health nurses to undertake physical health interventions: A pre-test/post-test evaluation. Journal of Nursing Management, 22, 383–393. doi:10.1111/jonm.12220 [CrossRef]
  • Israel, T., Willging, C.E. & Ley, D. (2016). Development and evaluation of training for rural LGBTQ mental health peer advocates. Journal of Rural Mental Health, 40, 40–62. doi:10.1037/rmh0000046 [CrossRef]
  • Kirkbakk-Fjaer, K., Andfossen, N.B. & Hedelin, B. (2015). Preceptors' expectations of nursing students' preparation before placement in psychiatry: Ability and will to reflect on and exercise knowledge. Issues in Mental Health Nursing, 36, 300–306. doi:10.3109/01612840.2014.978424 [CrossRef]
  • Lauritzen, C., Reedtz, C., Van Doesum, K.T. & Martinussen, M. (2014). Implementing new routines in adult mental health care to identify and support children of mentally ill parents. BMC Health Services Research, 14, 58–67. doi:10.1186/1472-6963-14-58 [CrossRef]
  • Posavac, E.J. (2015). Program evaluation: Methods and case studies (8th ed.). New York, NY: Routledge
  • Pratt, C.W., Gill, K.J., Barrett, N.M. & Roberts, M.M. (2014). Psychiatric rehabilitation (3rd ed.). Philadelphia, PA: Elsevier.
  • Saks, A.M., Salas, E. & Lewis, P. (2014). The transfer of training. International Journal of Training and Development, 18, 81–83. doi:10.1111/ijtd.12032 [CrossRef]
  • Thongpriwan, V., Leuck, S.E., Powell, R.L., Young, S., Schuler, S.G. & Hughes, R.G. (2015). Undergraduate nursing students' attitudes toward mental health nursing. Nurse Education Today, 35, 948–953. doi:10.1016/j.nedt.2015.03.011 [CrossRef]
  • U.S. Department of Health and Human Services. (2011, October1). State plan under title XIX of the social security act. Retrieved from
  • White, J., Hemingway, S. & Stephenson, J. (2014). Training mental health nurses to assess the physical health needs of mental health service users: A pre- and post-test analysis. Perspectives in Psychiatric Care, 50, 243–250. doi:10.1111/ppc.12048 [CrossRef]

Community Support Services (CSS) Curriculum Overview

SessionContent Description

Described the six reimbursable CSS services

Illuminated the goals, values, and principles of psychiatric rehabilitation


Discussed the Olmstead settlement and its impact on supportive housing

Practiced effective communication and engagement skills


Described motivational interviewing strategies

Examined a wellness approach to assessment


Focused on using a comprehensive assessment to identify areas of need

Described the benefits of an individualized rehabilitation plan

Practiced how to break down an individual's goal into measurable objectives and interventions

Elucidated key components in required documentation to meet billing requirements


Illustrated the use of evidence-based practices to guide staff interventions

Explained the benefits of conducting a task/structured group to enhance skills building

Provided a structured approach to teaching skills that promote independence


Highlighted integrated dual disorder treatment

Provided a cumulative review of all sessions

Participant Demographics

Variablen (%)
Total (N = 420)Direct Care Staff (n = 358)Supervisory Staff (n = 62)
  Female328 (78.1)276 (77.1)52 (83.9)
  Male92 (21.9)82 (22.9)10 (16.1)
Age (years)
  22 to 2438 (9)37 (10.3)1 (1.6)
  25 to 34191 (45.4)173 (48.2)18 (29)
  35 to 4486 (20.4)68 (18.9)18 (29)
  45 to 5462 (14.7)49 (13.6)13 (21)
  55 to 6438 (9)29 (8.1)9 (14.5)
  ≥656 (1.4)3 (0.8)3 (4.8)
  Caucasian238 (56.4)194 (53.9)44 (71)
  African American97 (23)87 (24.2)10 (16.1)
  Hispanic39 (9.2)36 (10)3 (4.8)
  Asian American15 (3.6)13 (3.6)2 (3.2)
  Multi-ethnic22 (5.2)19 (5.3)3 (4.8)
  Other11 (2.6)11 (3.1)0 (0)
  High school diploma7 (1.7)7 (1.9)0 (0)
  Some college46 (10.9)45 (12.5)1 (1.6)
  Undergraduate degree217 (51.4)209 (58.1)8 (12.9)
  Graduate degree152 (36)99 (27.5)53 (85.5)
Clinical licensure
  No328 (77)304 (83.5)24 (38.7)
  Yes98 (23)60 (16.5)38 (61.3)
Years of experience (mean [SD])21.63 (8.27)6.53 (7.21)15.1 (9.33)

Dr. Zazzarino is Lecturer, Ms. Reilly is Assistant Professor, and Dr. Clay is Lecturer, Department of Psychiatric Rehabilitation and Counseling Professions, School of Health Professions, Rutgers, The State University of New Jersey, Scotch Plains, New Jersey.

The authors have disclosed no potential conflicts of interest, financial or otherwise.

Address correspondence to Anthony Zazzarino, Lecturer, PhD, LPC, ACS, CPRP, Department of Psychiatric Rehabilitation and Counseling Professions, School of Health Professions, Rutgers, The State University of New Jersey, 1776 Raritan Road, Room 504, Scotch Plains, NJ 07076; e-mail:

Received: April 23, 2018
Accepted: September 28, 2018
Posted Online: January 03, 2019


Sign up to receive

Journal E-contents