The incidence of age-based disorders underscores the fact that caring for older adults is an intricate mosaic of treatment complexities. Older individuals are at high risk for developing psychiatric disorders including dementia (10%), delirium (1% to 2%), and depression (10% to 15%) (Coffey & Cummings, 1996). Risk estimates escalate even further for older adults with depression (40%) and delirium (10% to 20%) as a result of neurologic damage (e.g., stroke) (Coffey & Cummings, 1996). Neuropsychiatrie symptoms often overshadow and complicate the therapeutic management of older adults' multiple, chronic health conditions, resulting in their increased morbidity and mortality.
Direct care professionals should demonstrate knowledge competency as an antecedent for delivering quality care in a safe and effective manner. Research indicates the current work force, however, is inadequately trained to effectively manage older adults' special neuropsychiatrie treatment needs (Gottlieb, 1998; Smith, et al., 1994; Stolley, Buckwalter, &c Harper, 1995). In particular, staff members do not receive training adequately preparing them to effectively care for older adults with neuropsychiatrie disabilities such as dementia or delirium. Additionally, a shortage of trained professionals specializing in geriatrics has been projected well into the 21st century (Gatz Sc Finkel, 1995). The demand will continue to grow for professionals who possess specialized knowledge of behavioral health care needs for the aging (Pew Health Professions Commission, 1991).
Published tools with regards to dementia measure important, yet distinct domains of know-ledge (Dieckmann, Zarit, Zarit, & Gatz, 1988; Spore, Smyer, & Cohn, 1991).
No tool currently exists as a comprehensive aging and mental health measure and no prior studies were found reporting the establishment of a universal aging knowledge instrument. This article describes the development of a tool measuring health care professionals' essential knowledge of aging and mental illness. The tool was developed to verify work force knowledge competencies related to normal aging, serious mental illness, and dementia. This article presents preliminary data describing the use of this instrument as a measure of staff geropsychiatric knowledge.
Background and Study Design
The Mary Starke Harper Aging Knowledge Exam (MSHAKE) tool was developed to measure staff knowledge before and after participating in a 16-hour, in-service educational offering. Prior to opening the Mary Starke Harper Geriatric Psychiatry Center in Tuscaloosa, Alabama, a 2-day in-service was offered to all 177 employees. The inservice used a lecture-discussion format. A geropsychiatrist and a doctorally prepared nurse certified in geriatric nursing, team-taught the course content. The in-service was developed because of a perceived need to ensure basic knowledge of aging among a staff with diverse educational/practice backgrounds. The majority of the participants had work experience in psychiatric management but not necessarily with elderly individuals. The training involved both direct and support care personnel and covered four major topics:
* Older adult demographics.
* Changes in normal aging.
* Depression, dementia, and delirium characteristics.
* Older adult care issues.
A pretest/posttest design was used. The MSHAKE was administered on Day 1 as a pretest, prior to the content delivery, and again as a posttest on Day 2 at the conclusion of the in-service. This design supported the examination of shortterm knowledge changes following a staff development educational program about aging. A variety of descriptive and inferential statistics were used to describe the sample and evaluate the effectiveness of the tool.
A practical knowledge test should assess critical knowledge, minimize complex terminology, be user-friendly for individuals with limited educational background, and be easily scored (Burns & Grove, 1997). A team of two psychiatrists and two nurses participated in the item development of the MSHAKE tool. The team selected items that reflect basic and essential knowledge necessary to all disciplines when caring for older adults, especially addressing the geropsychiatric population. Following three focus group discussions, 25 true/false items were selected to create the MSHAKE tool. The items were evenly distributed and focused on the following three content areas: knowledge of basic aging, dementia, and mental illness. Pilot testing of the 25 items was completed with five individuals in other health care settings. On the basis of this pilot test, items were revised prior to the tool's administration.
Validity of the instrument was assessed in a number of ways. A factor analysis was performed to determine if the items clustered into factors might become subscales. The analysis revealed no definitive clustering, and the instrument was used as a single scale intended to measure essential, fundamental geropsychiatric knowledge. Inter-item reliability was determined through the use of Cronbach's alpha. The standardized alpha on the pretest was 0.7206 and 0.7084 on the posttest, as shown in Table 1. These values reflect an acceptable level of internal consistency (Polit & Hungler, 1999). The reliability procedures of Statistical Package for the Social Sciences (SPSS) Version 9.0 through analysis of variance demonstrated a residual that represented low subject variability (0.1614). Therefore, the MSHAKE has high sensitivity, implying that the power is high (Burns & Grove, 1997). To further verify the tool's content validity, the opinions of 15 nationally recognized geriatric and mental health experts were solicited. A listing of qualified reviewers was obtained from national aging and mental health publications, as well as personal recommendations from colleagues. Of the 15 solicited participants, 12 responded. They represented various United States geographic regions, and their backgrounds included a variety of professions with an aging focus, such as medicine, nursing, psychology, and social work. The respondents held positions in aging research, academia, geropsychiatric clinical practice, health care administration, and aging policy development. At least two of the panel members were published in the field of pyschometric instrumentation.
Content analysis of the returned surveys was completed to identify suggested modifications. The reviewers were asked to rate the relevance and clarity of each item, as well as provide editorial recommendations. Suggestions for additional items were also requested. Finally, they were asked to rate the tool's comprehensiveness, as well as the potential for its adoption as a measure of staff knowledge. Given its early stage of development, the tool was rated by the majority of the respondents as being an adequate representation of essential, foundational aging knowledge. While all items were rated as relevant, the following two items related to depression and anxiety were suggested as improvements: Older adults have a lower prevalence of depressive disorders and the most common psychiatric disorder among elderly individuals is an anxiety disorder.
A majority of the panel indicated a willingness to adopt and refine this instrument. The group provided extensive editing suggestions for improving the items' clarity. Table 2 represents the original tool used for this study as well as the revised items suggested as a result of the survey.
Pretest and posttest were completed by 171 participants. Of this sample, 136 (79.5%) were involved in direct patient care and 35 (20.5%) were employed in indirect support care positions. Participants consisted of staff psychiatrists, nurse practitioners, social workers, nurses, mental health workers, housekeepers, recreation therapists, clerical workers, administrators, and medical records personnel. Educational levels of the participants ranged from 9 to 27 years of education, with a mean of 13.6 years. The most frequently reported level of education was 12 years for 46.8% of the sample. Eighty-one percent (n = 139) of the participants were women, 19% (n = 32) were men, 66% (n = 112) were Black, and 34% (n = 59) were White.
The scores on the pretest ranged from 5 to 24, with a mean of 17.64. The posttest scores range from 8 to 24, with a mean of 20.33 (Table 3). A paired t test was used to examine changes in the MSHAKE tool from pretest to posttest. A statistically significant increase in the scores was found (t = -14.03, p = .000). The data were then examined for differences between pretest and posttest scores based on level of education and position using repeated measures ANOVA. For this examination, education was split into three levels: Level 1 is O to 12 years (n = 83), Level 2 is 13 to 16 years (« = 69), and Level 3 is more than 16 years of education (n = 19). There were significant differences between the pretest and posttest (F = 96.985, p = .000) scores for level of education. In addition, there were significant interaction effects (F - 7.430, p - .001). On further examination, it was found that the change for Levels 1 and 2 were reasonably linear but there appears to be a ceiling effect for Level 3 (Table 4).
AASHAKE RELIABILITY DATA
A significant difference was also found between the pretest and posttest scores (F = 163.247, p = .000) based on employment position. Position was a two-level variable with Level 1 (n = 135) being those involved in direct patient care and Level 2 (n = 35) being those in indirect care/support services. There were significant interaction effects as well (F = 5.003, p = .027). The greatest change was evidenced among the indirect patient-care staff. In examining the data closely, it was found that the pretest scores were fairly normally distributed. However, posttest scores for the patient care staff were skewed to the left and the support staff was skewed to the right.
In addition to the repeated measures ANOVA analysis, the data were examined using a simple gain score computed by subtracting a pretest score from a posttest score. Several studies have documented the use of a gain score as a way of indicating differences in knowledge between pretesting and posttesting (Miller, Jensen, & Achterberg, 1999; Reding, et aï., 1996; Schwanz, Donnelly, Sloan & Young, 1994; Zimmerman, et al., 1987). The gain scores for this study ranged from -3 to 12, with a mean of 2.96 and a standard deviation of 2.76. The median and the mode were both 3.00. Eighty percent of the subjects (n = 137) achieved improved knowledge pretest to posttest regardless of educational background, while 12% (n = 21) indicated no change and 8% (n = 13) demonstrated a decrease in knowledge.
Gain scores were examined for differences based on gender and ethnicity. No difference was found for gender. However, a significant difference (F = 7.088, p = .009) was found for ethnicity, with Black participants demonstrating the greatest gain in knowledge related to aging. The group's performance by test item was also examined. Table 5 provides each item's percent correct on the pretests and posttests.
The MSHAKE samples essential, fundamental knowledge related to aging, mental illness, and dementia.
The instrument uses simple, direct language to survey knowledge about complex diseases such as dementia in schizophrenic patients or complicated interactions between medical and psychiatric diseases. Test performance data indicates that the MSHAKE effectively measures staff knowledge in a group with past experience in psychiatric management but not necessarily with older adults. It was not surprising to find that there were significant differences between the pretest and posttest scores based on levels of education. The significant interaction effects noted between levels of education and the pretest and posttest scores might be attributed to the closeness in time between test administrations. However, the correct responses were not shared with the participants.
AAARY STARKE HARPER AGING KNOWLEDGE EXAM (MSHAKE)
PRETEST, POSTTEST, AND GAIN SCORES
Literacy was not an important variable, as increases in knowledge were more strongly attributed to staff position versus level of education. There were significant interaction effects between pretest and posttest scores by position. It was not surprising that the greatest knowledge gain was found in the group that was in indirect care or support services. Given the instrument's early stage of development, this finding provides initial validation of construct validity based on the known-groups technique.
While the extent of improvement varied, most of the items, 20 of the 25, demonstrated some margin of improvement from pretest to posttest. The items demonstrating the greatest improvement clustered around basic aging and care management knowledge. This is significant because of possible patient safety implications. For example, knowledge related to aspiration and choking with dementia patients (No. 19), restraining wandering patients (No. 18), sundowners with dementia (No. 24), and elderly suicide prevalence (No. 25) showed greatest improvement (Table 5). Notable improvements were also shown in normal aging content (e.g., Nos. 8, 13, 16). However, not all items showed improvement, notably Nos. 15, 20, 21, and 23 reflect stable percentages of correct responses pretest to posttest.
GAIN SCORE BY EDUCATION
Additionally there was a decreased score for item No. 6, and item No. Í5 demonstrated a stable, yet low, performance related to reorienting an agitated dementia patient. The stable and decreased scores were not unexpected because the educational program did not specifically address individual test items. However, the scores on these items indicate a need to focus on these topics in future in-service training.
The instrument's practical utility is a major asset. The 25-item examination is completed in 20 minutes and accommodates a wide variety of reading levels while comprehensively testing basic aging knowledge. This knowledge-screening instrument could function as part of a staff knowledge competency verification system or an in-service evaluation tool for a diversely educated and trained work force functioning in the geriatric mental health continuum of care. The MSHAKE is also useful as a pretest for identifying knowledge gaps that should be emphasized in inservice education.
The statistical outcomes with this tool, in this study population, support this as a test of short-term knowledge gain. What is not known is how this knowledge gain translates into improved patterns of clinical practice. Future plans for the tool include using the MSHAKE revised version, within the state's mental health system, as a mechanism for assessing staff knowledge of aging. This will enable the tool's ongoing refinement and development.
This article provides preliminary documentation corroborating the use of the MSHAKE as an effective measure of essential, fundamental knowledge of aging and mental health. While this instrument requires additional refinement, the findings support its continuing usage and development. Widespread use of the tool in a variety of clinical settings will further establish its statistical validity and reliability as well as its practice impact. Health care workers in nursing homes, acute inpatient settings, home health settings, and outpatient settings increasingly manage elderly patients with neuropsychiatrie disability. The MSHAKE provides supervisors, regulators, and reimbursement agencies with a method to assess knowledge of all staff providing care to these patients.
- Burns, N. & Grove, S.K. (1997). Practice of nttrsing research: Conduct, critique and utilization (3rd ed.)- Philadelphia: W.B. Saunders.
- Coffey, C.E. & Cummings, J.L., (Eds.). (1996). Textbook of geriatric neuropsychiatry. Washington, DC: American Psychiatric Press.
- Dieckmann, L., Zarit, S.H., Zarit, J.M., & Gatz, M. (1988). The Alzheimer's disease knowledge test. The Gerontologtst, 28(3), 402-407.
- Gatz, M. & Finkel, S. (1995). Emerging issues in mental health and aging. Washington, DC: American Psychological Association.
- Gottlieb, G. (Ed). (1998). Managed behavioral healthcare - Standards, guidelines and competencies for older adults with menial illness. Rockville, MD: Center for Mental Health Services, SAMSHA.
- Miller, C.K., Jensen, G.L., & Achterberg, C.L. (1999). Evaluation of a food label nutrition intervention for women with type 2 diabetes mellitus. Journal of the American Dietetics Association, 99(3), 323-328.
- Pew Health Professions Commission, (1991). Healthy America: Practitioners for 20Oi: An agenda for action for U.S. health professional schools. Durham, NC: Duke University Medical Center.
- Polit, D. & Hungler, B. (1999). Nursing research principles and methods (6th ed.). Philadelphia: Lippincott.
- Reding, D.J., Fischer, V, Gunderson, P., Lappe, K., Anderson, H., & Calvert, G. (1996). Teens teach skin cancer prevention. Journal of Rural Health, 12(4), 265-272.
- Schwartz, R. W., Donnelly, M.B., Sloan, D.A., & Young, B. (1994). Knowledge gain in a problem-based surgery clerkship. Academic Medicine, 69(2), 148-151.
- Smith, M., Buckwalter, K.C., Garand, L., Mitchell, S., Albanese, M.,& Kreiter, C. (1994). Evaluation of a geriatric mental health training program for nursing personnel in rural long-term care faculties. Issues in Mental Health Nursing, li, 149-I68.
- Spore, D.L., Smyer, M.A., & Cohn, M.D. (1991). Assessing nursing assistants' knowledge of behavioral approaches to mental health problems. The Gerontologist, 31(3"), 309-317.
- Stolley, J.M., Buckwalter, K.C., & Harper, M.S., (1995). Unity in diversity: Geropsychiatric nursing survey outcomes. Journal of Gerontological Nursing, 2/(I), 31-36.
- Zimmerman, E., Carter, M.C., Sears, J.H., Lawson, J. S., Howard, C.P., & Hassanein, R.E.S. (1987). Diabetic camping: Effect on knowledge, attitude, and self concept. Issues in Comprehensive Pediatrie Nursing. 10(2), 99-111.
AASHAKE RELIABILITY DATA
AAARY STARKE HARPER AGING KNOWLEDGE EXAM (MSHAKE)
PRETEST, POSTTEST, AND GAIN SCORES
GAIN SCORE BY EDUCATION