Journal of Gerontological Nursing

Research Brief 

Real-Time Observation of Apathy in Long-Term Care Residents With Dementia: Reliability of the Person–Environment Apathy Rating Scale

Ying-Ling Jao, PhD, RN; Jacqueline Mogle, PhD; Kristine Williams, PhD, RN, FNP-BC, FGSA, FAAN; Caroline McDermott, MS, RN; Liza Behrens, MSN, RN, CCRC

Abstract

Apathy is prevalent in individuals with dementia. Lack of responsiveness to environmental stimulation is a key characteristic of apathy. The Person–Environment Apathy Rating (PEAR) scale consists of environment and apathy subscales, which allow for examination of environmental impact on apathy. The interrater reliability of the PEAR scale was examined via real-time observation. The current study included 45 observations of 15 long-term care residents with dementia. Each participant was observed at three time points for 10 minutes each. Two raters observed the participant and surrounding environment and independently rated the participant's apathy and environmental stimulation using the PEAR scale. Weighted Kappa was 0.5 to 0.82 for the PEAR-Environment subscale and 0.5 to 0.8 for the PEAR-Apathy subscale. Overall, with the exception of three items with relatively weak reliability (0.50 to 0.56), the PEAR scale showed moderate to strong interrater reliability (0.63 to 0.82). The results support the use of the PEAR scale to measure environmental stimulation and apathy via real-time observation in long-term care residents with dementia. [Journal of Gerontological Nursing, 44(4), 23–28.]

Abstract

Apathy is prevalent in individuals with dementia. Lack of responsiveness to environmental stimulation is a key characteristic of apathy. The Person–Environment Apathy Rating (PEAR) scale consists of environment and apathy subscales, which allow for examination of environmental impact on apathy. The interrater reliability of the PEAR scale was examined via real-time observation. The current study included 45 observations of 15 long-term care residents with dementia. Each participant was observed at three time points for 10 minutes each. Two raters observed the participant and surrounding environment and independently rated the participant's apathy and environmental stimulation using the PEAR scale. Weighted Kappa was 0.5 to 0.82 for the PEAR-Environment subscale and 0.5 to 0.8 for the PEAR-Apathy subscale. Overall, with the exception of three items with relatively weak reliability (0.50 to 0.56), the PEAR scale showed moderate to strong interrater reliability (0.63 to 0.82). The results support the use of the PEAR scale to measure environmental stimulation and apathy via real-time observation in long-term care residents with dementia. [Journal of Gerontological Nursing, 44(4), 23–28.]

In the United States, 48.5% of residents in nursing homes and 39.6% of residents in residential care communities, including assisted living, have dementia (Harris-Kojetin, Sengupta, Park-Lee, & Valverde, 2013). Among individuals with dementia, 29% to 86% live with apathy (van Reekum, Stuss, & Ostrander, 2005). Apathy is a neuropsychiatric symptom of dementia, typically characterized as a deficit of motivation (Robert et al., 2009). Individuals with apathy demonstrate a lack of goal-directed behaviors, initiative, and interest, as well as decreased responsiveness to environmental stimulation and a flat mood (Robert et al., 2009). Apathy is associated with faster cognitive and functional decline (Starkstein, Jorge, Mizrahi, & Robinson, 2006), lower quality of life (Samus et al., 2005), and poor health outcomes (Vilalta-Franch, Calvó-Perxas, Garre-Olmo, Turró-Garriga, & López-Pouse, 2013). However, apathy is not well-identified or addressed in long-term care (LTC; Tagariello, Girardi, & Amore, 2009).

Apathy Assessment Tools and Challenges

Assessing apathy with a valid and reliable tool is a fundamental step toward addressing apathy. Over the past 2 decades, several instruments have been developed to measure apathy in neurodegenerative disorders; however, a gold standard has not been established, and there are limitations with existing tools (Radakovic, Harley, Abrahams, & Starr, 2015). The Neuropsychiatric Inventory–Apathy subscale (NPI–Apathy; Cummings et al., 1994) and the Apathy Evaluation Scale (AES; Marin, Biedrzycki, & Firinciogullari, 1991) are two widely used apathy scales with adequate reliability; however, they have not been well validated for individuals with dementia (Radakovic et al., 2015). Most apathy scales were developed and validated for community and clinic settings; however, few scales have been validated in LTC settings. Validated assessment tools are limited in application in LTC settings, omitting non-verbal residents with moderate to advanced dementia who may not have a family informant. Lastly, although response to environmental stimulation is a criterion for apathy diagnosis (Robert et al., 2009), few tools capture the environment experienced by individuals with dementia.

The Person–Environment Apathy Rating Scale

The Person–Environment Apathy Rating (PEAR) scale was recently developed to measure environmental stimulation and apathy and has been validated in LTC residents with dementia (Jao, Algase, Specht, & Williams, 2016). The PEAR scale is the first apathy scale that measures environmental stimulation experienced on the individual level. The PEAR scale measures the impact of environmental stimulation on apathy, providing evidence that is paramount to inform the development of environmental interventions to manage apathy (Jao, Algase, Specht, & Williams, 2015).

The psychometrics of the PEAR scale have been established in a prior study via video observation in LTC settings (Jao et al., 2016). The Environment subscale, content validity, interrater reliability (weighted Kappa = 0.49 to 0.94), intrarater reliability (weighted Kappa = 0.63 to 0.94), and internal consistency have been established (Cronbach's alpha = 0.84). The Apathy subscale demonstrated good convergent validity with the NPI–Apathy subscale (r = 0.71) and the Passivity in Dementia scale (r = 0.81) and moderate discriminant validity with the NPI–Depression subscale (r = 0.46).

It is necessary to expand the application of the PEAR scale to real-time observations. Video observations require extensive resources and training and may not be useful in small-scale pilot studies or clinical practice. Therefore, the purpose of the current study was to measure the interrater reliability of the PEAR scale for LTC residents with dementia using real-time observations.

Method

Design, Sample, and Setting

The current observational study collected 45 real-time observations of 15 LTC residents with dementia. Participants were recruited from one assisted living facility and one nursing home in central Pennsylvania. Inclusion criteria included participants who were: (a) 65 or older, (b) English-speaking, (c) diagnosed with dementia, and (d) residents of the facility for ≥1 month. Participants were excluded if they were diagnosed with: (a) acute, terminal illness, or (b) neurodegenerative or psychiatric disorders other than dementia that could impact neurobehavioral or affective function, such as acute stroke, Huntington's disease, Parkinson's disease, or head trauma. Participants could be at any stage of dementia, with or without apathy. Potential participants were identified by staff at the facility. Eligibility was determined by researchers via medical record reviews. The study protocol was approved by the Institutional Review Board. Prior to enrollment, consent was obtained from the resident's legally authorized representative and assent was obtained from the resident.

Study Procedures

For each resident, three real-time observations were conducted at different time points throughout the day: morning, afternoon, and during mealtime. No observations were made when residents were in bed or napping. Observations were completed ≥1 hour apart. Each observation was 10 minutes in length. During the observation, two raters independently observed and rated participants' apathy level and their surrounding environmental stimulation using the PEAR scale. The rater team comprised the principal investigator of the current study (Y.-L.J.) and three research assistants (two doctoral students [C.M., L.B.] and one undergraduate honors student in nursing). All research assistants reported having training in general research and experience in dementia research. Prior to the study, all research assistants were trained by the principal investigator using videos and real-time observations.

During each observation, raters were positioned between 3 and 10 feet from participants, so that they could see participants' facial expressions, whether eye contact was made, and the fronts of participants' upper bodies, as well as hear their voices (e.g., talking, laughing) while minimizing interference in participants' ongoing activities. The two raters did not communicate during observations. After the independent rating, the two raters reconciled, generating a final rating for each item on the Apathy and Environment subscales. Each rater's initial ratings were used to determine interrater reliability. Medical records at admission, within the past 4 weeks, and in the most recent Minimum Data Set (MDS) record were reviewed to collect medical information on participants' demographics and related information. MDS data were available only for nursing home residents, not those in assisted living.

Measurement

The PEAR comprises two subscales: Environment and Apathy. Each subscale consists of six items rated on a 1 to 4 scale. A higher rating on the Environment subscale indicates better environmental stimulation. A higher rating on the Apathy subscale indicates that the individual is more apathetic. The details of the PEAR scale have been published previously (Jao et al., 2016).

The Environment subscale defines environmental stimulation as any external stimulation that could trigger an individual's affective responses or motivate physical actions. Stimulation includes sensory, physical, and social stimuli present in the participant's visual field or the room the participant is in. The Environment subscale consists of six items, developed from a literature review of the mechanism of goal-directed behaviors (Levy & Dubois, 2006) and the environmental features relevant to aging, cognition, and apathy (Jao et al., 2016; Kaplan & Kaplan, 1936).

  • Stimulation Clarity. The least clear stimulation is chaotic with multiple competing stimuli; the clearest stimulation contains one straightforward stimulus.
  • Stimulation Strength. The weakest stimulation is undetectable; the strongest stimulation is loud, interesting, or innovative, and could easily trigger the individual's reactions.
  • Stimulation Specificity. The least specific stimulation contains no stimulation delivered to the individual; the most specific stimulation is directly delivered to the individual and addresses the individual's needs, preferences, or experiences.
  • Interaction Involvement. The least interactive stimulation contains no interpersonal interaction; the most interactive stimulation contains interpersonal interaction that actively involves the individual.
  • Physical Accessibility. The least accessible stimulation is far from the individual with barriers and is inaccessible for the individual without assistance; the most accessible stimulation is close to the individual and easy to access without assistance.
  • Environmental Feedback. The lowest level of environmental feedback is restrictive and prohibits individual's self-expression or engagement; the highest level of environmental feedback prompts the individual's expression and engagement.

The Apathy subscale quantifies the symptoms of apathy via observable behavioral and affective symptoms using six items. A higher rating indicates a higher apathy level.

  • Facial Expression. The lowest level is intense expression with excessive facial expression (e.g., laughing, weeping); the highest level is minimal expression with no observable facial expression.
  • Eye Contact. The lowest level is sustained eye contact with a specific target; the highest level is eyes closed.
  • Physical Engagement. The lowest level is enthusiastic engagement; the highest level is minimal engagement with no engagement in activities.
  • Purposeful Activity. The lowest level is self-initiated purposeful activity; the highest level is no engagement in activities.
  • Verbal Tone. The lowest level is loud or extreme intonation; the highest level is silent.
  • Verbal Expression. The lowest level is self-initiated verbal expression; the highest level is no verbal expression.

Data Analysis

Descriptive statistics were used to describe participants' demographics, baseline characteristics, and total scores on the Apathy and Environment subscales. SPSS version 24 was used for statistical analysis. Inter-rater reliability was analyzed using weighted Kappa and Fleiss-Cohen weighting, which enable differentiation of levels of disagreement (Fleiss & Cohen, 1973).

Results

Among the 15 resident participants, 60% lived in an assisted living facility and the remainder lived in a nursing home. The residents were Caucasian and predominantly women (60%). Participants ranged in age from 75 to 93 (mean age = 85.8 years) (Table 1). The final reconciled ratings of 45 observations of 15 residents on the PEAR subscales are summarized in Table 2. The average rating of the Environment subscale was 19.7 of 24, indicating a relatively high level of environmental stimulation. The average rating for the Apathy subscale was 13.5 of 24, indicating relatively low apathy levels in the current sample.

Participant Demographics and Baseline Characteristics (N = 15)

Table 1:

Participant Demographics and Baseline Characteristics (N = 15)

Description of Ratings (N = 45 Observations)

Table 2:

Description of Ratings (N = 45 Observations)

The interrater reliability of the Environment and Apathy subscales is reported in Table 3. The weighted Kappa of the Environment subscales was 0.50 to 0.82. Most items showed moderate to strong reliability, with weighted Kappa >0.6 (McHugh, 2012). Two items that showed relatively weak reliability were stimulation strength and physical accessibility, with weighted Kappa of 0.56 and 0.50, respectively. The weighted Kappa of the Apathy subscale was 0.5 to 0.8. Five of the six items showed moderate to strong interrater reliability. The only item that showed low reliability was facial expression, with weighted Kappa of 0.5.

Interrater Reliability of the Person–Environment Apathy Rating Scale (N = 45 Observations)

Table 3:

Interrater Reliability of the Person–Environment Apathy Rating Scale (N = 45 Observations)

Discussion

Overall, using the PEAR scale via real-time observation demonstrated moderate to strong interrater reliability for most items on the Environment and Apathy subscales. Three items, stimulation strength, physical accessibility, and facial expression, demonstrated relatively weak reliability.

These findings support the stability of the PEAR scale by adding a new approach. Compared to video observation in a previous study (Jao et al., 2016), real-time observation in the current study may cause challenges related to reliability in various ways. The observation time was extended from 1 to 2 minutes in the previous study to 10 minutes in the current study. In addition, real-time observation allowed a broader vantage point and therefore involved the collection of more observable visual data. The longer observation times and broader observation angles are likely to include variations, including fluctuation in resident apathy levels and more complex environmental stimulation, making the rating increasingly challenging. Furthermore, real-time observation did not allow repeated observations to confirm ratings.

Despite these challenges, the PEAR scale via real-time observation demonstrated reliability across raters in the current study. Additional rating instruction and training for raters may improve the reliability for real-time observations, especially for observations over longer periods of time or within a complex environmental context. For example, when the behavior or environment varies within the observation period, training should guide the raters as to whether to provide multiple ratings or rate the lowest observed level, highest level, or level that lasts the longest. In addition, when environmental stimulation is complex, which stimulation to focus on and how to distinguish primary stimulation from background noise may need to be included in rater training.

The PEAR scale via video or real-time observation demonstrates moderate to strong interrater reliability for most items, making either approach appropriate. Notably, for the Environment subscale, stimulation strength and physical accessibility showed relatively low interrater reliability for video and real-time observation. Thus, improving reliability on these two items is needed for either observation method. The two approaches have slightly different levels of reliability for the Apathy subscale. Eye contact showed strong reliability in real-time observation but relatively low reliability in video observation. This discrepancy may have occurred because eye contact assessment via video observation relies on whether the video clearly captures the participants' eyes. If participants turn their heads or the video captures only the sides of participants' faces, assessing eye contact is challenging. In contrast, real-time observation allows flexible viewing to evaluate participants' visual targets, making eye contact easier to assess. The difficulty in assessing eye contact with video observation might be addressed by improving video quality, broadening video angle, and/or supplementing with direct observation. Unlike eye contact, however, facial expression showed moderate reliability via video observation but weak interrater reliability via real-time observation. This weak interrater reliability might be addressed by ensuring raters are positioned close enough to the participant to observe subtle facial expressions.

Besides reliability, other practical factors should be considered regarding the PEAR scale observation method. These factors include the assessment purposes, resources, and rater training and background. Video observation allows for repeated viewing to confirm ratings, for future observation of other assessments (e.g., other behavioral symptoms in dementia), and more advanced statistical analysis. However, it requires extensive time, training, and cost to record and store videos. Thus, video observation is more practical for large-scale studies. Real-time observation, on the other hand, allows the environmental context to be comprehensively captured. However, it could complicate the rating and does not allow for repeated observations; thus, quality rater training is essential. Real-time observation is less expensive and less time consuming, making it feasible for pilot studies and clinical practice.

Limitations

The current study was a pilot study with a small, homogeneous sample from only two facilities, which limits the generalizability of the results. Participants' underlying characteristics, including cognitive and functional levels, were determined by chart review. The two facilities have brief and different assessments of residents' cognitive and functional levels, limiting the capacity to report participants' characteristics. Despite these limitations, the results provide evidence from trained raters in real time to support the interrater reliability of the PEAR scale. Further instruction and training on the assessment of stimulation strength, physical accessibility, and facial expression may enhance rating reliability. Finally, the current study did not evaluate the validity of using the PEAR scale in real-time observations. Although the validity of the PEAR scale has been established in video observation in prior research (Jao et al., 2016), it would be helpful to confirm the validity in real-time observations, through a method such as comparing the rating of real-time observations with video observations. Establishing the validity of using the PEAR scale in real-time observations is an area for future research.

Clinical Implications of the Pear Scale

Despite these limitations, the results of the current study support the use of the PEAR scale in real-time observations to assess apathy and environmental stimulation in LTC residents with dementia. The findings of the current study support expanded use of the PEAR scale in future research and clinical practice using real-time observation. Using real-time observation shortens the amount of time and reduces the cost required for data collection and analysis, thus making its use more feasible than collecting and rating video recordings. Currently, the PEAR scale is used primarily for research. However, it is believed that with training and rating practice, the PEAR scale could be used by clinical care providers, although the feasibility of its use by clinical care providers needs to be confirmed. The belief that it could be used by providers is due to the rating instruction of the scale, which provides detailed descriptions and examples to guide clinical use. In addition, the scale assesses apathy via observation, which makes it feasible to assess apathy even if individuals with dementia are unable to provide self-report data and family informants are not accessible. These features address the challenges of assessing apathy in nursing home residents with moderate to advanced dementia.

The PEAR scale is one of few apathy assessment tools that address environmental factors (Lanctôt et al., 2017), and it could have critical clinical implications for apathy assessment and management. Given that response to environmental stimulation is one criterion for apathy diagnosis (Robert et al., 2009), assessing apathy in consideration of environmental stimulation more accurately quantifies apathy (Jao et al., 2016). In addition, the PEAR scale could help identify environmental determinants of apathy to aid the identification of high-risk populations and guide development of environment interventions to manage apathy.

Conclusion

The current research was a pilot observational study using the PEAR scale in the LTC setting. Importantly, the PEAR scale enables evaluation of the impact of care environments on apathy in the LTC setting, and may be used to identify key environmental factors that can be transformed to develop effective interventions aimed to prevent and manage apathy in dementia in LTC.

References

  • Cummings, J.L., Mega, M., Gray, K., Rosenberg-Thompson, S., Carusi, D.A. & Gornbein, J. (1994). The neuropsychiatric inventory comprehensive assessment of psychopathology in dementia. Neurology, 44, 2308–2314. doi:10.1212/WNL.44.12.2308 [CrossRef]
  • Fleiss, J.L. & Cohen, J. (1973). The equivalence of weighted kappa and the intraclass correlation coefficient as measures of reliability. Educational and Psychological Measurement, 33, 613–619. doi:10.1177/001316447303300309 [CrossRef]
  • Harris-Kojetin, L., Sengupta, M., Park-Lee, E. & Valverde, R. (2013). Long-term care services in the United States: 2013 overview. Vital and Health Statistics, 3(37), 1–107.
  • Jao, Y.L., Algase, D.L., Specht, J.K. & Williams, K. (2015). The association between characteristics of care environments and apathy in residents with dementia in long-term care facilities. The Gerontologist, 55, S27–S39. doi:10.1093/geront/gnu166 [CrossRef]
  • Jao, Y.L., Algase, D.L., Specht, J.K. & Williams, K. (2016). Developing the Person-Environment Apathy Rating for persons with dementia. Aging and Mental Health, 20, 861–870. doi:10.1080/13607863.2015.1043618 [CrossRef]
  • Kaplan, S. & Kaplan, R. (1936). Cognition and environment: Functioning in an uncertain world. Ann Arbor, MI: Ulrich's.
  • Lanctôt, K.L., Agüera-Ortiz, L., Brodaty, H., Francis, P.T., Geda, Y.E., Ismail, Z. & Abraham, E.H. (2017). Apathy associated with neurocognitive disorders: Recent progress and future directions. Alzheimer's & Dementia, 13, 84–100. doi:10.1016/j.jalz.2016.05.008 [CrossRef]
  • Levy, R. & Dubois, B. (2006). Apathy and the functional anatomy of the prefrontal cortex—Basal ganglia circuits. Cerebral Cortex, 16, 916–928. doi:10.1093/cercor/bhj043 [CrossRef]
  • Marin, R.S., Biedrzycki, R.C. & Firinciogullari, S. (1991). Reliability and validity of the apathy evaluation scale. Psychiatry Research, 38, 143–162. doi:10.1016/0165-1781(91)90040-V [CrossRef]
  • McHugh, M.L. (2012). Interrater reliability: The kappa statistic. Biochemia Medica, 22, 276–282. doi:10.11613/BM.2012.031 [CrossRef]
  • Radakovic, R., Harley, C., Abrahams, S. & Starr, J.M. (2015). A systematic review of the validity and reliability of apathy scales in neurodegenerative conditions. International Psychogeriatrics, 2, 903–923. doi:10.1017/S1041610214002221 [CrossRef]
  • Robert, P., Onyike, C.U., Leentjens, A.F.G., Dujardin, K., Aalten, P., Starkstein, S. & Byrne, J. (2009). Proposed diagnostic criteria for apathy in Alzheimer's disease and other neuropsychiatric disorders. European Psychiatry, 24, 98–104. doi:10.1016/j.eurpsy.2008.09.001 [CrossRef]
  • Samus, Q.M., Rosenblatt, A., Steele, C., Baker, A., Harper, M., Brandt, J. & Lyketsos, C.G. (2005). The association of neuropsychiatric symptoms and environment with quality of life in assisted living residents with dementia. The Gerontologist, 45, 19–26. doi:10.1093/geront/45.suppl_1.19 [CrossRef]
  • Starkstein, S.E., Jorge, R., Mizrahi, R. & Robinson, R.G. (2006). A prospective longitudinal study of apathy in Alzheimer's disease. Journal of Neurology, Neurosurgery and Psychiatry, 77, 8–11. doi:10.1136/jnnp.2005.069575 [CrossRef]
  • Tagariello, P., Girardi, P. & Amore, M. (2009). Depression and apathy in dementia: Same syndrome or different constructs? A critical review. Archives of Gerontology and Geriatrics, 49, 246–249. doi:10.1016/j.archger.2008.09.002 [CrossRef]
  • van Reekum, R., Stuss, D.T. & Ostrander, L. (2005). Apathy: Why care?Journal of Neuropsychiatry, 17, 7–19. doi:10.1176/jnp.17.1.7 [CrossRef]
  • Vilalta-Franch, J., Calvó-Perxas, L., Garre-Olmo, J., Turró-Garriga, O. & López-Pousa, S. (2013). Apathy syndrome in Alzheimer's disease epidemiology: Prevalence, incidence, persistence, and risk and mortality factors. Journal of Alzheimer's Disease, 33, 535–543. doi:10.3233/JAD-2012-120913 [CrossRef]

Participant Demographics and Baseline Characteristics (N = 15)

Characteristicn (%)
Facility type
  Assisted living9 (60)
  Nursing home6 (40)
Gender
  Female9 (60)
Race
  Caucasian15 (100)
Marital status
  Widowed8 (53.3)
  Married6 (40)
  Never married1 (6.7)
Hearing impairment
  None4 (26.7)
  Minimal6 (40)
  Moderate5 (33.3)
No visual impairment12 (80)
Alzheimer's disease6 (40)
Depression8 (53.3)
Problems with short-term memory15 (100)

Description of Ratings (N = 45 Observations)

MeasurementMean (SD, Range)
PEAR-Environment Total19.7 (3.0, 10 to 23)
  Stimulation clarity3.6 (0.7, 1 to 4)
  Stimulation strength3.0 (0.6, 1 to 4)
  Stimulation specificity2.9 (0.8, 1 to 4)
  Interaction involvement2.9 (1.2, (1 to 4)
  Physical accessibility3.9 (0.3, 2 to 4)
  Environmental feedback3.5 (0.7, 2 to 4)
PEAR-Apathy Total13.5 (4.1, 6 to 24)
  Facial expression2.6 (0.8, 1 to 4)
  Eye contact1.7 (1.1, 1 to 4)
  Physical engagement2.1 (0.7, 1 to 4)
  Purposeful activity1.6 (0.8, 1 to 4)
  Verbal tone2.9 (0.9, 1 to 4)
  Verbal expression2.5 (1.3, 1 to 4)

Interrater Reliability of the Person–Environment Apathy Rating Scale (N = 45 Observations)

Rating ItemWeighted KappaStandard Error
Environment
  Stimulation clarity0.630.14
  Stimulation strength0.560.18
  Stimulation specificity0.680.08
  Interaction involvement0.820.05
  Physical accessibility0.500.16
  Environmental feedback0.720.07
Apathy
  Facial expression0.500.11
  Eye contact0.800.09
  Physical engagement0.710.08
  Purposeful activity0.660.08
  Verbal tone0.800.06
  Verbal expression0.790.08
Authors

Dr. Jao is Assistant Professor, Dr. Mogle is Assistant Professor, Ms. McDermott is Doctoral Student, and Ms. Behrens is Project Coordinator, Pennsylvania State University College of Nursing, University Park, Pennsylvania; and Dr. Williams is Professor, University of Kansas School of Nursing, Kansas City, Kansas.

The authors have disclosed no potential conflicts of interest, financial or otherwise.

Address correspondence to Ying-Ling Jao, PhD, RN, Assistant Professor, Pennsylvania State University College of Nursing, 307E Nursing Sciences Building, University Park, PA 16802; e-mail: yuj15@psu.edu.

Received: July 03, 2017
Accepted: December 11, 2017
Posted Online: February 14, 2018

10.3928/00989134-20180131-02

Sign up to receive

Journal E-contents