Research in Gerontological Nursing

The articles prior to January 2010 are part of the back file collection and are not available with a current paid subscription. To access the article, you may purchase it or purchase the complete back file collection here

Literature Review 

Direct Observation of Behavior: A Review of Current Measures for Use with Older Adults with Dementia

Kim J. Curyto, PhD; Kimberly Van Haitsma, PhD; Deedre K. Vriesman, BA

Abstract

This review article reports on methods of direct observation of behaviors for use in long-term care settings, particularly with older adults who have dementia. This article provides information on the theoretical roots, administration methods, and psychometric properties of measures of direct observation of individual behavior. It is hoped that this review will help gerontological nurses make informed choices about the direct observation measures that suit their specific needs, highlight the role of direct observation in quality improvement for dementia care, and facilitate a balance between identifying a gold standard and allowing flexibility to assess project-specific behaviors.

 

Click here to read a letter to the editor about this article.

Abstract

This review article reports on methods of direct observation of behaviors for use in long-term care settings, particularly with older adults who have dementia. This article provides information on the theoretical roots, administration methods, and psychometric properties of measures of direct observation of individual behavior. It is hoped that this review will help gerontological nurses make informed choices about the direct observation measures that suit their specific needs, highlight the role of direct observation in quality improvement for dementia care, and facilitate a balance between identifying a gold standard and allowing flexibility to assess project-specific behaviors.

 

Click here to read a letter to the editor about this article.

When documenting changes in functioning, evaluating the effects of an intervention on an individual’s life, or measuring the effectiveness of a program of care, there are a number of methods to assess specific areas of behavior. These methods include self-report, caregiver report, and family report of person-specific behaviors. Self-report of behaviors by an older adult with dementia is often not feasible in light of significant cognitive decline and communication difficulties, and ratings made by others can introduce measurement errors. These include poorly defined behaviors, unstandardized and variable opportunity for observation of behavior, unclear reference periods in looking retrospectively at an individual’s behavior, and errors in recall of the behavior and its context. Retrospective report is also limited in providing accurate information on the frequency and duration of behaviors.

Theory Underlying Direct Observation of Behavior

An area of measurement that has gained acceptance in capturing the behavior of older adults with dementia is direct observation. Although direct observation has a long history of use in the field of behavior analysis, its use with the dementia population is relatively new and often not guided by theory. The test developer’s perspective determines what components of behavior are measured and how they are recorded. A number of direct observation methods exist, and choosing among them without easy access to information on their theory, administration method, and psychometric properties can be difficult.

Trait and State Phenomena

One theoretical concept for gerontological nurses to consider when using direct observation is whether the behavior targeted is considered a trait or state phenomenon. Trait phenomenon assumes the behavior represents something stable about the individual. State phenomenon assumes the behavior represents an individual’s specific response to a particular situation. For example, an individual’s expression of pleasure may be reflective of the activity in which they are engaged, such as when a group of people are involved in gardening. Here, the assumption is that the behavior of individuals varies less between people and more between environments, and a measure of behavior should be sensitive to changes in surroundings. However, pleasure may be reflective of an individual’s typical personality style and approach to life. If behavior is considered stable, then it would be expected to vary more between people and be stable between settings and over time.

Choosing a Direct Observation Method

A tension exists when choosing direct observation methods. On one hand, professionals are searching for a gold standard by which samples, interventions, and programs can be evaluated and compared for quality improvement purposes. However, many researchers have chosen to construct their own measures that are specifically suited to the goals of their studies, rather than to use already available methods that have documented psychometric properties. Measurement of the specific behavior targeted by an intervention can increase the power to detect change and is more reflective of the results of individualized interventions. With the increasing emphasis on person-centered care, this tension will continue.

Clinicians and applied researchers strive to know what specific interventions produce specific changes in specific residents under specific circumstances. Because interventions focus on slightly different goals for modifying behavior, there is a lack of an agreed-on gold standard for measuring behaviors in evaluating interventions.

Basing the targeted behaviors to be modified on a theoretical model of dementia care would facilitate the creation of an accepted (or challenged) measure or method for direct observation that would provide a means for comparing the effects of multiple interventions and making meaningful conclusions about the behavior observed. This requires a well-defined target construct. According to Davis, Buckwalter, and Burgio (1997), the development of a direct observation measure requires identification of the target population, definition of the domain of the construct to be measured, explication of the purposes for which the measure will be used, and construction of items that adequately sample from the construct domain being measured. The authors concluded that observational measures of behavior problems have not sampled from all dimensions of the construct due to measures and behavior dimensions being adapted to specific stages of the dementia process.

This article reports on methods of direct observation of behaviors for use in long-term care settings, particularly with older adults who have dementia. It is hoped that this review will help gerontological nurses make informed choices about the direct observation measures that suit their specific needs, as well as facilitate a balance between identifying a gold standard and allowing flexibility to assess project-specific behaviors.

Review of Direct Observation Measures

A literature review was conducted of direct observation measures of individual behavior for use in assessing individuals with dementia. The literature from 1990 through 2007 in the AgeLine, PsycLIT, PubMed, and Sociological Abstracts databases was searched using the keywords dementia and observation, behavior observation, behavior assessment, behavior evaluation, observational measurement, observational assessment, quality of life assessment, affect observation, pain observation, and pain assessment. In addition, relevant references to observational measures cited in the literature obtained from these searches were gathered.

As mentioned above, methods of behavior observation for use with older adults with dementia are often developed specifically for the purposes of a particular project. A number of project-specific methods of observing behavior may be used to evaluate the effects of an intervention, such as interdisciplinary behavior management (Hughes & Medina-Walpole, 2000), music (Ragneskog, Kihlgren, Karlsson, & Norberg, 1996), psychosocial management (Doyle, Zapparoni, O’Connor, & Runci, 1997), behavioral activity (Swift, Harrigan, Cappelleri, Kramer, & Chandler, 2001), and domus philosophy of care (Dean, Briggs, & Lindesay, 1993; Lindesay, Briggs, Lawes, MacDonald, & Herzberg, 1991). These methods were developed for study-specific purposes, are not easily applicable for generalized use, and do not comprehensively assess methodological rigor. For the purposes of this review, observational measures that can be readily used in other studies, have been evaluated methodologically for use with individuals with dementia, and are in English were included. The measures are described briefly in this article, and their methodological rigor is documented in Tables 1 through 4.

Characteristics and Psychometric Properties of Direct Observation Measures of Individual BehaviorCharacteristics and Psychometric Properties of Direct Observation Measures of Individual BehaviorCharacteristics and Psychometric Properties of Direct Observation Measures of Individual BehaviorCharacteristics and Psychometric Properties of Direct Observation Measures of Individual BehaviorCharacteristics and Psychometric Properties of Direct Observation Measures of Individual BehaviorCharacteristics and Psychometric Properties of Direct Observation Measures of Individual Behavior

Table 1: Characteristics and Psychometric Properties of Direct Observation Measures of Individual Behavior

Characteristics and Psychometric Properties of Direct Observation Measures of Individuals’ Quality of Life (QOL)Characteristics and Psychometric Properties of Direct Observation Measures of Individuals’ Quality of Life (QOL)

Table 4: Characteristics and Psychometric Properties of Direct Observation Measures of Individuals’ Quality of Life (QOL)

To aid readers, definitions of the kinds of validity assessed are provided. Content validity recognizes the need to ensure that the entire domain of behaviors targeted is represented. Convergent validity confirms predicted relationships between the targeted measure and other methods and measures of related behaviors and constructs. It includes concurrent validity, which focuses on the relationship between two measures that are thought to measure the same behavior. Divergent validity is demonstrated when no relationship is found between measures thought to be unrelated. Construct validity is difficult to define because it is thought of as any process of validation of the theory underlying what is measured. This can be done by looking at the pattern of convergent and divergent validity in an attempt to evaluate whether the tool appears to be measuring the construct it claims to. Determinations of construct validity are often based on cluster and factor analysis, as well as the ability of the measure to discriminate between known groups or conditions in predicted ways.

Behavior observation measures are presented and discussed using the following domains: negative and positive behavior, negative and positive affect, functional behavior, and quality of life.

Negative and Positive Behavior

A number of measures of direct observation of individual behaviors are currently used in older adults who have dementia (Table 1). The Agitation Behavior Mapping Instrument (ABMI) is a measure of direct observation of agitated behaviors in the natural environment (Cohen-Mansfield, Werner, & Marx, 1989, 1992; Cohen-Mansfield & Libin, 2004). The tool includes the dimensions of aggressive, physically nonaggressive, and verbally agitated behaviors. Frequency and intensity of agitated behavior is recorded using either a checklist or laptop computer. Behavioral mapping is used to provide the environmental context and triggers within which to frame the behavior. Raters observe for 3 minutes at preselected times or events, and the authors recommend at least 23 observations per individual being assessed. Intensive training of observers is needed to achieve reliability. Good interobserver agreement was obtained with the measure, as well as convergent validity with the Cohen-Mansfield Agitation Inventory (CMAI) (Cohen-Mansfield, Marx, & Rosenthal, 1989).

The Agitated Behavior Scale (ABS) was originally developed to measure agitation through direct observation in individuals following traumatic brain injury (Corrigan, 1989). The measure was revised for use with individuals with dementia (Bogner, Corrigan, Bode, & Heinemann, 2000; Hall & Hare, 1997) on the basis of the assumption that agitation is the same phenomena across conditions. Raters record the presence or absence of attentional and mood difficulties, physical and verbal agitation, uncooperative and aggressive behavior, self-abuse, and positive affect and behaviors. Observations occur for 5 seconds of every minute for 10 to 20 minutes at preselected times. Training includes three sessions using videotapes to establish definitions of agitated and positive behaviors. Interrater reliability is well established, and internal consistency is demonstrated in the brain injury population. Item selection was based on item differentiation using ratings of individuals with traumatic brain injury, with no demonstration of content validity in individuals with dementia. Convergent validity was established in those with brain injury. Factor analyses demonstrate a consistent general construct and two latent variables of “directed” and “nondirected” agitation. Construct validity was demonstrated in the dementia population through sensitivity of positive behaviors to a video-respite intervention.

The Computer-Assisted Data Collection (Burgio et al., 1994; Holm, Rogers, Burgio, & McDowell, 1999; Pollock et al., 1997; Rogers et al., 1999) focuses on the effects of multiple environmental factors, including staff behavior, on resident behavior. A laptop computer is used to record the frequency and duration of disruptive vocalizations, duration of activities of daily living (ADLs), number of caregiver assists, requests for help, and variables identifying environmental context. Between 12 and 24 samples of behavior, 30 minutes in duration, are observed over a 2-week to 3-week period, distributed randomly from 8:00 a.m. to 8:00 p.m or structured to include morning, afternoon, and evening time periods. Intensive training and practice is required to obtain adequate reliability. Interobserver reliability was demonstrated, and convergent validity was established with cognitive functioning (i.e., Mini-Mental State Examination [MMSE]; Folstein, Folstein, & McHugh, 1975) and ADL functioning.

The Cohen-Mansfield Agitation Inventory (CMAI-R) revised the staff-reported CMAI to be used in real-time observation of a specific care event (Whall, 1999). The observation measure was developed to better understand and describe need-driven, dementia-compromised behaviors. Observations are videotaped, and raters record the frequency and duration of agitation. Observations of less than 30 minutes (7 to 10 minutes) during a care event are recommended to increase the chance of observing needdriven, dementia-compromised behaviors and avoid rater drift. Training entails the use of adult learning principles to rate videotapes and is accomplished in two to three brief sessions of less than 30 minutes each. Aides and experts are required to reach and maintain a 0.80 level of agreement. Interobserver reliability was demonstrated to be fair. Good convergent validity was established using staff-reported behavior and agitation ratings.

The Disruptive Behavior Scale (DBS) was developed to assess the intensity of observed disruptive behavior in geropsychiatric patients (Beck, Heithoff, et al., 1997; Beck et al., 1998). Disruptive behavior is defined in terms of its result, leading to negative consequences for the resident, caregiver, or another resident. Physically aggressive and nonaggressive, vocally aggressive, and agitated behaviors are included. The frequency of disruptive behaviors is documented each hour during 8-hour shifts over 2 to 7 consecutive days. Intensity of behaviors documented is determined by item weights. Training includes discussion and review of videotapes, 7 days of practice, and weekly retraining sessions. Reliability is monitored every other month by comparing the DBS forms with disruptive behaviors recorded in the medical record. Interrater reliability was good when data was log-transformed to improve agreement on high-frequency behaviors. Content validity was ensured. The combined frequency and intensity score better predicted nurses’ perceived disruptiveness than did the presence or absence of disruptive behaviors, frequency only, or intensity only. Convergent validity was established through a number of significant correlations with other measures. All four categories of disruptive behaviors were significantly intercorrelated.

The Empirical Behavioral Pathology in Alzheimer’s Disease Rating Scale (E-BEHAVE-AD) is an observational scale based on the caregiver report measure BEHAVE-AD (Auer, Monteiro, & Reisberg, 1996; Reisberg et al., 1987). The intensity of agitated behavior is recorded using a checklist that includes categories of paranoid/delusional ideation, hallucinations, activity disturbance, aggressivity, affective disturbance, and anxieties/phobias. Clinicians make observations during an informal dialogue in a 20-minute clinical interview. This is based on the theory that expressed emotion in the caregiver is a predictor of agitated behavior in individuals with dementia. Minimal training is needed because illustrative examples accompany the scale. Interrater reliability was good. Convergent validity was established with caregiver report (BEHAVE-AD; Reisberg et al., 1987). In general, caregivers reported more behavioral symptoms than the clinicians observed.

The Environment-Behavior Interaction Code (EBIC) (Morgan & Stewart, 1998; Stewart, Hiscock, Morgan, Murphy, & Yamamoto, 1999) is based on theories of personenvironment interaction and personal space and provides codes for ongoing interaction between the person and environment. Using a handheld computer, a nonparticipant observes and records in real time the frequency, intensity, and location of disruptive (including aversive, harmful, agitated), neutral, and positive behavior, according to topography (i.e., physical social, physical asocial, verbal, vocal, nonverbal) and a priori environmental influence (i.e., positive, neutral, negative). The initiator and recipient, degree of attentiveness, and presence of activity and restraints are also recorded. Observations are made during 4 to 18 periods (25 to 30 minutes in length) during a meal and when interactions are likely. Training is rigorous. Tests of interobserver reliability were good. Moderate test-retest reliability was demonstrated. Content and construct validity were also demonstrated.

The Engagement Scale targets the observation of engagement in older adults with dementia to evaluate the effects of an activities intervention (Judge, Camp, & Orsulic-Jeras, 2000). Engagement is targeted because its absence is the most frequently exhibited problem behavior and can lead to other behavioral problems. The duration of four main categories of engagement are recorded: constructive, passive, non-engagement, and self-engagement. Observations are made for 10 minutes, twice in the morning and twice in the afternoon, during regularly scheduled activities. Good interrater agreement was obtained. The construct was validated, as the scale discriminated between baseline and posttest results of those in a Montessori treatment group and between observations made during a Montessori group and a general activity programming group, and higher constructive engagement was demonstrated during a Montessori activity than in general program activities.

McCann, Gilley, Hebert, Beckett, and Evans (1997) constructed a direct observation measure of behavior to compare with staff-reported behavior in long-term care facilities. Observers record the frequency of the resident’s level of alertness; facial affect; ratings of aggressive, injurious, and disruptive behaviors; and the context surrounding these behaviors. Each resident is observed for 5 consecutive minutes during five randomly chosen times, with each minute considered a discrete interval for recording presence or absence of the behaviors. Training includes didactic sessions and practice observations until good interrater reliability is achieved. The range of behaviors targeted for measurement were developed from clinical and empirical data on psychiatric inpatients and outpatient clinical evaluations of individuals with Alzheimer’s disease, and were revised based on direct observations of nursing home residents with dementia. Convergent validity was established with staff-rated behaviors. Direct observation was demonstrated to be a more sensitive measure of behaviors than are staff ratings for all behaviors except cursing and physical aggression, which staff report at a higher rate, likely because of their effects on staff.

The Natural Observer was created to evaluate the effects of intergenerational programming with nursing home residents who have dementia (Ward, Los Kamp, & Newman, 1996). The system of observation targets positive observable behavior, such as eye contact, smiling, touching, rhythmic hand movement, verbal expression, hand extension, head nodding, and laughing. Raters record the presence of behaviors on a checklist only once during an observation. Observations are made for 30 seconds, either videotaped or recorded by hand, and are conducted every other week for 6 weeks. Some training is required but is not described. Interrater reliability is good. The ratings made in real time and using videotape recordings demonstrated similar effects of intergenerational programming but were not formally compared. Trends were found toward more observed positive behaviors during activities with children, but differences were not significant, other than less head nodding in the presence of children. The authors concluded that the measure needed to include behaviors indicating attentiveness.

The Observational Scale was developed to measure the full range of problematic behaviors in individuals with dementia (Ward, Murphy, Procter, & Weinman, 1992). This scale records the presence of attention to environment, posture, mobility, communication, informal recreation, personal care, eating, aggression, repetitive activity, inappropriate activity, and mood extremes. Observations are made during 10 periods (14 minutes in length) throughout the week during preselected and randomly selected times, alternating 10 seconds of observation and 10 seconds of recording. Training entails agreement of category definitions and criteria when observing videotaped behavior. Agreement between observers was good. Low frequency categories, such as physical aggression and lying down, demonstrated poor agreement. Videotaped recordings of patients on the wards were used to derive the categories of behavior, ensuring content validity. Convergent validity was not established for the full range of behaviors observed, but observations combined to produce a feeding score were associated with physical deterioration, staff-reported feeding score, and physical disability score.

The Observer computer software system is used to manage the data obtained from recording streams of behaviors on a handheld event recorder (Van Haitsma, Lawton, Kleban, Klapper, & Corn, 1997) to study the ecology of older adults with dementia. ADLs, pathological and social behaviors, and affective states (described under Apparent Affect Rating Scale) are directly observed in real time; their frequency and duration are recorded continuously using a handheld recorder and are identified within social context and physical location. Sixteen observations are made in 10-minute periods over 3 weeks, during early and late morning, late afternoon, and evening, avoiding mealtime and morning and evening care. Training involves monitoring reliability and honing criteria after discussion of discrepancies for 40 to 50 observations. Interobserver agreement was good. Convergent validity was demonstrated with cognitive functioning (Mattis Dementia Rating Scale; Mattis, 1988), Multi-Observational Scale for Elderly Subjects (MOSES; Helmes, Csapo, & Short, 1987) withdrawal and depression scales, activity participation ratings, and staff-rated physical agitation (CMAI; Cohen-Mansfield, Marx, et al., 1989), but not with verbal agitation and aggression. Staff report of depression was negatively related to observed participation in enriching activities but not social interaction. The nurse aide ratings and activity therapist ratings of activity participation differed in terms of convergence with active gazing (aides’ ratings not related) and group activity (therapists’ ratings not related).

The Overt Agitation Severity Scale (OASS) was developed as a rating scale of agitation that would reduce the overlap between agitated behavior and symptoms of anxiety, mood, and other disorders (Yudofsky, Kopecky, Kunik, Silver, & Endicott, 1997). The frequency and intensity of body movements and vocalizations are recorded. Observations are made for 15 minutes from a distance of 20 or more feet at four preselected times of agitation and nonagitation. Interobserver reliability is good, and internal consistency was demonstrated across times and raters. Content validity was established. There was strong convergence with the Pittsburg Agitation Scale (Rosen et al., 1994). Divergent validity was established through low but significant correlations with staff-rated aggression.

The Pittsburg Agitation Scale (PAS) was developed as a concise, easy-to-use rating scale of agitation that can be completed by clinical staff on the basis of direct observation (Rosen et al., 1994). Observers rate aberrant vocalizations, motor agitation, aggressiveness, and resting care on a scale of 0 to 4, according to intensity, disruptiveness within the environment, and ease with which the behavior can be redirected. The scale does not describe specific behaviors or provide a complete profile of behavior problems. Scores reflect the most aberrant behavior during the period of observation, which is typically the target of clinical intervention. The intensity score is specific to the environment, meaning a behavior may be scored differently based on the environment in which it was observed. Two to five people are rated simultaneously for 1 to 8 hours. For the study, observations were made during three 8-hour shifts per day over 7 days. The PAS takes less than 1 minute to complete and can be administered by clinical staff with no training other than reading brief instructions. Interrater reliability was obtained. Convergent validity was established with real-time, computer-recorded behaviors of aberrant vocalizations, motor agitation, and aggressiveness in a restricted sample of individuals with dementia and behavior disturbances. Construct validity was demonstrated through its sensitivity to behaviors that lead to physical restraint or as-needed medication.

The Patient Behaviour Observation Instrument (PBOI) (Bowie & Mountain, 1993, 1997) was aimed at improving the categorization and measurement of behavior in individuals with dementia. Using a handheld computer, observers record the duration of ADLs; participation in/reception of care; social engagement; abnormal motor activity; and antisocial, inappropriate, and neutral behavior. Each individual is observed for 5 minutes every hour between 8:00 a.m. and 9:00 p.m. An unspecified but “extensive” amount of practice is needed for familiarization. Interobserver reliability and content validity were established. Construct validity was established through its sensitivity in distinguishing between wards rated on environmental quality.

The Resistiveness to Care Scale in individuals with dementia of the Alzheimer’s type (RTC-DAT) was created to evaluate behavioral approaches for reducing resistiveness to care, the most frequent disruptive behavior evoked in those with dementia by caregiving encounters (Mahoney et al, 1999). The RTC-DAT is a context-sensitive instrument measuring rapid changes in individual behavior. It records the duration and intensity of each behavior and is completed by someone who is not a direct caregiver. Observations are typically made during the first 5 minutes of 4 to 12 instances of care provision; 5 minutes being the average duration of the shortest ADL: toilet use. Observers received intensive training until percentage agreement exceeded 95%. Interrater reliability and internal consistency were established. Content validity was ensured a priori. Convergent validity was established with the Discomfort Scale for individuals with dementia of the Alzheimer’s type (DS-DAT; Hurley, Volicer, Hanrahan, Houde, & Volicer, 1992), observer analogue rating of resistiveness, speech assessment, nurse-rated dementia severity, and the Katz Index of ADLs (Katz, Ford, Moskowitz, Jackson, & Jaffee, 1963), but not with cognitive functioning (MMSE; Folstein et al., 1975). Three factors were found, but the authors recommended using one score, as the factors do not have satisfactory reliability to be used as subscales.

The Scale for Observation of Agitation in Persons with Dementia of the Alzheimer’s type (SOAPD) is a measure of observed agitation designed to be uncomplicated to administer (Camberg et al., 1999; Hurley et al., 1999). The SOAPD targets behaviors that reflect an unpleasant state of excitement and communication of personal discomfort, are unrelated to known physical needs, and remain after reducing internal and external stimuli, making a distinction between agitation and the evoked behavior of resistiveness. The duration of agitated body movements and vocalizations is recorded by hand, and intensity is rated for two items. Observations are made during four 5-minute rating periods at 20-minute intervals over 3 or more hours at preselected times, such as after morning care, before lunch, after lunch, and before dinner for 3 days. The SOAPD requires intensive manualized training and practice observations. Interobserver agreement was adequate for the total score and ranged from fair to good for the seven items. Internal consistency was adequate. Content validity was ensured, and convergent validity was established with staff analogue rating of agitation and the CMAI (Cohen-Mansfield, Marx, et al., 1989). Two factors emerged: physical and verbal agitation. However, the scale was not sensitive to kind of activity intervention.

Smallwood, Irvine, Coulter, and Connery (2001) developed a Short Observational Tool (SOT) to assess five behavior dimensions of individuals with dementia, including neutral behavior, motor behavior, self-care, external behavior, and inappropriate behavior. Fifteen-minute videotaped observations are sampled twice during preselected times of the day, and behavior is recorded every 30 seconds. The tool demonstrated good interrater reliability. Divergent validity between three of the five behavioral dimensions of the tool was established, as 8 of the 10 correlations between dimensions were not significant, and motor and inappropriate behavior were the only dimensions positively correlated.

Timewand observation (Bridges-Parlet, Knopman, & Thompson, 1994) is a method that focuses on assessment of physically aggressive behavior, distinguishing it from agitation in terms of its unique therapeutic and safety issues and separate treatment indicators. A handheld bar code reader is used to record the frequency and sequential order of behaviors and environmental characteristics. The frequency of aggressive and other idiosyncratic (person-specific) behaviors, ADLs, and the environmental context in which the behaviors occur are recorded every minute as present or absent. Observations are typically made during four 2-hour intervals within a 1-week period during times when aggressive behavior typically did and did not occur. Tests of interobserver reliability were adequate to good. Convergent validity was loosely established with behaviors recorded in a staff diary.

Negative and Positive Affect

The direct observation method has also been devoted to measuring a specific aspect of behavior—the nonverbal communication of affect. Traditionally, self-report is used to assess emotional experience, but this is difficult to undertake in individuals with severe cognitive impairment who do not reliably report internal states verbally. The direct observation of nonverbal behaviors and facial expressions can be used to measure emotion in these individuals (Table 2).

Characteristics and Psychometric Properties of Direct Observation Measures of Individuals’ AffectCharacteristics and Psychometric Properties of Direct Observation Measures of Individuals’ AffectCharacteristics and Psychometric Properties of Direct Observation Measures of Individuals’ AffectCharacteristics and Psychometric Properties of Direct Observation Measures of Individuals’ Affect

Table 2: Characteristics and Psychometric Properties of Direct Observation Measures of Individuals’ Affect

The Apparent Affect Rating Scale (AARS) was constructed to assess the full range of affect in individuals with dementia (Lawton, Van Haitsma, & Klapper, 1996; Lawton, Van Haitsma, Perkinson, & Ruckdeschel, 1999). Observers record the duration of positive and negative emotional states, targeting facial expressions, body language, and nonword vocalizations. Ratings are made using either a checklist or a handheld event recorder. The authors recommend 9 to 19 observations (5 minutes in length) over 2 or more days. Specified times or events are chosen for observation on the basis of the specific aims for its use. One month of training is required, although less training (i.e., 30 minutes plus nine practice observations) can be provided to certified nurse assistants (CNAs) using the checklist format. Interobserver reliability was good, although lower between research assistants and CNAs. Convergent and divergent validity were demonstrated through significance in 13 of the 15 hypothesized relationships for negative affect, 9 of the 13 hypothesized relationships for positive affect; of the relationships hypothesized not to differ from zero, none of the 15 involving negative affect and only 2 of the 20 involving positive affect were significantly related. Construct validity was established by demonstrating significant differences in observation of affect states in various settings, and some support was demonstrated for the two-factor structure of affect without the inclusion of anger.

The Apparent Emotion Rating Scale (AERS) was constructed to assess positive and negative emotional states in individuals with cognitive impairment (Snyder et al., 1998). It is similar in conceptualization of emotion to the AARS but differs in format and scoring. Observations are made in one 5-minute to 10-minute observation during a predetermined time of day. A small amount of training is needed. Interrater reliability estimates were demonstrated for all emotions except anxiety; test-retest reliability was adequate, but internal consistency was lacking. Convergent validity was demonstrated with depression and morale. Construct validity was established through patterns of correlations among the emotions rated and negative relationship with cognitive impairment.

The Checklist of Nonverbal Pain Indicators (CNPI) provides a user-friendly way of documenting nonverbal behavioral pain indicators. It is based on the University of Alabama-Birmingham (UAB) Pain Behavior Scale (Richards, Nepomuceno, Riles, & Suer, 1982) and modified for nonverbal older adults with cognitive impairment, eliminating items that require ambulation and standing (Feldt, 2000; Nygaard & Jarland, 2006). Training includes instructions from nursing staff, which are not detailed. Observations are made during a period of rest and transfer over a 2-day period, although the period of rest was dropped due to infrequent pain behaviors. Interrater reliability and test-retest reliability was established. Content validity was ensured through an extensive literature review of existing pain scales that focus on pain behaviors typical in older adults with dementia, including vocalizations, facial grimacing, bracing, rubbing, restlessness, and vocal complaints. CNPI scores at rest and during movement were correlated with a self-report pain scale, and scores during movement were correlated with staff analogue rating of agitation, but its relationship with an interview-based assessment of pain was not reported.

The Discomfort Scale for persons with Dementia of the Alzheimer’s Type (DS-DAT), also known as the Hospice Approach to Discomfort Scale or the “Hurley Scale” (Hurley et al., 1992; Krulewitch et al., 2000; Volicer et al., 1994), documents the intensity of the presence of discomfort during direct observation, resulting in a numerical score of 0 to 27. The scale has been adapted to help identify when to begin the Assessment of Discomfort in Dementia protocol (Kovach, Weissman, Griffie, Matson, & Muchka, 1999), which is a systematic tool to help nurses make a differential assessment and develop a treatment plan for both physical pain and affective discomfort in individuals with dementia. Those administering the DS-DAT memorize the items during rigorous training sessions. Training for use of the entire protocol includes skills training and a practice period of 1 to 2 months. Observations are made for 5-minute intervals during a specific time or event that might cause discomfort. Good interrater reliability and internal consistency were documented. Content validity was ensured. Convergent validity was established with disease progression, physical disability, speech assessment, and cognitive functioning. Construct validity was demonstrated through the scale’s ability to distinguish between a fever state and health and through decline in discomfort following the use of the treatment protocol. However, the raters were not blind, as they were the same nurses who administered the protocol.

The Feeling-Tone Questionnaire (FTQ) was designed to detect symptoms of depression among older adults with communication impairment, specifically those with severe dementia, and assesses a broad range of feelings, including positive, neutral, and negative (Toner, Teresi, Gurland, & Tirumalasetti, 1999). The developers took a multisource approach; information is obtained through self-report and direct observation. The FTQ is administered during any 5-minute to 10-minute period, and display of emotion and interaction style is observed and rated according to intensity. Interobserver reliability and test-retest reliability were adequate, and internal consistency was good. Content validity was established. Convergent validity was demonstrated with an observation measure of affective behavior designed as a validation measure specifically for the study, and with physicians’ ratings of depressive symptoms.

The Maximally Discriminative Facial Movement Coding System (MAX) uses computerized technology to record affect (Magai, Cohen, Culver, Gomberg, & Malatesta, 1997; Magai, Cohen, Gomberg, Malatesta, & Culver, 1996). This system uses an anatomically and theoretically based coding scheme of facial movement, recorded on a continuing, realtime basis. The duration and frequency of the targeted affective states of positive and negative affect are recorded using a laptop computer. It is notable that anxiety/fear is not assessed. Two observers are needed: one to code the brow/eye region and the other to code the mouth region. Observations are made during a 20-minute family visit. Training with practice videotapes is required until the two raters achieve a kappa of at least 0.80. Good interrater reliability was demonstrated for each emotion. Convergent validity was established with family ratings of interest, joy, and anger; caregiver ratings of interest and joy; between observations of joy and premorbid attachment security; and between observations of sadness and premorbid hostility. However, the predicted relationship between premorbid hostility and expressions of anger, contempt and disgust were not confirmed. The communication value of facial expressions was validated, as no significant differences were found across cognitive stages of functioning for all emotions, except a significant reduction of joy in the last stage of cognitive decline.

The Modified Behavior Rating Scale (MBRS) was developed to directly observe several classes of emotional responses in individuals with severe dementia (Spaull, Leach, & Frampton, 1998). Raters record the amount of passive contentment, attending to the environment, happy mood, interest level, time spent showing interest, bodily movement, and interaction with others on the basis of video recordings. Thirty-second observations are typically made, following 30 seconds of recording, during a 10-mintue to 20-minute period. Significant interrater reliability was obtained for all responses. The MBRS was sensitive to changes in behavior before, during, and after a Snoezelen intervention (Chung & Lai, 2002). In contrast, comparisons of Dementia Care Mapping (DCM) (Bredin, Kitwood, & Wattis, 1995) scores before and after the intervention did not demonstrate a significant change in well-being.

The Non-communicative Patient’s Pain Assessment Instrument (NOPPAIN) (Horgas, Nichols, Schapson, & Vietes, 2007; Snow et al., 2004) is an observational measure of pain designed to be brief, to require little training, and to be administered by a nursing assistant. It includes intensity ratings of pain words, pain facial expressions, pain noises, bracing, rubbing, and restlessness. Pain is observed during a 10-minute period of rest and movement. Training includes six videotapes and practice observations to achieve rater agreement. Interrater reliability and test-retest reliability were established. Content validity was ensured. Convergent validity was established with other pain behavior ratings, and with self-reported pain only in cognitively intact individuals, not in cognitively impaired individuals. Construct validity was demonstrated through discrimination of videotapes demonstrating different levels of pain as determined by experts.

The Pain Assessment in Advanced Dementia (PAINAD) scale was developed as a clinically relevant and easy-to-use pain assessment tool for individuals with advanced dementia (Lane et al., 2003; Leong, Chong, & Gibson, 2006; Warden, Hurley, & Volicer, 2003). The scale rates five pain affect and behavior states on an intensity scale of 0 to 2, and it was later transposed to a categorical scale on the basis of its relationship with other pain measures. A 2-hour training curriculum for licensed practical nurses and nursing assistants is used. Observations are made on three preselected occasions during 5-minute intervals (no activity, pleasant activity, and caregiving activity). Interrater reliability was demonstrated using nonsignificant t tests to compare mean ratings among raters. Cronbach’s alpha coefficient was low, demonstrating poor internal consistency, which is not unusual for a 5-item scale. Content validity was ensured. Convergent validity was documented with the DS-DAT, visual analogue scales, and nurse ratings of pain. The PAINAD was able to discriminate between no activity, pleasant activity, and unpleasant activity, and demonstrated decreases in pain before and after receiving pain medication. Factor analysis confirmed that the PAINAD measures one single construct.

The Positive Response Schedule (PRS) for Severe Dementia was developed as an observational measure aimed at documenting affective well-being in individuals with severe dementia (Perrin, 1997). It is based on engagement theory (Godlove, Richard, & Rodwell, 1982; McFadyen, 1984) and Ekman’s theory of emotional expression (Ekman, Levenson, & Friesen, 1983). The presence of behavior is assessed by documenting only the first occurrence in a given observation. The percentage of observations in which a behavior was present is calculated. Observations are conducted in vivo or are videorecorded for a period of approximately 20 to 60 minutes, based on the length or hypothesized effect of the intervention to be evaluated. Observations are made during 20 to 30 seconds and are followed by 10-second to 30-second recording periods. Interobserver reliability was initially poor but improved after modifications. Convergent validity was not addressed, as no known psychological indexes thought to measure engagement existed at the time. Content validity is understood in the context of engagement theory and nonverbal expression. The PRS was found to be sensitive to changes following a Snoezelen intervention but was not statistically examined.

The Pain Assessment for the Dementing Elderly (PADE) is an observation measure of pain in individuals with advanced dementia (Villanueva, Smith, Erickson, Lee, & Singer, 2003). It is based on the assumption that nonverbal behavior can indicate the presence of pain that interferes with daily activities and can be reliably observed. The measure consists of 24 items rated on a Likert scale to assess physical signs, functional abilities, and caregiver judgment. Care providers require minimal training. Observations are made for 5 to 10 minutes during unspecified times or events over 10 nonconsecutive days. The tool demonstrated good internal consistency and interrater reliability and acceptable test-retest reliability. Content validity was established. Convergent validity was established for two of the three subscales of the CMAI, and construct validity was demonstrated through the measure’s ability to discriminate between individuals taking psychoactive medications and those not.

The purpose of the UAB Pain Behavior Scale is to offer a reliable observational pain scale designed for professionals to use on rounds and to be sensitive to small change (Richards et al., 1982). It was not developed for use with older adults but has been identified as relevant and potentially useful. According to the authors, “minimal” training is needed for accurate results. Raters record the frequency and/or intensity of pain behaviors. Observations are conducted for 5 minutes during periods of activity. The measure has good interrater and test-retest reliability. Content validity was drawn from clinical descriptions shared among pain centers, but the tool is based on the most salient items determined by the authors. The tool demonstrated good convergent validity with other pain measures at discharge from a pain treatment program but not before. Divergence from measures of well behavior was established.

Functional Behavior

ADLs are often measured using either self-report or caregiver report. Directly observing an individual’s ability to perform ADLs is beneficial to determine the length of time spent on tasks and to limit caregiver bias by directly assessing proficiency. Occupational therapists have a number of assessment measures that systematically assess ADLs by observing functioning using standardized instructions and tasks. This review will not include these measures but will focus on observation measures used to document functioning observed in the natural environment (Table 3).

Characteristics and Psychometric Properties of Direct Observation Measures of Individuals’ Activities of Daily Living (ADLs)

Table 3: Characteristics and Psychometric Properties of Direct Observation Measures of Individuals’ Activities of Daily Living (ADLs)

The Beck Dressing Performance Scale (BDPS) is used in Strategies to Promote Independence in Dressing (SPID), which focuses on eliminating excess disability by supporting cognitively impaired individuals’ retained ability to dress (Beck, 1988; Beck, Heacock, et al., 1997; Vogelpohl, Beck, Heacock, & Mercer, 1996). The BDPS was designed to address the limitations caused by cognitive deficits related to dressing and the assistance most often needed by cognitively impaired individuals. The BDPS aims to rate the amount of support and time needed for each dressing component. Observations of dressing are videotaped for 4 to 5 minutes twice per week, for a total of 4 to 12 observations. Raters were trained until interobserver agreement was at or above r = 0.80. Content validity of the multiple steps involved and the hierarchy of kinds of assistance needed was determined through task analysis. No significant relationship was found with overall cognitive functioning (MMSE; Folstein et al., 1975; or Neurobehavioral Cognitive Status Examination [NCSE]; Kiernan, Mueller, Langston, & Van Dyke, 1987), but a significant negative relationship to language functioning (NCSE language) was found. The BDPS documented improvement in independence with the SPID intervention and a slight but significant increase in dressing time (of 1 minute), validating the construct underlying the measure.

The InfoAid system (Holmes, Teresi, Lindeman, & Glandon, 1997; Holmes, Teresi, Ramirez, & Goldman, 1997) was developed to monitor the amount of staff time spent on ADLs with residents with dementia. The system is designed to continuously record direct and indirect care services provided, by whom, for whom, and the duration by scanning bar coded sheets with a bar code reader or data wand. Caring for multiple residents simultaneously and multiple-person assists are accounted for. Bar code sweeping becomes part of the service-providing act and is thought to be less prone to the sampling biases associated with other techniques. Traditional assessment of reliability is difficult because the behaviors monitored are not expected to be consistent or stable. Assessment of interobserver reliability is needed but is complicated with considerations of invasion of privacy. Observation of personal care was significantly related to staff report of ADLs, but observation of mobility assistance, psychological support, medication administration, training, and assessment/planning was unrelated to ADL staff report. Duration of personal care was not related to age, gender, continence, speech, behavior, or mood. InfoAid was able to detect more provision of personal care in special care units than in general long-term care units.

The Observational List for early signs of Dementia (OLD) is used by general practitioners to assess early signs of dementia in the areas of forgetting, repeating, language, understanding, orientation to time and place, daily living, social withdrawing, confabulation, and dependence (Hopman-Rock, Tak, & Staats, 2001). The measure is completed by a general practitioner after observation of patients. Notably, interrater reliability was not assessed. Internal consistency was demonstrated. An expert panel created the list of behaviors observed, ensuring content validity. Convergent validity was established with a cognitive screen and informant rating scale and a verbal memory scale but was not found with informant ratings of ADLs and instrumental ADLs. Divergent validity was established through no relationship with Geriatric Depression Scale (GDS) (Yesavage et al., 1982–1983) scores, as predicted, because individuals with depression were excluded from the study. Construct validity was established through a factor analysis, yielding two factors. Incremental validity was established as the OLD identified patients who exhibited signs that were missed and those with no signs that were falsely identified based on the general practitioner’s opinion.

The Self-Feeding Assessment Tool (SFAT) was developed to distinguish small differences in function during the feeding process and differentiate ability to initiate action from ability to respond to command (Osborn & Marshall, 1993). Kind of assistance is ranked on a 5-point scale, including unassisted, verbal prompt, nonverbal prompt, physical guiding, and full assistance. Individuals are observed for an undetermined length of time both at the beginning and the end of two meals because level of assistance often changed during the course of a meal. Interrater reliability of a videotaped simulation demonstrated perfect agreement. Content validity was supported and compared with more detailed, standard ADL scales. Convergent validity was demonstrated between cognitive impairment and both feeding ability and performance. Construct validity was demonstrated by the ability of the SFAT to discriminate between self-feeding capability and actual performance in drinking liquids and eating pureed food. In addition, those who had been in the nursing home the longest had the lowest mealtime performance.

Quality of Life

Most of those who are interested in assessing quality of life (QOL) in individuals with dementia choose to combine an observational measure with caregiver report measures, interviews, chart data, and self-report when possible. Few have attempted to standardize such attempts at assessing as broad a construct as QOL using observational measures (Table 4). A number of measures focus on the overall milieu in an attempt to assess QOL, and these measures will be included in a future review article on the topic.

The Observing Quality of Life in Dementia (OQOL-D) scale was designed as a useful and practical measure of resident QOL to be used in a partnership between researchers and service providers to inform dementia care practice by evaluating the effects of interventions, tracking progress, and evaluating care plans (Edelman, 2005; Edelman & Fulton, 2004). Nurses, CNAs, and activity staff act as staff participants in the environment and observe residents over 4 to 5 consecutive days during six to eight selected times or events and rate their experience on a 7-point scale, ranging from extremely pleasant to extremely unpleasant. The length of the observation is not standard and can be adjusted to fit the goal of the project. Training is provided to reach a minimum of 80% rater agreement, which is reassessed every 6 months. Convergent validity has been demonstrated with DCM (Bredin et al., 1995), staff and resident Quality of Life in Alzheimer’s Disease (QOL-AD; Logsdon, Gibbons, McCurry, & Terri, 2000, 2002) ratings, depression, cognitive functioning, and dependent ADLs. The staff provided ongoing input and feedback about the ease of use of the instrument and its relevance to dementia-related quality of life, indicating content validity. The ratings are understood by service provider staff in relation to their practical and personal knowledge of the residents, and they respond appropriately.

The Psychological Well-Being in Cognitively Impaired Persons (PWB-CIP) scale contains 11 items that measure positive and negative affect states and engagement behaviors (Burgener & Twigg, 2002; Burgener, Twigg, & Popovich, 2005). It is a relatively narrow measure of QOL, focusing only on affect and behavior. The primary caregiver is the observer and completes the scale for 5 to 10 minutes after observing the individual over 24 hours. Internal consistency was good at both baseline and follow up. Stability in scores was demonstrated. Content validity was ensured through use of Lawton’s (1997) theoretical model of QOL, literature review, and national expert input, and was demonstrated with a content validity index. Convergent validity was demonstrated with measures of cognition, depression and personal distress, quality of relationship with caregiver, pleasant activities, productive behaviors, and personality at baseline and 18-month follow up. The construct of well-being as a distinct concept was verified by its stability in contrast with increased depression and functional disability over time. Factor analysis revealed two constructs: positive and negative well-being. Internal consistency was demonstrated for both subscales.

The QUALIDEM is a dementia-specific observational assessment of QOL (Ettema, Droes, de Lange, Mellenbergh, & Ribbe, 2007a, 2007b) that was developed from the authors’ experience of 40% of nursing home residents being unable to answer self-report questions. The measure is based on an adaptation-coping model and evaluates nine subscales of QOL: care relationship, positive affect, negative affect, restless tense behavior, self-image, social relations, social isolation, feeling at home, and having something to do. CNAs score the QUALIDEM for 15 minutes after a 2-week observational period. Training includes lecture and written instructions. Good internal consistency, interrater reliability, and test-retest reliability were established. Content validity was achieved. Construct validity was demonstrated through Mokken scale analysis to demonstrate unidimensionality of subscales, and moderate relationships between subscales support the concept of different but related QOL domains, with the exception of a negative relationship between “feeling at home” and “having something to do.” Convergent and divergent validity were established with predicted significant negative relationships between selected QUALIDEM subscales and nonsignificant relationships between other selected subscales and a behavioral rating scale, behavioral and psychological symptom questionnaire, and a depression rating (Cornell Scale for Depression in Dementia; Alexopoulos, Abrams, Young, & Shamoian, 1988). Positive relationships were demonstrated with nurse-rated QOL for most subscales and with family-rated QOL for some subscales, but the QUALIDEM did not correlate well with selfreport of QOL other than positive affect.

Discussion

Direct observation of behavior provides a useful assessment of individual functioning and experience in individuals who are unable to report this verbally due to deficits in memory and language. In fact, this method is often considered the gold standard, more objective than caregiver report, when evaluating the functioning of older adults at mid to late stages of dementia. In using such methods, a number of things should be considered, including how often and when to observe, the behavior or range of behaviors to target, and the psychometric properties desired, depending on the assumptions held about the behavior observed (e.g., stable trait, sensitive to environmental change).

Direct observation is more costly and time consuming than are self-report, interview, and caregiver report methods. The training needed by raters to reliably record direct observation of behaviors is typically more extensive, especially when using computerized methods and when recording the frequency, duration, or context of behaviors (versus recording the presence or absence of behaviors). Use of direct observation also requires a method for monitoring raters. It is difficult to blind observers as they interact with the environment as intelligent participants; therefore, ensuring continued adherence to rating directions and to avoid rater shift and other biases is vital. Finally, the actual use of direct observation is often more labor intensive than are other report measures or interviews. Such costly methods are best used to investigate research questions around significant, potentially modifiable health outcomes. Investigating ways of using caregivers as participant observers or new technologies that can passively gather data using motion sensors should be considered further, as these methods may be less stressful and intrusive for older adults with dementia, may reduce reactivity, and may be less costly than using independent raters.

It is especially important that quality standards and measures exist for individuals with dementia, more than for any other client group, because they have less ability to voice their concerns due to memory and language difficulties, are part of a cohort less likely to challenge authority, and are subject to ageism. All of the measures described in this article have some information to offer that would be beneficial for use in quality improvement efforts in dementia care. However, some measures yield themselves for use in quality improvement more easily than others. Observation measures that examine specific areas of behavior, such as agitation or ADL functioning, would be useful if they represent the only behavior targeted for change. Future research is needed to identify a clinically meaningful comparison for interpretation of the measurement of such specific behaviors, either through documenting annual rate of change or clinically meaningful cut-off scores that provide useful distinctions on quality of care. Similarly, scales that focus on a range of individual behavior and experience capture quality more comprehensively and would also be highly useful. Measures aimed at observing interactions or the milieu as a whole, rather than individual behavior (to be covered in a separate review), also lend themselves to natural identification of interventions and may be matched to different programmatic goals.

Most computerized methods of recording behavior are fairly wieldy and time consuming and would be infeasible for caregivers to use because of the rigorous training needed. However, observing and recording behaviors in real time allows for linking antecedents and consequences of behavior, which would be clinically meaningful and could be used to identify interventions for quality improvement. Bar code readers and data wands were designed to be used by staff to document services provided by formal caregivers and have potential for use in documenting other quality indicators, as well as the context of individual resident behavior. In addition, the development of new sensor technology that can passively record resident behavior offers a promising way to more easily document behavior within its context in real time.

Summary

Ideally, quality improvement measures should encompass several perspectives using a number of methods to obtain the most valid view of quality of care and quality of life. This would include a combination of direct observation, self-report of individuals with dementia, and reports of their families and formal caregivers. In reality, limited time and resources often do not allow for multiple assessments, and direct observation measures are often the first method dropped from consideration. Hopefully, this review has demonstrated how the use of direct observation measures can contribute substantially to the understanding of the experience of older adults with dementia and to the identification of interventions for quality improvement.

References

  • Alexopoulos, GS, Abrams, RC, Young, RC & Shamoian, CA1988. Cornell Scale for Depression in Dementia. Biological Psychiatry, 23, 271–284. doi:10.1016/0006-3223(88)90038-8 [CrossRef]
  • Auer, SR, Monteiro, IM & Reisberg, B1996. The Empirical Behavioral Pathology in Alzheimer’s Disease (E-BEHAVE-AD) Rating Scale. International Psychogeriatrics, 8, 247–266. doi:10.1017/S1041610296002621 [CrossRef]
  • Beck, C1988. Measurement of dressing performance in persons with dementia. The American Journal of Alzheimer’s Care and Related Disorders & Research, 33, 21–25.
  • Beck, C, Frank, L, Chumbler, NR, O’Sullivan, P, Vogelpohl, TS & Rasin, J et al. 1998. Correlates of disruptive behavior in severely cognitively impaired nursing home residents. The Gerontologist, 38, 189–198.
  • Beck, C, Heacock, P, Mercer, SO, Walls, RC, Rapp, CG & Vogelpohl, TS1997. Improving dressing behavior in cognitively impaired nursing home residents. Nursing Research, 46, 126–132. doi:10.1097/00006199-199705000-00002 [CrossRef]
  • Beck, C, Heithoff, K, Baldwin, B, Cuffel, B, O’Sullivan, P & Chumbler, NR1997. Assessing disruptive behavior in older adults: The Disruptive Behavior Scale. Aging & Mental Health, 11, 71–79. doi:10.1080/13607869757407 [CrossRef]
  • Bogner, JA, Corrigan, JD, Bode, RK & Heinemann, AW2000. Rating scale analysis of the Agitated Behavior Scale. Journal of Head Trauma Rehabilitation, 15, 656–669.
  • Bowie, P & Mountain, G1993. Using direct observation to record the behaviour of long-stay patients with dementia. International Journal of Geriatric Psychiatry, 8, 857–864. doi:10.1002/gps.930081009 [CrossRef]
  • Bowie, P & Mountain, G1997. The relationship between patient behavior and environmental quality for the dementing. International Journal of Geriatric Psychiatry, 12, 718–723. doi:10.1002/(SICI)1099-1166(199707)12:7<718::AID-GPS608>3.0.CO;2-3 [CrossRef]
  • Bredin, K, Kitwood, T & Wattis, J1995. Decline in quality of life for patients with severe dementia following a ward merger. International Journal of Geriatric Psychiatry, 10, 967–973. doi:10.1002/gps.930101109 [CrossRef]
  • Bridges-Parlet, S, Knopman, D & Thompson, T1994. A descriptive study of physically aggressive behavior in dementia by direct observation. Journal of the American Geriatrics Society, 42, 192–197.
  • Burgener, S & Twigg, P2002. Relationships among caregiver factors and quality of life in care recipients with irreversible dementia. Alzheimer Disease and Associated Disorders, 16, 88–102. doi:10.1097/00002093-200204000-00006 [CrossRef]
  • Burgener, SC, Twigg, P & Popovich, A2005. Measuring psychological well-being in cognitively impaired persons. Dementia, 4, 463–485. doi:10.1177/1471301205058303 [CrossRef]
  • Burgio, LD, Scilley, K, Hardin, JM, Janosky, J, Bonino, P & Slater, SC et al. 1994. Studying disruptive vocalization and contextual factors in the nursing home using computer-assisted realtime observation. Journals of Gerontology. Series B, Psychological Sciences and Social Sciences, 49, P230–P239.
  • Camberg, L, Woods, P, Ooi, WL, Hurley, A, Volicer, L & Ashley, J et al. 1999. Evaluation of Simulated Presence: A personalized approach to enhance well-being in persons with Alzheimer’s disease. Journal of the American Geriatrics Society, 47, 446–452.
  • Chung, JCC & Lai, CKY2002. Snoezelen for dementia. Cochrane Database of Systematic Reviews, Issue 4, Article CD003152. doi: 10.1002/14651858.CD003152
  • Cohen-Mansfield, J & Libin, A2004. Assessment of agitation in elderly patients with dementia: Correlations between informant rating and direct observation. International Journal of Geriatric Psychiatry, 19, 881–891. doi:10.1002/gps.1171 [CrossRef]
  • Cohen-Mansfield, J, Marx, MS & Rosenthal, AS1989. A description of agitation in a nursing home. Journals of Gerontology. Series A, Biological Sciences and Medical Sciences, 44, M77–M84.
  • Cohen-Mansfield, J, Werner, P & Marx, MS1989. An observational study of agitation in agitated nursing home residents. International Psychogeriatrics, 1, 153–165. doi:10.1017/S1041610289000165 [CrossRef]
  • Cohen-Mansfield, J, Werner, P & Marx, MS1992. The social environment of the agitated nursing home resident. International Journal of Geriatric Psychiatry, 7, 789–798. doi:10.1002/gps.930071104 [CrossRef]
  • Corrigan, JD1989. Development of a scale for assessment of agitation following traumatic brain injury. Journal of Clinical and Experimental Neuropsychology, 11, 261–277. doi:10.1080/01688638908400888 [CrossRef]
  • Davis, LL, Buckwalter, K & Burgio, LD1997. Measuring problem behaviors in dementia: Developing a methodological agenda. Advances in Nursing Science, 201, 40–55.
  • Dean, R, Briggs, K & Lindesay, J1993. The domus philosophy: A prospective evaluation of two residential units for the elderly mentally ill. International Journal of Geriatric Psychiatry, 8, 807–817. doi:10.1002/gps.930081003 [CrossRef]
  • Doyle, C, Zapparoni, T, O’Connor, D. & Runci, S1997. Efficacy of psychosocial treatments for noisemaking in severe dementia. International Psychogeriatrics, 9, 405–422. doi:10.1017/S1041610297004547 [CrossRef]
  • Edelman, P2005. [Validation of the OQOL-D] Unpublished raw data
  • Edelman, P & Fulton, B2004. Measuring dementia-specific quality of life. Continuing Care, 233, 19–21.
  • Ekman, P, Levenson, RW & Friesen, WV1983. Autonomic nervous system activity distinguishes among emotions. Science, 221, 1208–1210. doi:10.1126/science.6612338 [CrossRef]
  • Ettema, TP, Droes, RM, de Lange, J, Mellenbergh, GJ & Ribbe, MW2007a. QUALIDEM: Development and evaluation of a dementia specific quality of life instrument. Scalability, reliability and internal structure. International Journal of Geriatric Psychiatry, 22, 549–556. doi:10.1002/gps.1713 [CrossRef]
  • Ettema, TP, Droes, RM, de Lange, J, Mellenbergh, GJ & Ribbe, MW2007b. QUALIDEM: Development and evaluation of a dementia specific qualify of life instrument—Validation. International Journal of Geriatric Psychiatry, 22, 424–430. doi:10.1002/gps.1692 [CrossRef]
  • Feldt, KS2000. The Checklist of Nonverbal Pain Indicators (CNPI). Pain Management Nursing, 11, 13–21. doi:10.1053/jpmn.2000.5831 [CrossRef]
  • Folstein, MF, Folstein, SE & McHugh, PR1975. “Mini-mental state”. A practical method for grading the cognitive state of patients for the clinician”. Journal of Psychiatric Research, 12, 189–198. doi:10.1016/0022-3956(75)90026-6 [CrossRef]
  • Godlove, C, Richard, L & Rodwell, D1982. Time for action: An observation study of elderly people in four different care environments. In Social Service Monographs: Research in Practice. Sheffield, United Kingdom: University of Sheffield.
  • Guy, W1976. ECDEU assessment manual for psychopharmacology. Rockville, MD: U.S. Department of Health, Education and Welfare.
  • Hall, L & Hare, J1997. Video Respite™ for cognitively impaired persons in nursing homes. American Journal of Alzheimer’s Disease, 12, 117–121. doi:10.1177/153331759701200304 [CrossRef]
  • Helmes, E, Csapo, KG & Short, JA1987. Standardization and validation of the Multidimensional Observation Scale for Elderly Subjects (MOSES). Journal of Gerontology, 42, 395–405.
  • Holm, MB, Rogers, JC, Burgio, LD & McDowell, BJ1999. Observational data collection using computer and manual methods: Which informs best?Topics in Health Information Management, 193, 15–25.
  • Holmes, D, Teresi, J, Lindeman, DA & Glandon, GL1997. Measurement of personal care inputs in chronic care settings. Journal of Mental Health & Aging, 31, 119–127.
  • Holmes, D, Teresi, J, Ramirez, M & Goldman, D1997. The measurement and comparison of staff service inputs in special dementia care units and in traditional nursing home units using a barcode methodology. Journal of Mental Health & Aging, 32, 195–207.
  • Hopman-Rock, M, Tak, EC & Staats, PG2001. Development and validation of the Observation List for early signs of Dementia (OLD). International Journal of Geriatric Psychiatry, 16, 406–414. doi:10.1002/gps.354 [CrossRef]
  • Horgas, AL, Nichols, AL, Schapson, CA & Vietes, K2007. Assessing pain in persons with dementia: Relationships among the non-communicative patient’s pain assessment instrument, selfreport, and behavioral observations. Pain Management Nursing, 8, 77–85. doi:10.1016/j.pmn.2007.03.003 [CrossRef]
  • Hughes, TL & Medina-Walpole, AM2000. Implementation of an interdisciplinary Behavior Management Program. Journal of the American Geriatrics Society, 48, 581–587.
  • Hurley, AC, Volicer, BJ, Hanrahan, PA, Houde, S & Volicer, L1992. Assessment of discomfort in advanced Alzheimer patients. Research in Nursing & Health, 15, 369–377. doi:10.1002/nur.4770150506 [CrossRef]
  • Hurley, AC, Volicer, L, Camberg, L, Ashley, J, Woods, P & Odenheimer, G et al. 1999. Measurement of observed agitation in patients with dementia of the Alzheimer’s type. Journal of Mental Health & Aging, 52, 117–133.
  • Judge, KS, Camp, CJ & Orsulic-Jeras, S2000. Use of Montessori-based activities for clients with dementia in adult day care: Effects on engagement. American Journal of Alzheimer’s Disease, 15, 42–46. doi:10.1177/153331750001500105 [CrossRef]
  • Katz, S, Ford, AB, Moskowitz, RW, Jackson, BA & Jaffe, MW1963. Studies of illness in the aged. The index of ADL: A standardized measure of biological and psychosocial function. Journal of the American Medical Association, 185, 914–919.
  • Kiernan, RJ, Mueller, J, Langston, JW & Van Dyke, C1987. The Neurobehavioral Cognitive Status Examination: A brief but differentiated approach to cognitive assessment. Annals of Internal Medicine, 107, 481–485.
  • Kleban, MH, Brody, EM & Lawton, MP1971. Personality traits in the mentally impaired aged and their relationship to improvements in current functioning. The Gerontologist, 11, 134–140.
  • Kovach, CR, Weissman, DE, Griffie, J, Matson, S & Muchka, S1999. Assessment and treatment of discomfort for people with late-stage dementia. Journal of Pain and Symptom Management, 18, 412–419. doi:10.1016/S0885-3924(99)00094-9 [CrossRef]
  • Krulewitch, H, London, MR, Skakel, VJ, Lundstedt, GJ, Thomason, H & Brummel-Smith, K2000. Assessment of pain in cognitively impaired older adults: A comparison of pain assessment tools and their use by nonprofessional caregivers. Journal of the American Geriatrics Society, 48, 1607–1611.
  • Lane, P, Kuntupis, M, MacDonald, S, McCarthy, P, Panke, JA & Warden, V et al. 2003. A pain assessment tool for people with advanced Alzheimer’s and other progressive dementias. Home Healthcare Nurse, 21, 32–37. doi:10.1097/00004045-200301000-00007 [CrossRef]
  • Lawton, MP1975. The Philadelphia Geriatric Center Morale Scale: A revision. Journal of Gerontology, 30, 85–89.
  • Lawton, MP1997. Assessing quality of life in Alzheimer disease research. Alzheimer Disease and Associated Disorders, 11Suppl. 6, 91–99.
  • Lawton, MP, Van Haitsma, K & Klapper, J1996. Observed affect in nursing home residents with Alzheimer’s disease. Journals of Gerontology. Series B, Psychological Sciences and Social Sciences, 51, P3–P14.
  • Lawton, MP, Van Haitsma, K, Perkinson, M & Ruckdeschel, K1999. Observed affect and quality of life in dementia: Further affirmations and problems. Journal of Mental Health & Aging, 51, 69–81.
  • Leong, IY, Chong, MS & Gibson, SJ2006. The use of a self-reported pain measure, a nurse-reported pain measure and the PAINAD in nursing home residents with moderate and severe dementia: A validation study. Age and Aging, 35, 252–256. doi:10.1093/ageing/afj058 [CrossRef]
  • Lindesay, J, Briggs, K, Lawes, M, MacDonald, A & Herzberg, J1991. The domus philosophy: A comparative evaluation of a new approach to residential care for the demented elderly. International Journal of Geriatric Psychiatry, 6, 727–736. doi:10.1002/gps.930061007 [CrossRef]
  • Logsdon, RG, Gibbons, LE, McCurry, SM & Terri, L2000. Quality of life in Alzheimer’s disease: Patient and caregiver reports. In Albert, S & Logsdon, RG (Eds.), Assessing quality of life in Alzheimer’s disease. New York: Springer.
  • Logsdon, RG, Gibbons, LE, McCurry, SM & Terri, L2002. Assessing quality of life in older adults with cognitive impairment. Psychosomatic Medicine, 64, 510–519.
  • Magai, C, Cohen, CI, Culver, C, Gomberg, D & Malatesta, C1997. Relation between premorbid personality and patterns of emotion expressions in mid-to late-stage dementia. International Journal of Geriatric Psychiatry, 12, 1092–1099. doi:10.1002/(SICI)1099-1166(199711)12:11<1092::AID-GPS690>3.0.CO;2-X [CrossRef]
  • Magai, C, Cohen, C, Gomberg, D, Malatesta, C & Culver, C1996. Emotional expression during mid-to late-stage dementia. International Psychogeriatrics, 8, 383–395. doi:10.1017/S104161029600275X [CrossRef]
  • Mahoney, EK, Hurley, AC, Volicer, L, Bell, M, Gianotis, P & Hartshorn, M et al. 1999. Development and testing of the Resistiveness to Care Scale. Research in Nursing & Health, 22, 27–38. doi:10.1002/(SICI)1098-240X(199902)22:1<27::AID-NUR4>3.0.CO;2-T [CrossRef]
  • Matthews, JNS & Morris, KP1995. An application of the Bradley-Terry type models to the measurement of pain. Applied Statistics, 44, 243–255. doi:10.2307/2986348 [CrossRef]
  • Mattis, S1988. Dementia Rating Scale: Professional manual. Odessa, FL: Psychological Assessment Resources.
  • McCann, JJ, Gilley, DW, Hebert, LE, Beckett, LA & Evans, DA1997. Concordance between direct observation and staff rating of behavior in nursing home residents with Alzheimer’s disease. Journals of Gerontology. Series B, Psychological Sciences and Social Sciences, 52, 63–72.
  • McFadyen, M1984. The measurement of engagement in the institutionalized elderly. In Hanley, I & Hodge, J (Eds), Psychological approaches to the care of the elderly. London, United Kingdom: Croom Helm.
  • Melzack, R & Torgerson, WS1971. On the language of pain. Anesthesiology, 34, 50–59.
  • Morgan, DG & Stewart, NJ1998. High versus low density special care units: Impact on the behaviour of elderly residents with dementia. Canadian Journal on Aging, 17, 143–165.
  • Nygaard, HA & Jarland, M2006. Checklist for Nonverbal Pain Indicators (CNPI): Testing of reliability and validity in Norwegian nursing homes. Age & Ageing, 35, 79–81. doi:10.1093/ageing/afj008 [CrossRef]
  • Osborn, CL & Marshall, MJ1993. Self-feeding performance in nursing home residents. Journal of Gerontological Nursing, 193, 7–14.
  • Perrin, T1997. The Positive Response Schedule for severe dementia. Aging and Mental Health, 12, 184–191. doi:10.1080/13607869757290 [CrossRef]
  • Pollock, BG, Mulsant, BH, Sweet, R, Burgio, LD, Kirshner, MA & Shuster, K et al. 1997. An open pilot study of citalopram for behavioral disturbances of dementia: Plasma levels and real time observations. American Journal of Geriatric Psychiatry, 5, 70–78.
  • Ragneskog, H, Kihlgren, M, Karlsson, I & Norberg, A1996. Dinner music for demented patients: Analysis of video-recorded observations. Clinical Nursing Research, 5, 262–277, discussion 278–282. doi:10.1177/105477389600500302 [CrossRef]
  • Reisberg, B, Borenstein, J, Salob, SP, Ferris, SH, Franssen, E & Georgotas, A1987. Behavioral symptoms in Alzheimer’s disease: Phenomenology and treatment. Journal of Clinical Psychiatry, 48Suppl., 9–15.
  • Richards, JS, Nepomuceno, C, Riles, M & Suer, Z1982. Assessing pain behavior: The UAB Pain Behavior Scale. Pain, 14, 393–398. doi:10.1016/0304-3959(82)90147-6 [CrossRef]
  • Rogers, JC, Holm, MB, Burgio, LD, Graneiri, E, Hsu, C & Hardin, JM et al. 1999. Improving morning care routines of nursing home residents with dementia. Journal of the American Geriatrics Society, 47, 1049–1057.
  • Rosen, J, Burgio, L, Kollar, M, Cain, M, Allison, M & Fogleman, M et al. 1994. The Pittsburg Agitation scale: A user-friendly instrument for rating agitation in dementia patients. The American Journal of Geriatric Psychiatry, 21, 52–59.
  • Smallwood, J, Irvine, E, Coulter, F & Connery, H2001. Psychometric evaluation of a short observational tool for small-scale research projects in dementia. International Journal of Psychiatric Psychiatry, 16, 288–292. doi:10.1002/gps.329 [CrossRef]
  • Snow, AL, Weber, JB, O’Malley, KJ, Cody, M, Beck, C & Bruera, E et al. 2004. NOPPAIN: A nursing assistant-administered pain assessment instrument for use in dementia. Dementia and Geriatric Cognitive Disorders, 17, 240–246. doi:10.1159/000076446 [CrossRef]
  • Snyder, M, Ryden, MB, Shaver, P, Wang, JJ, Savik, K & Gross, CR et al. 1998. The Apparent Emotion Rating Instrument: Assessing affect in cognitively impaired elders. Clinical Gerontologist, 184, 17–29. doi:10.1300/J018v18n04_03 [CrossRef]
  • Spaull, D, Leach, C & Frampton, I1998. An evaluation of the effects of sensory stimulation with people who have dementia. Behavioural and Cognitive Psychotherapy, 26, 77–86. doi:10.1017/S1352465898000083 [CrossRef]
  • Stewart, NJ, Hiscock, M, Morgan, DG, Murphy, PB & Yamamoto, M1999. Development and psychometric evaluation of the Environment-Behavior Interaction Code (EBIC). Nursing Research, 48, 260–268. doi:10.1097/00006199-199909000-00006 [CrossRef]
  • Swift, R, Harrigan, E, Cappelleri, J, Kramer, D & Chandler, L2001. Validation of the behavioural activity rating scale (BARS)™: A novel measure of activity in agitated patients. Journal of Psychiatric Research, 36, 87–95. doi:10.1016/S0022-3956(01)00052-8 [CrossRef]
  • Toner, JA, Teresi, JA, Gurland, BJ & Tirumalasetti, F1999. The Feeling-Tone Questionnaire: Reliability and validity of a direct patient assessment screening instrument for the detection of depressive symptoms in cases of dementia. Journal of Clinical Geropsychology, 51, 63–78. doi:10.1023/A:1022994930394 [CrossRef]
  • Van Haitsma, K, Lawton, MP, Kleban, MH, Klapper, J & Corn, J1997. Methodological aspects of the study of streams of behavior in elders with dementing illness. Alzheimer’s Disease and Associated Disorders, 11, 228–238.
  • Villanueva, MR, Smith, TL, Erickson, JS, Lee, AC & Singer, CM2003. Pain Assessment for the Dementing Elderly (PADE): Reliability and validity of a new measure. Journal of the American Medical Directors Association, 4, 1–8.
  • Vogelpohl, TS, Beck, CK, Heacock, O & Mercer, SO1996. “I can do it!” Dressing: Promoting independence through individualized strategies”. Journal of Gerontological Nursing, 223, 39–42.
  • Volicer, L, Collard, A, Hurley, A, Bishop, C, Kern, D & Karon, S1994. Impact of special care unit for patients with advanced Alzheimer’s disease on patients’ discomfort and costs. Journal of the American Geriatrics Society, 42, 597–603.
  • Ward, CR, Los Kamp, L & Newman, S1996. The effects of participation in an intergenerational program on the behavior of residents with dementia. Activities, Adaptation & Aging, 204, 61–76. doi:10.1300/J016v20n04_05 [CrossRef]
  • Ward, T, Murphy, E, Procter, A & Weinman, J1992. An observational study of two long-stay psychogeriatric wards. International Journal of Geriatric Psychiatry, 7, 211–217. doi:10.1002/gps.930070311 [CrossRef]
  • Warden, V, Hurley, AC & Volicer, L2003. Development and psychometric evaluation of the Pain Assessment in Advanced Dementia (PAINAD) scale. Journal of the American Medical Directors Association, 4, 9–15.
  • Whall, AL1999. The measurement of need-driven dementia-compromised behavior: Achieving higher levels of interrater reliability. Journal of Gerontological Nursing, 259, 33–37.
  • Yesavage, JA, Brink, TL, Rose, TL, Lum, O, Huang, V & Adey, M et al. 1982–1983. Development and validation of a geriatric depression screening scale: A preliminary report. Journal of Psychiatric Research, 17, 37–49. doi:10.1016/0022-3956(82)90033-4 [CrossRef]
  • Yudofsky, SC, Kopecky, HJ, Kunik, M, Silver, JM & Endicott, J1997. The Overt Agitation Severity Scale for the objective rating of agitation. Journal of Neuropsychiatry and Clinical Neuroscience, 9, 541–548.

Characteristics and Psychometric Properties of Direct Observation Measures of Individual Behavior

Author (Year) Recording Training Observation Reliability Validity
Agitation Behavior Mapping Instrument (ABMI)—Phenomena Observed: Agitated behaviors (frequency, intensity); environmental context and triggers (presence)
Cohen-Mansfield & Libin (2004) Computer Unspecified 72 3-minute observations every 30 minutes for 13 hours over 3 consecutive days

95.3% agreement for behavior

96% agreement for intensity rating

Convergent: correlated with Cohen-Mansfield Agitation Inventory (CMAI) individual items and total scores, r = 0.17 to 0.55 (p < 0.05, p < 0.001)

Cohen-Mansfield et al. (1992) By hand 1,000 3-minute observations every hour for 24 hours at random times over 2 to 3 months

Interrater r = 0.92 for social environment

Interrater r = 0.80 for triggers

Interrater r = 0.97 for direction of agitation

Interrater r = 0.71 for reaction

Cohen-Mansfield et al. (1989)

Interrater r = 0.93 for behavior

Agitated Behavior Scale (ABS)—Phenomena Observed: Agitated behaviors and positive behaviors (intensity)
Bogner et al. (2000) By hand Three sessions, unspecified 50 10-minute observations over three 8-hour shifts

Interrater r = 0.91

Hall & Hare (1997) 21 5-second observations every minute over 21 minutes

Interrater r = 0.98 to 0.99

Construct: positive behaviors different before versus during and during versus after intervention, t = −4.00, 2.62 (p < 0.01), respectively; no differences in agitated behaviors, t = −0.66 to 0.73 (p > 0.05)

Corrigan (1989)

Interrater r = 0.29 to 0.67 (p < 0.01) for those with brain injury

Cronbach’s alpha = 0.84 to 0.92 for those with brain injury

Content: literature review, expert input, item selection based on observation, for those with brain injury

Convergent: related to staff-reported agitation, r = 0.42 to 0.70 (p < 0.0001), staff comparison ratings to other individuals, r = 0.67 to 0.78 and to their own baseline, r = 0.63 to 0.72 (p < 0.0001)

Construct: factor analysis showed a one-factor solution (0.49 proportion of variance)

Computer-Assisted Data Collection—Phenomena Observed: ADLs (duration), disruptive vocalizations (frequency, duration), caregiver assists (frequency), requests for help (frequency), environmental context (presence, duration)
Rogers et al. (1999) Computer 3 months of practice observations 25 10-to 20-minute observations during morning care for 5 weeks

ADLs kappa = 0.78 to 0.95

Disruptive behavior kappa = 0.74

Staff assists kappa = 0.70 to 0.74

Requests for help kappa = 0.69

Holm et al. (1999) Unspecified number of 4-to 11-minute observations over 25 days

kappa = 0.71 to 0.95

Pollock et al. (1997) Three to five 1-hour observations over 3 days

kappa = 0.89 to 0.96

Burgio et al. (1994) Four 3-hour observations over 2 weeks

Environmental context kappa = 0.67 to 1.0

Verbal disruption kappa = .081

Convergent: related to cognitive status (MMSE), r = −0.70 to −0.77 (p < 0.01), and ADL functioning, r = 0.72 (p < 0.01)

Cohen-Mansfield Agitation Inventory-Revised—Phenomena Observed: Agitated behavior (frequency, duration)
Whall (1999) Video Two to three 30-minute practice observations to reach interrater r = 0.80 7-to 10-minute observations

Interrater r = 0.72 to 0.81

Convergent: related to behavior inventory, r = 0.79, and confusion inventory, r = 0.61

Disruptive Behavior Scale (DBS)—Phenomena Observed: Disruptive behaviors (frequency, intensity)
Beck et al. (1998) By hand 2 hours of lecture, 7 days of practice observations, weekly retrainings for 3 months Two 8-hour observations for 2 days every other week for 48 weeks

No significant differences across time (p > 0.05);

Convergent: related to age, r = −0.21 to −0.28; MMSE, r = −0.21 to −0.41; ADLs, r = 0.32; MOSES depression scale, r = 0.31 to 0.32; number of diagnoses, r = −0.20 to −0.33 (p < 0.05).

Beck, Heithoff, et al. (1997) 21 8-hour observations over 7 days

Interrater r = 0.80 (p < 0.0001)

Content: literature search and gerontological nurse input

Convergent: use of both frequency and intensity of behavior most predictive of staff-perceived disruptiveness, R2 = 0.322

Empirical Behavioral Pathology in Alzheimer’s Disease Rating Scale (E-BEHAVE-AD)—Phenomena Observed: Psychotic and agitated behavior (intensity)
Auer et al. (1996) By hand Unspecified 1 observation during 20-minute clinical interview

Interrater r = 0.73 to 1.0

Convergent: correlated with BEHAVE-AD, r = 0.51 (p < 0.01)

Environment-Behavior Interaction Code (EBIC)—Phenomena Observed: Disruptive behaviors (frequency, intensity), neutral and positive behaviors (frequency), environmental context (presence)
Stewart et al. (1999) Computer 120 hours of lecture and practice observations to maintain 60% to 75% agreement 4 to 10 30-minute intervals over 1 day during selected activity times

kappa = 0.65

78% agreement

Disruptive behavior test-retest (2 weeks), r = 0.56 to 0.95

Positive behavior test-retest (2 weeks), r = 0.50 to 0.97

Neutral behavior test-retest (2 weeks), r = 0.68 to 0.93

Content: field observation, literature review, expert panel

Construct: discriminated between groups at high and low risk of disruptive behavior, F = 5.33 to 7.76 (p < 0.05)

Morgan & Stewart (1998) 10 to 18 25-to 30-minute intervals over 1 to 2 weeks

kappa = 0.60 to 0.65

78% to 80% agreement

Disruptive behavior test-retest (2 weeks), r = 0.85 to 0.95

Positive behavior test-retest (2 weeks), r = 0.85 to 0.87

Neutral behavior test-retest (2 weeks), r = 0.65 to 0.93

Construct: discriminated between high-and low-density units,. t = 3.51 (p < 0.001)

Engagement Scale—Phenomena Observed: Engagement (duration)
Judge et al. (2000) By hand Not mentioned Four 10-minute observations over 1 day

>90% agreement

Construct: higher in Montessori group compared with regular activity group, F(1,17) = 4.7 to 14.8 (p < 0.05), and to own baseline, F(2,34) = 6.3 to 15.7 (p < 0.01); constructive engagement specifically higher when involved in Montessori activity than when not, t(8) = 3.6 (p < 0.01)

McCann Observation—Phenomena Observed: Disruptive behavior (frequency); facial affect (frequency); environmental context (presence)
McCann et al. (1997) By hand 1 month of lecture, including 2 weeks of practice observations 60 5-minute observations over 12 days

Interrater r > 0.90

Content: clinical data and direct observation

Convergent: related to staff behavior ratings, χ2 = 4.2 to 25.3 (p < 0.05), r = 0.14 to 0.46 (p < 0.05)

Construct: identified higher incidence of behavior than did staff ratings, McNemar’s χ2 = 13.56 to 101.32 (p < 0.001), except cursing and physical aggression, McNemar’s χ2 = 8.48 to 10.3 (p < 0.01)

Natural Observer—Phenomena Observed: Positive behavior (presence)
Ward et al. (1996) Video and by hand Unspecified Six 30-second observations over 3 months

90.7% agreement

Construct: no difference between activities with and without children, only less head nodding with children (p < 0.05)

Observational Scale—Phenomena Observed: Problem and positive behaviors (presence); personal care, eating, and mobility (presence)
Ward et al. (1992) By hand Unspecified 420 10-second observations over 10 14-minute periods in 1 week

95% agreement (range = 25% to 97%

Content: direct observations

Convergent: not demonstrated for most observed behaviors, but composite observation-based feeding score related to physical deterioration, feeding score, and physical disability score (p < 0.05)

Observer-Behavior Streams—Phenomena Observed: Pathological behavior (duration, frequency); ADLs (duration, frequency); social and enriching behaviors (duration, frequency); environmental context (presence, duration)
Van Haitsma et al. (1997) Computer 2,500 minutes of practice observations 16 10-minute intervals over 3 weeks

85.5% agreement (behaviors)

97.8% agreement (location)

96.2% agreement (position)

Interrater r = 0.80 to 0.90

Convergent: related to staff physical agitation ratings: r = −0.16 to 0.22 with pathological behavior; related to staff activity ratings: r = 0.14 to 0.46 with social and enriching behavior, r = −0.15 to −0.33 with pathological behavior; related to MOSES withdrawal scale: r = −0.17 to −0.48 with social behavior, r = 0.19 to 0.22 with pathological behavior; related to MOSES depression and irritability scale: r = −0.15 to −0.21 with enriching behavior; related to cognitive functioning: r = −0.27 with pathological behavior, r = −0.38 to −0.39 with sleeping and null behavior, r = 0.19 to 0.29 with social and enriching behavior

Overt Agitation Severity Scale (OASS)—Phenomena Observed: Agitation (frequency, intensity)
Yudofsky et al. (1997) By hand Not mentioned Four 15-minute observations over 16 hours

Interrater r = 0.90 (p < 0.01)

Split-half r = −0.10 to 0.97

Content: literature review

Convergent: associated with Pittsburg Agitation Scale, r = 0.81 to 0.82 (p < 0.01)

Divergent: low correlation with aggression scale, r = 0.28 to 0.29 (p < 0.01)

Pittsburg Agitation Scale (PAS)—Phenomena Observed: Agitation (intensity)
Rosen et al. (1994) By hand Unspecified 21 1-hour or 8-hour observations over 7 days

Interrater r = 0.82 (0.54 to 0.88) for nurses

Interrater r = 0.93 (0.85 to 1.0) for research assistants

Convergent: related to real-time recorded behavior, r = 0.61 to 0.89 (p < 0.01)

Construct: differentiated between behavior requiring restraint or as-needed medication and behavior that did not, t = 12.4 (p < 0.0001)

Patient Behaviour Observation Instrument (PBOI)—Phenomena Observed: Antisocial, inappropriate, and neutral behaviors (frequency, duration), abnormal motor activity (frequency, duration), ADLs and reception of care (frequency, duration), social engagement (frequency, duration)
Bowie & Mountain (1993, 1997) Computer Unspecified number of practice observations 13 5-minute observations every hour over 1 day

Interrater r = 0.90 to 0.95 (behavior)

Interrater r = 0.96 (care)

Interrater r = 0.94 (motor)

Interrater r = 0.82 (social)

Interrater r = 0.98 (neutral)

kappa = 0.82 to 1.0 (behavior)

kappa = 0.84 to 0.90 (care)

kappa = 0.93 (motor)

kappa = 0.82 (social)

kappa = 0.74 (neutral)

Content: video analysis of behavior

Construct: differentiated between wards rated on environmental quality (z = −4.58 to −2.87, p < 0.01)

Resistiveness to Care Scale in individuals with dementia of the Alzheimer’s type (RTC-DAT)—Phenomena Observed: Resistiveness to care (duration, intensity)
Mahoney et al. (1999) Video Practice observation to exceed 95% agreement 4 to 12 5-minute observations during care giving

kappa = 0.61 to 0.93

Cronbach’s alpha = 0.82 to 0.87

Item total r = 0.15 to 0.77

Content: direct observation, expert input, content validity index = 1.0 (p < 0.05)

Convergent: related to DS-DAT, r = 0.50 (p < 0.001); Visual Analogue Scale, r = 0.74 (p < 0.001); symptom severity, eta = 0.49 (p = 0.06); ADLs, eta = −0.32 (p = 0.08); speech assessment, eta = 0.46 (p < 0.01), but not with MMSE, eta = 0.21 (p = 0.42)

Construct: factor analyses found three factors: warding off/taking on, vocal/emotional resistance, and protective/reflexive resistance, 30.7%, 11.9%, and 9.7% variance explained, respectively

Scale for Observation of Agitation in Persons with Dementia of the Alzheimer’s type (SOAPD)—Phenomena Observed: Agitation behavior (intensity, duration)
Hurley et al. (1999) By hand 20 hours lecture and workshops, 2 weeks practice observations, manualized 2,136 5-minute observations over 3 days in 1 week

kappa = 0.55 to 0.90

Cronbach’s alpha = 0.70

Item total r = 0.20 to 0.60

Content: literature review, expert panel input, content validity index rating = 1.0 (p < 0.05)

Convergent: correlated with CMAI, r = 0.33 to 0.39 (p < 0.05) and Visual Analogue Scale, r = 0.75 to 0.90 (p < 0.001)

Construct: factor analysis found physical and verbal subscales, 16.4% and 35.7% variance explained, respectively.

Camberg et al. (1999) 40 5-minute observations over 1 week

Interrater r = .084

kappa = 0.55 to 0.90

Cronbach’s alpha = 0.82

Convergent: related to a Visual Analogue Scale, r = 0.91 (p < 0.001)

Construct: no differences between simulated presence, placebo, and usual care (p = 0.97)

Short Observational Tool (SOT)—Phenomena Observed: Agitated, neutral, and positive behavior (frequency)
Smallwood et al. (2001) Video Not mentioned Two 15-minute observations over 2 days

87% agreement

Test-retest r = 0.26 to 0.71

Divergent: five distinct behavior dimensions: 8/10, r = −0.03 to −0.32 (p > 0.05); 2/10, r = −0.75 to 0.45 (p < 0.05)

Timewand—Phenomena Observed: Physical aggression (frequency), idiosyncratic behavior (frequency), ADLs (frequency), environmental context (presence)
Bridges-Parlet et al. (1994) Computer Not mentioned Four 2-hour observations over 1 week

96% agreement for all codes

88% agreement for behavior

Convergent: 16 of 28 aggressive behaviors recorded also documented in staff diaries of aggression

Characteristics and Psychometric Properties of Direct Observation Measures of Individuals’ Affect

Author (Year) Recording Training Observation Reliability Validity
Apparent Affect Rating Scale (AARS)—Phenomena Observed: Positive and negative affect (duration)
Lawton et al. (1999) By hand, computer 1 month (lecture, informal discussions, practice videotapes, 35 to 40 practice observations, manualized Six 5-minute observations for each of four contexts

kappa = 0.78 to 0.89 for research assistants

kappa = 0.29 to 0.66 for research assistants and CNAs

Construct: demonstrated predicted differences in four different contexts, F = 3.10 to 133.31 (p < 0.01)

Lawton et al. (1996) 16 10-minute observations over 4 weeks kappa = 0.76 to 0.89

Convergent and Divergent: affective states hypothesized relationship between hand and computer versions, with MOSES depression and irritability subscales, Raskin Depression Scale (Guy, 1976), CMAI, Adult Personality Rating Scale (Kleban, Brody, & Lawton, 1971) confirmed r = 0.20 to 0.44 (p < 0.001), r = 0.05 to 0.17 (p > 0.001)

Construct: factor analyses demonstrated support for two-factor model (loadings = 0.63 to 0.99, −0.02 to −0.67).

Apparent Emotion Rating Scale (AERS)—Phenomena Observed: Positive and negative affect (presence)
Snyder et al. (1998) By hand 2 hours lecture One 5-to 10-minute observation

82% to 100% agreement

kappa = 0.28 to 1.0

Cronbach’s alpha = 0.55 to 0.58

Test-retest r(2 to 6 months) = 0.285 (p < 0.001)

Convergent: correlated with other measures of affect (GDS and Philadelphia Geriatric Center Morale Scale; Lawton, 1975), r = −0.19 to 0.30 (p < 0.001)

Construct: inverse correlations between positive and negative emotions, r = −0.09 to −0.30 (p < 0.05); correlations between positive emotions, r = 0.17 to 0.43 (p < 0.01) and negative emotions, r = 0.10 to 0.18 (p < 0.01); more cognitively impaired adults demonstrated lower affective scores, t = −4.13 (p < 0.001)

Checklist of Nonverbal Pain Indicators (CNPI)—Phenomena Observed: Pain (frequency)
Feldt (2000) By hand Practice observations until interrater reliability was established; instruction from nursing staff on the unit One observation over a period of rest and transfer

Interrater r = 0.93

kappa = 0.63 to 0.82 (p < 0.05)

K-R 20 = 0.54 to 0.64

Content: literature review

Convergent: correlated with self-report of pain, r = 0.30 to 0.50 (p < 0.05)

Nygaard & Jarland (2006) One observation over period of movement

73.9% to 89.1% agreement

kappa = 0.23 to 0.66

Test-retest (1 to 2 days) kappa = 0.45 to 069, 73.9% to 93.5% agreement

Convergent: related to a visual analogue scale, r = 0.69 to 0.88 (p < 0.001)

Discomfort Scale for persons with Dementia of the Alzheimer’s Type (DS-DAT)—Phenomena Observed: Discomfort (intensity)
Krulewitch et al. (2000) By hand Three training sessions for memorization of the scale One 5-minute observation

Convergent: poor correlation with other pain tools (self and caregiver report), statistics unspecified

Kovach et al. (1999) 12 hours lecture, workshops, 1 to 2 months practice observations

86% agreement among nurses

Content: literature review, expert input

Construct: reduction in behaviors following treatment (t = 6.56, p < 0.0001)

Volicer et al. (1994) Three 5-minute observations over 3 months

Interrater r = 0.89 to 0.98

Cronbach’s alpha = 0.86 to 0.88

Convergent: correlated with severity of dementia, r = 0.88; Katz Index of ADLs, r = 0.52; speech assessment, r = −0.87; MMSE, r = −0.59

Hurley et al. (1992)

Construct: ability to distinguish fever state versus health

Feeling-Tone Questionnaire (FTQ)—Phenomena Observed: Affective disorder (intensity)
Toner et al. (1999) By hand Not mentioned One 5-to 10-minute observation

Interrater r = 0.99

Cronbach’s alpha = 0.90 to 0.91

Test-retest r(1 to 2 days) = 0.81

Convergent: correlated with physician depression ratings, r = 0.43 to 0.71; observer ratings of mood, r = 0.39

Maximally Discriminative Facial Movement Coding System (MAX)—Phenomena Observed: Positive and negative affect (frequency, duration)
Magai et al. (1997) Computer Practice video-tapes, practice observations until kappa > 0.80 is reached One 20-minute observation

Convergent: positive emotion related to attachment style, r = 0.26 to 0.40; only sadness related to hostility, r = 0.50 (p < 0.01)

Divergent: confirmed no relationship between negative emotion and attachment style

Magai et al. (1996)

kappa = 0.86 to 1.0

Convergent: 3/4 emotions correlated with family ratings, r = 0.31 to 0.61, and 2/4 emotions correlated with caregiver ratings, r = 0.36 to 0.56

Construct: no significant differences across cognitive stage for all emotions (p > 0.05), except joy, F(3,43) = 2.4 (p < 0.05)

Modified Behavior Rating Scale (MBRS)—Phenomena Observed: Positive affect (frequency); bodily movement (frequency); interactions with others (frequency)
Spaull et al. (1998) By hand Not mentioned 10 to 20 30-second observations over a 10-to 20-minute period

Interrater r = 0.61 to 0.83

Construct: sensitive to Snoezelen intervention, F(2,6) = 31.12 (p < 0.001), whereas Dementia Care Mapping was not

Non-communicative Patient’s Pain Assessment Instrument (NOPPAIN)—Phenomena Observed: Pain (intensity)
Horgas et al. (2007) By hand Unspecified number of practice observations to achieve 90% agreement One 10-minute observation of physical activity

90% to 100% agreement

kappa = 0.72 to 1.0

Interrater r = 0.72 to 1.0

Test-retest (1 week) r = 0.68 to 0.95

Cronbach’s alpha = 0.80 to 0.97, 70% to 100% agreement over 1 week

Convergent: related to self-reported pain in cognitively intact (not impaired) adults, r = 0.66 (p < 0.001); and to other pain behavior ratings, r = 0.63 to 0.65 (p < 0.001)

Snow et al. (2004)

kappa = 0.87

Content: Expert panel input

Construct: discriminated between videos demonstrating different levels of pain as determined by geropsychologist and palliative care physician, goodness of fit index (Matthews & Morris, 1995) = 18.14 (p > 0.05)

Pain Assessment in Advanced Dementia (PAINAD)—Phenomena Observed: Pain (intensity)
Warden et al. (2003) By hand 2 hours lecture, practice observations to reach 100% agreement, manualized Three 5-minute observations

Interrater t(17, 18) = 0.0, 1.7 (p > 0.05)

Cronbach’s alpha = 0.70 (range = 0.50 to 0.83)

Content: literature review, expert input, observations.

Convergent: correlated with DS-DAT and two visual analogue scales (0.75 to 0.76, p < 0.001)

Construct: discriminated between pleasant, unpleasant, and no stimulation (F1,17 = 10.93, p < 0.001), improvement from before to after pain medication (t = 9.6, p < 0.001); a one-factor solution explained 50.1% of variance

Leong et al. (2006)

Convergent: related to nurse-reported pain, r = 0.84 (p < 0.001); self-reported analogue scale, r = 0.30 (p < 0.005); and depression score, r = 0.29 (p < 0.005)

Positive Response Schedule (PRS) for severe dementia—Phenomena Observed: Well-being and ill-being (frequency); body movement (frequency)
Perrin (1997) By hand Unspecified 40 to 120 20-second observations over 20 to 60 minutes

80% (79% to 82%) agreement

Convergent: stated as impossible to determine

Content: based on engagement theory

Pain Assessment for the Dementing Elderly (PADE)—Phenomena Observed: Pain (intensity)
Villanueva et al. (2003) By hand 1 hour lecture 30-35 5-minute observations over 10 days

Interrater r = 0.54 to 0.95

Cronbach’s alpha = 0.24 to 0.88

Test-retest r(20 days) = 0.34 to 0.89

Content: literature review, expert interviews, observations, staff feedback

Convergent: correlated with verbal and aggressive CMAI scores, r = 0.30 to 0.42 (p < 0.01)

Construct: discriminated between those taking psychoactive medications and those not, z = −3.19 to 3.25 (p < 0.01)

University of Alabama-Birmingham (UAB) Pain Behavior Scale—Phenomena Observed: Pain (intensity, frequency)
Richards et al. (1982) By hand Unspecified One 5-minute observation

Interrater r = 0.94 to 0.96 (p < 0.01)

Test-retest r(2 days) = 0.89 (p < 0.01)

Construct: significant decline after treatment, T = 69 (p < 0.05)

Convergent: not related to McGill (Melzack & Torgerson, 1971) and analogue pain scales at admission (0.17, 0.16, not significant) but was related to the analogue scale at discharge, r = 0.55 (p < 0.05)

Divergent: negatively related to walking, sitting, and standing, r = −0.29 to −0.38 (p < 0.05); not related with biking, r = −0.10 (p > 0.05)

Characteristics and Psychometric Properties of Direct Observation Measures of Individuals’ Activities of Daily Living (ADLs)

Author (Year) Recording Training Observation Reliability Validity
Beck Dressing Performance Scale (BDPS)—Phenomena Observed: Ability in dressing (assistance needed, duration)
Beck, Heacock, et al. (1997) Video 2 weeks practice observations to reach interrater r = 0.80 4 to 12 4-to 5-minute observations over 2 to 6 weeks

Interrater r = 0.80 to 0.89

Content: task analysis

Convergent: not related to MMSE (p > 0.05), related to NCSE language scores, r = −0.33 (p < 0.05)

Construct: 75% improvement (p < 0.000005) and increased duration (p < 0.05) during intervention

InfoAid—Phenomena Observed: ADL care (duration)
Holmes et al. (1997) Computer Unspecified 21 8-hour observations over 7 days

Convergent: related to staff-reported ADL, r = 0.41 (p < 0.05), F = 20.70 and cognitive ability, F = 12.21 (p < 0.01)

Divergent: not related to age, gender, continence, speech, behavior, or mood, F = 0.11 to 3.33 (p > 0.05)

Construct: discriminated between special care units (SCUs) and non-SCUs, F = 18.84 (p < 0.01)

Observational List for early signs of Dementia (OLD)—Phenomena Observed: Memory, language, orientation, ADLs, social withdrawal (presence)
Hopman-Rock et al. (2001) By hand Not mentioned One 5-minute observation

Cronbach’s alpha = 0.88

Content: literature review, expert panel input

Convergent: related to cognitive screen (F = 11.8), verbal memory (F = 8.3 to 10.5), informant cognitive rating (F = 2.6), age, x2 = 9.02 (p < 0.05), but not related to informant ADL and instrumental ADL (F = 0.35 to 0.77 (p > 0.05)

Divergent: not related to GDS depression score (F = 1.23) (p > 0.05)

Construct: found two factors; 48% and 14% variance explained; different than general practitioner opinion (p < 0.01)

Self-Feeding Assessment Tool (SFAT)—Phenomena Observed: Performance and ability during eating (assistance needed)
Osborn & Marshall (1993) By hand Not mentioned Four observations over two meals

100% agreement

Content: expert input, comparison to standard scales

Convergent: related to cognitive impairment, r = −0.62 (p < 0.05) for ability, r = −0.56 (p < 0.05) for performance, and related to length of stay, r = −0.57 (p < 0.05)

Construct: capability and performance scores different for drinking and self-feeding (p < 0.05)

Characteristics and Psychometric Properties of Direct Observation Measures of Individuals’ Quality of Life (QOL)

Author (Year) Recording Training Observation Reliability Validity
Observing Quality of Life in Dementia (OQOL-D)—Phenomena Observed: QOL (intensity)
Edelman & Fulton (2004) By hand One 8-hour session including lecture and practice observations to reach 80% agreement

>80% agreement

Content: staff input

Edelman (2005) 7 to 8 observations over 4 to 5 days

Convergent: related to Dementia Care Mapping, r = 0.73 to 0.98 (p < 0.0005); staff QOL-AD ratings, r = 0.62 (p <0 .0005); resident QOL-AD, r = 0.50 (p < 0.0005); proxy measure of QOL, r = 0.55 (p < 0.0005); depression, r = −0.49 (p < 0.0005); MMSE, r = 0.37 (p < 0.0005); number of dependent ADLs, r = −0.36 (p > 0.01)

Psychological Well-Being in Cognitively Impaired Persons (PWB-CIP)—Phenomena Observed: Positive and negative well-being (presence)
Burgener & Twigg (2002) By hand Not mentioned

Cronbach’s alpha = 0.74

Test-retest F(18 months) = 0.48 (p > 0.05)

Content: literature review

Convergent: related to depression, r = −0.59; MMSE, r = 0.49; caregiver education, r = 0.39; relationship quality, r = −0.51; personal distress, r = −0.39; domestic upset, r = −0.43; negative feelings, r = −0.39; and pleasant events, r = 0.43 (p < 0.01); not related to social contacts, r = 0.16 (p > 0.05)

Construct: stability of well-being, F = 0.48 (p > 0.05), in face of change in depression and functional ability over time, F = 3.75 to 76.23 (p < 0.05)

Burgener et al. (2005) One 24-hour observation period

Cronbach’s alpha = 0.74 to 0.84

Test-retest F(18 months) = 0.22 to 1.80 for 9/11 items (p > 0.05)

Content: expert input, content validity index = 0.80

Convergent: related to a faces rating scale, r = 0.49 to 0.65; extroversion, r = 0.53 to 0.61; hostility, r = −0.40 to −0.50; task orientation, r = 0.35 to 0.41; MMSE, r = 0.30 to 0.53; depression, r = −0.30 to −0.42; productive behaviors, r = 0.36 to 0.58; social contacts, r = 0.26; pleasant events, r = 0.35 (p < 0.05)

Construct: factor analysis found two factors: positive well-being and negative well-being, 16% and 35% variance explained, respectively

QUALIDEM—Phenomena Observed: Positive and negative QOL (intensity)
Ettema et al. (2007a) By hand 1 hour lecture and reading 2 weeks of observations

Cronbach’s alpha = 0.50 to 0.89

rho = 0.51 to 0.90

Interrater r = 0.47 to 0.79

Test-retest (1 week) r = 0.74 to 0.89

Content: literature review, dementia patient focus group, participant observation, expert panel input

Construct: subscales demonstrated unidimensional constructs H coefficients = 0.34 to 0.62, moderate relationships demonstrated between majority of subscales, r = 0.16 to 0.59

Ettema et al. (2007b)

Convergent: related to all 39 predicted relationships with behavior and mood ratings, r = 0.17 to 0.75; related to 23 of 24 nurse QOL ratings, r = 0.16 to 0.44 (p < 0.05); 15 of 24 family QOL ratings, r = 0.21 to 0.44 (p < 0.05); 2 of 16 self-report QOL ratings, r = 0.17 to 0.24 (p < 0.05)

Divergent: not related to 21 of 24 predicted relationships with behavior and mood ratings, r < 0.20

Authors

Dr. Curyto is Research Scientist and Ms. Vriesman is Research Assistant, Center for Senior Care, Pine Rest Christian Mental Health Services, Grand Rapids, Michigan, and Dr. Van Haitsma is Senior Research Scientist, Polisher Research Institute, Madlyn & Leonard Abramson Center for Jewish Life, North Wales, Pennsylvania.

The authors thank the Harry Stern Family Center for Innovations in Alzheimer’s Care and the Alzheimer’s Association for their generous support in funding this work.

Address correspondence to Kim J. Curyto, PhD, Research Scientist, Center for Senior Care, Pine Rest Christian Mental Health Services, 300 68th Street SE, PO Box 165, Grand Rapids, MI 49501-0165; e-mail: Kimberly.Curyto@PineRest.org.

10.3928/19404921-20080101-02

Sign up to receive

Journal E-contents