Journal of Gerontological Nursing

Nursing Research: Threats to Reliability and Validity Gerontology

Rebecca Shaw, PHD, RN

Abstract

INTRODUCTION

Several authors have described the difficulties inherent in conducting research using elderly subjects (Gift, 1987; Kovach, 1989, Minte, 1981; Sexton, 1983; Uman, 1990). These difficulties have an impact on the reliability, internal validity, and external validity of gerontological nursing research. This article will examine the factors to be considered when planning or critiquing research using elderly subjects. The classic works by Campbell and Stanley (1963), Cook and Campbell (1979), and Nunnally (1978) serve as a basis for discussion.

Reliability is the extent to which a tool or research procedure tests a concept without measurement error (Nunnally, 1978). Several factors affect the reliability of data gathered in research on the aged (Figure 1). For example, presbyopia, retinal atrophy, cataracts, and glaucoma cause decreased visual acuity, affecting the ability to read instructions on questionnaires (Burnside, 1988; Gift, 1987). Additionally, poor hearing due to inner ear atrophy can reduce the ability of potential subjects to understand explanations about research projects or to respond accurately to interview questions.

Reliability can also be influenced by fatigue and discomfort. Elderly subjects may not be able to sit through a long session of data collection. Muscle atrophy, joint calcification, decreased circulation to coronary arteries, a decrease in the force of the heart, dry conjunctiva, hypothyroidism, and a decrease in fat cells all contribute to fatigue and discomfort (Burnside, 1988). The number of sweat glands declines with aging, resulting in inefficient body temperature regulation.

Due to neurological and metabolic changes, older people often have difficulty understanding the purpose of a research project or making appropriate responses. They tend to be concrete thinkers, and although intellectual capabilities normally do not decline, the elderly may be unable to screen out irrelevant information (Gift, 1987).

To improve the reliability of measurement procedures, research material should be simple and wellorganized, and the study should be carefully explained to subjects. Short, simple sentences and words are more likely to elicit the appropriate response (Sexton, 1983). Questionnaires printed with a clear, large font on buff-colored nonglare paper are more readable. Research activities that are as brief and simple as possible will prevent fatigue and discomfort (Sexton, 1983). Well-padded chairs and a comfortable room temperature are also recommended.

Table

INTERNALVAUDfIY

Internal validity is the extent to which it is possible to infer that an intervention results in an observed difference, or that a subject's performance on a dependent measure results from the effects of an independent measure (Cook, 1979). Statistical conclusion validity (the ability of the researcher to draw conclusions based on the data gathered) is an important facet of internal validity.

The length of the entire research instrument affects statistical conclusion validity. Although questionnaires should include enough items to ensure reliability (Cook, 1979), too many items could be so stressful for elderly respondents that they might give up or answer carelessly due to fatigue or discomfort. This, in turn, could influence the accuracy (ie, reliability) with which questions are answered (Colsher, 1989).

Another way in which reliability and statistical conclusion validity can be adversely affected is through response set bias or acquiescence. Elderly subjects may hesitate to admit that they are having trouble understanding a questionnaire, or they may be concerned about the researcher's reactions to their responses. To prevent these problems, positively and negatively worded questions should be interspersed throughout the research instrument. In addition, multiple means to measure variables of interest should be used (Topf, 1986). A social desirability instrument can also be used to assess for acquiescence.

The problem with both of these suggestions is that they add to the length of the questionnaire. The researcher is faced with a…

INTRODUCTION

Several authors have described the difficulties inherent in conducting research using elderly subjects (Gift, 1987; Kovach, 1989, Minte, 1981; Sexton, 1983; Uman, 1990). These difficulties have an impact on the reliability, internal validity, and external validity of gerontological nursing research. This article will examine the factors to be considered when planning or critiquing research using elderly subjects. The classic works by Campbell and Stanley (1963), Cook and Campbell (1979), and Nunnally (1978) serve as a basis for discussion.

Reliability is the extent to which a tool or research procedure tests a concept without measurement error (Nunnally, 1978). Several factors affect the reliability of data gathered in research on the aged (Figure 1). For example, presbyopia, retinal atrophy, cataracts, and glaucoma cause decreased visual acuity, affecting the ability to read instructions on questionnaires (Burnside, 1988; Gift, 1987). Additionally, poor hearing due to inner ear atrophy can reduce the ability of potential subjects to understand explanations about research projects or to respond accurately to interview questions.

Reliability can also be influenced by fatigue and discomfort. Elderly subjects may not be able to sit through a long session of data collection. Muscle atrophy, joint calcification, decreased circulation to coronary arteries, a decrease in the force of the heart, dry conjunctiva, hypothyroidism, and a decrease in fat cells all contribute to fatigue and discomfort (Burnside, 1988). The number of sweat glands declines with aging, resulting in inefficient body temperature regulation.

Due to neurological and metabolic changes, older people often have difficulty understanding the purpose of a research project or making appropriate responses. They tend to be concrete thinkers, and although intellectual capabilities normally do not decline, the elderly may be unable to screen out irrelevant information (Gift, 1987).

To improve the reliability of measurement procedures, research material should be simple and wellorganized, and the study should be carefully explained to subjects. Short, simple sentences and words are more likely to elicit the appropriate response (Sexton, 1983). Questionnaires printed with a clear, large font on buff-colored nonglare paper are more readable. Research activities that are as brief and simple as possible will prevent fatigue and discomfort (Sexton, 1983). Well-padded chairs and a comfortable room temperature are also recommended.

Table

FIGURE 1 PHYSICAL CHANGES INAOIHG WITH USf ARCH IMPLICATIONS FIGURE 2

FIGURE 1 PHYSICAL CHANGES INAOIHG WITH USf ARCH IMPLICATIONS FIGURE 2

Table

FIGURE 1 PHYSICAL CHANGES INAOIHG WITH USf ARCH IMPLICATIONS FIGURE 2

FIGURE 1 PHYSICAL CHANGES INAOIHG WITH USf ARCH IMPLICATIONS FIGURE 2

Subjects who are hard of hearing require special handling. The researcher should sit at the side on which the subject's hearing is better, maintain eye contact, and speak clearly in low tones (Gift, 1987). Butler and Lewis also suggest the use of both written and oral instructions (Butler, 1982).

When working directly with elderly research subjects, the environment should be as free of distractions as possible (Gift, 1987). If a psychomotor skill is involved in the project, an adequate amount of time to complete the task is necessary to prevent subjects from feeling threatened by the testing situation.

All of these suggestions are recommended because the changes common in the aged affect the reliability of data collection. Internal and external validity are also important in gerontological nursing research (Figure 2).

Table

THPMATS TO INTERNAL AND EXTEPNAL VALIDITY WITH SUGGESTIONS FOR RESEARCH DESIGN

THPMATS TO INTERNAL AND EXTEPNAL VALIDITY WITH SUGGESTIONS FOR RESEARCH DESIGN

INTERNALVAUDfIY

Internal validity is the extent to which it is possible to infer that an intervention results in an observed difference, or that a subject's performance on a dependent measure results from the effects of an independent measure (Cook, 1979). Statistical conclusion validity (the ability of the researcher to draw conclusions based on the data gathered) is an important facet of internal validity.

The length of the entire research instrument affects statistical conclusion validity. Although questionnaires should include enough items to ensure reliability (Cook, 1979), too many items could be so stressful for elderly respondents that they might give up or answer carelessly due to fatigue or discomfort. This, in turn, could influence the accuracy (ie, reliability) with which questions are answered (Colsher, 1989).

Another way in which reliability and statistical conclusion validity can be adversely affected is through response set bias or acquiescence. Elderly subjects may hesitate to admit that they are having trouble understanding a questionnaire, or they may be concerned about the researcher's reactions to their responses. To prevent these problems, positively and negatively worded questions should be interspersed throughout the research instrument. In addition, multiple means to measure variables of interest should be used (Topf, 1986). A social desirability instrument can also be used to assess for acquiescence.

The problem with both of these suggestions is that they add to the length of the questionnaire. The researcher is faced with a dilemma: the need to improve reliability by avoiding the use of long, difficult tools and procedures versus the need to improve reliability through the collection of a complete set of data. This dilemma should be considered and noted in the discussion section of the research report. Additionally, it might be possible to collect some data through observation or chart review, or to use matrix sampling, which is randomly assigning subgroups of samples to respond to randomly selected parts of the same questionnaire (Morris, 1978).

Apart from statistical conclusion validity, several other threats to internal validity are of concern in gerontological research. The first is ambiguity about the direction of causal influence, most typically a problem with cross-sectional studies. Longitudinal designs are recommended because changes in variables over time support the claim of causality (Cook, 1979; Kovach, 1989). However, longitudinal designs are also a problem with elderly subjects because they involve two other threats to internal validity: history and attrition.

History concerns events that influence subjects over time as they participate in a research project. For example, in longitudinal investigations of depression in older adults, several life events may affect results, such as chronic illness, family deaths, or financial uncertainty. Random selection of subjects reduces the effects of this threat to internal validity, because one can assume that life events would be occurring similarly (on the average) across the groups in the study.

Attrition from a study may also be a problem in longitudinal studies (Herzog, 1988). For example, in the Normative Aging Study described by Rose, Bosse, and Szretter (1976), subjects dropped out for a variety of reasons, such as socioeconomic status, difficulty understanding the research, social isolation, and declining interest.

To minimize the effects of attrition, Gift and Palmer (1987) suggest obtaining as many subjects and samples as possible. The problem of attrition can also be treated by evaluating specific reasons why subjects drop out of the study and by examining possible competing variables (Norris, 1987). The most obvious causes of attrition in older subjects are morbidity and mortality. Both variables are probably representative of the aged cohort (Goudy, 1985); a survival analysis can be used to assess their effects.

Statistical conclusion validity, ambiguity about the direction o( causal influence, history, and attrition are important factors to consider when designing a gerontological research project or critiquing its internal validity.

EXTERNAL VAUDITY

Selection bias influences external as well as internal validity. According to Cook and Campbell (1979), external validity is the ability of the researcher to generalize from a study sample to a larger population. Selection bias is important because the characteristics of a sample may affect the way that subjects react to an intervention or other independent variable. Selection of subjects is a problem in gerontological research due to the carelessness with which researchers sometimes define "the elderly" population and the possibility that older potential subjects might be excluded from studies (or exclude themselves from studies) because of chronic illness or debilitation.

Researchers inaccurately define the population of interest in an investigation when they do not make a distinction about the specific elderly age group they are studying. The over-65 age group is essentially made up of two generations: the young-old, aged 65 to 84 years, and the old-old, aged 85 years or older (Burnside, 1988). These two cohorts probably experienced significantly different life events (Gift, 1987; Kovach, 1989; Mintz, 19.81; Sexton, 1983; Uman, 1990). For example, the present cohort of young-old were children and young adults during the Great Depression and World War ?. In contrast, the old-old were children and young adults during World War I. Although the two groups share common life events and characteristics, they differ enough that they may respond differently to questionnaires in a particular study.

Selection bias can also affect external validity when potential subjects are excluded from a study because they refuse to participate or are too ill or frail to participate (Streib, 1983). For example, Rose et al (1976) noted that the population used in the Normative Aging Study (Veterans Affairs outpatients) was relatively free of disease at the outset of the study, excluding less healthy potential subjects. The frail elderly are routinely excluded from research samples even though their experiences are of interest in gerontological research (Streib, 1983). Chronically ill and debilitated subjects may lack the energy or resources to participate in a study, or their families may desire to protect them and interfere with their consent to be involved in a study. The opposite can also occur, in that potential subjects may participate because they are isolated from the rest of society, bored, or lonely (Sexton, 1983).

Similarly, some potential elderly subjects are too disoriented to participate or refuse to participate for a variety of reasons, such as suspiciousness or unwillingness to answer certain questions (Colsher, 1989; Herzog, 1988). Even when random samples are used, the refusal or exclusion rate could be large enough that the sample would not be representative of the target population.

To counteract selection bias, youngold and old-old subjects should be carefully delineated and a large, random sample obtained. Furthermore, demographic data on subjects can be compared with census data for the geographical area in which the study is conducted to determine whether the sample is similar to the target population. Selection bias should also be recognized in the report of the study.

Another threat to external validity is the measurement effect, in which the tools used may be inappropriate for the specific target population. Of special concern is the possibility that the tools used may have been tested on younger populations (Salthouse, 1982). This is essentially a question of construct validity because older people would probably respond differently than younger people due to physical differences or different life events or perspectives (Kovach, 1989). Instruments that have not been evaluated for use on the elderly should be subjected to pilot-testing first (Uman, 1990).

CONCLUSIONS

A growing body of gerontological nursing research provides guidelines for clinical care. Older subjects are unique. Issues related to reliability, internal validity, and external validity have been outlined in this article to provide an overview of the factors that should be considered when designing gerontological nursing research.

REFERENCES

  • Burnside, LM. Nursing and the aged: A self-care approach, St. Louis: McGraw-Hill, 1988.
  • Butler, R., Lewis, M. Aging and mental health: Positive psychosocial and biomedical approaches, 3rd ed. St. Louis: CV Mosby, 1982.
  • Campbell, D.T., Stanley, J.C. Experimental and quasi-experimental designs for research. Chicago: Rand McNaIIy, 1963.
  • Colsher, P.L. Data quality and age: Health and psychobehavioral correlates of item nonresponse and inconsistent responses. J Gerontol 1989; 44.P45-P52.
  • Cook, T.D., Campbell, D.T. Quasi-experimentation design and analysis issues for field settings. Chicago: Rand McNaIIy College Publishing Co, 1979.
  • Gift, AG., Palmer, M.H. Planning clinical nursing research with a geriatric population. Clinical Nurse Specialist 1987; 1:56-60.
  • Goudy, WJ. Sample attrition and multivariate analysis in the Riterment History Study. J Gerontol 1985; 40:358-367.
  • Herzog, A.R., Rodgers, W.L. Age and response rates to interview sample surveys. J Gerontol 1988; 43:S200-S205.
  • Kovach, CR., Knapp, XR. Age, cohort, and time-period confounds in aging research. fournal of Gerontological Nursing 1989; 15(3):11-15.
  • Mintz, J., Steuer, J., Jarvik, L. Psychotherapy with depressed elderly patients: Research considerations. Journal of Counseling and Clinical Psychology 1981; 49:542-548.
  • Morris, L.L., Fitz-Gibbon, CT. How to deal with goals and objectives. Beverly Hills, CA: Sage, 1978.
  • Noms, RH., Murrell, S.A. Transitory impact of life-event stress on psychological symptoms in older adults, journal of Health and Behavior 1 987; 28: 1 97-21 1 .
  • Nunnally, J.C. Psychometric theory. St. Louis: McGraw-Hill, 1978.
  • Rose, CL., Bosse, R., Szretter, W.T. The relationships of scientific objectives to population selection and attrition in longitudinal studies- Gerontologist 1976; 16:508-516.
  • Salthouse, T.A. Adult cognition. New York: Springer, 1982.
  • Sexton, D.L. Some methodological issues in chronic illness research. Nurs Res 1983; 32:378-380.
  • Streib, G.E The frail elderly: Research dilemmas and research opportunities. Gerontologist 1983; 23:40-44.
  • Topf, M. Response sets in questionnaire research. Nurs Res 1986; 35:119-121.
  • Uman, G.C., Urman, H.N. The challenges of conducting clinical nursing research with elderly populations. AORN J 1990; 52:400-406.

FIGURE 1 PHYSICAL CHANGES INAOIHG WITH USf ARCH IMPLICATIONS FIGURE 2

FIGURE 1 PHYSICAL CHANGES INAOIHG WITH USf ARCH IMPLICATIONS FIGURE 2

THPMATS TO INTERNAL AND EXTEPNAL VALIDITY WITH SUGGESTIONS FOR RESEARCH DESIGN

10.3928/0098-9134-19920801-08

Sign up to receive

Journal E-contents