Journal of Nursing Education

The articles prior to January 2012 are part of the back file collection and are not available with a current paid subscription. To access the article, you may purchase it or purchase the complete back file collection here

Ten Criteria for Evaluating Qualitative Research Proposals

Ann Kuckelman Cobb, PhD, RN; Julia Nelson Hagemaster, PhD, RN

Abstract

ABSTRACT

With the proliferation of interest in qualitative research in nursing comes the attendant problem of how to evaluate it appropriately. Qualitative research has its own unique history, philosophical foundations, and methodologies that separate it from the quantitative approach. Although the literature is crowded with guidelines for evaluating the latter, little is offered for the qualitative reviewer. The Research Proposal Evaluation Form: Qualitative Methodology is a partial solution to this dilemma. It provides a framework for critiquing the proposal phase of a qualitative study and can be an important guide both for the educator and for the novice researcher.

Abstract

ABSTRACT

With the proliferation of interest in qualitative research in nursing comes the attendant problem of how to evaluate it appropriately. Qualitative research has its own unique history, philosophical foundations, and methodologies that separate it from the quantitative approach. Although the literature is crowded with guidelines for evaluating the latter, little is offered for the qualitative reviewer. The Research Proposal Evaluation Form: Qualitative Methodology is a partial solution to this dilemma. It provides a framework for critiquing the proposal phase of a qualitative study and can be an important guide both for the educator and for the novice researcher.

Introduction

In discussions with colleagues, and in reviewing literature, we find that there is some confusion about exactly what constitutes qualitative research. Definitions range from "anything without numbers" to a series of identifying characteristics that should be present if a given project is to be designated qualitative. Some of this confusion stems from the fact that nursing has borrowed methodologies from several different disciplines that employ qualitative approaches, such as anthropology, sociology, psychology, and history. The field of education has similarly adopted these methodologies (Reichardt & Cook, 1979). Within each of the parent disciplines, distinctions are made simply on the basis of what specific methods are used. Thus, the terms "fieldwork," "participant-observation," "grounded theory," "ethnography" and "phenomenology" are sufficient to convey meaning about the kind of research to be conducted. The term qualitative research has taken on a slightly different and more general meaning as nursing, education, and other disciplines attempt to synthesize methods from several fields. The labels "qualitative paradigm" and "quantitative paradigm" (Reichardt & Cook, 1979) have emerged as a way to summarize these distinctions, and methodology is only one attribute of each. Some (Taylor & Bogdan, 1984) have argued that phenomenology, ethnography, and ethnomethodology are essentially the same thing, and distinctions should be discarded. This supports the use of the more generalized term qualitative paradigm. Others (Goodwin & Goodwin, 1984; Reichardt & Cook, 1979) also contend that adherence to one paradigm does not preclude the use of methodologies from the other. The distinctions between the two approaches are not absolute, they say, and recommend that both types of methods can and, in some cases, should be included in the same study. Whether the study is qualitative or quantitative, then, would depend on the overall approach (paradigm components), not just the use of one or another method.

For the purposes of this paper, we define qualitative research in the more generalized, paradigmatic sense and include comments on methods from several disciplines. Qualitative research is a type of investigation in which: 1. there is attention to the social context in which events occur and have meaning; 2. there is an emphasis on understanding the social world from the point of view of the participants in it; 3. the approach is primarily inductive; 4. major data collection techniques include interviewing, participantobservation, examination of personal documents and other printed materials; 5. procedures and tools for data gathering are subject to ongoing revision in the field situation; 6. the concern is primarily with discovery and description, although verification is also possible; 7. hypotheses are usually developed during the research, rather than a priori; and 8. analysis is presented for the most part in narrative rather than numerical form. Given this definition, it should be clear that the inclusion of several open-ended questions within an otherwise quantitative research design does not constitute qualitative research (Agar, 1980). At the same time, a qualitative study that incorporates most of the defining characteristics outlined above, does not preclude the inclusion of some quantitative measures and numerical expressions (Benoliel, 1985). The distinction, then, is not dichotomous, but lies along a continuum.

Limitations of Current Evaluation Frameworks

Twenty years ago Glaser and Strauss (1965) questioned the applicability of the "canons of quantitative research" as criteria for judging qualitative inquiry, especially grounded theory. They suggested that ". . . criteria of judgment be based on generic elements of qualitative methods for collecting, analyzing and presenting data and for the way in which people read qualitative analysis."

More recently, Aamodt (1983) re-stated this concern and offered a preliminary set of criteria. She outlines the four dimensions of discovery, assumptions, context and contributions to nursing, and within each demonstrates ways in which qualitative researchers differ from their quantitative colleagues. She herself acknowledges that her attempts at this stage are introductory.

Another partial but useful evaluation framework is that of Le Compte and Goetz (1982), which is an extended discussion of problems of reliability and validity in ethnographic research. While it can serve as a useful guide for evaluating these particular issues in a qualitative study, it is not otherwise comprehensive.

Knafl and Howard (1984) provide a well-articulated discussion of the various purposes of qualitative research and argue that evaluation should be based on the match between purpose and other elements of a study They point out that existing guidelines relevant to qualitative research tend to assume that the author's goal is always to generate theory, whereas instrumentation, illustration, and sensitization may be alternative purposes. Their interest is, however, in research reports rather than proposals, which is the focus of this paper.

Present Framework for Evaluation

While we agree that there is a need for a set of criteria which do not attempt to force qualitative studies into a quantitative framework, we also agree with Agar (1980) and Miles and Huberman (1984) that qualitative researchers must make their work understandable to the "dominant culture" in the scientific community. Our major evaluative categories, then, do not depart greatly from what might be found in any scientific proposal; however, the questions forming the content of each category address the concerns unique to qualitative research. It is also true that in the development of a qualitative research proposal a student should be guided by an experienced researcher, but student proposals are often evaluated by committees which may include members less versed in qualitative research. Making explicit these criteria, and developing a check list to guide the critique parallels the use of similar tools in quantitative proposal reviews.

1. EXPERTISE: Each of the parent disciplines that contributes to qualitative methodology has seminal publications that explicate, critique, expand, or illustrate the process. The nurse researcher who states in a proposal that she is using an "ethnographic approach," for example, should demonstrate an in-depth understanding of what ethnography entails (Agar, 1980). The nurse researcher should convince the reviewer that she or he is aware of the historical origins of the method, the controversies surrounding its use, and how it has come to be modified for use in urban settings as opposed to small, isolated societies. This should be done succintly and clearly. The nurse researcher should also know the goals, demands, and expected outcomes of ethnographic research and whether or not his or her proposal is congruent with them. The nurse researcher must be familiar with and able to critique the work of other nurses who have utilized the method.

It is not sufficient at this point in time for a nurse to say in a proposal that she or he is "using the grounded theory approach" or the "phenomenological method." If the latter, does she or he understand exactly what "bracketing" is, why it is important, and how it must be dealt with in the study? Perhaps one of the things most damaging to the credibility of a qualitative proposal is for the writer to indicate the use of a particular method, but fail to convince the reader that she or he has real expertise in it.

2. PROBLEM AND/OR RESEARCH QUESTION: In several types of qualitative approaches, the assumption is that one does not know the real or relevant questions until one is in the field situation. Broad research questions in a proposal are quite acceptable (Oiler, 1982; Schatzman & Strauss, 1973). However, the researcher needs to be specific enough to convince the evaluator that there is indeed a problem or substantive area of investigation and that a particular qualitative method is likely to be the most productive in obtaining necessary data. There is often the anticipation of modification of a research question as the study proceeds, and although this can be acknowledged in the proposal, it is helpful to include comments on the nature and direction of the expected change. A set of subquestions may also be included relevant to the broader concern. The variation in approaches to stating research questions/problems is illustrated in the following statement by Miles and Huberman (1984):

The formulation of research questions can precede or follow the development of a conceptual framework, but in either case represents the facets of an empirical domain that the researcher most wants to explore. Research questions can be general or particular, descriptive or explanatory. They can be formulated at the outset or later on, and can be refined or reformulated in the course of fieldwork.

No matter how or when the research question is stated, the amount of time allocated for conducting the study must fit the scope of the problem and the realities of the field situation. If the study is to be carried out locally, with a population similar in culture and language to that of the researcher, less time will be needed than if it is conducted in another city or country where gaining entree and maintaining rapport may be more problematic.

3. PURPOSE: A clearly stated purpose is critical to the reader's understanding of qualitative research. For example, providing baseline data for theory building is a much different goal from providing a descriptive base for nursing practice (Knafl & Howard, 1984). Although the same data gathering techniques may be used, the way in which the data are analyzed and presented will be different.

4. LITERATURE REVIEW: Different qualitative approaches vary regarding the question of when a literature review should be conducted and whether a conceptual framework should precede and guide the collection and analysis of data. In phenomenological investigations the literature review is delayed until analysis is completed. This assures that the study is truly grounded in the data. Supporting evidence for hypotheses or conceptual frameworks emerging from the study may then be derived from an examination of the literature. In this approach, an important first stage process is bracketing (Knaack, 1984; Oiler, 1982) or the ability to step back from the normal, natural way of perceiving (Frank, 1981).

Schatzman and Strauss (1973), in outlining the grounded theory method, indicate that a specific theory need not guide the research, although a framework "... no more elaborate than a scheme of general but grounded concepts commonly applied by the discipline . . ." (p. 12) is appropriate. For an ethnography, the literature review may be a significant part of the research proposal in which the writer demonstrates knowledge of previous studies as well as frameworks that have been used in analysis (Robertson & Boyle, 1984). Historical research, on the other hand, has usually begun with a problem and moved directly to data collection, using or developing a conceptual framework only during the final, or synthesization, stage (Christy, 1974). More recently, however, there has been a trend toward using theories or models in the initial phase of historical investigation (Matejski, 1979).

5. CONTEXT: A proposal writer must show that she or he has both informal and official access to a target population as well as requisite skills for operating in a setting, including language or interpreters where necessary. Gaining entree to a particular setting or context is discussed at length in most publications on qualitative research (Agar, 1980; Schatzman & Strauss, 1973; Wax, 1971) since it is prerequisite to the conduct of fieldwork. Similarly, historians must show that they have access to the documents necessary to a given study.

Another significant aspect of context is the researcher's understanding of the use of the self as a research tool and the importance of the researcher-as-learner role (Agar, 1980). Much of data gathering depends upon skillful observing, listening, and communicating - abilities for which nursing perhaps more than any other discipline is well prepared. The researcher-learner, informant-teacher roles are congruent with the goal in qualitative research of understanding social reality from the subject's point of view.

6. SAMPLE: Sampling in qualitative research is different from the randomized sampling of quantitative inquiry. Phenomenological studies digress perhaps to the greatest degree, with selection criteria being simply the subject's experience with a particular phenomenon and the ability to communicate it (Knaack, 1984). The number of people included in this type of investigation may be very limited (Ornery, 1983). Life histories are a type of singlesubject research which may be phenomenological (Frank, 1981), ethnographic (Crapanzano, 1980), or historical. Bogdan and Taylor (1975) offer four recommendations to guide selection of a subject for a life history. These include a willingness to see the project through; ability to verbalize experiences; interest of the investigator; and absence of consanguinai, personal or professional ties between the subject and the investigator.

Group and culture studies often use samples based on convenience and/or special interests of the researcher. Agar (1980) calls these, respectively, "opportunistic samples," and "judgmental samples," and point out that it is not only people, but also events and processes that are sampled (Miles & Huberman, 1984). The proposal writer should be aware of these differences, and be able to address the reasons why sampling in a qualitative study is purposive rather than random. Further, while it is not always possible to define and select a sample in advance, the researcher might identify potential types of persons, events, or processes to be sampled. Schatzman and Strauss (1973 ) suggest "casing" and "mapping" a field site as a way to begin identifying a sample.

7. DATA COLLECTION: Phenomenological researchers caution against an overly strict adherence to specific methods of data collection, preferring them to be "openended and suggestive." However, participant-observation, interviewing, and written or verbal self-reports are all possibilities (Knaack, 1984). Ethnography and grounded theory depend heavily on participant-observation and interviewing, while historical research utilizes both personal and public documents, and sometimes interviews. There are numerous publications discussing these techniques (Bogdan & "Taylor, 1975; Langness & Frank, 1980; Schatzman & Strauss, 1973; Spradley, 1979; Taylor & Bogdan, 1984; Wax, 1971), and a well-written qualitative proposal should demonstrate familiarity with key methodological references.

During the pre-fieldwork phase of qualitative research, the investigator needs to establish what the procedures for data collection will be, including how much instrumentation will be devleoped and when. There are arguments pro and con for designing instrumentation prior to fieldwork, and Miles and Huberman (1984) summarize several. Agar (1980) sees an ethnography as moving from informal to more structured, and while it may not be possible to specify instrumentation plans in detail in a proposal, the researcher can at least indicate what general strategies, such as surveys and event sampling, might be used in the later stages of the study. This requires a demonstration of skills relevant to these methods.

Table

FIGURERESEARCH PROPOSAL EVALUATION FORM QUALITATIVE METHODOLOGY

FIGURE

RESEARCH PROPOSAL EVALUATION FORM QUALITATIVE METHODOLOGY

Field notes are crucial to the data collection phase of any qualitative study. They should be transcribed at the earliest opportunity, categorized, coded, and easily retrieved. The writer should outline a tentative system for keeping data organized and accessible. Tape recording of interviews may be most appropriate in some situations, while on-the-spot note taking may be superior where affect and non-verbal communication are of importance. The investigator should provide a rationale for using one or the other, or both.

A qualitative research proposal should address problems of reliability and validity peculiar to this approach. Le Compte and Goetz (1982) examine these questions in depth and suggest that while absolute validity and reliability are impossible in any research model, the ethnographer can employ a number of techniques to reduce undesirable effects dring both data collection and analysis. Most of their suggestions are appropriate to phenomenological study and grounded theory as well. Historiographers use the method of external criticism to determine the reliability of the information found within the document itself (Christy, 1975).

8. DATA PROCESSING AND PLANS FOR ANALYSIS: A key characteristic of qualitative research of all types is the continuous, ongoing analysis of the data. The "constant comparative method" (Glaser, 1965) so often cited is not the only technique for accomplishing this. Other grounded theorists (Schatzman & Strauss, 1973) discuss the craftmanship involved in processing and analyzing data, and note that the critical operation is the discover of significant classes, their properties, and the links among them. Ethnographers constantly process data by not only recording observations and interviews, but also by making notations about ideas for follow-up, writing preliminary explanations (hypotheses), and searching for patterns. Miles and Huberman (1984) present an entire chapter on processes of analysis during data collection including memoing and summary sheets.

Coding is one means of processing data. Most qualitative studies use code words derived directly from the data, which may be noted in pencil in the margins of field notes and interviews. The codes change as categories are elaborated or collapsed. Some researchers, however, may have a preliminary coding scheme derived from a conceptual framework, or from a formal coding scheme such as Murdock's (1950) Ethnographic Atlas. If a formal coding scheme is to be used, the writer should present it in the proposal and indicate under what circumstances it will be modified. Further, the researcher should discuss whether or not an attempt will be made to increase reliability by double coding (Miles & Huberman, 1984).

Bogdan and Taylor (1975) suggest there is no precise formula that will enable the researcher to construct hypotheses and to recognize themes. If an analytical framework is to be derived directly from the data, the proposal should indicate this, and the writer should show familiarity with similar processes used in other studies. If a conceptual framework has guided the development of the research questions, the writer should show how the framework will inform the analysis phase of the study.

9. HUMAN SUBJECTS: The protection of human subjects poses unique questions in certain kinds of qualitative research. Federal guidelines were originally written to guarantee informed consent of participants in biomedical and experimental psychological research (Wax, 1977) and, although the regulations have become more flexible, many institutions continue to keep close watch on any kind of research involving human subjects. A study using participant-observation raises the issue of who in the field situation must be informed that a study is in process (Agar, 1980). The proposal writer should discuss how the research will be explicated in accord with local institutional requirements, and include examples of contracts and/or consent forms for interviews.

10. IMPORTANCE TO NURSING: Any nurse who is writing a qualitative proposal will be concerned with the potential contribution of the study to nursing. This may be discussed in relation to the original statement or purpose (e.g. theory-building, description, etc.), and the problem or substantive area under investigation.

Summary and Conclusions

We have outlined ten criteria for the evaluation of qualitative research proposals, and have presented these in a check list format (Figure). While it has not been possible to describe in detail all of the different qualitative approaches, we have tried to summarize general points which apply to most types of qualitative research, and to point out any significant differences among them in relation to the ten criteria. This effort is one attempt to respond to the problem of the inappropriate use of quantitative criteria to evaluate qualitative research proposal.

References

  • Aamodt, A. (1983). Problems in doing nursing research: Developing a criteria for evaluating qualitative research. Western Journal of Nursing Research, 5(4), 399.
  • Agar, M. (1980). The professional stranger: An informal introduction to ethnography. New York, Academic Press.
  • Benoliel, J. (1984). Advancing nursing science: Qualitative approaches. Community Nursing Research, 17, 1-8.
  • Bogdan, R. & Taylor, S. (1975). Introduction to qualitative research methods. New York, John Wiley and Sons.
  • Bruyn, S. (1966). The human perspective. Englewood Cliffs, NJ, Prentice-Hall.
  • Christy, T. (1975). The methodology of historical research: A brief introduction. Nursing Research, 24, 189-192.
  • Crapanzano, V. (1980). Tuhami: Portrait of a Moroccan. Chicago, University of Chicago Press.
  • Duffey, M. & Muhlenkamp, A. (1974). A framework for theory analysis. Nursing Outlook, 22(9), 571.
  • Glaser, B. (1965). The constant comparative method of qualitative analysis. Social Problems, 12, 436-445. Reprinted in McCaIl, G. & Simmons, J. (Eds.). Issues in participant observation: A text and reader. Reading, Mass, Addison-Wesley Publishing Company (pp. 216-227).
  • Glaser, B. & Strauss, A (1965). The discovery of substantive theory: A basic strategy undelying qualitative research. The American Behavioral Scientist, 8(6), 5-12.
  • Goodwin, L. & Goodwin, W. (1984). Qualitative vs. quantitative research or qualitative and quantitiative research? Nursing Research, 33(6), 379.
  • Holm, K. (1983). Single subject research. Nursing Research, 32(4), 253.
  • Kerlinger, F. (1973). Foundations of behavioral research, New York, Holt, Rinehart, & Winston, Inc.
  • Knaack, P. (1984). Phenomenological research. Western Journal of Nursing Research, 6(1), 107-114.
  • Knafl, K. (1984). Interpreting and reporting qualitative research. Research in Nursing and Health, 7, 18.
  • Langness, L. & Frank, G. (1981). Lives - an anthropological approach to biography, Novate, CA, Chandler & Sharp Publishers.
  • LeCompte, M. & Goetz, J. (1982). Problems of reliability and validity in ethnographic research. Review of Educational Research, 52(1), 31-60.
  • Leininger, M. (1985). Nature, rationale, and importance of qualitative research methods in nursing. In Leininger, M. (ed.) Qualitative Research Methods in Nursing (p. 2). New York, Grune & Stratton.
  • Lewis, F. & Foltz, A. (1981). Reviewed research abstracts for the sixth congress: The process, a critique and future directions. Oncology Nursing Forum, 8(3), 75-78.
  • Mandelbaum, D. (1973). The study of life history: Gandhi. Current Anthropology, 14(3), 177.
  • Matejski, M. (1979). Humanities: The nurse and historical research. Image, 11(3), 80-85.
  • Miles, M. & Huberman, A (1984). Qualitative data analysis: A sourcebook of new methods. Beverly Hills, CA Sage Publications.
  • Murdock, G. (1950). Outline of cultural materials. New Haven, CT, Yale University Press.
  • Nunnally, J (1978). Psychometric theory (pp. 86-113). New York, McGraw-Hill Book Company.
  • Oiler, C. (1982). The phenomenological approach in nursing research. Nurisng Research, 31(3), 178-181.
  • Omery, A. (1983). Phenomenology: A method for nursing research. Advances in Nursing Science. 49-63.
  • Reichardt, C. & Cook, T (1979). Beyond qualitative versus quantitative methods. In Cook, T. & Reichardt, C. (Eds.), Qualitative and quantitative methods in evaluation research, (pp. 7-32). Beverly Hills, CA, Sage Publications.
  • Robertson, M. & Boyle, J. (1984). Ethnography: Contributions to nursing research. Journal of Advanced Nursing, 9, 43-49.
  • Schatzman, L. & Strauss, A. (1973). Field Research: Strategies for a natural sociology. Englewood Cliffs, NJ: Prentice-Hall Methods of Social Science Series.
  • Spradley, J. (1979). The ethnographic interview (p. 59). New York, Holt, Rinehart & Winston.
  • Taylor, S. & Bogdan, R. (1984). Introduction to qualitative research methods. Second edition. New York, John Wiley & Sons.
  • Wax, M. (1977). On fieldworkers and those exposed to fieldwork. Human Organization, 36(3), 321-328.
  • Wax, R. (1971). Doing fieldwork: Warnings and advice. Chicago, University of Chicago Press.
  • Wiseman, J. (1974). The research web. Urban Life and Culture, 3(3), 325-326.

FIGURE

RESEARCH PROPOSAL EVALUATION FORM QUALITATIVE METHODOLOGY

10.3928/0148-4834-19870401-04

Sign up to receive

Journal E-contents