Journal of Nursing Education

Methodology Corner 

The Nominal Group Technique: Generating Consensus in Nursing Research

Simon Cooper, PhD, RN; Robyn Cant, PhD; Elise Luders, MMid, RN, RM; Donna Waters, PhD, RN; Amanda Henderson, PhD, RN; Kerry Hood, PhD, RN; Kerry Reid-Searl, PhD, RN, RM; Colleen Ryan, MHPE, RN; Marion Tower, PhD, RN; Georgina Willetts, MEd, RN, RM

Abstract

The purpose of this article is to describe the Nominal Group Technique and its application as a consensus-generating approach in nursing research. The approach incorporates face-to-face meetings to explore opinions, generate ideas, and determine priorities. The nominal group technique process, which is based on a study designed to develop a nursing student clinical placement (clinical practicum) evaluation tool, is described. Advantages of the approach include creative face-to-face discussions with minimal resource demands. The nominal group technique is beneficial and can be used to achieve consensus in nursing research, but a lack of anonymity may preclude the process in some investigations. [J Nurs Educ. 2020;59(2):65–67.]

Abstract

The purpose of this article is to describe the Nominal Group Technique and its application as a consensus-generating approach in nursing research. The approach incorporates face-to-face meetings to explore opinions, generate ideas, and determine priorities. The nominal group technique process, which is based on a study designed to develop a nursing student clinical placement (clinical practicum) evaluation tool, is described. Advantages of the approach include creative face-to-face discussions with minimal resource demands. The nominal group technique is beneficial and can be used to achieve consensus in nursing research, but a lack of anonymity may preclude the process in some investigations. [J Nurs Educ. 2020;59(2):65–67.]

The Nominal Group Technique (NGT) is designed to explore opinions, generate ideas, and determine priorities (McMillan, King, & Tully, 2016). Although the method is well recognized across medicine and in psychology, the approach is unusual in the nursing discipline. Early nursing research from Chapple and Murphy (1996) evaluated teaching and learning experiences using this technique, and more contemporary work from Harvey and Holmes (2012) used clinical experts to examine triage in the emergency department.

The Delphi technique is an alternative consensus-generating approach (Foth et al., 2016) often used for guideline developments involving expert panel members. However, the Delphi approach most commonly followed circulates a questionnaire anonymously among members, as opposed to face-to-face meetings in nominal groups where a greater exploration of the field of focus is possible (McMillan et al., 2016). In comparison, focus groups are a broader discussion relating to a concern without the requirement to reach consensus.

This article describes the use of the NGT, incorporating an illustrative example from current work in which we aimed to develop a nursing student clinical placement (clinical practicum) evaluation tool.

Background

The NGT was originally developed in the 1960s for group analyses, decision making, and evaluation in the aerospace and industrial fields and later in education, policy, and health care areas (e.g., Van de Ven, 1974). The technique involves face-to-face activities and discussion in small stakeholder groups ranging from two to 14 members. The technique derives its name from the fact that the group is in name only, as individuals work alone for much of the process, with facilitation by a group leader (Chapple & Murphy, 1996). The recommended group size is a maximum of seven members who meet over a period of 1.5 to 3 hours (McMillan et al., 2016).

The NGT was highly applicable to our current work in developing a clinical placement evaluation tool that rated nursing students' experiences. Nominal groups should include those with relevant lived experience (Sanders, 2008); in the case of clinical placements, this was students, educators, and supervisors. It was important to involve all stakeholders as active codesign partners to generate ideas, suggest prototypes, gather feedback, and change the outcomes.

Delbecq and Van de Ven (1971) described a traditional approach to the NGT as a six-stage meeting consultation. These were:

  • Individual generation of ideas.
  • Recording of all participants' ideas (in a round-robin format).
  • Group discussion of all generated ideas (to organize the list and remove duplications).
  • A preliminary vote to select the most important ideas.
  • Group discussion relating to the vote outcomes (including additions and further merging of overlaps).
  • Final individual voting on the priority of items.

The following section describes the application of the NGT to the consensus development of a clinical placement evaluation tool in order to demonstrate the utility of the process.

The NGT Method

We applied the NGT in a study that aimed to develop a nursing student clinical placement evaluation tool with strong evidence of validity and reliability. We expanded the process into a codesign approach, an important feature as we wanted to increase engagement, sustainability, and transferability of the resulting Placement Evaluation Tool (PET). The core project design steps were as follows:

Step 1: Preview

A review of the literature was conducted to identify existing placement evaluation tools. Ten tools were identified, with a total of 188 distinct question items. An expert panel of six clinical nurse academics then ranked all items for relevance and clarity for an Individual-Content Validity Index (I-CVI) (Polit & Beck, 2006). Items that did not reach acceptable levels on the index < 0.78 were excluded.

Step 2: Consultation

NGT meetings were held in two Australian university nursing schools to generate additional question items. Participants included nursing students from all year levels, university lecturers, and clinical nurse educators from the health care sector (n = 20). In each meeting, a priority list of potential questions was generated, based on group consensus. The meeting procedure is described in Table A (available in the online version of this article), and Table B (available in the online version of this article) provides examples of generated items.

An Example of the Nominal Group Technique: The Face-to-Face Meeting Procedure in the Development of the Placement Evaluation Tool Project

Table A:

An Example of the Nominal Group Technique: The Face-to-Face Meeting Procedure in the Development of the Placement Evaluation Tool Project

Nominal Group Meeting Outcome: Examples of Generated Items Ranked From One Face-to-Face Meeting in the PET Project (N = 11 Participants)

Table B:

Nominal Group Meeting Outcome: Examples of Generated Items Ranked From One Face-to-Face Meeting in the PET Project (N = 11 Participants)

Step 3: Questionnaire Development

The first author (S.C.) undertook an independent thematic analysis of the ranked questions (from step 2) to develop a preliminary tool. This was followed by a 5-hour meeting with four of the clinical academics to select, adapt, and thematize items using a tabletop approach, based on all items identified as important. The preliminary tool was used as a reference point and cross-check. Individual items were grouped under key factors from prior studies: culture of clinical environment, supervision, learning, and outcomes. Items were critiqued (reworded if necessary) to generate a 20-item questionnaire. A 5-point Likert scale based on agreement was added. Two additional quality checks were subsequently instituted. The first was a content validity analysis by 10 clinical educators from three Australian states and 12 nursing students from two states in order to calculate the I-CVI prior to final selection. The expected I-CVI of > 0.78 was exceeded in all but three ratings of relevance or clarity, which were resolved with minor changes to wording. Second, the draft tool was presented to a meeting of 37 deans of nursing (Australia and New Zealand), with minor modifications made.

Step 4: Student Survey

The resulting version of the PET is being pilot tested in an online survey of nursing students at six universities and one college in eastern Australia (July through December 2019). Students across all year levels have been invited to rate their most recent clinical placement and provide feedback on the tool itself. The results will allow us to confirm validity, reliability, and feasibility of the PET.

Discussion

Consensus-generating approaches are used across a range of disciplines to explore opinions, generate ideas, and determine priorities. The NGT face-to-face format enables deep and insightful explorations of the field of interest. The approach should not be used in isolation, however, as it is just one method of generating data with strong evidence of validity and reliability (Foth et al., 2016). In the clinical placement study described, data were triangulated from a literature review and two nominal group meetings, together with surveys of content validity, expert review, and a pilot trial.

There are a range of considerations when planning a nominal group approach, not least is the feasibility of face-to-face meetings and the validity of the approach. The level of experience and expertise of participants, as opposed to the number, is thought to improve the validity of the data generated (Potter, Gordon, & Hamer, 2004). In our study, two meetings were incorporated, and a convenience sample of 20 students and educators was selected. A purposeful selection of participants may improve representativeness.

Further, to ensure consensus, the facilitator is tasked with leading the group discussion while taking an unbiased approach and posing open questions to reduce their influence (Black et al., 1999). Nominal group outcomes should also be verified, for example by presenting results to additional participants or by comparison with parallel studies.

Achieving Consensus

The degree and form of consensus achieved within a nominal group is also important. Consensus refers to the relative ranking and weighting of ranked items, not to the degree to which all participants agree with all the ranked items. Ranking and rating can be achieved in a variety of ways which may include the strength of the cumulative vote score (Sink, 1983) and/or the number of votes per item. The strength of the cumulative vote could be calculated by asking participants to allocate a number to each item with larger numbers indicating greater importance (McMillan et al., 2014). For example, if the aim is to identify the top six items, a six would be a participant's highest ranking. Alternatively, it may be more intuitive for participants to rank items from one to six, with one as the most important (Table B).

In the clinical placement study, we asked participants to identify the top 10 items, which, in retrospect, proved challenging and was unnecessary. Ranking of up to five ideas is likely to be more practical and is common in the literature (Dening, Jones, & Sampson, 2013). The number of votes given per item (i.e., the overall popularity) should also be examined and may influence priority rankings if the scores are identical (McMillan et al., 2014). As these authors suggest, we incorporated more than one analyses approach to ensure participants' priorities were reflected and that high scores alone were balanced against voting frequency. To achieve this, we selected the top ranked items based on selection by one third or more of participants and the mean group ranks. However, the process cannot be purely quantitative, and later stage discussions with participants/experts may rightly change the priority, inclusion criteria, and cut points.

Advantages and Disadvantages of the NGT

The main advantage of the NGT over other consensus generating approaches is the opportunity for rich, all-encompassing creative face-to-face discussions where participants can openly articulate their ideas. Particular strengths of the approach include the ability to debate, challenge, and discuss viewpoints from the participants' standpoints to enable the development of ideas based on personal experience. The commencing stage—individual generation of ideas—ensures that group conformity may be avoided; the round robin stage ensures that all opinions are considered and recorded; and final stages enable individual voting on priorities (Chapple & Murphy, 1996).

Further, the process is minimally resource intensive with efficient generation of results. Participants can gain a sense of accomplishment as results are presented back to them and the attrition rate is lower than for a Delphi process (Foth et al., 2016). Regardless, however, there is a potential for bias if the project design lacks rigor including a prearranged number of rounds and number of participants, appropriate feedback procedures, and valid consensus on the outcomes (Foth et al., 2016).

Disadvantages include the lack of anonymity, the potential for individuals to dominate discussions (either peers or the group facilitator), and a possible loss of interest and reduced concentration span over time. Depending on the representativeness and quality of data, there is also potential disadvantages to the face-to-face time commitment, the lack of reflection time, and the likelihood that the nominal group process alone may not generate outcomes that are generalizable (Chapple & Murphy, 1996).

Conclusion

The NGT is a consensus-generating approach that can enable priorities, ideas, and opinions to be systematically explored among a small group of participants. The technique requires rigorous design with several stages to filter information and achieve consensus. Such creative face-to-face discussions using minimal resources are beneficial and can be used effectively for research in nursing and in other disciplines. The lack of anonymity may preclude the process in some investigations.

References

  • Black, N., Murphy, M., Lamping, D., McKee, M., Sanderson, C., Askham, J. & Marteau, T. (1999). Consensus development methods: A review of best practice in creating clinical guidelines. Journal of Health Services Research & Policy, 4(4), 236–248. doi:10.1177/135581969900400410 [CrossRef]
  • Chapple, M. & Murphy, R. (1996). The nominal group technique: Extending the evaluation of students' teaching and learning experiences. Assessment & Evaluation in Higher Education, 21(2), 147–160. doi:10.1080/0260293960210204 [CrossRef]
  • Delbecq, A. L. & Van de Ven, A. H. (1971). A group process model for problem identification and program planning. Journal of Applied Behavioral Science, 7(4), 466–492.
  • Dening, K. H., Jones, L. & Sampson, E. L. (2013). Preferences for end-of-life care: A nominal group study of people with dementia and their family carers. Palliative Medicine, 27(5), 409–417. doi:10.1177/0269216312464094 [CrossRef]
  • Foth, T., Efstathiou, N., Vanderspank-Wright, B., Ufholz, L.-A., Dütthorn, N., Zimansky, M. & Humphrey-Murto, S. (2016). The use of Delphi and Nominal Group Technique in nursing education: A review. International Journal of Nursing Studies, 60, 112–120. doi:10.1016/j.ijnurstu.2016.04.015 [CrossRef]
  • Harvey, N. & Holmes, C. A. (2012). Nominal group technique: An effective method for obtaining group consensus. International Journal of Nursing Practice, 18(2), 188–194. doi:10.1111/j.1440-172X.2012.02017.x [CrossRef]
  • McMillan, S. S., Kelly, F., Sav, A., Kendall, E., King, M. A., Whitty, J. A. & Wheeler, A. J. (2014). Using the nominal group technique: How to analyse across multiple groups. Health Services and Outcomes Research Methodology, 14(3), 92–108 doi:10.1007/s10742-014-0121-1 [CrossRef]
  • McMillan, S. S., King, M. & Tully, M. P. (2016). How to use the nominal group and Delphi techniques. International Journal of Clinical Pharmacy, 38(3), 655–662.
  • Polit, D. F. & Beck, C. T. (2006). The content validity index: Are you sure you know what's being reported? Critique and recommendations. Research in Nursing & Health, 29(5), 489–497.
  • Potter, M., Gordon, S. & Hamer, P. (2004). The nominal group technique: A useful consensus methodology in physiotherapy research. New Zealand Journal of Physiotherapy, 32, 126–130.
  • Sanders, L. (2008). An evolving map of design practice and design research. Interactions, 15(6), 13–17.
  • Sink, D. S. (1983). Using the nominal group technique effectively. National Productivity Review, 2(2), 173–184. doi:10.1002/npr.4040020209 [CrossRef]
  • Van de Ven, A. H. (1974). Group decision making and effectiveness: An experimental study. Kent State University Press.

An Example of the Nominal Group Technique: The Face-to-Face Meeting Procedure in the Development of the Placement Evaluation Tool Project

Time Allowed
1. Introduction to the project aim and the Nominal Group Technique process.

Describe the aim and objectives of the meeting:

‘To build consensus as to the content of a clinical placement evaluation questionnaire’ with the objective of ‘generating a list of items that broadly measure the quality of a clinical placement (primary and acute care) from a student's perspective’.

Facilitator generated stories of placement experiences with photo illustrations.

Encourage participant placement experience stories.

30 mins
2. Silent generation of potential survey evaluation questionsIndividually composed and formatted in writing on cue cards.20 mins
3. Round robin listing of items without discussion or clarification. (Recorded by a scribe on flip-chart paper. Participants asked not to engage in side conversations etc)

Participants to read a single response from one of their cue cards (without providing a rationale or relating it to other participants' responses).

Items are not repeated unless a group member identified that it generated a new foci/emphasis.

Continue until all participants complete presenting their questions.

30 mins
4. Group discussion and clarification of items.

A brief discussion regarding the nominated responses for the purpose of clarification (not evaluation) ensuring each response is understood from a common perspective.

Removal of duplicates following additional discussion.

15 mins
5. Rank and order items without discussion.(Note alternat ive/additional approaches to ranking are available – see end of table and discussion)

Participants to individually select 10 items from the generated list that they felt were the most important.

Score by ranking from 1–10 (1 as the most important, 10 as least important)

Participants provided with post-it notes with number 1–10 printed on them enabling them to rank/label their preferred items on the paper flip charts. Performed in silence.

Sum the ranks – the total score (the lowest numbers equating to the most important), the mean score for each item and the number who voted for each item.

Data management – Use a spread-sheet to record the items and ranked totals for each item.

60 mins
6. Provisional items presented to the group based on group rankings*.

Review and discuss the final listings

*note that final discussions may need to be based on initial analyses of rankings as time limits may preclude detailed calculations.
25 mins
180 mins
Alternative/additional singular approaches to ranking

Participants to individually allocate a number to each selected item with larger numbers indicating greater importance e.g. for five ideas the most important idea scores five.

Calculate the mean score for each item i.e. the total score divided by the number of votes.

Consider the voting frequency i.e. the number who voted for each item. Consider using a cut point such as prioritising items where one third or more of participants had rated an item.

Nominal Group Meeting Outcome: Examples of Generated Items Ranked From One Face-to-Face Meeting in the PET Project (N = 11 Participants)

Question ItemRANK Mean ScoreTotal Score (n of raters)
Items ranked as important (ranked by 4 or more participants)
I had opportunities to extend my learning1.56 (4)
My overall experience was positive3.715 (4)
The staff were willing to work with students5.734 (6)
I received constructive feedback that helped improve my practice6.024 (4)
I was able to engage with my facilitator on a regular basis6.024 (4)
I felt comprehensively orientated to the clinical area7.028 (4)
I felt supported to work in my scope of practice7.731 (4)
………………and so on

Items of lesser priority (ranked by 3 or less participants)
I felt I was an effective member of the ward teamN/A3 (2)
I feel like students were valued within the placementN/A5 (1)
I was able to pursue learning opportunities as they aroseN/A5 (1)
…………….and so on

Items that did not receive any ranking
I completed the expected clinical skills00
I would speak highly of this placement00
I want to apply to new grad program00
…………. and so on
Authors

Dr. Cooper is Associate Dean, Research, School of Nursing and Health Professions, Dr. Cant is Associate Professor, Ms. Luders is Lecturer, School of Nursing and Health Professions, Federation University Australia, Churchill, Victoria; Dr. Waters is Professor, Head of School and Dean, The University of Sydney, Susan Wakil School of Nursing and Midwifery, Camperdown, New South Wales; Dr. Henderson is Associate Professor, Acting Head of School, School of Nursing Midwifery and Paramedicine, University of the Sunshine Coast, Maroochydore, Queensland; Dr. Hood is Head of School, Nursing at Holmesglen Institute, Melbourne, Victoria; Dr. Reid-Searl is Deputy Dean Simulation, School of Nursing, Midwifery and Social Sciences, CQUniversity, Rockhampton, Central Queensland; Ms. Ryan is Industry Liaison Educator, School of Nursing, Midwifery and Social Sciences, CQUniversity Australia, Noosa Campus, Noosaville, Queensland; Dr. Tower is Director, Undergraduate and Pre-registration Nursing & Midwifery, School of Nursing, Midwifery & Social Work, University of Queensland, St. Lucia, Queensland; and Ms. Willetts is Associate Professor, Head of Discipline & Course Director in Nursing, Department of Health Professions Faculty of Health, Arts and Design, Swinburne University, Hawthorn, Victoria, Australia.

The authors have disclosed no potential conflicts of interest, financial or otherwise.

The authors thank Adele Callaghan, RN, PhD candidate, for ideas that helped to generate this study. This research received funding from the Council of Deans of Nursing and Midwifery (Australia and New Zealand).

Address correspondence to Simon Cooper, PhD, RN, Associate Dean, Research, School of Nursing and Health Professions, Federation University Australia, Room 2W-249, Gippsland Campus, Churchill, VIC 3842, Australia; e-mail: s.cooper@federation.edu.au.

10.3928/01484834-20200122-02

Sign up to receive

Journal E-contents