The Basics of the Delphi Method.

The Delphi method is an exercise in group communication among a panel of geographically dispersed experts (Adler and Ziglio, 1996). The technique allows experts to deal systematically with a complex problem or task. The essence of the technique is fairly straightforward. It comprises a series of questionnaires sent either by mail or via computerized systems, to a pre-selected group of experts. These questionnaires are designed to elicit and develop individual responses to the problems posed and to enable the experts to refine their views as the groups work progresses in accordance with the assigned task. The main point behind the Delphi method is to overcome the disadvantages of conventional committee action. According to Fowles (1978) anonymity, controlled feedback, and statistical response characterize Delphi. The group interaction in Delphi is anonymous, in the sense that comments, forecasts, and the like are not identified as to their originator but are presented to the group in such a way as to suppress any identification.

In the original Delphi process, the key elements were (1) structuring of information flow, (2) feedback to the participants, and (3) anonymity for the participants. Clearly, these characteristics may offer distinct advantages over the conventional face-to-face conference as a communication tool. The interactions among panel members are controlled by a panel director or monitor who filters out material not related to the purpose of the group (Martino, 1978). The usual problems of group dynamics are thus completely bypassed. Fowles (1978) describes the following ten steps for the Delphi method:

1. Formation of a team to undertake and monitor a Delphi on a given subject. 2. Selection of one or more panels to participate in the exercise. Customarily, the panelists are experts in the area to be investigated. 3. Development of the first round Delphi questionnaire 4. Testing the questionnaire for proper wording (e.g., ambiguities, vagueness) 5. Transmission of the first questionnaires to the panelists 6. Analysis of the first round responses 7. Preparation of the second round questionnaires (and possible testing) 8. Transmission of the second round questionnaires to the panelists 9. Analysis of the second round responses (Steps 7 to 9 are reiterated as long as desired or necessary to achieve stability in the results.)

10. Preparation of a report by the analysis team to present the conclusions of the exercise

Delbecq et al., (1975) argue that the most important issue in this process is the understanding of the aim of the Delphi exercise by all participants. Otherwise the panelists may answer inappropriately or become frustrated and lose interest. The respondents to the questionnaire should be well informed in the appropriate area (Hanson and Ramani, 1988) but the literature (Armstrong, 1978; Welty, 1972) suggest that a high degree of expertise is not necessary. The minimum number of participants to ensure a good group performance is somewhat dependent on the study design. Experiments by Brockhoff (1975) suggest that under ideal circumstances, groups as small as four can perform well.

Before deciding whether or not the Delphi method should be used, it is very important to consider thoroughly the context within which the method is to be applied (Delbecq et al. 1975). A number of questions need to be asked before making the decision of selecting or ruling out the Delphi technique (Adler and Ziglio, 1996):

What kind of group communication process is desirable in order to explore the problem at hand? Who are the people with expertise on the problem and where are they located? What are the alternative techniques available and what results can reasonably be expected from their application?

Only when the above questions are answered can one decide whether the Delphi method is appropriate to the context in which it will be applied. Adler and Ziglio (1996) further claim that failure to address the above questions may lead to inappropriate applications of Delphi and discredit the whole creative effort.

The outcome of a Delphi sequence is nothing but opinion. The results of the sequence are only as valid as the opinions of the experts who made up the panel (Martino, 1978). The panel viewpoint is summarized statistically rather than in terms of a majority vote.

The Delphi method has got criticism as well as support. The most extensive critique of the Delphi method was made by Sackman (1974) who criticizes the method as being unscientific and Armstrong (1978) who has written critically of its accuracy. Martino (1978) underlines the fact

that Delphi is a method of last resort in dealing with extremely complex problems for which there are no adequate models. Helmer (1977) states that sometimes reliance on intuitive judgement is not just a temporary expedient but in fact a mandatory requirement. Makridakis and Wheelright (1978) summarize the general complaints against the Delphi method in terms of (a) a low level reliability of judgements among experts and therefore dependency of forecasts on the particular judges selected; (b) the sensitivity of results to ambiguity in the questionnaire that is used for data collection in each round; and (c) the difficulty in assessing the degree of expertise incorporated into the forecast. Martino (1978) lists major concerns about the Delphi method:

Discounting the future: Future (and past) happenings are not as important as the current ones, therefore one may have a tendency to discount the future events. The simplification urge: Experts tend to judge the future of events in isolation from other developments. A holistic view of future events where change has had a pervasive influence cannot be visualized easily. At this point cross-impact analysis is of some help. Illusory expertise: some of the experts may be poor forecasters. The expert tends to be a specialist and thus views the forecast in a setting which is not the most appropriate one. Sloppy execution: there are many ways to do a poor job. Execution of the Delphi process may loose the required attention easily. Format bias: it should be recognized that the format of the questionnaire may be unsuitable to some potential societal participants. Manipulation of Delphi: The responses can be altered by the monitors in the hope of moving the next round responses in a desired direction.

Goldschmidt (1975) agrees that there have been many poorly conducted Delphi projects. However, he warns that it is a fundamental mistake to equate the applications of the Delphi method with the Delphi method itself, as too many critics do. There is, in fact, an important conceptual distinction between evaluating a technique and evaluating an application of a technique.

On the other hand there have been several studies (Ament, 1970; Wissema, 1982; Helmer, 1983) supporting the Delphi method. A study conducted by Milkovich et al. (1972) reports the use of the Delphi method in manpower forecasting. The results of the comparison indicated high agreement between the Delphi estimate and the actual number hired and less agreement between quantitative forecasts and the number hired. Another study by Basu and Schroeder (1977)

reports similar results in a general forecasting problem. They compared Delphi forecasts of fiveyear sales with both unstructured, subjective forecasts and quantitative forecasts that used regression analyses and exponential smoothing. The Delphi forecasting consisted of three rounds using 23 key organization members. When compared against actual sales for the first two years, errors of 3-4% were reported for Delphi, 10-15% for the quantitative methods, and of approximately 20% for the previously used unstructured, subjective forecasts.

In general, the Delphi method is useful in answering one, specific, single-dimension question. There is less support for its use to determine complex forecasts concerning multiple factors. Such complex model building is more appropriate for quantitative models with Delphi results serving as inputs (Gatewood and Gatewood, 1983). This point is supported by Gordon and Hayward (1968) who claim that the Delphi method, based on the collation of expert judgement, suffers from the possibility that reactions between forecasted items may not be fully considered. The need for the cross impact matrix method of forecasting integrated with the Delphi method is pointed out by many researchers (Gordon and Hayward, 1968; Gatewood and Gatewood, 1983; Adler and Ziglio, 1996). An improvement in forecasting reliability over the Delphi method was thought to be attainable by taking into consideration the possibility that the occurrence of one event may cause an increase or decrease in the probability of occurrence of other events included in the survey (Helmer, 1978). Therefore cross impact analysis has developed as an extension of Delphi techniques.

Technological Forecasting Judgment-Based Technological Forecasting Techniques Delphi Method Perhaps the best known of the various judgmental approaches to technological forecasting, the Delphi method uses a panel of individuals who make anonymous, subjective judgments about the probable time when a specific technological capability will be available. The results of these estimates are aggregated by a process administrator and fed back to the group, which then uses the feedback to generate another round of judgments. After several iterations, the process is stopped and areas of agreement or disagreement are noted and documented. Let us now look more closely at some characteristics of this process.

1. Opinion Gathering and Distribution The key features of the Delphi process are quite simple. Panelists may submit their judgments by mail or may be gathered together in a single room. In either case, the opinions are written and anonymous. if the panelists are in a face-toface meeting, no discussion of the subjects to be covered is allowed. In this way, loud or aggressive panelists cannot sway the votes of others. The written ballots are collected by the process administrator, who aggregates the responses in a statistical format (i.e., prepares a distribution of the responses). This information is given to all panel members, who can then see how their individual judgments compare with the anonymous views of others on the panel. 2. Iterative Balloting The process is iterative. Additional ballots are passed out to the panel members and they vote again, altering or not altering their previous judgments as they wish. Panelists who found their original vote in the outer reaches of the distribution may, if they feel uncertain about the matter, move their vote toward the majority. There is no pressure on them to do so, however, and those who feel sure about their original vote may choose not to move it at all. 3. Reasons and Consensus This iterative process is sometimes accompanied by anonymous written arguments concerning why some specific judgment is correct or incorrect. The process continues until a consensus is reached (for example, some proportion of respondents-say, 75 percent-cast votes within a predetermined range, such as a 20-year period) or until a predetermined number of iterations has been accomplished, usually four or live. If consensus is reached, a statistical measure of the result, usually the median or mode, is used to represent the actual forecast. if consensus is not reached, the distribution of the final iteration is often displayed with a note that it does not represent a consensus. 4. Group Composition The composition of the group depends on the nature of the capabilities to be forecast, If they are general and abstract, a heterogeneous group is desirable. if highly technical or specific, then specialists in that area and generalists in outside but relevant areas should comprise the group. Attempts should be made to reach a balance between specialists and generalists, and between theoreticians and pragmatists. With a homogeneous group, 10 to 15 persons will probably be adequate. For a heterogeneous group, more may be necessary for representation, but numbers are not the critical factor. The Delphi method has application beyond its use in technological forecasting. It has been widely used as an aid in policy decision making (e.g., [31]). Its three main characteristics-anonymity, statistical formatting of results, and controlled feedback-make it an acceptable and reliable process for extracting numeric data from subjective opinion. Also, no limits are placed on the factors a Delphi panelist may consider in deciding how to vote. As a result, the process is particularly effective when opinions and judgments must be based on a broad and complex set of underlying factors.

It is most relevant, at this point, to note that the judgmental forecasting methods, like managerial methods in general, are not all equally compatible with different cultures. In nations with a strong tradition of discipline rather than a problem orientation, the Delphi technique is not apt to be workable. Experts in a field are not likely to admit that nonexperts have a right to an opinion, let alone the right to have their opinions considered in a forecast. Judgment-Based Technological Forecasting Techniques Cross-Impact Analysis This procedure is an extension of the Delphi method. Cross-impact analysis extends the confidence that can be placed on the forecasts of related events. It allows inclusion of an additional set of factors even beyond those usually considered by the respondents in a normal Delphi process. The purpose of cross-impact analysis is to study the mutual influence of events explicitly and systematically, and to include those influences when forecasting technical capabilities. The set of events being studied for cross-impact potential is subjected to a Delphi analysis where a probability and a date of occurrence is assigned to each event. The events are then entered in a cross-impact matrix, as in Figure 15, where each event's impact is measured against each related event. A revised forecast can be prepared manually or by computer programs that have been developed explicitly for this purpose.

Technological Forecasting in Use In spite of the wealth of models and approaches that are available for technological forecasting, the main elements involved in the determination of future technology are

economics, sociopolitics, and existing technology. The forecaster must not ignore any one of these three areas when attempting to forecast technology because they interact in a complex fashion to influence and determine the future. The key to valid technological forecasting is the careful inclusion of realistic and informed judgment into the forecasting methodology. In terms of choosing a forecasting strategy, one of the crucial factors is the potential economic value of the forecast compared with the cost of making the forecast. Some methods are much more expensive than others, and some tend to give better results for certain situations. The ease of entering, storing, fitting curves, and manipulating large amounts of data in microcomputers makes it tempting to "try everything," a strategy that tends to create more confusion than understanding. Technological forecasting can even become a way of procrastinating-the timid manager avoiding the need to act by insisting on more and more forecasts. The "best" technique depends in part on the environment in which the firm is operating. Many forecasting methods assume a relatively stable environment with constant trends. if such is actually the case, then statistical projection methods may be most appropriate, especially for the short range. Yet a major aspect of technological forecasting rests on the presumption that invention and change will produce technologies that are not simple extensions of the past. If this were not true, much technological forecasting would be unnecessary. In general, available data should be plotted as a time series to determine underlying patterns that may aid in choosing between alternative forecasting methods. A search for causal relationships is also helpful. Data are generally available for short-range forecasting situations only, which is why extrapolation techniques work best for shortrun problems. For long-range forecasts, the judgmental methods are more suitable because, as time periods are extended, the dangers of unjustifiable extrapolation grow rapidly. Among judgmental methods, Delphi is probably the best known and most often used. Judgmental techniques are also more appropriate when hard data are lacking or when there is insufficient time or funds to collect and analyze hard data. Judgmental techniques are also appropriate when the number of significant problem variables is large and their relationships are complex or not well understood. The various judgmental approaches each have their particular strengths and weaknesses. See Table 1 for a summary of these strengths and weaknesses.

Group approaches have special advantages and disadvantages. For one thing, the sharing of information and insights significantly improves the validity of the forecast. Improvement is obtained through combining abilities, illuminating inconsistencies and contradictions, checking errors, and working through fuzzy and indistinct thinking. While it has already been mentioned that groups provide a synergism that builds on the ideas and thoughts of each person, groups also provide synthesis where pieces of previous thoughts and ideas are combined to form a new thought or idea. Some disadvantages of open discussion (confrontation) groups are: 1. The Halo (or Horns) Effect: A person's reputation (or lack of reputation), or the respect (disrespect) in which a person is held can influence the group's thinking. 2. Bandwagon Effect: Pressure to agree with the majority. 3. Personality Tyranny: A dominant personality forces the group to agree with his or her thinking. 4. Time Pressure: Some people may rush their thinking and offer a forecast without sufficient reflection in order not to delay the group. 5. Limited Communication: in large groups, not everyone may have an opportunity to provide input. The more aggressive group members or those with the loudest voices may have an exaggerated effect on the group's opinion. As with methods for fostering creativity, the methods for technological forecasting have an imperfect past and a highly conditional future. But questions about the value of

individual methods do not extend to the general subject. Technological forecasting is not a luxury to be enjoyed if it can be afforded, it is a necessity that is recognized and done explicitly or unrecognized and conducted implicitly. Not to forecast is inherently a forecast that the future will be precisely like the past-a forecast certain to be false. Technological forecasting is a steadily developing art. Several of the methods covered above, both statistical and judgmental, have been computerized, either to make numeric data collection and manipulation more convenient or, through interactive programs, to ease the problems of collecting judgmental data. Current research seems to be focused mainly on developing more sophisticated judgmental methods, and on combining judgmental and statistical methods.