III.B. 1. The Policy Delphi
A Seer upon perceiving a flood should be the first to climb a tree.
The Policy Delphi was first introduced in 1969 and reported on in 1970.' It represented a significant departure from the understanding and application of the Delphi technique as practiced to that point in time. Delphi as it originally was introduced and practiced tended to deal with technical topics and seek a consensus among homogeneous groups of experts. The Policy Delphi, on the other hand, seeks to generate the strongest possible opposing views on the potential resolutions of a major policy issue. In the author's view, a policy issue is one for which there are no experts, only informed advocates and referees. An expert or analyst may contribute a quantifiable or analytical estimation of some effect resulting from a particular resolution of a policy issue, but it is unlikely that a clear-cut (to all concerned) resolution of a policy issue will result from such an analysis; in that case, the issue would cease to be one of policy. In the face of the policy issue, systems analysis, operations research, and other related disciplines can do no more than supply a factual basis for advocacy. The expert becomes an advocate for effectiveness or efficiency and must compete with the advocates for concerned interest groups within the society or organization involved with the issue.
The Policy Delphi also rests on the premise that the decision maker is not interested in having a group generate his decision; but rather, have an informed group present all the options and supporting evidence for his consideration. The Policy Delphi is therefore a tool for the analysis of policy issues and not a mechanism for making a decision. Generating a consensus is not the prime objective, and the structure of the communication process as well as the choice of the respondent group may be such as to make consensus on a particular resolution very unlikely. In fact, in some cases the sponsor may even request a design which inhibits consensus formulation.
The Committee and the Delphi Process
Traditionally the approach in most organizations to the examination and exploration of policy issues has been the committee process. Certainly it is well documented by a number of writers on the functioning of government organizations, that the committee system is a structure that evolved initially to promote the advocacy process associated with policy analyses.2,3 The committee approach brings people together across organizational lines in order that all views at similar organizational levels in the whole organization may be represented, and a meaningful view arrived at after the differing interests have been adequately expressed and advocated. It is the contention here, however, that from a pragmatic viewpoint, the committee approach in government and most other organizations no longer functions as effectively in the realm of policy formulation as it once may have.
Many organizations today have become bigger, serve more functions, and span a much wider range of complex interacting functions. Committees that truly represent all interests on an issue are often quite large and unwieldy. By the time one has reached the point of twenty or more people constrained to reach a view in a limited amount of time, a complete and free exchange of views among all concerned is often too time consuming or impossible within the scope of the allocated effort for the job.
With increasing size of organizations, the ratio of the number of people at the top echelons to those in the lower echelons has decreased over the years, particularly in government. This implies that those at the top must spend more time devoted to day-to-day management functions and less time for committee participation on the longer-range issues associated with policy. As a result, the responsibility for committee participation falls more and more on lower-level people. Individuals at the lower levels are less likely to be advocates of anything until they have had ample time to clear it with their supervisors. This often forces the committee into added weeks of delay whenever any new point is made and usually results in the early or premature termination of new considerations that might result from the advocacy process.
If an organization is top heavy a similar problem also develops. Power becomes too diffuse and no one feels he has the authority or jurisdiction to act as an advocate on the broader issues that usually arise at the policy level. There are so many narrowly defined functional responsibilities that everyone is taking care not to tread on their neighbors' territory.
The complexity of issues today usually calls for a great deal of additional staff to supplement the committee process. More often than not, this time or support is not allocated to or available for committee participants. In an atmosphere of budget cuts, belt tightening, and competition for limited funds, it may appear advantageous not to advocate, not to be noticed, and especially not to be held accountable for views, promises, or positions which require effort to document or substantiate. In addition, in most organizations today, we have individuals who are not familiar with many of the new decision aids coming out of operations research and systems analyses but who have an intuitive feel for the complexities of the particular business or function the organization is involved in. We also have a good many individuals who have been trained in many of the modern management techniques and who are sometimes a little too confident that these approaches can be applied to every problem. The lack of effective communication between these two groups has brought about the ineffectiveness of many committee exercises,
It is the above factors, or any combinations of these factors, which have motivated attempts to seek substitutes for the committee process. Contrary to the above, the earlier writings on Delphi have usually presented a separate but canonical set of problems associated with committees that tend to reflect psychological characteristics of committee processes:
Given a small committee of around ten individuals with sufficient time to consider and explore the issues, and some assurance that the privacy of their respective remarks will be respected outside of the committee room, it is doubtful that any of the above issues would greatly inhibit the process. However, as the size of the committee increases, the time available decreases, and the organizational considerations listed above present themselves, the psychological problems will also come into play.
Delphi, however, is not a replacement for the committee process. The proposition presented here is that the Policy Delphi can be utilized to revise the effectiveness of the committee approach.
A Policy Delphi can be given to anywhere from ten to fifty people as a precursor to a committee activity. Its goal in this function is once again not so much to obtain a consensus as to expose all the differing positions advocated and the principal pro and con arguments for those positions. In many policy areas, a larger number of respondents, in the area of twenty or more, is commensurate with the number of differing interests that must often be considered in the increasingly complex issues facing organizations.
Once the Delphi has been accomplished, a small workable committee can utilize the results to formulate the required policy, This then is the author's view of the role of the Policy Delphi-a mechanism for reviving the advocacy process in organizations through improving the effectiveness of lateral policy formulating committees. In this way, Policy Delphis operate as precursors to the committee activity.4
The Policy Delphi, therefore, is not in any way a substitute for studies, analyses, staff work, or the committee. It is merely an organized method for correlating views and information pertaining to a specific policy area and for allowing the respondents representing such views and information the opportunity to react to and assess differing viewpoints. Because the respondents are anonymous, fears of potential repercussions and embarrassment are removed and no single individual need commit himself publicly to a particular view until after the alternatives have been put on the table. Even in those cases where the Policy Delphi uses only the committee or sponsoring body as the respondent group, it has the advantage of eliminating the principal bottleneck in the committee procedure by providing a clear delineation of specific differing views, thereby providing an opportunity for the committee members to prepare their respective cases adequately.
A Policy Delphi should be able to serve any one or any combination of the following objectives:
The ability of the Delphi technique to improve current practices for handling the first objective seems quite clear. Whether or not it can meet or fulfill any portion of the other objectives probably depends on whether the design team can distinguish the motivation of the respondents in making a particular judgment on an option, More specifically, when a difference in judgment does occur on an option, is it based upon uncertainty and/or lack of information with respect to consequences, or is it based upon differences among the self-interests as represented by the respondent group? If the Delphi can be designed to make this distinction it should be able to serve these latter objectives of examining and distinguishing consequences and acceptabilities. Because in some cases people are not fully aware of the motivating factors behind their views, the exposing of these factors could require fairly sophisticated approaches, such as multidimensional scaling.
The Mechanics of a Policy Delphi
A Policy Delphi is a very demanding exercise, both for the design team and for the respondents. There are six phases that can be identified in the communication process that is taking place. These are:
In principle the above process would require five rounds in a paper-andpencil Delphi procedure. However, in practice most Delphis on policy try to maintain a three- or four-round limit by utilizing the following procedures: (1) the monitor team devoting a considerable amount of time to carefully preformulating the obvious issues; (2) seeding the list with an initial range of options but allowing for the respondents to add to the lists; (3) asking for positions on an item and underlying assumptions in the first round.
With the above simplifications it is possible to limit the process to three rounds. However, new material raised by the respondents will not get the same complete treatment as the initial topics put forth by the monitor team. Still, very successful Delphis have been carried out within a three-round format. Ultimately, however, the best vehicle for a Policy Delphi is a computerized version of the process in which the round structure disappears and each of these phases for a given issue is carried through in a continuous process.5
It is also necessary on a Policy Delphi that informed people representative of the many sides of the issues under examination are chosen as participants. These individuals will not be willing to spend time educating the design team, by way of the Delphi, on the subject matter of concern. The respondents must gain the feeling that the monitors of the exercise understand the subject well enough to recognize the implications of their abbreviated comments. Therefore, the initial design must ensure that all the "obvious" questions and subissues have been included and that the respondent is being asked to supply the more subtle aspects of the problem.
In some instances, the respondent group may overconcentrate its efforts on some issues to the detriment of the consideration of others. This may occur because the respondent group finally obtained was not as diversified as the total scope of the exercise required it should be. With proper knowledge of the subject material, the design team can stimulate consideration of the neglected issues by interjecting comments in the summaries for consideration by the group. It is a matter of the integrity of the design team to use this privilege sparingly to stimulate consideration of all sides of an issue and not to sway the respondent group toward one particular resolution of an issue. If, however, the respondent team is as diversified as required by the material, there should be no need to engage in this practice.
A Policy Delphi deals largely with statements, arguments, comments, and discussion. To establish some means of evaluating the ideas expressed by the respondent group, rating scales must be established for such items as the relative importance, desirability, confidence, and feasibility of various policies and issues. Furthermore, these scales must be carefully defined so that there is some reasonable degree of assurance that the individual respondents make compatible distinctions between concepts such as "very important" and "important." This is further complicated by the fact that many of the respondents may not have to think through their answers in order to remain consistent in answering different parts of the questionnaire.
The Delphi technique is not just another polling scheme, and the practices that are standard in polling should not be transferred to Delphi practice without close scrutiny of their applicability. Consider, for example, a poll of different groups in an organization asking for their budget projections over the next five years. This is a comparatively straightforward request which does not ask any one group to place itself in context or to worry about consistency with other groups in the organization. A Delphi on the same subject would ask each group to make projections for every group's budget and, in addition, to project separately a feasible total budget for the organization as a whole.
The normal budget process in an organization is essentially a poll. A few research laboratories have in recent years attempted a budget review process via the Delphi mode, but unfortunately these are never reported in the literature because of the proprietary nature of the subject material. In principle, it would appear that the Delphi offers more opportunity for people to support budget items outside of their current management function and often to obtain a better appreciation of the budget trade-offs that have to be made.
There are many different voting scales that have been utilized on policy type Delphis; however, there are four scales, or voting dimensions, that seem to represent the minimum information that must be obtained if an adequate evaluation is to take place. On the resolutions to a policy issue it is usually necessary to assess both desirability and feasibility. One will usually find a significant number of items which are rated desirable and unfeasible or undesirable and feasible. These types of items willusually induce a good deal of discussion among the respondents and may lead to the generation of new options. The underlying assumptions or supporting arguments are usually evaluated with respect to importance and validity or confidence. In this case a person may think an invalid item is important (because others believe it to be true) or that a true item is rather unimportant. It is usually unwise to attempt to ask for a vote on more than two dimensions of any item. However, if one has established a significant subset of items utilizing these scales then further questions can be introduced focusing on the significant subset. For example, there is the possibility of taking desirable options and asking the probability for each, given certain actions are taken.
Typical examples of these scales follow. Note that no neutral answer is allowed other than No Judgment (which is always allowed on any question). A neutral position offers very little information in policy debates and it is usually desirable to force the respondent to think the issue out to a point where he can take a nonneutral stance. In other words, the lack of a neutral point promotes a debate which is in line with developing pros and cons as one primary objective. This design choice has sometimes upset those who feel consensus is the only valid Delphi objective.
The first and foremost problem in conducting a Policy Delphi occurs with the initial steps in the process. If the respondents feel strongly about the issues, and this should be the case, they will generate a large amount of written material. If they are provided a certain number of items to deal with on the first round then each of them will make approximately the same number of written comments or additions in response. These must be abstracted carefully and duplications among the respondents eliminated. On the average, the written material in the questionnaire for the second round will be five to ten times that of the first round.
After the votes are taken on the second round, the material should be rearranged by the average vote on the third round. In other words, referring to the preceding scales, the options should be reordered by Desirability and the supporting arguments reordered for each option by Importance. When the votes are in, the resulting summary for the third round should clearly point out which items exhibited polarized distributions, which ones exhibited a flat distribution across the whole range, skewed distributions, or on which items only a very small sample of the respondents were able to make a judgment. For these items, additional comments should be solicited. If possible, the revote should be put off until a fourth round when everyone can see the additional remarks. In a three-round exercise a revote is taken on the third round.
In many cases it may be desirable to keep track of certain subgroups making up the respondent group as a whole. This provides a mechanism to check whether polarized views reflect the affiliations or the backgrounds of the respondents. Depending on the application, this information can be fed back to the group. Schneider(4) in his article on Policy Delphis proposed a very concise "Measure of Polarization" among the subgroups. Take all two-by-two combinations of subgroups and add the absolute value difference of the average vote on a given item. This sum of first differences is now an index which provides an appropriate ranking of the degree to which differences exist for each item relative to the group of items as a whole. The same measure may be applied to each individual who voted on the item when a subgroup breakdown is not appropriate. Note that in opposition to average and standard deviation this measure is a strong function of the number who voted when applied on an individual basis.
Some additional guidelines on carrying out the Policy Delphi process are as follows:
As can be seen, there are many things to be considered in running a Policy Delphi, or any other Delphi for that matter. The Delphi concept seems so simple that many people have thought it an easy thing to do. Consequently there have probably been more poorly done Delphis than ones that have been well done.
One additional aspect of the Policy Delphi which usually argues for four or more rounds arises in the situation where the respondents feel very strongly about their respective views. In such a case they sometimes have an attitude where they cannot imagine that there are rational and intelligent people who hold a contrary view. Even with a vote on the first round on a given issue, the reaction of this type of respondent to the vote presented on the second round is that the individuals holding the opposite view to his just don't understand the problem completely. A few simple comments will clear up their ignorance. It is only until the third round comes back that this type of respondent feels the shock resulting from a realization that the other side also feels it has some valid points to be made. Therefore, it is only at the third round that this type of respondent begins to put a great deal of careful effort into the points he is making and to consider more carefully what the other side is saying. The material generated out of this type of process could have a significant impact on the group views if carried back in a fourth round.
The selection of respondents is one of the most difficult tasks. However, this problem applies to any committee or study effort. The sponsor is likely to have a certain candidates in mind. The design team should try to structure the problem in order to get a comprehensive coverage of the topic. It is also a good idea to mix in a couple of lateral thinkers and devil's-advocate types, just on a matter of general principle-i.e. those individuals who always manage to come up with the unexpected.
It is possible on a Policy Delphi to observe two very different phenomena taking place. One is when the exercise starts with disagreement on a topic and ends with agreement. This can be very useful to those sponsoring the study if it does occur, but, as has been said, is not a necessary result. Another process is to start with agreement on a topic and end with disagreement. In a sense this can be viewed as an educational process taking place among the respondents who suddenly realize, as a result of the process. that the issue is not as clear-cut or simple as they may have thought. Unfortunately, to this point in time there has not been sufficient exploration of the use of the Delphi technique as an educational process. Schneider(4) also discusses this point. As he pointed out, Delphi could be used by a planning agency to interface more effectively with representatives of the community and serve an educational function for both groups.
Another unexplored use of the Policy Delphi is the investigation of the performance of past policy actions. Too many organizations do not have an appropriate mechanism for taking stock of what they have accomplished. Understanding of what has occurred is often lacking and can lead to future mistakes in policy formulation.
Examples of Policy Delphis
One of the first Delphis that bordered on being policy oriented was an exercise undertaken in 1968 by the National Industrial Conference Board. It was titled "An Experimental Public Affairs Forecast." It involved 70 people representing the following areas of expertise:
The vast majority had titles of chief executive or director. All were con-sidered by the Conference Board to be distinguished in their field.
The overall objective of the study was to obtain a rank ordered list of National Priorities or Areas of Major Concern to the Nation, areas which could create major public problems in the seventies and eighties and should receive attention by U. S. leadership. The top ten in that list in order of priority were: (1) division in U. S. society; (2) international affairs; (3) education; (4) urban areas; (5) law and order; (6) science, technology, management of change; (7) economy; ($) resources; (9) values; (10) population.
The Delphi was completed before the presidential campaign and one may note a degree of correspondence between the priorities set by this exercise and the Republican campaign themes. While the Delphi dealt with policy considerations, it was largely oriented to putting the pieces of the problem together by collecting information and views from a diverse set of respondents. Therefore it largely reflected a Kantian-type exercise. The bulk of the material produced was a collection of commentaries on the problem areas with sonic estimate of when particular problems would arise. Each item was handled in terms of the following categories of information:
The Delphi appeared to be quite adequate in meeting the needs of its sponsors; however, the exercise has never been described in the literature so one can only infer this from the final report, which unfortunately did not receive public distribution beyond those immediately involved and some individuals working in the Delphi area at that time. One major fault of the study was the decision made by the staff people not to abstract the comments of the panelists but to retain the full text. In part this decision was probably influenced by the distinguished nature of the respondent group. The result was a very large volume of material which is a little painful to wade through to gather the particular nuggets of wisdom that were produced. One goal of a Delphi design shouid, therefore, always be to obtain a filtering of the essential from the superfluous.
The next Delphi in the policy area was one conducted by Emory Curtis as a consultant to San Mateo County in California, This one involved around 80 community people representative of the many different constituent groups making up the public body. A great deal of effort went into obtaining a broad-based distribution of respondents. They were provided a large number of policy options dealing with the structure and functions of the county government, and asked to vote on these for relative agreement on a seven-point agreement scale. Additional items were added as a result of the first round. However, the one shortcoming of this exercise was the lack of exploration of the factors underlying disagreement when it did occur. The exercise produced some new options and exhibited consensus where it occurred but provided no mechanism for effectively resolving disagreement. However, it represented one of the first attempts to use the Delphi in policy areas related to community government.
In 1970 a Delphi was conducted by the Office of Emergency Preparadness and the Rand Corporation on the subject of Civil Defense Policy. This Delphi introduced a number of unique features. It exhibited the structure of a Hegelian inquiring system as opposed to the earlier Lockean- and Kantiantype Delphis. First, it recognized in the design that strong disagreement already existed on a number of the issued involved. For a number of items the respondents were asked to choose sides by circling "could/could not," "should
should not," these being choices in the wording of the items. They were also asked to develop the strongest arguments on the various sides of a given issue. The sponsor was not interested in having the group make a decision for him, but in having the group develop, compare, and evaluate the best possible arguments on each side of an issue.
The details of this exercise are well documented in the literature.(1) As typical of these types of Delphis, the respondents generated about eight times the amount of material they were initially given on the first round, which contained some seventy items for evaluation. Basically policy options were evaluated on scales of desirability and feasibility, while supporting points were evaluated on importance or validity. This Delphi was really the first to incorporate a structured debating-type format, which appears to be the useful approach for the exploration of policy issues.
In 1970 Professor J. B. Schneider at the University of Washington adopted the same approach to the exploration of transportation planning as it applied to highway development in the Seattle area. His report of the exercise' is an excellent example of applying these techniques to urban planning problems. He also contributed some very useful observations on the methodology for handling disagreement in that particular context.
Following the same line of development, Joel Goodman of the College of Marine Studies, University of Delaware, conducted a policy-type Delphi on the Coastal Zone Land Use Planning Issue. This involved a large number of people representing government business, public groups, and specialists. This exercise was done in 1971 and 1972. It converged the following types of items into different sections of the questionnaires: respondent characteristics; respondent attitudes; arguments pro and con; general policy and budget items; specific policy issues; specific programs; strategic issues.
Some sample questions from that exercise follow:
Indicate by check mark who should assume responsibility for establishing:
A large number of the current Delphis have started to incorporate policy issues even when that was not the primary concern. Such issues have the psychological advantage of making the exercise of more interest to the respondents. The policy orientation has been introduced in some different ways. Instead of asking individuals to extrapolate data into the future in terms of their best estimate of what they think will occur, a policy approach would be to ask what would be a desirable and possible extrapolation as well as an undesirable and possible extrapolation. Based upon those estimates one can ask what are the factors that could make the curve go one way or the other.
A Delphi study conducted by the Federal Department of Public Works in Canada illustrates the incorporation of policy options into an essentially non-Policy Delphi. The department's major role is providing accommodation for federal civil servants, and the Delphi was undertaken as part of a model for forecasting government employment with the purpose of determining future accommodation needs. But the department's mandate extends beyond simply providing buildings to house federal employees. It is concerned with the total work environment of the civil service.
Consequently, the Public Works Delphi also explored the existing procedure for space allocation, which at present is based on the average salary of all employees using that space, and asked respondents to comment on that process. In the first round, after reviewing the present process, the respondents were asked to list what they felt were the strengths and weaknesses in the process, and asked to suggest possible options for change. In the second round these options were voted upon according to Desirability and Feasibility, keeping in mind that a particular option could be desirable and unfeasible at the same time, or vice versa. Some examples of suggestions for change according to Desirability were:
The Delphi also looked at possible parameters for measuring building performance that would go beyond the usual cost benefit measures, such as the ratio of rentable square feet to total square feet. Specific suggestions or concepts for consideration fell into the following categories:
The respondents were asked to vote on the Desirability and Feasibility of specific suggestions and to suggest ways in which some of these concepts could be measured..
Public Works then used the Delphi exercise not only to fulfill its immediate objective of forecasting federal government employment but also to explore policy options relating to its mandate of fulfilling broader social, economic, and environmental objectives.
Another excellent example of a Delphi mixing policy issues with future forecasts was one done by the Canadian Department of Health and Welfare on the Future of Genetic Counseling Services in Canada. The exercise involved some sixty respondents ranging from research geneticists to public health workers. The design was well balanced between "technical issues" of what was possible at what point in time and "policy issues" of who could, would, or should do what. In this latter area, the issue of a genetic registry and its potential abuses as well as uses were explored. The Delphi used the same sort of scales that were mentioned earlier. However, it tended to redefine a scale such as "importance" for each question it was used on. This had the merit of minimizing what the respondent had to remember, since each question was largely self-contained. It also minimized the chance of confusion by placing the scale within the context of the particular question. Furthermore, it allowed more variety in the sequencing of questions. Most other designs, by grouping questions of a given type under one explanation, can produce a feeling of monotony as the respondent goes through the exercise.
The Problems of a Policy Delphi
We have already mentioned the danger that a Policy Delphi can be misinterpreted as a decisionmaking tool as opposed to a decision-analysis tool. Everyone at heart is a decisionmaker, or wishes to be, and it is all too easy on the part of the designer to appeal to this unrequited desire, It should be a matter of intellectual honesty for designers to make clear just what the objective of the exercise is. If we have a problem in organizations today, especially governmental ones, it is that the responsibility for a given decision is not clearly focused on one individual. A decision should be made by one individual, and the role of the Policy Delphi and other tools is to provide the best possible information and ensure that all the options are on the table. To do this the Delphi must explore dissension. Both Dalkey and Helmer in the early writings on Delphi expressed the need to establish clearly the existent basis for observed dissension. However, this implies a good deal more work for the design team and has often been neglected in the majority of the early exercises. When a strong minority view exists and is not explored, the dissenters will often drop out, leading to an "artificial" consensus on the final product.
Once a Policy Delphi has been started, there is no way to guarantee a specific outcome if it is to be an honest exercise. This is something the sponsor must be well aware of. Occasionally a sponsor, particularly in a policy exercise, will desire that the group not reach a consensus on any particular option. While it is consistent with the objective of a Policy Delphi to choose a respondent group such that a consensus is unlikely to occur, it can never be guaranteed that it will not be a result. However, there is a fine line between Delphi as an analysis tool and Delphi as an educational or persuasion device. It is possible to consider using a Delphi to educate at least a part of a respondent group on options they may not be aware of. Unfortunately, very little work has been done on the use of Delphi in an educational mode even though most designers would agree that educational processes take place in most exercises.
A Policy Delphi is a forum for ideas. In opening up the options for review, items may arise which can be disconcerting to members of the group. If a sensitive area is under review and an attempt has been made to have diverse representation in the group, then premature leakage of the results can occur. In such a case, individuals outside the exercise may misinterpret what is taking place. This problem of lifting items out of context occurs all the time in the committee process. A workable approach to this problem in the Delphi process is to incorporate members of the press into the respondent group when dealing with major public policy items.
As with any policy process, there are many ways to abuse the use of the Policy Delphi: the manner in which comments are edited, the neglect of items, the organization of the results. However, such a process is a rather dangerous game and not likely to go unnoticed by some segment of the respondents. There are very few greater wraths than that of a respondent who discovers himself to be engaged in a biased exercise. Furthermore, Delphi has reached the point where there is no longer any excuse on a professional basis for making many of the mistakes found in earlier exercises. The person seeking to undertake a Delphi today should be reasonably familiar with what has taken place in the field.
1 Murray Turoff, "The Design of a Policy Delphi," Technological Forecasting and Social Change 2, No. 2 (1970).
2 Charles F. Schultze, "The Politics and Economics of Public Spending," Brookings Institution, Washington, DC., 1968.
3 Numerous references to Lindblom's writings on committee processes appear in the work cited in reference 2.
4 Jerry B. Schneider, "The Policy Delphi: A Regional Planning Application," Technological Fowashng and Social Change 3, No. 4 (1972).
5 Murray Turoff, "Delphi Conferencing," Technological Forecasting and Social Change, 3, No. 2 972).