Skip to navigation – Site map

Action without change? On the use and usefulness of pilot experiments in environmental management

Raphaël Billé

Abstracts

One of the most common approaches to environmental action consists in designing and implementing pilot experiments, which aim at testing new practices and organisations. These technical or socio-political innovations are then supposed to spread. However, the way such a juxtaposition of experiments is expected to lead to a global transformation of our development paths and of their environmental dimension remains quite mysterious, if not mythified. Analysing the pilot approach as a strategy of change, this article offers a concise view on what is at stake in the replication hypothesis underlying the approach.

Top of page

Editor's notes

This paper is a revised version of an article originally published in French by VertigO-La revue électronique en sciences de l’environnement (Billé, 2009).

Publication history

Published: 10 July 2010.

Full text

Introduction

1The implementation of sustainable development calls for drastic changes in our current development trends. Such changes will entail radical transformations of the economic, cultural and other relations we entertain with our environment and of our management practices. Although most of the path that will lead us towards the utopia of sustainability is yet to be mapped out and followed, it is clear that a change of trajectory, practices and policies is necessarily at the centre of all debates.

2Several avenues are available to address this change issue. One of the most widespread in the field of research and action focused on sustainable environmental management is to set up pilot experiments in order to experiment with new practices and organisation forms. Those technical as well as socio-political innovations are supposed to spread and eventually to be adopted by other stakeholders in different, often contrasting contexts. Pilot experiments, as “recipes for mobilization” (Lecomte, 1986), have a well-established tradition in public action. They have a great success in the field of environmental management, be it in the case of traditional problems, such as fisheries management, or of recently emerging issues, like avoided deforestation. The way in which this spectacular juxtaposition of more or less novel experiences is expected to give rise to a global transformation of our development modes and of their environmental “content” is therefore a crucial issue.

3It thus seems a paradox that the social, political and organisational dynamics associated with those pilot experiments are so little studied and mostly, so little understood (Saunders, 2003). Of course, there are numerous studies, amongst others in anthropology, which have analyzed the process of diffusion of sociotechnical innovations: Olivier de Sardan (1995), for example, shows their potential contribution, although he essentially limits his study to agropastoral innovations. However, the experience we have acquired in research as well as in project development and evaluation, shows us that de facto, those contributions are usually ignored in the field of environment, either actively, or by remaining largely unknown, or by not fully responding to the needs of the stakeholders. In the end, the way in which a pilot experiment exerts or fails to exert an influence on “routine” practices remains quite mysterious. A substantial number of evaluations have shown that those experiments possess intrinsic characteristics that prevent them from provoking more important changes in practices, that is from being replicated. Nevertheless, the “piloting strategy” is still called upon under very different sets of circumstances in order to reach various goals, generally without any serious justification.

4In the present paper, we will therefore examine the use of pilot experiments as a strategy for change. Our goal is to propose a concise analysis of this subject, from both a theoretical and a practical point of view. We will start by defining the problem more precisely and then discuss the appeal of pilot experiments and examine in more detail the issues surrounding their replication, which lies at the heart of the strategy of change implicitly linked to “piloting”.

The nature of pilot experiments

5A pilot experiment, as we define it, is an action undertaken by one or more public and/or private stakeholders in order to test novel practices or technologies, i.e. innovations in the sense of Olivier de Sardan (1995): “every graft of novel technology, knowledge or form of organization (generally as local adaptation of borrowed or imported innovations) on existing technologies, knowledge or forms of organization”. Following the hypothesis of the “’proximity contamination’ of the change process” (Behar, 1997), commonly referred to as the “ripple effect” hypothesis, the main characteristic of a pilot experiment is to be implemented on a smaller scale than that of the ultimate objective: local or regional scale for national objectives, pilot farms for watershed-wide objectives, individual work unit for objectives at the whole administration or company scale, etc.. The Brestian Roadstead Contract (France) is a typical example. Although its eventual objective was the reduction of pollution from agricultural sources at the scale of the entire watershed (2 800 km2, 137 municipalities), its preliminary phase led to the establishment of, among others, pilot experiments in the basin of the Kerouallan River, involving only 600 hectares of agricultural land spread over just two municipalities (cf. Turpin, Bouraoui et Tranvoiz, 1999 ; Billé, 2004).

6Pilot projects are a specific, but widely used variant of the pilot experiment. They deserve particular attention because they combine the advantages and disadvantages of a project on one hand and those of “piloting” on the other hand. Therefore, they have all the characteristics of a pilot experiment, paired with those of a project as a particular form of organisation of action. Generally speaking, a project is a “temporary organisation” (Turner et Müller, 2003) that is commonly defined as follows:

 “Endeavour in which human, material and financial resources are organised in a novel way, to undertake a unique scope of work, of given specification, within constraints of cost and time, so as to achieve beneficial change defined by quantitative and qualitative objectives” (Turner, 1993).

  • 1  Without going as far as implying a cause and effect relationship, it is noteworthy that integrated (...)

7We will concentrate here on the “pilot” dimension since the shortcomings of the project approach as such have been exhaustively described elsewhere (cf. for example Lecomte, 1986 or Bako-Arifari and Le Meur, 2001 in the context of development aid; Billé and Mermet, 2002a or Billé, 2007 in the context of integrated coastal zone management). Some key findings are (1) that projects have often shown their limitations in sustaining their existence from the moment the financing comes to term; (2) that they don’t easily conform to the longer time scales characteristic of social change and collective action; (3) that they are extremely sensitive to even small changes in local conditions (for example loss of a leader) or in the external context; finally (4) that they contribute to the fragmentation of public action. It is well known in organisation theory that projects create a need for integration: integration of resources in order to implement the project, integration of the different parts of the project, integration of the project in the general activity (Turner et Müller, 2003). This is of course a significant drawback, and even a paradox in the case of integrated management projects1.

8Regardless, the pilot experiment contains by its very nature many of the characteristics of the project: temporary intervention with set goals and means, frequently relying on small ad hoc structures.

An indisputable power of seduction

9Their very logic hence seems to enable pilot experiments to circumvent bureaucratic slowness, which is often considered necessary. Pilot experiments are reputed to be more flexible and better suited to adapt to the uncertainty of processes and objectives (Turner et Müller, 2003) than routine operation-based action. This is why pilot experiments offer leeway for social and/or technical innovation. They obtain results at a smaller scale, with moderate costs and a more consensual participation of concerned stakeholders than large-scale projects tackling the same issues (Saunders, 2003).

10In the environmental field, key stakeholders often need to be persuaded in the first place, since those calling for change are rarely those able to implement it. In this regard, pilot experiments have a strong power of seduction for decision makers. As Saunders (2003) shows, many resistances can be overcome simply by classifying a political innovation, desired by certain stakeholders, as a pilot experiment: “do not worry, it is just an experiment!”. Stakeholders who are recalcitrant to change are more ready to accept a pilot experiment if they still feel in control of processes on a larger scale, where the real stakes are at hand. Going back to the case of the roadstead of Brest, where pollution from agricultural sources, and in particular from intensive pig farming, is a major environmental issue and a constant source of conflict, a wholesale transformation of the regional agricultural model would have been required. However, this was a politically sensitive, if not taboo, issue. In that context, the Chamber of agriculture and the research organisms depending on the Ministry of Agriculture chose a small sub-basin in which to work with farmers and suggest the use of modified, environmentally less harmful yet economically viable practices, such as the reduction of technological inefficiencies or the introduction of new feed practices for animals. The experiment was considered a success: it demonstrated the feasibility of the necessary transformations, in partnership with the concerned sector, although it did not have any repercussions for the practices in question at a larger scale.

11More precisely, pilot experiments generally have a double tactical advantage. First, the “ripple effect” remains a convincing argument in the eyes of many decision makers and is often a winning argument in grant applications. The ideal progression “national ambition / local pilot actions / duplication and generalization” constitutes the framework of many initiatives in environmental management and indeed of the majority of environmental projects undertaken in the context of development aid. The domestic or international replicability of projects is even a selection criterion of both the French global environment facility (Fonds Français pour l’Environnement Mondial, FFEM) and the Global Environment Facility (GEF). Thus, for example, the support of international cooperation in the management of protected areas in a developing country will often be initiated through the implementation of “innovative” practices in one national park with the implicit expectation – fingers crossed – that the proven efficiency of those practices will be sufficiently persuasive for them to be spread nationwide to the extent of becoming the norm. It requires more effort and justifications to stray from this scheme, to question it or to proceed with caution than to adopt it strictly, even if its limits are well known. Second, considering the urgency, the scope and the complexity of the problems at hand and the difficult changes they entail, the pair of “pilot experiments / later generalization” gives scientists and environmental advocates something “to gnaw on”. It leaves the impression of “being on the right path” and that “one can’t rely on past trends since we are about to change things”. As Mermet (1996) points out, “the administration must not discourage its administered (or its own agents….) and usually likes to adopt a reasonable optimism, once the scarecrows have been brandished. After that, all energy and attention is focused on the management of a few promising pilot operations, which eventually induces a global sense of optimism”.

12Our observations lead us to conclude that, although pilot experiments achieve various degrees of success, in general, they rarely initiate the dynamics they are supposed to “pilot”: they may be replicable, but are rarely replicated.

Replicability: the myth of the “ripple effect”

Stimulation of collective action

13The hope for a “proximity contamination of the change process”, which is the foundation of the concept of pilot experiments, relies on a heavy assumption in the field of collective action: that the combination of an objective benefit of a given change and of an external impulse brings a human group to move towards this change.

14However, research in the general field of collective action, and in the environmental field in particular, does not support the hypothesis that “good practices” are bound to spread. The claim that the “objective” and collective interest of a group of humans, or even that of its individual members, to adopt a particular mode of environmental management is a sufficient condition for them to do so, ignores most theories on collective action. For instance, Olson (1965) writes:

 “…even if all of the individuals in a large group are rational and self-interested, and would gain if, as a group, they acted to achieve their common interest or objective, they will still not voluntarily act to achieve that common or group interest,,,”

15except in specific conditions, such as with a small group of individuals, coercion or “some other specialdevice”. This of course also ignores a large body of work on the phenomenon of the spread of changes and innovation mentioned above. Different coastal zone management projects which we studied, for example in Madagascar (Billé et Mermet, 2002b), demonstrate the heaviness of the “common interest” hypothesis of collective action, as a look at the management of fisheries worldwide also reveals. Ostrom gives a specific set of conditions under which this type of collective action becomes possible. In another domain and at a different scale, the difficulty the international community experiences in organizing itself in order to reduce its greenhouse gas emissions despite the collective economic interest clearly demonstrated by the Stern report (Stern et al., 2006), speaks volumes.

The Hawthorne effect

16What is more, the simple presence of an observer, an advisor or a facilitator in a human group modifies its behaviour, through the presence of an external view. Not only does it modify it, but the direction of the modification of the group’s behaviour generally goes in the sense which the group perceives as being favoured by the external agent, regardless of whether his intervention is appropriate or not. This phenomenon, by which “people react positively to the fact that they are being taken care of in order to improve their situation, particularly when they are in a position of weakness” (Bernoux, 1990) is called the “Hawthorne effect”, in reference to workshops of the Western Electric Company (Chicago, USA). During the 1920s and 1930s, a series of experiments on work productivity were performed under the supervision of Elton Mayo, professor at the Harvard Business School. Those experiments showed that the productivity of the workers increased as soon as they were aware of participating in a pilot experiment, regardless of whether their work conditions improved or deteriorated. The results of the experiments were therefore not due to experimental factors, but to the fact that workers were conscious of participating in an experiment. This is why, according to Bernoux, “so many initiatives of reorganization have brilliantly started only to quickly lose their initial momentum”.

17Although this effect was initially reported in an industrial context, it can be presumed to accurately describe the behaviour of a human group (village, fishermen or farmers association,  etc.) in which an external agent momentarily intervenes with the objective of inducing a change towards a more sustainable environmental management. Whatever the pertinence of the intervention of the external agent and regardless of the support it receives from the group, there is a high probability that the group will temporarily adopt “good practices”, at least in appearance, in order to please him. This can severely bias the conclusions drawn with respect to the pertinence of an intervention or the reproducibility of a new mode of management.

From exceptional conditions, exceptional results

18The cases we have studied or in which we have participated clearly show that “pilot” results are all the more easily obtained, the more exceptionally favourable conditions are chosen.

19Therefore the first step in setting up a pilot experiment is usually to identify a space (territory, community, sub-basin, administrative unit, etc.) in which the conditions before the intervention seem favourable enough to offer the best promise of success. One of the fundamental parameters often taken into account is the presence of key individuals, talented and charismatic leaders thriving towards innovation (Saunders, 2003). Later on, the anticipated upscaling of the experience is hindered by personalities less driven by innovation, less motivated and less prone to change, be it out of lack of conviction, for reasons of personal agenda (such as career opportunities), because of decisions on the allocation of available resources, or others. In the “pilot operation Menabe” lead by the regional environmental program of the Indian Ocean Commission, the pilot region was chosen amongst other reasons because of the existence of a regional development committee (“Comité régional de développement”), unique in Madagascar and without any official existence in the national politico-administrative system, considered reliable, energetic and with strong leadership under the authority of a motivated local dignitary (Billé, 2004).

Decreasing resources applied to increasing difficulties

20Not only are pilot experiments conducted in exceptional and well-chosen conditions, they also benefit from exceptional financial, human and technical means. Depending on the case, these can be specifically devoted financial resources, technical assistance, the presence of mediators, or particular political support. The presence of local mediators or the availability of financial resources (travel expenses and per diem) to support stakeholder participation in the various consultative processes of development aid projects are examples of such exceptional resources.

21What is more, this concentration of resources is employed in a context where resistance to change is weak. As we have already seen, pilot experiments that promote cultural change, change of practices and innovative organisation modes tend to be tolerated by stakeholders who are the bearers of the “traditional” and dominant modus operandi as long as they remain pilot ventures. Therefore, they do not have to face the opposition that usually appears in the latter stages, at least not at its full strength or diversity. The resistance to change on behalf of those who have a real or perceived interest in maintaining the status quo only begins in the phase of replication and upscaling. At that stage, studies have identified what Saunders (2003) refers to as “innovation enclaves”, around which the management systems remain essentially unaltered. The existence of such enclaves is accepted by the majority of stakeholders as long as they are not destined to become spearheads of change.

22The Kerouallon experiment is an archetype for all three of those factors (Hawthorne effect, exceptional conditions, decreasing resources). “An elected representative of the urban community of Brest met them [the involved farmers] individually in order to explain to them the objectives of the study and sign the cooperation convention” (Turpin et al., 1999). One easily imagines – and this was the intent – how this could favour the rest of the project, since each farmer felt acknowledged or even favoured. The project leaders moreover add that “the teams in the field pledged to employ only permanent agents in order to avoid having the farmers repeatedly give the same information to several persons”. This appears to be another factor likely to favourably influence the course of the project. One could even suppose that due to regular contact, farmers and project agents will begin to develop a relationship, which would further benefit the quality of information exchanges, the mutual trust and the motivation of the farmers. The question is to what extent favourable conditions for the implementation of a pilot experiment can be created without invalidating its reproducibility? What value as an example can an experiment in intensive agriculture have, that presents all the conditions of trust, dialogue, information and implication of farmers, which to say the least, is not the typical situation?

An evaluation challenge

23We already mentioned the seduction power of pilot experiments as modes or organisation of action. This often leads to pilot experiments being accompanied by an exaggerated sense of optimism or even euphoria on the part of the stakeholders. Many of them, and in particular the operators of public action, derive great satisfaction from focusing the discussion on successful local operations, or even better, on promising fledgling operations. As we have already pointed out, they absorb all energies and convey the unverifiable impression that “one is going in the right direction”, to the point of inducing a global optimism far beyond the actual scope of the experiment. The Hawthorne effect and the distortions related to the observation that “every experiment begins by succeeding” (Conseil Scientifique de l'Evaluation, 1996) are generally ignored. Furthermore, the mere fact of participating in an experiment conveys to the stakeholders the exhilarating, but often mistaken feeling of innovating, even where the same recipes have been used time and again without any real success for decades (see for example Lecomte, 1986, on the successive versions of community-based management).

24Pilot experiments also have a tendency to invalidate evaluation processes, for two sets of reasons. First, the stated objective is often to mobilise and create a new dynamic more than to directly change the performance of an environmental management system. The problem is that the emergence of a mobilization or action dynamics is difficult to measure in abstracto: it is only in the light of the future behaviour of the concerned human groups that it is possible to know whether a durable dynamic has been created. In addition, there is the problem of scale. It is difficult to evaluate micro-experiences with regard to their impact on the way in which the environment is managed. Moreover, pilot experiments are often associated with the concept of adaptive management, itself akin to the method of learning by doing. Even though the usefulness of those concepts for the general case is in no doubt and they are often justified in practice, they can through gradual drifting lead to an understanding of action, in which mistakes and failures become an integral part of the process. In such a context, an evaluation is fundamentally invalidated, including cases in which those errors equate to severe and largely predictable failures.

25Finally, we saw before that pilot projects, and in a larger sense pilot experiments, contribute to a fragmentation of public action. This phenomenon leads to an “illegibility” (fr. “illisibilisation”, Mermet, 2005) of environmental management systems, which makes it extremely difficult to discern the web of responsibilities, to assign effects to practices or to identify trends.

26Exaggerated optimism, invalidation of evaluation and illegibility of environmental management systems all seriously hinder efforts to evaluate pilot experiments, and thereby to capitalize on them, despite this being an essential condition for any kind of collective learning.

Conclusion: a strategy for change with severe limitations

27Heavy assumptions on collective action, the Hawthorne effect, favourable conditions, exceptional resources, weak resistance and difficulties in evaluating: all these do not automatically deprive pilot experiments of any interest. However, those factors are inevitably detrimental to the reproducibility of the experiments and make the findings and lessons learnt from them, either by the project bearers or by outside groups supposed to adopt those changes, debatable. The underlying theory of change is thus particularly weak.

28While there are no grounds to claim that the “pilot strategy” can never work, it is evident that typically, a pilot experiment will not “naturally” evolve towards generalisation. If a “ripple effect” is really desired, it has to be precisely and strategically devised, as much, and probably even more, than the experiment at the “source” of the change. If the aim is to reach certain environmental objectives at a given scale, an experiment at a smaller scale, paired with a vague promise of generalization in case of success, cannot be considered sufficient. There are many theoretical arguments warning that this change in scale is not easy to accomplish, supported by many empirical examples of failure in that respect. Considering this, the burden of proof should in a way be inverted. The implicit hypothesis should be that a pilot experiment may be valid for itself, and for its bearer (who may for example want to test a new instrument or a new practice), but rarely for other cases; unless a precise justification can be put forward, clearly stating the resources – including legislative – required to generalize the experiment. Otherwise, the field of sustainable development will continue, like other fields, to be a playground for an “infinite multiplication of exceptions rather than the transformation of the rule” (Behar, 1997).

Acknowledgements

29The author wishes to thank anonymous referees as well as Alexandre Magnan, Romain Pirard and Julien Rochette for their comments on earlier versions of this article.

Top of page

Bibliography

Bako-Arifari, N., P.-Y. Le Meur, 2001, Une anthropologie sociale des dispositifs du développement, 52 p., in J.-F. Baré (dir.) L'évaluation des politiques de développement. Approches pluridisciplinaires, L'Harmattan, Paris, pp.121-173.

Behar, D., 1997, La politique de la ville mérite-t-elle d'être évaluée ? Urbanisme, novembre-décembre, pp. 55-58.

Bernoux, P., 1990, La sociologie des organisations, Seuil (1ère édition : 1985), Points - Essais, Paris, 382 p.

Billé, R., 2004, La Gestion Intégrée du Littoral se décrète-t-elle ? Une analyse stratégique de la mise en œuvre, entre approche programme et cadre normatif, thèse de doctorat en sciences de l’environnement (gestion), ENGREF, Paris, octobre, 473 p. + annexes.

Billé, R., 2007, A dual-level framework for evaluating Integrated Coastal Management beyond labels, Ocean and Coastal Management, 50, pp. 796-807.

Billé, R., L. Mermet, 2002a, Sectoralization of an integrated coastal management programme: A case study in Madagascar,Journal of Environmental Planning and Management, 45, 6, pp. 913-926.

Billé, R., L. Mermet, 2002b, Integrated Coastal Management at the regional level : Lessons from Toliary, Madagascar, Ocean and Coastal Management, 45, 1, pp. 41-58.

Conseil Scientifique de l'Evaluation, 1996, Petit guide de l'évaluation des politiques publiques, La Documentation Française, Paris, 123 p.

Lecomte, B.J., 1986, Project Aid. Limitations and Alternatives, OECD Development Centre, Paris.

Mermet, L., 1996, Les études d'évaluation entre stratégie et méthodologie. L'exemple des politiques publiques en matière de zones humides, Gérer et Comprendre, 46, pp. 55-64.

Mermet, L., 2005, Et si les "gagnants-gagnants" avaient "gagné-perdu"? Pour une comptabilité analytique des enjeux de la négociation, Négociations,1, pp. 11-26.

Olivier de Sardan, J.-P., 1995, Anthropologie et développement. Essai en socio-anthropologie du changement social, Karthala, Paris, 221 p.

Olson, M., 1965, The logic of collective action. Public goods and the theory of groups, Harvard University Press, Cambridge, Massachusetts, 186 p.

Ostrom, E., 1990, Governing the commons. The evolution of institutions for collective action, Cambridge University Press, 280 p.

Saunders, M., 2003, The "problems with pilots" : debating notes, The Evaluator, March.

Stern, N., S. Peters, V. Bakhshi, A. Bowen, C. Cameron, S. Catovsky, D. Crane, S. Cruickshank, S. Dietz, N. Edmonson, S.-L. Garbett, L. Hamid, G. Hoffman, D. Ingram, B. Jones, N. Patmore, H. Radcliffe, R. Sathiyarajah, M. Stock, C. Taylor, T. Vernon, H. Wanjie, and D. Zenghelis, 2006, Stern Review: The Economics of Climate Change, HM Treasury, London.

Turner, J.R., 1993, The handbook of project based management, McGraw-Hill, London, 560 p.

Turner, J.R., R. Müller, 2003, On the nature of the project as a temporary organization, International Journal of Project Management, 21, pp. 1-8.

Turpin, N., F. Bouraoui, M. Tranvoiz, 1999, Pratiques agricoles et qualité de l'eau dans un petit bassin d'élevage intensif. Une expérience pilote réalisée dans le cadre du contrat de baie "Rade de Brest", Ingénieries - EAT,19, pp. 19-30.

Top of page

Notes

1  Without going as far as implying a cause and effect relationship, it is noteworthy that integrated coastal zone management emerged and was formally and procedurally first adopted in countries in which the project approach to public action has the longest history.

Top of page

References

Electronic reference

Raphaël Billé, « Action without change? On the use and usefulness of pilot experiments in environmental management », S.A.P.I.EN.S [Online], 3.1 | 2010, Online since 23 June 2010, connection on 22 November 2017. URL : http://sapiens.revues.org/979

Top of page

About the author

Raphaël Billé

Institute for Sustainable Development and International Relations, Sciences Po, 27 rue Saint Guillaume, 75337 Paris Cedex 07, France, Courriel : raphael.bille@iddri.org

By this author

Top of page

Copyright

Licence Creative Commons

Top of page
  • Logo DOAJ - Directory of Open Access Journals
  • Logo Institut Veolia
  • Revues.org