Agreements by Date
Agreements by Subject
Agreements by Lineage
Problem or suggestion?
Other Treaty Lists
List of Text Files
Download IEA Dataset (Excel)
A Research Guide
for Identifying Indicators of the
Effectiveness of International Environmental Agreements
© Ronald B. Mitchell, 2000
NOTE: This Research Guide was written many years ago and requires significant updating. It may be
useful, however, in its present form for those initiating research on international environmental agreements.
The Nature of the Problem
Identifying Goals, Requirements, and Relevant indicators
Identifying If Relevant Indicators Are Available
Identifying Treaty Secretariats and Data Sources
Cataloguing Data Source Information
This guide provides a strategy for conducting research into the effectiveness of international environmental
treaties. The fundamental goal of the strategy delineated in the following pages is to guide the researcher through a
systematic set of steps for identifying indicators relevant to evaluating the effectiveness of an environmental treaty
and determining whether those indicators are available. The guide seeks to allow the researcher to identify systematic
answers to the following questions:
- What are the environmental goals and behavioral requirements delineated in the treaty?
- What are useful operational indicators (or proxies) corresponding to these goals and requirements that, if
available, would provide the basic data needed by an analyst to evaluate a treaty's effectiveness?
- Of these "relevant" indicators, which are currently available in a specified (and extensive if not
exhaustive) set of likely sources?
- Are there other indicators of environmental quality or environmentally-related behavior that, although not initially
identified as relevant indicators, might nonetheless be useful in evaluating the treaty's effectiveness?
In short, the guide provides a systematic way of identifying environmental and behavioral indicators that are both
available and relevant to evaluating an environmental treaty's effectiveness.
The Nature of the Problem
Evaluation of environmental treaties has, to date, been stymied by structural obstacles that hinder, even if they do
not prevent, identification of indicators that are both relevant to and available for conducting such evaluations.
On the one hand, a wide variety of groups have taken on the task of identifying and often making available data
relevant to a particular treaty. Thus, for example, many secretariats provide tables of data that could be used in
evaluating the treaty or treaties for which they are responsible. Many governments, non-governmental organizations, and
private corporations have also begun providing similar data, sometimes covering a wide range of different treaties.
Rarely, however, do these groups have incentives to undertake in-depth and unbiased assessments of the effectiveness of
On the other hand, those interested in evaluating the effectiveness of environmental treaties are often unaware of
and unskilled at identifying the range of available data relevant to a particular treaty. No systematic and
comprehensive catalog of these data sources is available, particularly not one which provides an overview of the
parameters that would help an analyst interested in such questions to determine whether a dataset with particular
characteristics is available or which would help them recognize the availability of datasets with certain
characteristics that make them advantageous for use in evaluating a particular treaty. Given the increasing and
frequently changing nature of what data is available, as well as the variety of places in which one might be able to
locate such information, data quite valuable to an analysis is frequently overlooked. Most scholars and practitioners
lack the skills, the time, or both to undertake a systematic search for currently available data on a particular treaty.
This often leads either to a discouraged decision not to evaluate a particular treaty because "the data simply
isn't there," or the use of indicators that are not particularly good proxies of the effectiveness of a treaty. In
the latter case, the relief at eventually finding an indicator that bears the slightest resemblance to the indicator
that the analyst would like to have data on often creates the temptation to build an argument that uses
less-than-fully-adequate data justified on the basis that it was available rather than that it was particularly relevant
to the question at hand.
In short, those providing the data that would be relevant to evaluating the effectiveness of a treaty usually do not
have the incentives or resources to undertake unbiased analyses. At the same time, the distributed, non-systematic, and
often random approaches to making such datasets available mean that it is at best difficult and time-consuming, and at
worst, impossible, for those with incentives to undertake independent analyses to easily identify what data relevant to
analyzing a particularly treaty or set of treaties is available.
This research guide seeks to facilitate efforts by scholars and students to conduct rigorous, high-quality,
positivist research into the effectiveness of international environmental treaties despite these contextual obstacles.
Summary of Procedures
Conducting systematic research on the effectiveness of international environmental treaties requires several distinct
steps. Mitchell and Bernauer lay out one approach to these problems in their "Empirical Research on International
Environmental Policy: Designing Qualitative Case Studies" (Journal of Environment and Development
7:1 (March 1998), 4-31).
- Identify environmental goals and behavioral requirements in the treaty text. Brainstorm as many observable, and
measurable, indicators as possible for each goal or requirement. These are described more extensively under
"Identifying Goals, Requirements, and Relevant Indicators" below.
- Identify the secretariat for the relevant treaty and determine whether data on these relevant indicators are
available from the secretariat, using their website or direct contact.
- Identify whether relevant indicators are available in a range of other likely sources.
- Catalog information regarding your data source.
When complete, the researcher should have identified one or more indicators of treaty effectiveness that are both
relevant and available or, alternatively, be able to claim that such relevant indicators were not available at the time
and in the sources used in the above procedures.
Identifying Goals, Requirements, and Relevant Indicators
This stage seeks to identify treaty goals and requirements so they can be used to generate a list of indicators
relevant to evaluating a treaty's effectiveness. These procedures are crucial to identifying the explicit goals and
rules that those negotiating the treaty.set for themselves as standards of success. In many cases, "common
wisdom" about what a treaty.seeks to accomplish (and hence how its success should be judged) can stray
significantly from the goals states actually set for themselves and the operational requirements they placed on
- Read the treaty through completely but quickly to gain an overall sense of the treaty's goals and requirements.
Then, re-read the treaty more slowly, and identify the explicit goals that motivated its creation, usually identified in
the preamble but also elsewhere, as well as the explicit behavioral requirements laid out in the substantive articles.
Make sure that you have a sense of what the treaty intends to do, in the words and definitions of that treaty. Seek to
answer the following questions:
Distinguish, as needed, between what the treaty.seems to want to accomplish and what it actually requires states to
- What were the treaty negotiators seeking to accomplish?
- What is the problem the treaty.seeks to solve?
- What are the specific substantive rules established by the treaty?
- What are the behavioral requirements placed on the signatories to the treaty?
- What behaviors are parties to the treaty required to report on?
Note that a single treaty often has several goals and numerous requirements, some quite different from each other
and others close but not quite the same. Identify as many distinct goals and requirements as possible since each can
provide one or more possible indicators of the treaty effectiveness.
"Brainstorm" a list of potential indicators for each goal and each requirement.
- For example, the Framework Convention on Climate Change seeks "to protect the climate system for present and
future generations" (Preamble) which is made more specific in Art. 2 as the "stabilization of greenhouse gas
concentrations in the atmosphere at a level that would prevent dangerous anthropogenic interference with the climate
system." However, the convention then requires only that developed states "adopt national policies and take
corresponding measures on the mitigation of climate change, by limiting its anthropogenic emissions of greenhouse gases
and protecting and enhancing its greenhouse gas sinks and reservoirs" (Art. 4).
Using the secondary literature on the treaty, identify other indicators of effectiveness by identifying those used
by other scholars, making use of and updating indicators used in those studies and avoiding unnecessary
"re-inventing of the wheel."
Taken together, this should produce a list of relevant indicators for a treaty that may be available.
- Begin by identifying specified indicators which the treaty delineates as standards against which it wants to hold
- Now, identify "related indicators" which are not specified in the treaty but would serve as a good
"proxy" of achievement of the goals that are specified. To give an example, the amount of oil pollution in the
ocean would not be a relevant indicator for the Framework Convention on Climate Change, but "greenhouse gas
concentrations in the atmosphere" and "sea-level rise" would be relevant specified indicators and
"average global temperature" could be a relevant related indicator even though it is not mentioned anywhere in
Identifying If Relevant Indicators Are Available
Armed with a brainstormed list of relevant indicators, you can now proceed to determining whether such data is
available. The following procedures seek to allow you to systematically determine whether an indicator that has been
deemed relevant to evaluating a particular treaty is available, and if not, to use a consistent search method that can
document, and hence provide confidence, that the indicator is not available in an extensive, although by definition not
comprehensive, set of likely sources.
- For each of your brainstormed relevant indicators, take each of the following steps until you have found adequate
- Go to the secretariat's web site, if there is one, and look for links to data sources.
- For each indicator, develop a list of search terms and search phrases to use in searching for the availability of an
indicator in the databases listed below and any others you can identify.
- If you find a data source, make sure to record the source AND the search term that uncovered it so that you can
re-find the data yourself and provide the information to others.
Starting databases to search:
Identifying Treaty.secretariats and Data Sources
This stage seeks to identify contacts for a treaty.secretariats that can assist the scholar in identifying both
relevant indicators and available data sources. Secretariat staff are very likely to have extensive knowledge regarding
which behavioral and environmental data relevant to a treaty is available and which is not. Because most scholars
working on a treaty will have been in contact with the secretariat, and because a wide range of data may be available
only to the secretariat and groups and contractors working for the secretariat, there may be far more information
available than would be easily identified through other sources. In many cases, secretariats will provide access to
closely-held data if the scholar promises to keep the information confidential and/or provides the completed analysis in
- Read the treaty to identify the name of the organization responsible for secretariat duties.
- Start with the Secretariat List established by Ronald Mitchell at
- If you cannot find the secretariat for a treaty there, search the Union of International Associations (
) web site. This site lists web (if available), fax, phone, and snail-mail contacts for the vast majority of
international organizations and treaty.secretariats.
- Examine other collections of international relations websites, such as those of the International Studies
) or the International Studies Network (
- If you still fail to identify secretariat contacts, search the web using different search engines, using popular and
official names for the treaty and secretariat to identify the main web site for the treaty in question.
- Once identified, always ensure that you have the official secretariat home page for the site and make sure to get
the exact address for contact information for the secretariat.
- For those for which web information does not appear available, look at the hard copy of the Yearbook of
International Organization, since this provides a more comprehensive listing of information than on the UIA web page.
- Once identified, spend time perusing the secretariat's website, or contact the secretariat by phone, fax, or
snail-mail to request information on issues of treaty effectiveness. Given the dearth of resources available to most
secretariats, an offer to undertake an analysis and provide the results to the secretariat free of charge upon
completion can often open many doors that would not otherwise be available.
If you identify a secretariat not already on CIESIN's website, please send CIESIN the following information:
- Relevant treaty name
- Secretariat official name and acronym
- URL for the secretariat home page, and treaty home page, if different
- Mailing address
- Phone number
- Fax number
- Email address
- Information officer, if any - preferably the name for the position, not the person
Cataloguing Data Source Information
Since it is often difficult to resurrect where you found data after the fact, it is important that you catalog the
following information at the time that you first find the data. It will be helpful to the scholarly community more
generally if you can also provide this information to CIESIN so that CIESIN can develop a larger meta-database of
information relevant to studying the effectiveness of environmental treaties.
Information to collect and catalog:
- Brief name of data: Provide a brief title that will allow quick identification
- Official title for data: Provide the fullest title possible, based on the original source document
- Source of data: Provide the name of the organization or individual that has produced the information.
Provide most complete citation available. Provide the URL or the hard copy citation, including exact pages, for the
data. For URLs that are identified through data compilations, provide exact sequence of choices to get to
data set and date of website access.
- Units of indicator: Provide the metric used to measure the indicator, e.g., tonnes of SO2, or violations
- Time period of metric for indicator: For example, annual, every five years, monthly
- Spatial scale of indicator: For example, global, regional, national, subnational, individual, continent, region,
country, substate aggregations, municipality, river basin, wetlands, national parks, hectare, etc.
- Actual, estimated or projected data: Delineate whether data is actual, estimated, or projected. Where
necessary, provide separate entries for different year spans, even if for same data set
- Years: Either separated by commas or by a hyphen, e.g., 1980-1996 or 1980, 1985, 1990, 1995
- Countries involved: Provide a list of all countries for which at least some data is available; e.g., Austria,
Czechoslovakia, Denmark, Germany, Finland, Hungary, Norway, Sweden, Switzerland, USSR.
- Other clarifications of data: E.g., (not legends or footnotes available from chart itself, but more general
clarifications of what data represents that are not captured in foregoing parameters)
- Treaty goal or requirement to which indicator is related: Specify the goal to which the indicator is related.
- Location of explanatory notes: Identify where clarifying notes such as footnotes are available providing further
clarification on the data.
- Contact person for data: Identify a contact person at the secretariat or elsewhere, if one is provided.