Review of Necessary Assumptions and Validity Conditions for "Natural Experiment" As a Policy Impact Identification and Assessment Methodology and Development of Alternative Methods for Identification by External Validity of Multiple Treatment Effects Those Single Natural Experiment Hardly Identify

Author Name KAINOU Kazunari (Fellow, RIETI)
Creation Date/NO. October 2018 18-J-030
Download / Links


"Natural Experiment" is one of the measures used for policy impact assessment ard regarded as an analytical methodology to identify particular treatment effect from phenomena that multiple factor complexly affect exploiting natural, economical and social events or condition change. In the application of "Natural Experiment", it is well-known that several points such as consistency of events or condition change and analytical issues should be checked and that adequate econometric methodology should be chosen in order to ensure internal and external validity of the results.

This research survayed and summarized 62 recent social science literature in major academic journals focusing the points such as what measures are taken to check internal and external validity and what econometric methodologies are applied and derives standardized procedures to apply "Natural Experiment" ensuring both internal and external validity based on the survey. Furthermore, based on the derived standardized procedures, the author developed new methodology to isolate particular treatment effect from mixed treatment effect of "Natural Experiment" confirming external validity of multiple similar results of the particular treatment effect from other "Natural Experiments".

In order to demonstrate practical applicability and to identify possible limitations of the new methodology, the author tried road-tests to separate and identify the effect of East Japan Great Earthquake and Fukushima No.1 Nuclear Power Plant Accident on the agricultural products wholesale marlet of neghbouring Prefectures because those incidents simultaneously occurred and simple "Natural Experiment" cannot separate and identify those effect. The author exploited other "Natural Experiment" of large earthquakes such as Chuets and Kumamoto to confirm the external validity of earthquake impacts over the agricultural product wholesake markets.

As a result of the road-test above, the author found there are some case say "Partial Evxtermal Validity" which useful bound information such as scale or duration of particular treatment effect are identified to be valid for all similar "Natural Experiments"although no complete external validity confirmed. Those new methodologies are emerged as a useful methodology for quantitative policy impact assessment where laboratory experiments and/or other approaches are hardly applicable due to some obstacles.