Effect of Shadow Education Vouchers and the Implications for Evidence-based Policymaking

In recent years, evidence-based policymaking (EBPM) has become widespread overseas (Ｎote 1). In Japan, also, the Statistical Reform Promotion Council announced its final recommendation in May 2017 indicating the basic policy for the promoting EBPM in Japan. In conjunction with this, the EBPM Promotion Committee was launched in August 2017, and, starting from the current fiscal year, the Director-General for Evidence-based Policymaking was appointed at each ministry.

Following these government policies, RIETI established a new framework for EBPM promotion to function as a hub for policy practitioners and researchers to conduct consulting activities during the process of policy formation and ex-post evaluation, among others, and reinforce its initiatives relating to EBPM. Furthermore, the study group of the Promoting Evidence-based Policy in Japan project (https://www.rieti.go.jp/en/projects/program_2016/pg-09/004.html), headed by Visiting Fellow Kazuo Yamaguchi, has been providing opportunities for vigorous discussions between the participants from various research organizations and government agencies, and continuing research into how to advance EBPM in Japan based on case studies from both Japan and abroad. In December 2017, the RIETI EBPM Symposium "Toward the Promotion of Evidence-based Policymaking" (https://www.rieti.go.jp/en/events/17121901/info.html) was held, and experts from various organizations exchanged opinions on how to disseminate and utilize EBPM in Japan by presenting actual cases from both Japan and abroad.

RIETI's research activities in regard to EBPM is not limited to the aforementioned projects. As part of the Reform of Labor Market Institutions project (https://www.rieti.go.jp/en/projects/program_2016/pg-07/001.html) headed by Program Director/Faculty Fellow Kotaro Tsuru, I recently published a discussion paper titled "The Effect of Shadow Education Vouchers after the Great East Japan Earthquake: Evidence from regression discontinuity design" (https://www.rieti.go.jp/en/publications/summary/18050007.html). The purpose of this paper is to estimate the effects of shadow education vouchers introduced by Chance for Children (https://cfc.or.jp/, "CFC"), a public interest incorporated association, in the areas devastated by the Great East Japan Earthquake.

In order to promote EBPM, accurate and efficient estimation of the effects of various measures will become essential. This article aims to present a summary of the results mentioned before while at the same time summarize the implications for EBPM.

CFC is an organization established in 2011 engaged in the shadow education voucher program in the areas devastated by the Great East Japan Earthquake and the Kumamoto earthquake. Figure 1 illustrates the shadow education voucher system. CFC selects voucher beneficiaries from the applicants who hope to receive the vouchers within CFC's budget by taking into account the applicant households' income, their year in school, and children's motivation for learning.

The beneficiaries can use the vouchers for a wide range of educational services including cram schools and extracurricular lessons. While the use of vouchers is expressly limited to educational purposes, the system is extremely flexible, and the beneficiaries can select services according to their individual needs. The amounts provided by the vouchers vary by the school year of the beneficiary; for example, third year students in junior high school and high school preparing for entrance exams receive 300,000 JPY per year.

In addition to financial support, voucher beneficiaries can also receive mentoring from university student volunteers in the form of advice and counseling on their future plan, etc. CFC is training university student volunteers through seminars and other means.

[Click to enlarge]

It is generally considered to be difficult to estimate the effects of such programs targeting children (Ｎote 2), but due to the constraints of its budget, CFC quantifies the need for support of its applicants based on household's income and the school year of the applicant into an eligibility index, and provides support in descending order of need. The paper also exploits this system in analyzing the effects.

We utilized a quasi-experimental method called regression discontinuity (RD) design. Figure 2 illustrates the conceptual image of RD design. The horizontal axis indicates the score calculated based on factors such as households' income and the school year, and the vertical axis depicts the outcome/output index including academic achievement and study hours. The threshold to accept or reject is referred to as the cutoff. In this case, children on the right side of the cutoff are given vouchers and the children on the left side of the cutoff are not.

Those eligible for vouchers and those who are not vary in such a wide range of attributes that we cannot estimate the effects of the vouchers by simply comparing the two groups. However, with this system design, estimation becomes possible as the children around the cutoff are assumed to have very similar attributes. This is the basic concept behind RD design. As indicated in Figure 2, if a difference in academic achievement are observed around the cutoff, we can assure that it indicates the effect of the voucher.

Figure 3 illustrates the effect of the shadow education voucher on academic achievement, which is translated into deviation value. As indicated in the results, in cases without vouchers, the increase in the deviation value was mostly flat at 0.1, whereas in cases with vouchers, it rose by 4.5. Moreover, the effects were greater among children living in relative poverty than those who were not.

As stated in the paper, while the results remain provisional, the following four points are considered to be the background factors behind this result.

First of all, the system was designed flexibly. As stated earlier, while the use of the shadow education vouchers is limited to educational purposes, beneficiaries can flexibly select educational services in line with their needs. As a result, suitable services are provided each beneficiary with. Second, we can also think that the children received high-quality services supplied by private sector which faces intense market competition. Third, it is possible that since children attending cram schools using the voucher could not be distinguished from other attendances without the voucher, the system hardly aroused any stigma. And fourth, as the system offered assistance from university student volunteers as well as financial support, it might improve the children's motivation for learning and mental stability, thereby enhancing the effects

At the end, we will point out the following implications for EBPM which can be induced in the process of estimating the effects of shadow education vouchers.

First, it is very important to gather ex-ante and ex-post data from both the treatment group and the control group. The analysis of the shadow education voucher system was made possible since CFC had collected data from not only the voucher beneficiaries but also from the non-beneficiaries. In policymaking, it is all too common to retain copious amounts of data on the treatment group while hardly any data are retained for the control group. It is also important to gather data before the treatment as well as after the treatment. The ex-ante data can make analysts control individual/corporate attributes and improve the precision of the effect.

The second point, which is related to the first point, is that an agreement must be reached on the outcome index, the improvement of which will be targeted by the measures (i.e., academic achievement and study hours, in this analysis). This will ensure that data will be gathered without omissions on the necessary indices.

Third, it is essential that the effect measurement methods are studied beforehand. Although we used the RD design method in this analysis, it was purely coincidental that we were able to use this method and was not intended. However, when determining policies for subsidies to corporations, for example, examining the effects would be difficult if the eligibility criteria are unclear, whereas if the criteria have been clarified, analysis using the RD design method, etc. would be possible.

These implications point to the importance of making preparations prior to the implementation of the measures. Whether it is data gathering or the study of analysis methods, much cost and trouble would be saved by making preparations before undertaking the measures.

EBPM is not intended as policymaking based on evidence alone. The term EBPM originates from evidence-based medicine ("EBM"). Sackett et al. (1996), for example, defined EBM as "use of current best evidence in making decisions about the care of individual patients" and stated that the practice of EBM means integrating individual clinical expertise with the best available external clinical evidence.

As stated earlier, the results of this analysis are provisional, and even if the analysis results were valid, the mechanism has not been made clear. The flexible system of vouchers may have been a good system; the utilization of private services may have been good; or it may have been the counseling support, which made the difference. None of these points have been made clear from this analysis. In many empirical studies, also, the results remain provisional. We must remember that even if further advances are made in EBPM, the insights of experts and experience of practitioners should never be taken lightly.

Nevertheless, if evidence is accumulated, it will be possible to reduce the uncertainty involved in decision making at the time of policy formation, and once the effects become clear, the measures may also be improved. We have entered the stage of building a better system through the cooperation of politicians, administrative officials, and outside researchers.