Evidence of "Evidence-based Policymaking"

The importance of evidence-based policymaking has been highlighted in recent years. International institutions led by the Organisation for Economic Co-operation and Development (OECD) and the World Bank are promoting this policymaking approach vigorously, while some advanced economies, such as the United States and the United Kingdom, are taking concrete steps to put it into practice (Note 1). The approach has been applied to various policy areas including healthcare, social security, labor, and education. Japan has recently begun to take similar steps in some policy areas such as science and technology. In a bid to promote evidence-based policymaking, the Ministry of Economy, Trade and Industry (METI) seems to be planning to mandate the presentation of empirical evidence, such as statistical data and research findings, as a requisite to introducing new policy programs. The move should be welcomed as it contributes to the effective use of limited financial, personnel, and other policy resources.

In order to ensure that evidence-based policymaking will deliver its intended effects, not only policymakers but also relevant research institutes have important roles to play. For instance, IZA, a leading think tank in Europe which has a partnership with RIETI, has a website section called "IZA World of Labor: Evidence-based policymaking," which is designed to provide policymakers with findings from academic policy research in the areas of labor economics. Its research findings have also been compiled and published as a book (Zimmermann and Kritikos, 2015). High-quality research findings are presented in a way that is easy to understand for the general public on various issues of great interest to Japan including non-standard employment (part-timers, agency workers, etc.), gender inequality, child care support, foreign labor, and employer-provided education and training.

RIETI also takes it as its important mission to put forward evidence-based policy proposals. Specific examples include Fujita ed. (2016), a recent publication from RIETI, which has a subtitle entitled "Evidence-based Policy Recommendations" and provides a bird's eye view of research findings in each area relevant to putting the Japanese economy on a sustainable growth path.

However, in order to promote evidence-based policymaking, we first need to have evidence as to how such an approach is perceived by Japanese policymakers, to what extent it is being utilized in a practical policymaking setting, and what factors, if any, stand as obstacles to taking such a policymaking approach.

With an aim to link RIETI's research findings to policy planning and formulation, we conducted surveys to find out how policymakers and policy researchers perceive evidence-based policymaking, asking their views on: 1) the necessity of evidence-based policymaking, 2) policymakers' awareness, 3) progress hitherto made in implementing evidence-based policymaking, and 4) factors inhibiting the implementation thereof. In doing so, we thought it would be desirable to be able to identify the perception gap between policy researchers (including think tank researchers and academic scholars engaging in policy research) and policymakers. Thus, we conducted two separate surveys on two different samples—(A) government policymakers and (B) policy researchers (RIETI fellows)—using almost identical questionnaires (Note 2). Since these are simple surveys conducted on small samples, I have considerable reservations about interpreting the survey results. But I would like to introduce some of our preliminary findings.

Aggregate survey results regarding the first three questions are shown in Figure 1. The number beside each bar, which takes the value of one through four, represents the average score rated by respondents in each group using a single-answer, multiple-choice format. The higher the value is, the greater is the tendency for the respondents to believe that "evidence-based policymaking is necessary," "policymakers are aware of evidence-based policymaking," or "evidence-based policymaking is implemented" as applicable, with a value of 2.5 indicating that the positive and negative responses are roughly equivalent (Note 3).

Figure 1: Evidence-based Policymaking

Note: Rated on a scale of one to four. The higher the value is, the higher is the level of necessity, awareness, or implementation as applicable.

Both policymakers and policy researchers showed similar patterns in their responses. Respondents in both groups are quite forthcoming in acknowledging 1) the necessity of evidence-based policymaking. However, they gave rather low scores to 2) policymakers' awareness of evidence-based policymaking, and their evaluations were even lower on 3) the implementation of evidence-based policymaking. Meanwhile, a comparison between the two groups show that policy researchers' evaluations on policymakers' awareness of evidence-based policymaking are lower than policymakers' self-evaluations, and the same tendency is observed on the degree of its implementation (Note 4).

One interpretation of this is that the differences represent perception gaps between the two groups on the quantity and quality of evidence deemed necessary. Whether in making budget requests or amending laws, there are many hurdles that need to be cleared, such as priority setting within each organization, scrutiny in the budget-making process, and deliberations at the Diet, making it impossible to formulate any policy without evidence. Furthermore, in a relatively recent move, government policy programs have been made subject to ex post evaluation, for instance, through the administrative project review system launched several years ago.

At the forefront of policymaking, aggregate statistical data, information obtained from interviews with companies, and overseas case examples tend to be used as evidence showing the need to maintain, alter, or scrap specific policy programs. In contrast, policy researchers are more inclined to look to detailed empirical analysis in assessing the effects of policy measures. For instance, they would try to determine whether an observed correlation is a causal relationship, or they would want to know the size of effects in quantitative terms. Also, there is a growing tendency to put emphasis on the estimation of causal relationships based on evidence from natural experiments and more recently on policy evaluation using randomized controlled trials (RCT).

The quantity and quality of evidence needed in actual policy formulation vary depending on the skill levels of senior decision-makers and examiners. This is a question of whether it should be considered sufficient enough to present specific examples and show the existence of a correlation, or if further evidence is required.

Figure 2 illustrates responses to the question concerning factors inhibiting evidence-based policymaking (Note 5). It is a multiple-answer question and each percentage value represents the ratio of respondents who selected the answer. "Policy decisions are made politically irrespective of evidence" is the most chosen answer by policymakers, followed by "Government officials are not sufficiently skilled to analyze statistical data and understand relevant research findings," and "Evidence-based policymaking is neither a customary practice nor in line with the organizational culture" in that order. This compares to "Government officials are not sufficiently skilled to analyze statistical data and understand relevant research findings," "Policymakers are too busy in daily works and unable to find time for evidence-based policymaking," and "Policy decisions are made politically irrespective of evidence" selected by policy researchers, showing some differences in the patterns of responses between the two groups (Note 6).

Figure 2: Obstacles to Evidence-based Policymaking

Note: Multiple answers are allowed and each percentage value represents the ratio of respondents who selected the answer.

It is interesting that policymakers, who appear very busy in the eyes of policy researchers, do not find their heavy workload as constraints on evidence-based policymaking (Note 7). "Government officials are not sufficiently skilled to analyze statistical data and understand relevant research findings" was selected by roughly two-thirds of the respondents in each group, suggesting that improving government officials' analytical skills is crucial to evidence-based policymaking. It is expected that not only the use of microdata collected for government statistics but also the applications of big data and artificial intelligence may become feasible in the coming years, and it may require much higher levels of skills to utilize them (Note 8).

Few respondents deny the necessity of evidence-based policymaking. However, when asked whether they think evidence-based policymaking is being implemented in Japan, both policymakers and policy researchers are far from positive in responses. In addition to evolving the existing policymaking and ex post evaluation mechanisms, government officials need to improve their skills to utilize academic research findings. At the same time, it is also important to enhance the interest of academic scholars and researchers in real-world government policies, and thereby accumulate and disseminate in an easy-to-understand way research findings useful for policymaking.

As aforementioned, the surveys were conducted on small samples, which are presumably biased in favor of evidence-based policymaking as respondents contained therein are either policymakers having contact with RIETI or researchers engaging in policy research at RIETI. There is no ruling out the possibility that policymakers who have no contact with policy research and scholars who have very few occasions to interact with policymakers may have different views. Also, the ratings provided by respondents on various aspects of evidence-based policymaking are based on their subjective judgments.

Also, even though both policymakers and researchers definitely affirm the importance of evidence, they may differ in their understanding of what constitutes sufficient evidence. Finding ways to bridge such gaps between policymakers and researchers is an important role that policy think tanks, including RIETI, are required to play.