Making policy proposals based on objective data is a crucial mission for RIETI. Four years ago, we launched an Evidence-Based Policy Making (EBPM) research project. Has the EBPM approach become established in Japan during the past four years? How should EBPM be used to respond to life-threatening crisis situations such as torrential rain disasters and COVID-19? We asked Fumio Ohtake, Specially Appointed Professor at Osaka University and member of the Japanese government's Advisory Council on Countermeasures against Novel Influenza and Other Diseases, who has played a leading role in RIETI's EBPM research from the start.
Interviewed by SABURI Masataka (Director, International Coordination and PR Strategy, RIETI)
Please tell us about the EBPM research you have undertaken at RIETI so far.
At RIETI, we have held research meetings on Evidence-Based Policy Making (EBPM) for the past four years. We also hold annual symposiums. The primary purpose of the research meetings has been to establish the EBPM approach to policy in Japan. Before that, however, we had to begin by defining what EBPM means. The Japanese government had previously carried out policy assessments, but these were focused on assessing whether budgets were allocated and used in accordance with the original plans. Yet the essential purpose of policy assessment should be to ascertain whether or not the established policy goals have been achieved. In other words, policy assessment should include outcomes, which measure the results of policy goals, and outputs, which measure the effectiveness of the projects themselves. However, we found that in many cases it was not outcomes or outputs that were being assessed, but rather inputs; that is, whether the project had been implemented according to plan. We therefore started by establishing a common understanding of goal setting for outcomes and outputs. To begin with, in many cases we found insufficient explanation, based on scientific knowledge and evidence, of the logical mechanisms that would cause a policy input, say Project A, to lead to the desired outputs and outcomes. This was the first step toward establishing EBPM as a general approach.
Next, we were faced with the problem of how to measure the effectiveness of a policy. This was often measured by comparing the situation before and after the policy was implemented, but this does not constitute an accurate policy assessment in terms of the causal inference approach used in fields such as statistics and economics. When testing the effectiveness of a particular policy, it is crucial to compare the results after policy implementation with counterfactual assumptions regarding what the situation would have been like if the policy had not been implemented. During the initial research meetings and symposiums, we therefore aimed to define the goals of EBPM, and to establish general methods for evaluating policy effectiveness. I feel that this approach is now beginning to sink in.
We also considered that it would be beneficial for those entrusted with policy implementation if we provided specific, practical examples in addition to evaluation methods. Through RIETI research meetings, symposiums and other forums, we have provided EBPM case studies across a range of fields such as education and disaster prevention, as well as COVID-19 countermeasures, which we presented at the symposium in December 2021.
Moreover, through discussions at RIETI research meetings, we have endeavored to emphasize our ideal vision for EBPM: conventional policy assessment focused on the a posteriori evaluation of implemented policies. However, as our mission is "Evidence-Based Policy Making," we consider it vital to establish a number of policy candidates to achieve a specific goal, and ascertain a priori which policy will be most effective, before moving on to full-scale implementation. We have repeatedly discussed the idea that EBPM should be carried out as a preparatory step during the policy formulation or budgeting stage and linked to full-scale policy implementation.
After four years of work, I now feel that the EBPM approach is gradually taking root in Japan.
What kind of research are you engaged in at the moment? Can you tell us a little about the results of this research?
I am currently engaged in two types of research related to EBPM. The first is the validation of policy effectiveness. This includes, among other work, the analysis of education policy effectiveness in collaboration with Amagasaki City in Hyogo Prefecture and Nara City in Nara Prefecture. For example, we have examined the impact of factors such as small class sizes, special abacus zones, temporary school closures due to COVID-19, and poverty on academic ability and soft skills. The preparatory work before beginning the actual research was quite demanding, as we needed to start by constructing datasets. The second type of research I am engaged in is closer to the original EBPM approach. It involves a priori analysis of what kind of policies will be effective, finding evidence, and linking it to policy decisions. Here, I would like to mention my research in Hiroshima Prefecture, where many lives were lost in torrential rain disasters. Hiroshima Prefecture has introduced a system of disaster prevention education, and a very large proportion of residents are aware of shelters and evacuation routes. However, not many residents actually evacuated when torrential rain struck the prefecture in 2018, leading to significant casualties. We therefore examined the effectiveness of policies using the behavioral economics concept of "nudges." We used a method known as a randomized controlled trial (RCT), based on the results of questionnaire surveys by 10,000 respondents. We sent respondents one of six types of messages, selected at random, and examined how much their evacuation awareness had changed. We also conducted a follow-up survey of the same respondents eight months later, to determine whether or not they were continuing to engage in disaster prevention behavior.
The main issue we encountered with the surveys was what kind of messages to send. From interviews and surveys of people who had evacuated during the torrential rain disaster, we found that the vast majority had evacuated only after seeing those around them evacuating or being warned to evacuate by those close to them. Phenomena such as this, where people determine their own actions with reference to the behavior of those around them, are referred to in behavioral economics as "social norms." It was therefore clear that expressions like "those around you are evacuating" would effectively promote evacuation, but such expressions could not be used in situations where many people have yet to evacuate. Such expressions were also unsuitable for community education activities, so we had to use a little ingenuity. We came up with the following message: "If you evacuate, you can save the lives of people close to you" (Table 1-A). Using this message in community education activities promotes the awareness that evacuating is an altruistic act that can save the lives of others. We also developed a negative message—"If you do not evacuate, you are putting people's lives at risk" (Table 1-B)—and messages indicating the benefit of being able to obtain supplies by evacuating to a shelter—"Evacuating to a shelter will help you secure food and blankets"; and "If you do not evacuate, then you may not be able to secure food or blankets" (Table 1-D and E).
|A. Influence gain nudge||In the past, most people who evacuated in response to evacuation orders during heavy rains did so because others around them were evacuating. If you evacuate, you can save the lives of people close to you.|
|B. Influence loss nudge||In the past, most people who evacuated in response to evacuation orders during heavy rains did so because others around them were evacuating. If you do not evacuate, you are putting people's lives at risk.|
|C. Reference point||When evacuation advisories are issued, due to heavy rains, it is necessary to evacuate as soon as possible. If you must remain at home, just in case, please wear something that can help identify you, as your life may be in danger.|
|D. Gain-framed relief goods||When evacuation advisories are issued, due to heavy rains, evacuating to a shelter will help you secure food and blankets.|
|E. Loss-framed relief goods||If you do not evacuate to an evacuation site when an evacuation order is issued, due to heavy rain, you may not be able to secure food or blankets.|
|F. Control||Every year, most rain falls between the beginning of the rainy season, around the start of June, and autumn, due to the influence of rainy season fronts and typhoons. In Hiroshima Prefecture, there have been many disasters, such as landslides, where mountains and steep slopes collapse. We should learn about the damage caused by heavy rainfall and protect our lives from disasters by developing the ability to make good decisions and taking action when danger is imminent.|
The results of the questionnaire survey showed that those who received messages A or B had a strong intention to evacuate (Figure 1). Moreover, the follow-up survey indicated that those who received message A still maintained disaster prevention behavior even eight months after the message was sent. The effect of the negative message, on the other hand, had disappeared after eight months. From a policy perspective, we found that positive messages that emphasize societal impact (e.g., "your evacuation will save lives") were extremely effective. As a result, Hiroshima Prefecture began extensive use of the "your evacuation will save lives" style of messaging. (Figure 2)
I am confident that this research represents a good example of EBPM using nudges. We developed a number of message candidates that we anticipated would be effective in realizing policy goals from a behavioral economics point of view, examined the effects of these messages, and implemented those that we ascertained were effective and deployable from an administrative standpoint.
Another example I would like to talk about is the messaging used for COVID-19 countermeasures. A state of emergency was declared in Japan from April 2020, and a policy goal was established of reducing interpersonal contact by 80% during the Golden Week holiday from the end of April to early May. The message most utilized at the time was "Stay home." We developed other candidate messages and evaluated which was the most effective (Table 2). Was it best to simply provide information? Or was it better to have an altruistic message like "By refraining from going out, avoiding the ‘3 Cs (closed spaces, crowded spaces, and close contact),’ washing your hands, and wearing a mask, you can protect the lives of people close to you" instead? Or was a selfish message—"By refraining from going out, avoiding the ‘3 Cs,’ washing your hands, and wearing a mask, you can protect your own life"—better? We also developed a range of other messages based on behavioral economics, including a loss-framed altruistic message, and a combination of altruistic and selfish messages. Throughout the 2020 Golden Week, we implemented an RCT using online surveys. As a result, we found that a gain-framed altruistic message (Table 2-B) was the most effective.
|A. Control||To prevent infection, reducing contact with others, avoiding the "3 Cs" (closed spaces, crowded spaces, and close contact), practicing proper hand washing, and wearing a mask are effective.|
|B. Gain-framed altruistic message||By refraining from going out, avoiding the "3 Cs," washing your hands, and wearing a mask, you can protect the lives of people close to you.|
|C. Loss-framed altruistic message||By going out, not avoiding the "3 Cs," and not washing your hands or wearing a mask, you will put the lives of people close to you at risk.|
|D. Selfish message||By refraining from going out, avoiding the "3 Cs," washing your hands, and wearing a mask, you can protect your own life.|
|E. Altruistic and selfish message||By refraining from going out, avoiding the "3 Cs," washing your hands, and wearing a mask, you can protect your life and the lives of people close to you.|
|F. Simple message||Stay home. You can protect the lives of people close to you.|
In 2021, we went on to examine measures to promote vaccination. In Japan, vaccinations for the elderly began in earnest from May 2021. We began research into effective messaging from around January 2021, so that our research results could be utilized in the actual vaccination drive. Through our research, it became increasingly clear that people are significantly influenced by social norms. There is not a strong desire for vaccination while the number of vaccinated people is still low, but as that number increases, so does the desire to get vaccinated. Of course, just like in the case of evacuations, vaccination programs are always going to start with only a few people. We developed and tested three candidates messages to send at this early stage (Table 3). As a result, we found that a gain-framed social impact message (message B) was the most effective in increasing motivation to get vaccinated, with a significant effect observed among the elderly in particular. On the other hand, we found that a loss-framed message (message C), while effective among some people, also represented a psychological burden.
|A. Comparison nudge||Seven to eight out of 10 people in your age group answered that they would receive this vaccine.|
|B.Influence-gain nudge||The more people receive this vaccine, the more people have the intention to do so. Your vaccination behavior can encourage the vaccination behaviors of those around you.|
|C. Influence-loss nudge||The more people receive this vaccine, the more people have the intention to do so. If you do not receive the vaccine, the people around you may also not do so.|
In this way, even before a policy is implemented, it is possible to test its effectiveness to some extent by using online surveys such as those where messages are distributed to respondents at random. I am also involved in research into messages to promote antibody testing for rubella. In this case, we not only carried out surveys but also tracked participants' actual behavior, finding a significant correlation between their intentions and behavior. I believe that if we establish a framework for constant implementation of surveys using this simple method, then we will be able to test the effectiveness of messages in real time as we deploy them in policy, even when faced with policy issues where the situation is rapidly changing, such as during the COVID-19 pandemic.
How should EBPM be developed in Japan in the future?
The EBPM method I have described enables a relatively simple and logical way to test for the best approach. For example, in practice, the composition of notices such as posters and notifications written by those responsible for policy implementation is left largely to the discretion of the person writing the notice. When providing information, it is easy for the person writing the notice to alter the design and composition of these notices slightly, without changing their actual content. It is relatively simple to obtain a comparison of outcomes, such as which notification elicited the greatest number of applications, and the effectiveness of different approaches can be rapidly verified. There are various other sophisticated methods used to verify causal inference, but in this way, nudges can be used first of all to provide an extremely accessible, low-budget method of testing effectiveness on the level of those responsible on the front line of policy implementation. I think it’s best to learn the basics of EBPM from this approach to begin with, also in terms of human resources formation and development. In this way, it’s possible to gain a first-hand understanding of the entire process, from the reasoning behind policy proposals and methods of verifying their effectiveness, to actual implementation based on these results. Of course, many government undertakings are very large in scale, and it is not possible to test their effectiveness so easily. However, even at the policy design stage, personnel with experience in the kind of approach described above should have an understanding of what sort of policy design will facilitate the verification of effectiveness later in the process, and provide useful hints for policy improvement.
I think that empirical verification using nudges from behavioral economics is a highly effective method of gaining expertise in EBPM. It does not require large-scale surveys or sophisticated statistical techniques, and I sincerely hope that it is practiced by as many people as possible.