The Significance of the Revision of the National Guidelines for Evaluating Government Funded R&D

The National Guidelines for Evaluating Government Funded R&D were revised at the Council for Science and Technology Policy and will be followed in earnest in 2013. This revision is quite significant, coming at a time when Japan is faced with a pressing need to strengthen its science, technology, and innovation policies to promote economic growth.

In what follows, I will introduce recent trends in Japan's science and technology policies and discuss the significance of the latest changes to the guidelines.

As the world economy continues to stagnate following the collapse of Lehman Brothers and the sovereign debt crisis in Europe, securing economic growth has become a priority policy issue for countries around the world. Particularly in developed nations, how to parlay the results of research and development (R&D) activities into innovation and achieve economic growth has become a major policy issue.

Meanwhile, fiscal reform has become a pressing concern not only in Japan but also in the United States and Europe. Against this backdrop, the efficient implementation of science, technology, and innovation policies has become another focus point. For example, in the OECD Science, Technology and Industry Outlook 2012 report, which the Organisation for Economic Co-operation and Development (OECD) publishes biennially (Note 1), case studies of efforts to strengthen the governance of science, technology, and innovation policies, such as integrating related laws and regulations, streamlining evaluation procedures, and establishing specialized evaluation organizations, are introduced.

The National Guidelines for Evaluating Government Funded R&D (hereinafter referred to as the National Guidelines) are revised as appropriate in tandem with the formulation of the Science and Technology Basic Plan (Note 2), which is drawn up every five years according to the Basic Act on Science and Technology. The latest revision is in line with the Fourth Science and Technology Basic Plan, formulated in 2011.

I would like to briefly touch on the Fourth Science and Technology Basic Plan. The plan has incorporated a new policy perspective that calls for promoting the entire process of innovation—from R&D to application—in an integrated manner. This marks a shift from the conventional "R&D seeds-push" approach, which focuses on the incubation of R&D seeds, to a "problem-solving" approach.

The problem-solving approach starts out with identifying a goal or a specific problem that needs to be solved and then R&D and innovation policies—those designed to promote R&D activities and disseminate fruits resulting therefrom into society—toward reaching that goal will be promoted in an integrated manner. This is a major change from past science and technology policies, which were centered around the promotion of R&D activities.

Also underlying the latest revision is the notion that Japan's innovation system as a whole, including the evaluation of R&D activities, might not have been effective as a mechanism for promoting innovation. In the past, the chief purpose of evaluating government-funded R&D activities was to establish accountability. It is thus fair to say that interim and ex-post evaluations were respectively intended to determine whether R&D projects were progressing as planned and whether their expected goals were achieved. Accountability will undoubtedly remain important in the future, but it is becoming increasingly critical to consider how to promote innovation in the evaluation of R&D activities.

In my view, the latest revision of the National Guidelines is significant in two aspects. First, it called for introducing program evaluation. Second, it pointed to the need to make program outcomes visible and measurable.

A "program" referred here is a policy package—composed of multiple R&D projects and measures to promote the dissemination of their results—that will be taken as the smallest unit of evaluation. What is important from an innovation policy standpoint is to implement R&D projects along with measures to facilitate the dissemination of their fruits—such as deregulation, subsidies, and tax breaks—as a package. In promoting R&D in the form of a packaged program, we must create an appropriate mechanism for evaluating such programs.

The Ministry of Economy, Trade and Industry (METI) has been making pioneering efforts to promote and evaluate the implementation of packaged R&D programs under the name of the "evaluation of R&D measures." What has been done so far, however, is limited to the "bundling of multiple R&D projects"—i.e., evaluating and managing closely-linked R&D projects as a single package—and there is a long way to go to develop such mere bundles of R&D projects into full-fledged programs with an eye on measures to disseminate R&D fruits (Note 3).

The term "Outcome" referred to in the guidelines is an indicator measuring the effects of R&D outcomes. For example, sales revenues from a product developed by utilizing a certain R&D outcome or new product specifications can be used as an Outcome index. In contrast, "Output" is an indicator measuring the level of activities, and its examples include the number of academic papers and the number of patent applications filed. Because of the relative ease at which to obtain and assess such data, the latter type of indicators have long been used as a primary evaluation measurement. However, under the new innovation policy that promotes R&D and the dissemination of the resulting outcomes in an integrated manner, it is essential to make such outcomes visible and measurable. What becomes important in this process is strengthening the ex-ante and follow-up evaluation.

As its name suggests, ex-ante evaluation is to evaluate the necessity of an R&D project prior to its implementation. Unlike interim and ex-post evaluations, where specific data are available to be evaluated (e.g., the progress of the project in the case of the former and a specific R&D outcome in the case of the latter), no such data exist in the case of an ex-ante evaluation. Therefore, it will instead evaluate the significance of an R&D project, its intended outcome, plans for measures to disseminate such outcome into society, and so forth.

Among all, the intended outcome of an R&D project and its relevant measures is the most important item in ex-ante evaluation because clarifying what is aimed to be achieved helps clarify how they relate to other projects and measures targeting similar outcomes, and enables the effective and efficient attainment of the intended outcome. In view of this, the new National Guidelines advocate strengthening ex-ante evaluation, suggesting the increased use of feasibility studies as an example of specific measures to accomplish this.

Follow-up evaluation is to examine and assess an R&D project comprehensively after a certain period of time has elapsed following its completion, for instance, with respect to whether or not the outcome of the project has been commercialized and what effects or impact it has had on the size of the relevant market, industrial competitiveness, human resource development in the relevant area, and people's livelihoods in general. Because of the significant time and costs involved in the collection and analysis of basic data, follow-up evaluation hitherto has only been conducted for large-scale R&D projects.

Follow-up evaluation is an attempt to find out what has been achieved as an outcome of R&D efforts, and from an accumulation of such evaluation experiences, we can derive important implications for our future science, technology, and innovation policies.

One relatively recent example is METI's report on the results of its follow-up evaluation of R&D for solar power generation (Note 4). This was an attempt to assess a series of R&D activities for solar power generation undertaken as part of the Sunshine Project, a long-term government program launched immediately after the first oil crisis, in a comprehensive manner, ranging from technological achievements made in the process to the final outcome. Although the technical achievements were generally highly rated, the report states that efforts to disseminate their benefits into society were not necessarily sufficient, having taken place mostly in the form of subsidies to promote the installation of residential solar power systems.

By continuing to add such follow-up evaluation results, we should accumulate our knowledge as to what policies are needed to facilitate the dissemination of R&D fruits and incorporate that knowledge in formulating innovation policies.

"Winning in technology but losing in business" is a phrase typically used to describe the Japanese electronics industry, which has been losing global market share despite its sophisticated technologies. In order to change this situation, the Fourth Science and Technology Basic Plan took a radical shift in policy direction toward those of the problem-solving approach including measures to facilitate the dissemination of R&D fruits.

To achieve sustainable economic growth under the adverse influence of a declining and aging population, Japan has to engage in bolder and more adventurous innovation than do other countries. As Japanese companies are cutting back on their R&D investments (Note 5), it is all the more important for the government to implement R&D innovation policies from a medium- to long-term viewpoint to generate innovation seeds. Needless to say, this must be done both efficiently and effectively under severe fiscal constraints.

It is sincerely hoped that the purport of the latest revision of the National Guidelines will be fully put into practice and become a strong tool for formulating and implementing science, technology, and innovation policies that can facilitate much-needed economic growth in Japan.