How Should We Measure the Development of Artificial Intelligence Technology?

Fellow, RIETI

In recent years, artificial intelligence (AI) has been attracting increasing attention. Figure 1, which shows year-to-year changes in the number of newspaper articles containing the term "artificial intelligence" in Japan indicates that in the four years since 2015, public interest in AI has grown rapidly.

Figure 1: Number of AI-related Newspaper Articles
Figure 1: Number of AI-related Newspaper Articles
Source: All newspaper articles recorded in Nikkei Inc.'s Nikkei Telecom database (as of November 22, 2018; the number in 2018 is an estimate).

Expectations and Concerns over AI

However, the attention focused on AI has two aspects—high expectations and concerns. On the one hand, AI technology is expected to play a key role in improving the productivity of various economic activities. In particular, if economic growth is to be achieved in Japan, which has entered the era of an aging society, low birth rate, and population shrinkage, it is essential to improve productivity. Therefore, there are high expectations for productivity improvements using AI.

On the other hand, there are concerns that the development and use of AI technology will substitute some employment, at least in the short term, thereby increasing inequality. In particular, a research paper by Carl Frey and Michael Osborne of Oxford University published in 2013 has created a huge media buzz around the world, including in Japan, with its estimate that AI is highly likely to deprive workers of jobs in some occupations (Frey and Osborne 2013).

Over the long term, there are also concerns that most jobs may be replaced by AI. For example, Erik Brynjolfsson and Andrew McAfee of the MIT Sloan School of Management emphasized in a book they co-authored and published in 2011 that insufficiently fast changes in social systems, companies' business strategies, and workers' skills to catch up with the rapid development of digital technologies are linked to inequality and weak job growth and warned that unless our society responds significantly to "a deeper structural change," both the wealthy and poor will suffer considerable consequences (Brynjolfsson and McAfee 2013). Three years later, in another book they co-authored, Brynjolfsson and McAfee expressed a stronger sense of crisis (Brynjolfsson and McAfee 2015). The problem of inequality brought about by AI is not limited to workers. There has also been talk about risks inherent in the monopoly of big data, which form the basis of the development of AI, by a small group of huge internet companies, represented by Google, Facebook and Amazon (The Economist 2018).

Initiatives to Measure the Development of AI Technology

Under these circumstances, if we are to consider appropriate methods and policies to deal with AI technology, it is necessary to accurately understand the current development stage of this technology and how it is being used. In other words, it is important to measure the development of AI technology, both qualitatively and quantitatively. Although it is not easy to determine the state of AI technology, there are some potential methods of analyzing its development using data concerning research papers and patents (e.g., Japan Patent Office 2015, OECD 2017, and Fujii and Managi 2018).

I conducted a quantitative analysis concerning the development of AI technology using bibliographical information from research papers and patents and data related to research projects funded by public competitive research funding under a joint research project with the National Graduate Institute for Policy Studies' SciREX Center, the National Institute of Science and Technology Policy (NISTEP), and the Japan Science and Technology Agency's Center for Research and Development Strategy (CRDS), which is being implemented as part of a project to promote "Science for Policy" (SciREX) under the science and technology innovation policy of the Ministry of Education, Culture, Sports, Science and Technology. Described below are the provisional analysis results I presented at a TIER-RIETI-KIET workshop called "AI: Asia – The next frontier in AI development," which was held on October 11, 2018.

First, let us look at the trends in the numbers of AI-related papers and patent applications. Figure 2 shows that the numbers of AI-related papers and patent applications have increased steeply around the world in the past five years or so.

Figure 2: Numbers of AI-related Papers and Patent Applications (by year of publication/application)
Figure 2: Numbers of AI-related Papers and Patent Applications (by year of publication/application)
Source: Academic documents recorded in the JGlobal database and bibliographical information concerning patents recorded in the Patstat database.

The development of AI research may be related with a trend in the allocation of research funds. Figure 3 shows the trend in the amount of public competitive research funds allocated to AI-related research projects by the year of project adoption. Until 2013, the annual total amount of public competitive research funds allocated to AI-related research projects in Japan remained lower than 200 million yen, but since 2014, the amount has increased rapidly, surpassing 900 million yen in 2016.

Figure 3: Allocation of Public Competitive Funds to AI-related Research Projects in Japan (Unit: million yen)
Figure 3: Allocation of Public Competitive Funds to AI-related Research Projects in Japan (Unit: million yen)
Source: SciREX Policy Intelligence Assistance System (SPIAS)

It is possible to conduct a more detailed analysis of the development process of AI technology by using information concerning papers, patents and research projects. According to the provisional analysis results based on Japanese data concerning these items, I found that co-authorship with companies has the effect of increasing the number of citations of AI-related papers. Citations of papers has the effect of increasing the frequency of citations of AI-related patents. An analysis conducted at the researcher level showed that the financial impact of industry-academia partnership on the number of papers produced by researchers and the number of paper citations is equivalent to around 4% of the research expenditure amount. In addition, it was confirmed that the importance of industry-academia joint research and patent-to-paper citations has continued to grow throughout the analysis period. These provisional analysis results suggest the importance of interactions between companies and academia for the development of AI technology. Meanwhile, Motohashi (2018) conducted an analysis concerning interaction between academia and companies in the AI field using U.S. science and technology papers and patent data.

Future Challenges

By using text information, such as titles and summaries of papers and patents, it may be possible to identify the development stage of AI technology in more detail. Therefore, in addition to the above analysis based on quantitative information, I am planning to conduct an analysis using text information.

However, if the state of AI-related research and technology development is to be identified more precisely, it will be insufficient to use data concerning papers and patents alone. In particular, we must not overlook the importance of algorithms and big data. Google, for example, has published papers concerning many AI-related algorithms, including one used in AlphaGo, which has become famous for beating the world champion of the Go board game. The company has also made some algorithms themselves open to the public for free. There are also activities to improve published papers and algorithms on online open software development platforms (e.g., GitHub and CRAN). The approach of measuring the development of AI-related algorithms using public information that is available online will also be effective in identifying the development of AI technology (OECD 2018).

Moreover, Nomura (2016) noted that choosing appropriate data sets and implementing appropriate training in terms of both quality and quantity, rather than devising unique algorithms, is more effective in improving accuracy and practicality and pointed out the importance of ideas for industrial applications. In order to precisely identify the state of industrial applications and potential of AI technology, it is essential to measure the state of accumulation and usage of data for the purpose of training AI. For example, the Research Institute of Economy, Trade and Industry conducted a questionnaire survey with Japanese manufacturing companies with respect to the state of usage of internal and external data (Motohashi 2016). It is important to deepen understanding on the state of development and usage of AI technology and the economic impact by expanding the scope of subjects of surveys like this, and by conducting an analysis in relation to patents and papers, as well as data related to open-source software.

December 4, 2018
  • Erik BRYNJOLFSSON and Andrew MCAFEE (MURAI Akiko, trans.) (2013). Kikai to no Kyoso, Nikkei Business Publications Inc. (originally published as Race Against the Machine, 2011)
  • Erik BRYNJOLFSSON and Andrew MCAFEE (2015). Za Sekando Mashin Eiji (MURAI Akiko, trans.), Nikkei Business Publications Inc. (originally published as The Second Machine Age: Work, Progress, and Prosperity in a Time of Brilliant Technologies, 2014)
  • Japan Patent Office (2015). "FY2014 Patent Application Technology Trend Survey Report: Artificial Intelligence"
  • INOUE Tomohiro (2016). Jinkochino to Keizai no Mirai: 2030 Nen Koyo Daihokai (Future of Artificial Intelligence and Economy: Great Collapse of Employment in 2030), Bunshun Shinsho
  • NOMURA Naoyuki (2016). Jinkochino ga Kaeru Shigoto no Mirai (Future of Jobs Being Changed by Artificial Intelligence), Nikkei Inc.
  • MOTOHASHI Kazuyuki (2016). "Survey of Big Data Use and Innovation in Japanese Manufacturing Firms," RIETI Policy Discussion Paper Series 16-P-012
  • OECD(2017)OECD Science, Technology and Industry Scoreboard 2017: THE DIGITAL TRANSFORMATION.
  • FUJII Hidemichi, MANAGI Shunsuke (2017). Trends and priority shifts in artificial intelligence technology invention: A global patent analysis RIETI Discussion Paper Series 17-E-066 (Fujii and Managi, Trends and priority shifts in artificial intelligence technology invention: A global patent analysis. Economic Analysis and Policy, 58, 60-69, 2018)
  • The Economist (2018). Competition in the digital age: How to tame the tech titans.
  • OECD (2018). Bridging the Digital Gender Divide: Include, Upskill, Innovate.
  • MOTOHASHI Kazuyuki (2018). "Co-occurrence of Science and Innovation in AI:
    Empirical Analysis of Paper-patent Linked Dataset in the United States," NISTEP DISCUSSION PAPER No.160

February 13, 2019