Re-estimating the Rate of Deflation in Japan

-- The original title of the paper is "How Fast Are Prices in Japan Falling?" Could you tell us your purpose for writing this paper?

WATANABE: In Japan, deflation has been continuing since the mid-1990s. While this is obviously a major problem not only for Japan but also for the global economy, the CPI shows that the deflation rate has been about 1% every year, or about 2% at the highest during the most rampant periods.

Although there have been a very limited number of cases of deflation in a developed country, one such case occurred in the United States during the Great Depression in the 1930s, where an annualized 7% price plunge was observed. Compared to this, the present deflation in Japan can be considered as a moderate price decline continuing over a very long period. The fact that deflation has not been so rapid in Japan while people have claimed that the country has been suffering from a stagnant economy for more than a decade is drawing close attention among researchers and businesspersons overseas as well as domestically.

The cause of this phenomenon has been discussed for years by policymakers and researchers. Their views vary, among which there is a skeptical view regarding the reliability of Japan's price statistics themselves, claiming that "the deflation rate is only apparently low due to the inaccuracy of the CPI calculation by the MIC." This perspective seems to be prevalent, particularly in Europe and the United States.

Thus, while deflation in Japan has been moderate, we intend to identify whether this inexplicable phenomenon of moderate deflation, in which there have been sluggish price falls, is attributable to statistical issues or other factors.

-- What methods did you adopt to verify the reliability of the statistics?

WATANABE: There are two possible methods for verifying the reliability of the statistics themselves. One method is to review each data point used for the preparation of the CPI by the MIC and examine whether they are appropriate. However, it is difficult for external researchers to access easily the details of the data and the statistics actually used for the CPI preparation. Even if such in-depth information was available, it would be difficult to determine their validity.

For the second method, we aim to prepare a price index of our own in consideration of the MIC methods for CPI preparation while using a data source that is different from that used for the CPI prepared by the MIC.

Analysis based on sales-related information amounting to 3.6 billion data entries over 10 years

-- Is it challenging to prepare a new price index?

WATANABE: In fact, we tried to calculate the price index a few years ago, but gave up during the process. The MIC has been publishing its CPI preparation methods, but the details of each method involve many technical issues which are not described in the text. We proceeded with this task while confirming the issues step by step by contacting the MIC, but unfortunately we had to discontinue the process because the task was too complicated and time-consuming.

However, given the current situation in which researchers and businesspersons in Europe and the United States have publicly expressed their skepticism about the reliability of Japan's CPI in overseas academic meetings, we considered it beneficial for not only researchers but also policymakers to verify the reliability of Japan's CPI using data other than that used by the MIC. Thus, we continued to approach the MIC for its cooperation.

Of course, it was not certain at the start of the research as to whether the final results would be favorable to the MIC, given the academic nature of the research. Obviously, it was not an option to carry out analyses that were biased in order to obtain the cooperation of the MIC. However, we have successfully obtained the cooperation of the relevant department of the MIC by sharing the view that "it is necessary in the first place to investigate the actual conditions."

Mr. Imai, a co-author of the paper, is in charge of consumer price-related systems in the Statistics Bureau of the MIC. It is unusual for researchers and the person who is actually preparing the statistics to work jointly with all of their proprietary information disclosed to each other as we did in this research. In this regard, the fact that our collaboration was very successful is a valuable achievement for similar analyses in the future.

-- What data did you use?

WATANABE: For the reproduction of the CPI, we had to collect a large volume of consumer shopping data. The POS data used in this research was jointly compiled by Nikkei Digital Media, Inc. and the University of Tokyo. This dataset is the sum of data collected on more than 200,000 commodity items sold at some 200 supermarkets across the nation between 2000 and 2010. These items mainly include food products, beverages, and other nondurable consumer goods (general merchandise such as shampoo), and the sales information was summed up through the so-called POS system.

In 2009, for instance, we surveyed 260 supermarkets and 230,000 items. The total number of data entries added up to 422 million in 2009, and reached 3.6 billion over the entire research period. However, both the number of stores and items have been partially replaced (Tab. 1). The number of stores that continued in cooperating throughout the research period was 103, and the total number of items sold there was 203,000 in 2000, and subsequently increased to 256,000 in 2009.

Table 1 shows that approximately 30% of the items were replaced every year. In 2009, for example, 75,495 new items were added while 87,212 existing items were withdrawn. The proportion of about 30% is greater than that in foreign countries, suggesting that marketable items are being replaced every year to a considerable extent.

Some items are typically consumed over a long period of time, such as sweet adzuki-bean jelly from a long-established Japanese confectionery company, whereas others are continually being replaced with new items, such as instant noodles. Unless such replacement for each item is accurately identified in the process of price research, the resulting prices may vary widely. We also wanted to examine the degree of the impact of such replacement.

Preparing a consumer price index on our own to verify the reliability of the CPI

-- Could you tell us the details of the analyses?

WATANABE: Using the enormous volume of data collected, we tried to reproduce the MIC's CPI in accordance with its CPI preparation methods. In parallel with this task, we also performed a total of 64 patterns of analyses based on the MIC methods while modifying these methods stepwise to determine, as a simulation, whether the results would differ depending on such modifications.

Finally, given the substantial criticism from overseas, particularly from the United States, regarding the reliability of Japan's statistics, we tried to prepare a price index using data obtained in accordance with the methods used by the U.S. Bureau of Labor Statistics (BLS), which is responsible for the country's preparation of the CPI. Mr. Imai is of course familiar with the practice of CPI preparation as an MIC official with responsibility over it, and he is also knowledgeable about the methods of the U.S. BLS. This was a great help to us in advancing the research.

-- How were the results?

WATANABE: In the first task of reproducing the price index in accordance with the MIC methods, the CPI was prepared using a different data source that was very similar to one published by the MIC. This suggests no problem with the data and procedures used in this research in terms of the reproducibility of the CPI. As for the second task of performing analyses while modifying the current MIC methods stepwise, although retailers such as supermarkets, for example, often hold special sales events, "special sales prices" are not included in price data unless they continue for at least eight days according to the MIC methods for calculating the CPI. Some people see this practice as a problem; therefore we made it a rule to include such prices in the price data if they continue for at least three days. The result, however, showed no significant difference compared to that of the first task.

Finally, the results of the use of the U.S. BLS methods also follows a similar trend to that resulting from the use of the MIC methods, except for the details (Fig. 1).

Figure 1: Year-on-year changes in the price index estimated by the U.S. methods [ Click to enlarge ]

As shown in this graph, the annualized deflation rate in Japan is about 2% to 2.5% at the highest level even though it was estimated using different methods, and no deflation has occurred that is comparable to the 7% to 8% price plunge in the United States during the Great Depression. In short, the MIC methods for CPI preparation involve no significant statistical error.

-- Where does the difference between the MIC and the U.S. methods lie?

WATANABE: The price index estimated using the U.S. methods (blue line), compared with that using the MIC methods (red and green lines), shows lower prices for the most recent months although it is basically tracing a similar trend (Fig. 1). Figure 2 was created focusing on this difference.

To put it simply, the primary difference between the U.S. and the MIC methods lies in the method of sampling. According to the U.S. methods, items to be included in the price data differ for each research study because they are randomly sampled each time the research is conducted. This method of sampling is superior in that it lowers the risk of eliminating marketable items from the research, but involves certain variation in the results due to its probabilistic nature.

Let me focus on the consumer price increase rate of 1%, which was the target set by the Bank of Japan (BOJ) for its monetary policy. Supposing a 1% price increase is achieved by the MIC methods, how is the inflation rate estimated using the U.S. methods? As shown in the graph, the inflation rate based on the U.S. methods is nearly zero, or 0.1%, on average, but ranges from -0.5% to +0.5%. That is, when the inflation rate is 1% based on the MIC methods, it varies between -0.5% and +0.5% within the 80% confidence interval according to the U.S. methods.

In other words, even if a 1% price increase is estimated by the MIC methods, the probability of a price increase or decrease is 50-50 according to the U.S. methods.

A cautious stance is essential in judgments or actions with regard to exit from deflation

-- What policy implications can be derived from these results?

WATANABE: We would like to emphasize the necessity of paying heed to these differences due to the statistical methods in discussions on the inflation rate required to end deflation. As shown in Figure 2, even if the inflation rate estimated by the MIC methods is +1%, it is highly likely that the rate estimated by the U.S. methods would remain negative.

In order to ensure that price changes are positive, the inflation rate must be at least +2% when estimated using the MIC methods. At this level, the rate will be positive even if it is estimated using the U.S. methods.

Utmost care should be taken in making judgments and taking action to pull out of deflation. It is recommended that various measures to combat deflation be retracted after price changes estimated by the MIC methods reach +2%.

-- The BOJ policy target was an inflation rate of 1% when this paper was written, and the target rate was raised to 2% following the start of the new government.

WATANABE: As suggested by the analysis results, we thought that an inflation rate of 1% as estimated by the current MIC methods was too low for the target and should be raised to 2%. We finished the paper with the hope that this would be proposed, and the target which I considered preferable was actually set by the newly-elected government and the BOJ. In this respect, their response was satisfactory.

The frequency of publishing price data should also be daily rather than monthly

-- Could you tell us about your future studies?

WATANABE: As it is referred to as a barometer of the economy, the CPI, which was the research subject this time, is considered as a critical indicator in effectively managing economic policies. Unfortunately, the CPI published by the MIC is on a monthly basis. Thus, we tried to prepare a daily price index using the POS data in the research. By designing a system to transfer all price information collected from 300 stores to my study during the night of the same day, the "price index of yesterday" can be obtained the following day. Price information includes special sales data under this system, which, in this context, is close to the U.S. practice. As the price index has been published on the University of Tokyo website since May 2013 and is also linked to RIETI sites, it is possible to monitor the daily fluctuations in prices such as those in foreign exchanges.

While such data as a daily price index has also been developed by Google Inc. as well as the Massachusetts Institute of Technology (MIT) in the United States, they calculate their price indexes based on price information collected through the Internet. Our trial is primarily characterized by the fact that it is based on actual data collected from as many as 300 stores.

With a strong interest in prices, I have been studying this through various cross-sectional methods. For instance, I conducted a study on the pricing process using a typical price comparison site Kakaku.com in 2008, and analyzed price rigidity using rent data in 2009 ( Note1 ). I hope to undertake new types of studies in the future by effectively using the above-mentioned data and indicators.