Machine Learning as Natural Experiment: Method and Deployment at Japanese Firms

From public policy to business, machine learning and other algorithms produce a growing portion of treatment decisions and recommendations. Such algorithmic decisions are natural experiments (conditionally quasi-randomly assigned instruments) since the algorithms make decisions based only on observable input variables. We use this observation to characterize the sources of causal-effect identification for a class of stochastic and deterministic algorithms. This identification result translates into consistent estimators of causal effects and the counterfactual performance of new algorithms. We apply our method to improve a large-scale fashion e-commerce platform (ZOZOTOWN). We conclude by providing public policy applications.