Introduction Recently, I came across a simple question that led me through a quite interesting rabbit hole. The question is very simple (and very niche), but the analysis involves key results from statistical theory and numerical analysis (the Central Limit Theorem, the Delta method, and the second-order Delta method, Monte Carlo integration).
Here is the seemingly innocent (and slightly bizarre) question:
Context: One is interested in evaluating the square of an expectation \(E\Big[g(X) \Big]^2 = \big(\int_{a}^{b} g(x) f_X(x) dx \Big)^2 = \mu^2\), where \(\mu\) is an unknown quantity “close to zero”. Note that the integral squared can be rewritten as \(\big(\int_{a}^{b} g(x) f_X(x) dx \Big)^2 = \int_{a}^{b} \int_{a}^{b} g(x) g(y) f_X(x) f_Y(y) dx dy \). Transforming the square of an integral into a double integral is a standard trick, that is for instance used when calculating the Gaussian integral. Let’s say one want to numerically approximate \(\mu^2\) using Monte Carlo integration. Question: Is it better to approximate \(\mu^2\) using the square of the Monte Carlo estimator \(\big(\frac{1}{N} \sum_{i=1}^{N} f(x_i) \Big)^2\), or the Monte Carlo estimator based on the double integral: \(\frac{1}{N} \sum_{i=1}^{N} f(x_i) f(y_i) \)? Here the sample \({x_i}\) and \({y_i}\) are i.i.d random variables with density f(.). The random variables \({x_i}\) and \({y_i}\) have the same density function, but they are drawn independently from each other. I. Consistency Notice that both estimators are consistent. That is, as we increase \(N\), the estimators converges in probability to the true value of the parameter \(\mu^2\).