Dec 21

Sum of random variables and convolution

Sum of random variables and convolution

Link between double and iterated integrals

Why do we need this link? For simplicity consider the rectangle A=\left\{ a\leq x\leq b,c\leq y\leq d\right\} . The integrals

I_{1}=\underset{A}{\int \int }f(x,y)dydx


I_{2}=\int_{a}^{b}\left( \int_{c}^{d}f(x,y)dy\right) dx

both are taken over the rectangle A but they are not the same. I_{1} is a double (two-dimensional) integral, meaning that its definition uses elementary areas, while I_{2} is an iterated integral, where each of the one-dimensional integrals uses elementary segments. To make sense of this, you need to consult an advanced text in calculus. The  difference notwithstanding, in good cases their values are the same. Putting aside the question of what is a "good case", we  concentrate on geometry: how a double integral can be expressed as an iterated integral.

It is enough to understand the idea in case of an oval A on the plane. Let y=l\left( x\right) be the function that describes the lower boundary of the oval and let y=u\left( x\right) be the function that describes the upper part. Further, let the vertical lines x=m and x=M be the minimum and maximum values of x in the oval (see Chart 1).

Double integral

Chart 1. The boundary of the oval above the green line is described by u(x) and below - by l(x)

We can paint the oval with strokes along red lines from y=l\left( x\right) to y=u\left(x\right) . If we do this for all x\in \left[ m,M\right] , we'll have painted the whole oval. This corresponds to the representation of A as the union of segments \left\{ y:l\left( x\right) \leq y\leq u\left( x\right) \right\} with x\in \left[ m,M\right] :

A=\bigcup\limits_{m\leq x\leq M}\left\{ y:l\left( x\right) \leq y\leq u\left( x\right) \right\}

and to the equality of integrals

(double integral)\underset{A}{\int \int }f(s,t)dsdt=\int_{m}^{M}\left(\int_{l\left( x\right) }^{u(x)}f(x,y)dy\right) dx (iterated integral)

Density of a sum of two variables

Assumption 1 Suppose the random vector \left( X,Y\right) has a density f_{X,Y} and define Z=X+Y (unlike the convolution theorem below, here X,Y don't have to be independent).

From the definitions of the distribution function F_{Z}\left( z\right)=P\left( Z\leq z\right) and probability

P\left( A\right) =\underset{A}{\int \int }f_{X,Y}(x,y)dxdy

we have

F_{Z}\left( z\right) =P\left( Z\leq z\right) =P\left( X+Y\leq z\right) =\underset{x+y\leq z}{\int \int }f_{X,Y}(x,y)dxdy.

The integral on the right is a double integral. The painting analogy (see Chart 2)

Integration for sum of two variables

Chart 2. Integration for sum of two variables

suggests that

\left\{ (x,y)\in R^{2}:x+y\leq z\right\} =\bigcup\limits_{-\infty <x<\infty}\left\{ y:y\leq z-x\right\}.


\int_{-\infty }^{z}f_{Z}\left( z\right) dz=F_{Z}\left( z\right)=\int_{R}\left( \int_{-\infty }^{z-x}f_{X,Y}(x,y)dy\right) dx.

Differentiating both sides with respect to z we get

f_{Z}\left( z\right) =\int_{R}f_{X,Y}(x,z-x)dx.

If we start with the inner integral that is with respect to x and the outer integral - with respect to y, then similarly

f_{Z}\left( z\right) =\int_{R}f_{X,Y}(z-y,y)dy.

Exercise. Suppose the random vector \left( X,Y\right) has a density f_{X,Y} and define Z=X-Y. Find f_{Z}. Hint: review my post on Leibniz integral rule.

Convolution theorem

In addition to Assumption 1, let X,Y be independent. Then f_{X,Y}(x,y)=f_{X}(x)f_{Y}\left( y\right) and the above formula gives

f_{Z}\left( z\right) =\int_{R}f_{X}(x)f_{Y}\left( z-x\right) dx.

This is denoted as \left( f_{X}\ast f_{Y}\right) \left( z\right) and called a convolution.

The following may help to understand this formula. The function g(x)=f_{Y}\left( -x\right) is a density (it is non-negative and integrates to 1). Its graph is a mirror image of that of f_{Y} with respect to the vertical axis. The function h_{z}(x)=f_{Y}\left( z-x\right) is a shift of g by z along the horizontal axis. For fixed z, it is also a density. Thus in the definition of convolution we integrate the product of two densities f_{X}(x)f_{Y}\left( z-x\right) . Further, to understand the asymptotic behavior of \left( f_{X}\ast f_{Y}\right) \left( z\right) when \left\vert z\right\vert \rightarrow \infty imagine two bell-shaped densities f_{X}(x) and f_{Y}\left( z-x\right) . When z goes to, say, infinity, the humps of those densities are spread apart more and more. The hump of one of them gets multiplied by small values of the other. That's why \left(f_{X}\ast f_{Y}\right) \left( z\right) goes to zero, in a certain sense.

The convolution of two densities is always a density because it is non-negative and integrates to one:

\int_{R}f_{Z}\left( z\right) dz=\int_{R}\left( \int_{R}f_{X}(x)f_{Y}\left(z-x\right) dx\right) dz=\int_{R}f_{X}(x)\left( \int_{R}f_{Y}\left(z-x\right) dz\right) dx

Replacing z-x=y in the inner integral we see that this is

\int_{R}f_{X}(x)dx\int_{R}f_{Y}\left( y\right) dy=1.
May 18

Density of a sum of independent variables is given by convolution

Density of a sum of independent variables is given by convolution

This topic is pretty complex because it involves properties of integrals that economists usually don't study. I provide this result to be able to solve one of UoL problems.

General relationship between densities

Let X,Y be two independent variables with densities f_X,f_Y. Denote f_{X,Y} the joint density of the pair (X,Y).

By independence we have

(1) f_{X,Y}(x,y)=f_X(x)f_Y(y).

Let Z=X+Y be the sum and let f_Z,\ F_Z be its density and distribution function, respectively. Then

(2) f_Z(z)=\frac{d}{dz}F_Z(z).

These are the only simple facts in this derivation. By definition,

(3) F_Z(z)=P(Z\le z)=P(X+Y\le z).

For the last probability in (3) we have a double integral

P(X+Y\le z)=\int\int_{x+y\le z}f_{X,Y}(x,y)dxdy.

Using (1), we replace the joint probability by the product of individual probabilities and the double integral by the repeated one:

(4) P(X+Y\le z)=\int\int_{x+y\le z}f_X(x)f_Y(y)dxdy=\int_R\int_{-\infty}^{z-x}f_X(x)f_Y(y)dxdy


The geometry is explained in Figure 1. The area x+y\le z is limited by the line y=z-x. In the repeated integral, we integrate first over red lines from -\infty to z-x and then in the outer integral over all x\in R.

Area of integration

Figure 1. Area of integration

(3) and (4) imply


Finally, using (2) we differentiate both sides to get

(5) f_Z(z)=\int_Rf_X(x)f_Y(z-x)dx.

This is the result. The integral on the right is called a convolution of functions f_X,f_Y.

Remark. Existence of density (2) follows from existence of f_X,f_Y, although we don't prove this fact.

Exercise. Convolution is usually denoted by (f*g)(z)=\int_Rf(x)g(z-x)dx. Prove that

  1. (f*g)(z)=(g*f)(z).

  2. \int_R|(f*g)(z)|dz\le \int_R|f(x)|dx\int_R|g(x)|dx.

  3. If X is uniformly distributed on some segment, then (f_X*f_X)(z) is zero for large z.