14
Dec 21

## Sum of random variables and convolution

### Link between double and iterated integrals

Why do we need this link? For simplicity consider the rectangle $A=\left\{ a\leq x\leq b,c\leq y\leq d\right\} .$ The integrals

$I_{1}=\underset{A}{\int \int }f(x,y)dydx$

and

$I_{2}=\int_{a}^{b}\left( \int_{c}^{d}f(x,y)dy\right) dx$

both are taken over the rectangle $A$ but they are not the same. $I_{1}$ is a double (two-dimensional) integral, meaning that its definition uses elementary areas, while $I_{2}$ is an iterated integral, where each of the one-dimensional integrals uses elementary segments. To make sense of this, you need to consult an advanced text in calculus. The  difference notwithstanding, in good cases their values are the same. Putting aside the question of what is a "good case", we  concentrate on geometry: how a double integral can be expressed as an iterated integral.

It is enough to understand the idea in case of an oval $A$ on the plane. Let $y=l\left( x\right)$ be the function that describes the lower boundary of the oval and let $y=u\left( x\right)$ be the function that describes the upper part. Further, let the vertical lines $x=m$ and $x=M$ be the minimum and maximum values of $x$ in the oval (see Chart 1).

Chart 1. The boundary of the oval above the green line is described by u(x) and below - by l(x)

We can paint the oval with strokes along red lines from $y=l\left( x\right)$ to $y=u\left(x\right) .$ If we do this for all $x\in \left[ m,M\right] ,$ we'll have painted the whole oval. This corresponds to the representation of $A$ as the union of segments $\left\{ y:l\left( x\right) \leq y\leq u\left( x\right) \right\}$ with $x\in \left[ m,M\right] :$

$A=\bigcup\limits_{m\leq x\leq M}\left\{ y:l\left( x\right) \leq y\leq u\left( x\right) \right\}$

and to the equality of integrals

(double integral)$\underset{A}{\int \int }f(s,t)dsdt=\int_{m}^{M}\left( \int_{l\left( x\right) }^{u(x)}f(x,y)dy\right) dx$ (iterated integral)

### Density of a sum of two variables

Assumption 1 Suppose the random vector $\left( X,Y\right)$ has a density $f_{X,Y}$ and define $Z=X+Y$ (unlike the convolution theorem below, here $X,Y$ don't have to be independent).

From the definitions of the distribution function $F_{Z}\left( z\right)=P\left( Z\leq z\right)$ and probability

$P\left( A\right) =\underset{A}{\int \int }f_{X,Y}(x,y)dxdy$

we have

$F_{Z}\left( z\right) =P\left( Z\leq z\right) =P\left( X+Y\leq z\right) = \underset{x+y\leq z}{\int \int }f_{X,Y}(x,y)dxdy.$

The integral on the right is a double integral. The painting analogy (see Chart 2)

Chart 2. Integration for sum of two variables

suggests that

$\left\{ (x,y)\in R^{2}:x+y\leq z\right\} =\bigcup\limits_{-\infty

Hence,

$\int_{-\infty }^{z}f_{Z}\left( z\right) dz=F_{Z}\left( z\right) =\int_{R}\left( \int_{-\infty }^{z-x}f_{X,Y}(x,y)dy\right) dx.$

Differentiating both sides with respect to $z$ we get

$f_{Z}\left( z\right) =\int_{R}f_{X,Y}(x,z-x)dx.$

If we start with the inner integral that is with respect to $x$ and the outer integral $-$ with respect to $y,$ then similarly

$f_{Z}\left( z\right) =\int_{R}f_{X,Y}(z-y,y)dy.$

Exercise. Suppose the random vector $\left( X,Y\right)$ has a density $f_{X,Y}$ and define $Z=X-Y.$ Find $f_{Z}.$ Hint: review my post on Leibniz integral rule.

### Convolution theorem

In addition to Assumption 1, let $X,Y$ be independent. Then $f_{X,Y}(x,y)=f_{X}(x)f_{Y}\left( y\right)$ and the above formula gives

$f_{Z}\left( z\right) =\int_{R}f_{X}(x)f_{Y}\left( z-x\right) dx.$

This is denoted as $\left( f_{X}\ast f_{Y}\right) \left( z\right)$ and called a convolution.

The following may help to understand this formula. The function $g(x)=f_{Y}\left( -x\right)$ is a density (it is non-negative and integrates to 1). Its graph is a mirror image of that of $f_{Y}$ with respect to the vertical axis. The function $h_{z}(x)=f_{Y}\left( z-x\right)$ is a shift of $g$ by $z$ along the horizontal axis. For fixed $z,$ it is also a density. Thus in the definition of convolution we integrate the product of two densities $f_{X}(x)f_{Y}\left( z-x\right) .$ Further, to understand the asymptotic behavior of $\left( f_{X}\ast f_{Y}\right) \left( z\right)$ when $\left\vert z\right\vert \rightarrow \infty$ imagine two bell-shaped densities $f_{X}(x)$ and $f_{Y}\left( z-x\right) .$ When $z$ goes to, say, infinity, the humps of those densities are spread apart more and more. The hump of one of them gets multiplied by small values of the other. That's why $\left(f_{X}\ast f_{Y}\right) \left( z\right)$ goes to zero, in a certain sense.

The convolution of two densities is always a density because it is non-negative and integrates to one:

$\int_{R}f_{Z}\left( z\right) dz=\int_{R}\left( \int_{R}f_{X}(x)f_{Y}\left( z-x\right) dx\right) dz=\int_{R}f_{X}(x)\left( \int_{R}f_{Y}\left( z-x\right) dz\right) dx$

Replacing $z-x=y$ in the inner integral we see that this is

$\int_{R}f_{X}(x)dx\int_{R}f_{Y}\left( y\right) dy=1.$
1
May 18

## Density of a sum of independent variables is given by convolution

This topic is pretty complex because it involves properties of integrals that economists usually don't study. I provide this result to be able to solve one of UoL problems.

## General relationship between densities

Let $X,Y$ be two independent variables with densities $f_X,f_Y$. Denote $f_{X,Y}$ the joint density of the pair $(X,Y)$.

By independence we have

(1) $f_{X,Y}(x,y)=f_X(x)f_Y(y).$

Let $Z=X+Y$ be the sum and let $f_Z,\ F_Z$ be its density and distribution function, respectively. Then

(2) $f_Z(z)=\frac{d}{dz}F_Z(z)$.

These are the only simple facts in this derivation. By definition,

(3) $F_Z(z)=P(Z\le z)=P(X+Y\le z)$.

For the last probability in (3) we have a double integral

$P(X+Y\le z)=\int\int_{x+y\le z}f_{X,Y}(x,y)dxdy$.

Using (1), we replace the joint probability by the product of individual probabilities and the double integral by the repeated one:

(4) $P(X+Y\le z)=\int\int_{x+y\le z}f_X(x)f_Y(y)dxdy=\int_R\int_{-\infty}^{z-x}f_X(x)f_Y(y)dxdy$

$=\int_Rf_X(x)\left(\int_{-\infty}^{z-x}f_Y(y)dy\right)dx$.

The geometry is explained in Figure 1. The area $x+y\le z$ is limited by the line $y=z-x$. In the repeated integral, we integrate first over red lines from $-\infty$ to $z-x$ and then in the outer integral over all $x\in R$.

Figure 1. Area of integration

(3) and (4) imply

$F_Z(z)=\int_Rf_X(x)\left(\int_{-\infty}^{z-x}f_Y(y)dy\right)dx$.

Finally, using (2) we differentiate both sides to get

(5) $f_Z(z)=\int_Rf_X(x)f_Y(z-x)dx$.

This is the result. The integral on the right is called a convolution of functions $f_X,f_Y$.

Remark. Existence of density (2) follows from existence of $f_X,f_Y$, although we don't prove this fact.

Exercise. Convolution is usually denoted by $(f*g)(z)=\int_Rf(x)g(z-x)dx$. Prove that

1. $(f*g)(z)=(g*f)(z)$.

2. $\int_R|(f*g)(z)|dz\le \int_R|f(x)|dx\int_R|g(x)|dx$.

3. If $X$ is uniformly distributed on some segment, then $(f_X*f_X)(z)$ is zero for large $z$.