Raising the bar

Sum of random variables and convolution

Link between double and iterated integrals

Why do we need this link? For simplicity consider the rectangle $latex A=\left\{ a\leq x\leq b,c\leq y\leq d\right\} .$ The integrals

$latex I_{1}=\underset{A}{\int \int }f(x,y)dydx$

and

$latex I_{2}=\int_{a}^{b}\left( \int_{c}^{d}f(x,y)dy\right) dx$

both are taken over the rectangle $latex A$ but they are not the same. $latex I_{1}$ is a double (two-dimensional) integral, meaning that its definition uses elementary areas, while $latex I_{2}$ is an iterated integral, where each of the one-dimensional integrals uses elementary segments. To make sense of this, you need to consult an advanced text in calculus. The  difference notwithstanding, in good cases their values are the same. Putting aside the question of what is a "good case", we  concentrate on geometry: how a double integral can be expressed as an iterated integral.

It is enough to understand the idea in case of an oval $latex A$ on the plane. Let $latex y=l\left( x\right) $ be the function that describes the lower boundary of the oval and let $latex y=u\left( x\right) $ be the function that describes the upper part. Further, let the vertical lines $latex x=m$ and $latex x=M$ be the minimum and maximum values of $latex x$ in the oval (see Chart 1).

Chart 1. The boundary of the oval above the green line is described by u(x) and below - by l(x)

We can paint the oval with strokes along red lines from $latex y=l\left( x\right) $ to $latex y=u\left(x\right) .$ If we do this for all $latex x\in \left[ m,M\right] ,$ we'll have painted the whole oval. This corresponds to the representation of $latex A$ as the union of segments $latex \left\{ y:l\left( x\right) \leq y\leq u\left( x\right) \right\} $ with $latex x\in \left[ m,M\right] :$

$latex A=\bigcup\limits_{m\leq x\leq M}\left\{ y:l\left( x\right) \leq y\leq u\left( x\right) \right\} $

and to the equality of integrals

(double integral)$latex \underset{A}{\int \int }f(s,t)dsdt=\int_{m}^{M}\left(\int_{l\left( x\right) }^{u(x)}f(x,y)dy\right) dx$ (iterated integral)

Density of a sum of two variables

Assumption 1 Suppose the random vector $latex \left( X,Y\right) $ has a density $latex f_{X,Y}$ and define $latex Z=X+Y$ (unlike the convolution theorem below, here $latex X,Y$ don't have to be independent).

From the definitions of the distribution function $latex F_{Z}\left( z\right)=P\left( Z\leq z\right) $ and probability

$latex P\left( A\right) =\underset{A}{\int \int }f_{X,Y}(x,y)dxdy$

we have

$latex F_{Z}\left( z\right) =P\left( Z\leq z\right) =P\left( X+Y\leq z\right) =\underset{x+y\leq z}{\int \int }f_{X,Y}(x,y)dxdy.$

The integral on the right is a double integral. The painting analogy (see Chart 2)

Chart 2. Integration for sum of two variables

suggests that

$latex \left\{ (x,y)\in R^{2}:x+y\leq z\right\} =\bigcup\limits_{-\infty <x<\infty}\left\{ y:y\leq z-x\right\}.$

Hence,

$latex \int_{-\infty }^{z}f_{Z}\left( z\right) dz=F_{Z}\left( z\right)=\int_{R}\left( \int_{-\infty }^{z-x}f_{X,Y}(x,y)dy\right) dx.$

Differentiating both sides with respect to $latex z$ we get

$latex f_{Z}\left( z\right) =\int_{R}f_{X,Y}(x,z-x)dx.$

If we start with the inner integral that is with respect to $latex x$ and the outer integral $latex -$ with respect to $latex y,$ then similarly

$latex f_{Z}\left( z\right) =\int_{R}f_{X,Y}(z-y,y)dy.$

Exercise. Suppose the random vector $latex \left( X,Y\right) $ has a density $latex f_{X,Y}$ and define $latex Z=X-Y.$ Find $latex f_{Z}.$ Hint: review my post on Leibniz integral rule.

Convolution theorem

In addition to Assumption 1, let $latex X,Y$ be independent. Then $latex f_{X,Y}(x,y)=f_{X}(x)f_{Y}\left( y\right) $ and the above formula gives

$latex f_{Z}\left( z\right) =\int_{R}f_{X}(x)f_{Y}\left( z-x\right) dx.$

This is denoted as $latex \left( f_{X}\ast f_{Y}\right) \left( z\right) $ and called a convolution.

The following may help to understand this formula. The function $latex g(x)=f_{Y}\left( -x\right) $ is a density (it is non-negative and integrates to 1). Its graph is a mirror image of that of $latex f_{Y}$ with respect to the vertical axis. The function $latex h_{z}(x)=f_{Y}\left( z-x\right) $ is a shift of $latex g$ by $latex z$ along the horizontal axis. For fixed $latex z,$ it is also a density. Thus in the definition of convolution we integrate the product of two densities $latex f_{X}(x)f_{Y}\left( z-x\right) .$ Further, to understand the asymptotic behavior of $latex \left( f_{X}\ast f_{Y}\right) \left( z\right) $ when $latex \left\vert z\right\vert \rightarrow \infty $ imagine two bell-shaped densities $latex f_{X}(x)$ and $latex f_{Y}\left( z-x\right) .$ When $latex z$ goes to, say, infinity, the humps of those densities are spread apart more and more. The hump of one of them gets multiplied by small values of the other. That's why $latex \left(f_{X}\ast f_{Y}\right) \left( z\right) $ goes to zero, in a certain sense.

The convolution of two densities is always a density because it is non-negative and integrates to one:

$latex \int_{R}f_{Z}\left( z\right) dz=\int_{R}\left( \int_{R}f_{X}(x)f_{Y}\left(z-x\right) dx\right) dz=\int_{R}f_{X}(x)\left( \int_{R}f_{Y}\left(z-x\right) dz\right) dx$

Replacing $latex z-x=y$ in the inner integral we see that this is

$latex \int_{R}f_{X}(x)dx\int_{R}f_{Y}\left( y\right) dy=1.$