Raising the bar

Vector autoregression (VAR)

Suppose we are observing two stocks and their respective returns are To take into account their interdependence, we consider a vector autoregression

(1)

Try to repeat for this system the analysis from Section 3.5 (Application to an AR(1) process) of the Guide by A. Patton and you will see that the difficulties are insurmountable. However, matrix algebra allows one to overcome them, with proper adjustment.

Problem

A) Write this system in a vector format

(2)

What should be in this representation?

B) Assume that the error in (1) satisfies

(3) for with some symmetric matrix

What does this assumption mean in terms of the components of from (2)? What is if the errors in (1) satisfy

(4) for for all

C) Suppose (1) is stationary. The stationarity condition is expressed in terms of eigenvalues of but we don't need it. However, we need its implication:

(5) .

Find

D) Find

E) Find

F) Find

G) Find

Solution

A) It takes some practice to see that with the notation

the system (1) becomes (2).

B) The equations in (3) look like this:

Equalities of matrices are understood element-wise, so we get a series of scalar equations for

Conversely, the scalar equations from (4) give

for .

C) (2) implies or by stationarity or Hence (5) implies

D) From (2) we see that depends only on (information set at time ). Therefore by the LIE

E) Using the previous post

(by stationarity and (3)). Thus, and (see previous post).

F) Using the previous result we have

G) Similarly,

Autocorrelations require a little more effort and I leave them out.

 

Vector autoregressions: preliminaries

Suppose we are observing two stocks and their respective returns are A vector autoregression for the pair is one way to take into account their interdependence. This theory is undeservedly omitted from the Guide by A. Patton.

Required minimum in matrix algebra

Matrix notation and summation are very simple.

Matrix multiplication is a little more complex. Make sure to read Global idea 2 and the compatibility rule.

The general approach to study matrices is to compare them to numbers. Here you see the first big No: matrices do not commute, that is, in general

The idea behind matrix inversion is pretty simple: we want an analog of the property that holds for numbers.

Some facts about determinants have very complicated proofs and it is best to stay away from them. But a couple of ideas should be clear from the very beginning. Determinants are defined only for square matrices. The relationship of determinants to matrix invertibility explains the role of determinants. If is square, it is invertible if and only if (this is an equivalent of the condition for numbers).

Here is an illustration of how determinants are used. Suppose we need to solve the equation for where and are known. Assuming that we can premultiply the equation by to obtain (Because of lack of commutativity, we need to keep the order of the factors). Using intuitive properties and we obtain the solution: In particular, we see that if then the equation has a unique solution

Let be a square matrix and let be two vectors. are assumed to be known and is unknown. We want to check that solves the equation (Note that for this equation the trick used to solve does not work.) Just plug

(write out a couple of first terms in the sums if summation signs frighten you).

Transposition is a geometrically simple operation. We need only the property

Variance and covariance

Property 1. Variance of a random vector and covariance of two random vectors are defined by

respectively.

Note that when variance becomes

Property 2. Let be random vectors and suppose are constant matrices. We want an analog of In the next calculation we have to remember that the multiplication order cannot be changed.

(applying )

 

Blueprint for exam versions

This is the exam I administered in my class in Spring 2022. By replacing the Poisson distribution with other random variables the UoL examiners can obtain a large variety of versions with which to torture Advanced Statistics students. On the other hand, for the students the answers below can be a blueprint to fend off any assaults.

During the semester my students were encouraged to analyze and collect information in documents typed in Scientific Word or LyX. The exam was an open-book online assessment. Papers typed in Scientific Word or LyX were preferred and copying from previous analysis was welcomed. This policy would be my preference if I were to study a subject as complex as Advanced Statistics. The students were given just two hours on the assumption that they had done the preparations diligently. Below I give the model answers right after the questions.

Midterm Spring 2022

You have to clearly state all required theoretical facts. Number all equations that you need to use in later calculations and reference them as necessary. Answer the questions in the order they are asked. When you don't know the answer, leave some space. For each unexplained fact I subtract one point. Put your name in the file name.

In questions 1-9 is the Poisson variable.

Question 1

Define and derive the population mean and population variance of the sum where is an i.i.d. sample from .

Answer. is defined by Using and (ST2133 p.80) we have

(by independence and identical distribution). [Some students derived instead of respective equations for sample means].

Question 2

Derive the MGF of the standardized sample mean.

Answer. Knowing this derivation is a must because it is a combination of three important facts.

a) Let Then so standardizing and gives the same result.

b) The MGF of is expressed through the MGF of :

(independence) (identical distribution)

c) If is a linear transformation of then

When answering the question we assume any i.i.d. sample from a population with mean and population variance :

Putting in c) and using a) we get

(using b) and )

This is a general result which for the Poisson distribution can be specified as follows. From ST2133, example 3.38 we know that . Therefore, we obtain

[Instead of some students gave .]

Question 3

Derive the cumulant generating function of the standardized sample mean.

Answer. Again, there are a couple of useful general facts.

I) Decomposition of MGF around zero. The series leads to

where are moments of and Differentiating this equation yields

and setting gives the rule for finding moments from MGF:

II) Decomposition of the cumulant generating function around zero. can also be decomposed into its Taylor series:

where the coefficients are called cumulants and can be found using . Since

and

we have

Thus, for any random variable with mean and variance we have

terms of higher order for small.

III) If then by c)

IV) By b)

Using III), and then IV) we have

For the last term on the right we use the approximation around zero from II):

[Important. Why the above steps are necessary? Passing from the series to the series for is not straightforward and can easily lead to errors. It is not advisable in case of the Poisson to derive from .]

Question 4

Prove the central limit theorem using the cumulant generating function you obtained.

Answer. In the previous question we proved that around zero

This implies that

(1) for each around zero.

But we know that for a standard normal its MGF is (ST2133 example 3.42) and hence for the standard normal

(2)

Theorem (link between pointwise convergence of MGFs of and convergence in distribution of ) Let be a sequence of random variables and let be some random variable. If converges for each from a neighborhood of zero to , then converges in distribution to

Using (1), (2) and this theorem we finish the proof that converges in distribution to the standard normal, which is the central limit theorem.

Question 5

State the factorization theorem and apply it to show that is a sufficient statistic.

Answer. The solution is given on p.180 of ST2134. For the joint density is

(3)

To satisfy the Fisher-Neyman factorization theorem set

and then we see that is a sufficient statistic for

Question 6

Find a minimal sufficient statistic for stating all necessary theoretical facts.

Answer. Characterization of minimal sufficiency A statistic is minimal sufficient if and only if level sets of coincide with sets on which the ratio does not depend on

From (3)

The expression on the right does not depend on if and only of The last condition describes level sets of Thus it is minimal sufficient.

Question 7

Find the Method of Moments estimator of the population mean.

Answer. The idea of the method is to take some populational property (for example, ) and replace the population characteristic (in this case ) by its sample analog () to obtain a MM estimator. In our case [Try to do this for the Gamma distribution].

Question 8

Find the Fisher information.

Answer. From Problem 5 the log-likelihood is

Hence the score function is (see Example 2.30 in ST2134)

Then

and the Fisher information is

Question 9

Derive the Cramer-Rao lower bound for for a random sample.

Answer. (See Example 3.17 in ST2134) Since is an unbiased estimator of by Problem 1, from the Cramer-Rao theorem we know that

and in fact by Problem 1 this lower bound is attained.

Estimation of parameters of a normal distribution

Here we show that the knowledge of the distribution of for linear regression allows one to do without long calculations contained in the guide ST 2134 by J. Abdey.

Theorem. Let be independent observations from . 1) is distributed as 2) The estimators and are independent. 3) 4) 5) converges in distribution to

Proof. We can write where is distributed as Putting and (a vector of ones) we satisfy (1) and (2). Since we have Further,

and

Thus 1) and 2) follow from results for linear regression.

3) For a normal variable its moment generating function is (see Guide ST2133, 2021, p.88). For the standard normal we get

Applying the general property (same guide, p.84) we see that

 

Therefore

4) By independence of standard normals

5) By standardizing we have and this converges in distribution to by the central limit theorem.

 

Distribution of the estimator of the error variance

If you are reading the book by Dougherty: this post is about the distribution of the estimator  defined in Chapter 3.

Consider regression

(1)

where the deterministic matrix is of size satisfies (regressors are not collinear) and the error satisfies

(2)

is estimated by Denote Using (1) we see that and the residual is estimated by

(3)

is a projector and has properties which are derived from those of

(4)

If is an eigenvalue of then multiplying by and using the fact that we get Hence eigenvalues of can be only or The equation
tells us that the number of eigenvalues equal to 1 is and the remaining are zeros. Let be the diagonal representation of Here is an orthogonal matrix,

(5)

and is a diagonal matrix with eigenvalues of on the main diagonal. We can assume that the first numbers on the diagonal of are ones and the others are zeros.

Theorem. Let be normal. 1) is distributed as 2) The estimators and are independent.

Proof. 1) We have by (4)

(6)

Denote From (2) and (5)

and is normal as a linear transformation of a normal vector. It follows that where is a standard normal vector with independent standard normal coordinates Hence, (6) implies

(7)

(3) and (7) prove the first statement.

2) First we note that the vectors are independent. Since they are normal, their independence follows from

It's easy to see that This allows us to show that is a function of :

Independence of leads to independence of their functions and

 

Sufficiency and minimal sufficiency

Sufficient statistic

I find that in the notation of a statistic it is better to reflect the dependence on the argument. So I write for a statistic, where is a sample, instead of a faceless or

Definition 1. The statistic is called sufficient for the parameter if the distribution of conditional on does not depend on

The main results on sufficiency and minimal sufficiency become transparent if we look at them from the point of view of Maximum Likelihood (ML) estimation.

Let be the joint density of the vector , where is a parameter (possibly a vector). The ML estimator is obtained by maximizing over the function with fixed at the observed data. The estimator depends on the data and can be denoted

Fisher-Neyman theorem. is sufficient for if and only if the joint density can be represented as

(1)

where, as the notation suggests, depends on only through and does not depend on

Maximizing the left side of (1) is the same thing as maximizing because does not depend on But this means that depends on only through A sufficient statistic is all you need to find the ML estimator. This interpretation is easier to understand than the definition of sufficiency.

Minimal sufficient statistic

Definition 2. A sufficient statistic is called minimal sufficient if for any other statistic there exists a function such that

A level set is a set of type for a constant (which in general can be a constant vector). See the visualization of level sets.  A level set is also called a preimage and denoted When is one-to-one the preimage contains just one point. When is not one-to-one the preimage contains more than one point. The wider it is the less information about the sample carries the statistic (because many data sets are mapped to a single point and you cannot tell one data set from another by looking at the statistic value). In the definition of the minimal sufficient statistic we have

Since generally contains more than one point, this shows that the level sets of are generally wider than those of Since this is true for any carries less information about than any other statistic.

Definition 2 is an existence statement and is difficult to verify directly as there are words "for any" and "exists". Again it's better to relate it to ML estimation.

Suppose for two sets of data there is a positive number such that

(2)

Maximizing the left side we get the estimator Maximizing we get Since does not depend on (2) tells us that

Thus, if two sets of data satisfy (2), the ML method cannot distinguish between and and supplies the same estimator. Let us call indistinguishable if there is a positive number such that (2) is true.

An equation means that belong to the same level set.

Characterization of minimal sufficiency. A statistic is minimal sufficient if and only if its level sets coincide with sets of indistinguishable

The advantage of this formulation is that it relates a geometric notion of level sets to the ML estimator properties. The formulation in the guide by J. Abdey is:

A statistic is minimal sufficient if and only if the equality is equivalent to (2).

Rewriting (2) as

(3)

we get a practical way of finding a minimal sufficient statistic: form the ratio on the left of (3) and find the sets along which the ratio does not depend on Those sets will be level sets of

Chi-squared distribution

This post is intended to close a gap in J. Abdey's guide ST2133, which is absence of distributions widely used in Econometrics.

Chi-squared with one degree of freedom

Let be a random variable and let

Question 1. What is the link between the distribution functions of and

The start is simple: just follow the definitions. Assuming that , on Chart 1 we see that Hence, using additivity of probability,

(1)

The last transition is based on the assumption that for all , which is maintained for continuous random variables throughout the guide by Abdey.

Question 2. What is the link between the densities of and By the Leibniz integral rule (1) implies

(2)

Exercise. Assuming that is an increasing differentiable function with the inverse and answer questions similar to 1 and 2.

See the definition of Just applying (2) to and we get

Since the procedure for identifying the gamma distribution gives

We have derived the density of the chi-squared variable with one degree of freedom, see also Example 3.52, J. Abdey, Guide ST2133.

General chi-squared

For with independent standard normals we can write where the chi-squared variables on the right are independent and all have one degree of freedom. This is because deterministic (here quadratic) functions of independent variables are independent.

Recall that the gamma density is closed under convolutions with the same Then by the convolution theorem we get

Gamma distribution

Definition. The gamma distribution is a two-parametric family of densities. For the density is defined by

Obviously, you need to know what is a gamma function. My notation of the parameters follows Feller, W. An Introduction to Probability Theory and its Applications, Volume II, 2nd edition (1971). It is different from the one used by J. Abdey in his guide ST2133.

Property 1

It is really a density because

 (replace )

Suppose you see an expression and need to determine which gamma density this is. The power of the exponent gives you and the power of gives you It follows that the normalizing constant should be and the density is

Property 2

The most important property is that the family of gamma densities with the same is closed under convolutions. Because of the associativity property it is enough to prove this for the case of two gamma densities.

First we want to prove

(1)

Start with the general definition of convolution and recall where the density vanishes:

(plug the densities and take out the constants)

(replace )

Thus (1) is true. Integrating it we have

We know that the convolution of two densities is a density. Therefore the last equation implies

and

Alternative proof. The moment generating function of a sum of two independent beta distributions with the same shows that this sum is again a beta distribution with the same , see pp. 141, 209 in the guide ST2133.

Gamma function

The gamma function and gamma distribution are two different things. This post is about the former and is a preparatory step to study the latter.

Definition. The gamma function is defined by

The integrand is smooth on so its integrability is determined by its behavior at and . Because of the exponent, it is integrable in the neighborhood of The singularity at is integrable if In all calculations involving the gamma function one should remember that its argument should be positive.

Properties

1) Factorial-like property. Integration by parts shows that

if

2) because

3) Combining the first two properties we see that for a natural

Thus the gamma function extends the factorial to non-integer

4)

Indeed, using the density of the standard normal we see that

(replacing )

(replacing )

Many other properties are not required in this course.

Analysis of problems with conditioning

These problems are among the most difficult. It's important to work out a general approach to such problems. All references are to J. Abdey,  Advanced statistics: distribution theory, ST2133, University of London, 2021.

General scheme

Step 1. Conditioning is usually suggested by the problem statement: is conditioned on .

Your life will be easier if you follow the notation used in the guide: use for probability mass functions (discrete variables) and for (probability) density functions (continuous variables).

a) If and both are discrete (Example 5.1, Example 5.13, Example 5.18):

b) If and both are continuous (Activity 5.6):

c) If is discrete, is continuous (Example 5.2, Activity 5.5):

d) If is continuous, is discrete (Activity 5.12):

In all cases you need to figure out over which to sum or integrate.

Step 2. Write out the conditional densities/probabilities with the same arguments
as in your conditional equation.

Step 3. Reduce the result to one of known distributions using the completeness
axiom.

Example 5.1

Let denote the number of hurricanes which form in a given year, and let denote the number of these which make landfall. Suppose each hurricane has a probability of making landfall independent of other hurricanes. Given the number of hurricanes , then can be thought of as the number of successes in independent and identically distributed Bernoulli trials. We can write this as . Suppose we also have that . Find the distribution of (noting that ).

Solution

Step 1. The number of hurricanes takes values and is distributed as Poisson. The number of landfalls for a given is binomial with values . It follows that .

Write the general formula for conditional probability:

Step 2. Specifying the distributions:

where

and

where

Step 3. Reduce the result to one of known distributions:

(pull out of summation everything that does not depend on summation variable
)

(replace to better see the structure)

(using the completeness axiom for the Poisson variable)