We will now work through an example where the principal components cannot easily determined by inspection.
Given 4 data points in 2 dimensions:
[mathjaxinline]\displaystyle \displaystyle \mathbf{x}^{(1)}[/mathjaxinline] [mathjaxinline]\displaystyle =[/mathjaxinline] [mathjaxinline]\displaystyle (0,2)[/mathjaxinline]
[mathjaxinline]\displaystyle \mathbf{x}^{(2)}[/mathjaxinline] [mathjaxinline]\displaystyle =[/mathjaxinline] [mathjaxinline]\displaystyle (0,-2)[/mathjaxinline]
[mathjaxinline]\displaystyle \mathbf{x}^{(3)}[/mathjaxinline] [mathjaxinline]\displaystyle =[/mathjaxinline] [mathjaxinline]\displaystyle (1,1)[/mathjaxinline]
[mathjaxinline]\displaystyle \mathbf{x}^{(4)}[/mathjaxinline] [mathjaxinline]\displaystyle =[/mathjaxinline] [mathjaxinline]\displaystyle (-1,-1)[/mathjaxinline]
By inspection, roughly estimate the direction in which the (empirical) variance is the largest. (There is no answer box for this question.)
Find the spectral decomposition of the [mathjaxinline]\mathbf{S}[/mathjaxinline]. That is, find the eigenvalues and their corresponding eigenvectors.
Enter the eigenvalues in decreasing order (so [mathjaxinline]\lambda _1>\lambda _2[/mathjaxinline].)
[mathjaxinline]\lambda ^{(1)}=\quad [/mathjaxinline]
=
[mathjaxinline]\lambda ^{(2)}=\quad [/mathjaxinline]
Find the eigenvectors [mathjaxinline]\mathbf v_{\lambda _1}[/mathjaxinline] and [mathjaxinline]\mathbf v_{\lambda _2}[/mathjaxinline]. (All scalar multiples will be accepted)
[mathjaxinline]\mathbf v_{\lambda _1}=\quad[/mathjaxinline]
[mathjaxinline]\mathbf v_{\lambda _2}=\quad[/mathjaxinline]
To estimate the direction in which the variance is the largest, we can calculate the covariance matrix of the data points and find its eigenvectors and eigenvalues.
First, we calculate the mean of the data points:
[mathjaxinline]\bar{\mathbf{x}} = \frac{1}{4} \left( \mathbf{x}^{(1)} + \mathbf{x}^{(2)} + \mathbf{x}^{(3)} + \mathbf{x}^{(4)} \right) = \frac{1}{4} \left( (0,2) + (0,-2) + (1,1) + (-1,-1) \right) = \left( 0,0 \right)[/mathjaxinline]
Next, we calculate the covariance matrix [mathjaxinline]\mathbf{S}[/mathjaxinline]:
[mathjaxinline]\mathbf{S} = \frac{1}{3} \left( (\mathbf{x}^{(1)} - \bar{\mathbf{x}}) (\mathbf{x}^{(1)} - \bar{\mathbf{x}})^T + (\mathbf{x}^{(2)} - \bar{\mathbf{x}}) (\mathbf{x}^{(2)} - \bar{\mathbf{x}})^T + (\mathbf{x}^{(3)} - \bar{\mathbf{x}}) (\mathbf{x}^{(3)} - \bar{\mathbf{x}})^T + (\mathbf{x}^{(4)} - \bar{\mathbf{x}}) (\mathbf{x}^{(4)} - \bar{\mathbf{x}})^T \right)[/mathjaxinline]
Plugging in the values:
[mathjaxinline]\mathbf{S} = \frac{1}{3} \left( \begin{pmatrix} 0 \\ 2 \end{pmatrix} \begin{pmatrix} 0 & 2 \end{pmatrix} + \begin{pmatrix} 0 \\ -2 \end{pmatrix} \begin{pmatrix} 0 & -2 \end{pmatrix} + \begin{pmatrix} 1 \\ 1 \end{pmatrix} \begin{pmatrix} 1 & 1 \end{pmatrix} + \begin{pmatrix} -1 \\ -1 \end{pmatrix} \begin{pmatrix} -1 & -1 \end{pmatrix} \right)[/mathjaxinline]
Simplifying:
[mathjaxinline]\mathbf{S} = \frac{1}{3} \begin{pmatrix} 2 & 2 \\ 2 & 2 \end{pmatrix} + \frac{1}{3} \begin{pmatrix} 2 & -2 \\ -2 & 2 \end{pmatrix} + \frac{1}{3} \begin{pmatrix} 2 & 2 \\ 2 & 2 \end{pmatrix} + \frac{1}{3} \begin{pmatrix} 2 & -2 \\ -2 & 2 \end{pmatrix}[/mathjaxinline]
[mathjaxinline]\mathbf{S} = \begin{pmatrix} \frac{4}{3} & \frac{4}{3} \\ \frac{4}{3} & \frac{4}{3} \end{pmatrix}[/mathjaxinline]
Now, we find the eigenvalues [mathjaxinline]\lambda^{(1)}[/mathjaxinline] and [mathjaxinline]\lambda^{(2)}[/mathjaxinline] of [mathjaxinline]\mathbf{S}[/mathjaxinline]:
[mathjaxinline]\text{det}(\mathbf{S} - \lambda \mathbf{I}) = 0[/mathjaxinline]
[mathjaxinline]\begin{vmatrix} \frac{4}{3} - \lambda & \frac{4}{3} \\ \frac{4}{3} & \frac{4}{3} - \lambda \end{vmatrix} = 0[/mathjaxinline]
[mathjaxinline]\left(\frac{4}{3} - \lambda\right)\left(\frac{4}{3} - \lambda\right) - \left(\frac{4}{3}\right)\left(\frac{4}{3}\right) = 0[/mathjaxinline]
[mathjaxinline]\lambda^2 - \frac{8}{3} \lambda + \frac{16}{9} - \frac{16}{9} = 0[/mathjaxinline]
[mathjaxinline]\lambda^2 - \frac{8}{3} \lambda = 0[/mathjaxinline]
[mathjaxinline]\lambda \left( \lambda - \frac{8}{3} \right) = 0[/mathjaxinline]
Solving for [mathjaxinline]\lambda[/mathjaxinline]:
[mathjaxinline]\lambda = 0 \quad \text{ or } \quad \lambda = \frac{8}{3}[/mathjaxinline]
The eigenvalues are [mathjaxinline]\lambda^{(1)} = \frac{8}{3}[/mathjaxinline] and [mathjaxinline]\lambda^{(2)} = 0[/mathjaxinline].
To find the corresponding eigenvectors, we solve the equation [mathjaxinline](\mathbf{S} - \lambda \mathbf{I})\mathbf{v} = \mathbf{0}[/mathjaxinline].
For [mathjaxinline]\lambda^{(1)}[/mathjaxinline]:
[mathjaxinline]\begin{pmatrix} \frac{4}{3} - \frac{8}{3} & \frac{4}{3} \\ \frac{4}{3} & \frac{4}{3} - \frac{8}{3} \end{pmatrix} \begin{pmatrix} v_{1} \\ v_{2} \end{pmatrix} = \begin{pmatrix} 0 \\ 0 \end{pmatrix}[/mathjaxinline]
[mathjaxinline]\begin{pmatrix} -1 & \frac{4}{3} \\ \frac{4}{3} & -1 \end{pmatrix} \begin{pmatrix} v_{1} \\ v_{2} \end{pmatrix} = \begin{pmatrix} 0 \\ 0 \end{pmatrix}[/mathjaxinline]
Simplifying:
[mathjaxinline]-v_{1} + \frac{4}{3}v_{2} = 0[/mathjaxinline]
[mathjaxinline}\frac{4}{3}v_{1} - v_{2} = 0[/mathjaxinline]
Solving this system of equations, we find the eigenvector [mathjaxinline]\mathbf{v}_{\lambda^{(1)}} = \begin{pmatrix} 1 \\ 1 \end{pmatrix}[/mathjaxinline].
For [mathjaxinline]\lambda^{(2)}[/mathjaxinline]:
[mathjaxinline]\begin{pmatrix} \frac{4}{3} & \frac{4}{3} \\ \frac{4}{3} & \frac{4}{3} \end{pmatrix} \begin{pmatrix} v_{1} \\ v_{2} \end{pmatrix} = \begin{pmatrix} 0 \\ 0 \end{pmatrix}[/mathjaxinline]
[mathjaxinline]\begin{pmatrix} \frac{4}{3} & \frac{4}{3} \\ \frac{4}{3} & \frac{4}{3} \end{pmatrix} \begin{pmatrix} v_{1} \\ v_{2} \end{pmatrix} = \begin{pmatrix} 0 \\ 0 \end{pmatrix}[/mathjaxinline]
Simplifying:
[mathjaxinline]\frac{4}{3}v_{1} + \frac{4}{3}v_{2} = 0[/mathjaxinline]
[mathjaxinline}\frac{4}{3}v_{1} + \frac{4}{3}v_{2} = 0[/mathjaxinline]
Solving this system of equations, we find the eigenvector [mathjaxinline]\mathbf{v}_{\lambda^{(2)}} = \begin{pmatrix} 1 \\ -1 \end{pmatrix}[/mathjaxinline].
Therefore, the spectral decomposition of [mathjaxinline]\mathbf{S}[/mathjaxinline] is:
[mathjaxinline]\mathbf{S} = \frac{8}{3} \begin{pmatrix} 1 \\ 1 \end{pmatrix} \begin{pmatrix} 1 & -1 \end{pmatrix} + 0 \begin{pmatrix} 1 \\ -1 \end{pmatrix} \begin{pmatrix} 1 & -1 \end{pmatrix}[/mathjaxinline]
[mathjaxinline]\mathbf{S} = \frac{8}{3} \begin{pmatrix} 1 & -1 \\ 1 & -1 \end{pmatrix}[/mathjaxinline]
The eigenvalues in decreasing order are [mathjaxinline]\lambda^{(1)} = \frac{8}{3}[/mathjaxinline] and [mathjaxinline]\lambda^{(2)} = 0[/mathjaxinline].
The eigenvectors are [mathjaxinline]\mathbf{v}_{\lambda^{(1)}} = \begin{pmatrix} 1 \\ 1 \end{pmatrix}[/mathjaxinline] and [mathjaxinline]\mathbf{v}_{\lambda^{(2)}} = \begin{pmatrix} 1 \\ -1 \end{pmatrix}[/mathjaxinline].
To estimate the direction in which the (empirical) variance is the largest, we can calculate the covariance matrix of the data points. The covariance matrix is given by:
[mathjaxinline]\mathbf{S} = \frac{1}{n-1} \sum_{i=1}^{n} (\mathbf{x}^{(i)} - \bar{\mathbf{x}})(\mathbf{x}^{(i)} - \bar{\mathbf{x}})^T[/mathjaxinline]
where [mathjaxinline]\bar{\mathbf{x}}[/mathjaxinline] is the mean vector.
Let's calculate the covariance matrix:
[mathjaxinline]\mathbf{S} = \begin{bmatrix} 0 & 0 & 1 & -1 \\ 2 & -2 & 1 & -1 \end{bmatrix} \begin{bmatrix} 0 & 2 & 1 & -1 \\ 0 & -2 & 1 & -1 \end{bmatrix} = \begin{bmatrix} 2 & -2 \\ -2 & 2 \end{bmatrix}[/mathjaxinline]
To find the spectral decomposition of the covariance matrix, we need to find the eigenvalues and eigenvectors.
Let's solve for the eigenvalues:
[mathjaxinline]\text{det}(\mathbf{S} - \lambda \mathbf{I}) = \begin{vmatrix} 2-\lambda & -2 \\ -2 & 2-\lambda \end{vmatrix}[/mathjaxinline]
[mathjaxinline]=(2-\lambda)(2-\lambda) - (-2)(-2)[/mathjaxinline]
[mathjaxinline]= \lambda^2 - 4\lambda + 4 + 4 = \lambda^2 - 4\lambda + 8[/mathjaxinline]
Solving the quadratic equation, we find:
[mathjaxinline]\lambda_1 = 2 + 2i[/mathjaxinline]
[mathjaxinline]\lambda_2 = 2 - 2i[/mathjaxinline]
Now let's find the eigenvector corresponding to [mathjaxinline]\lambda_1[/mathjaxinline]:
For [mathjaxinline]\lambda_1 = 2 + 2i[/mathjaxinline], substituting back into [mathjaxinline](\mathbf{S} - \lambda \mathbf{I})\mathbf{v} = \mathbf{0}[/mathjaxinline], we get:
[mathjaxinline]\begin{bmatrix} 2-(2+2i) & -2 \\ -2 & 2-(2+2i) \end{bmatrix} \begin{bmatrix} v_1 \\ v_2 \end{bmatrix} = \begin{bmatrix} 0 \\ 0 \end{bmatrix}[/mathjaxinline]
Simplifying, we have:
[mathjaxinline]\begin{bmatrix} -2i & -2 \\ -2 & -2i \end{bmatrix} \begin{bmatrix} v_1 \\ v_2 \end{bmatrix} = \begin{bmatrix} 0 \\ 0 \end{bmatrix}[/mathjaxinline]
This leads to the equation:
[mathjaxinline]-2iv_1 - 2v_2 = 0[/mathjaxinline]
Solving this equation, we find that [mathjaxinline]v_1 = iv_2[/mathjaxinline]. Let's choose [mathjaxinline]v_2 = 1[/mathjaxinline], then [mathjaxinline]v_1 = i[/mathjaxinline].
Therefore, the eigenvector corresponding to [mathjaxinline]\lambda_1[/mathjaxinline] is [mathjaxinline]\mathbf{v}_{\lambda_1} = \begin{bmatrix} i \\ 1 \end{bmatrix}[/mathjaxinline].
Now let's find the eigenvector corresponding to [mathjaxinline]\lambda_2[/mathjaxinline]:
For [mathjaxinline]\lambda_2 = 2 - 2i[/mathjaxinline], repeating the same steps, we find that [mathjaxinline]\mathbf{v}_{\lambda_2} = \begin{bmatrix} -i \\ 1 \end{bmatrix}[/mathjaxinline].
Thus, the eigenvalues in decreasing order are [mathjaxinline]\lambda_1 = 2 + 2i[/mathjaxinline] and [mathjaxinline]\lambda_2 = 2 - 2i[/mathjaxinline].
The corresponding eigenvectors are [mathjaxinline]\mathbf{v}_{\lambda_1} = \begin{bmatrix} i \\ 1 \end{bmatrix}[/mathjaxinline] and [mathjaxinline]\mathbf{v}_{\lambda_2} = \begin{bmatrix} -i \\ 1 \end{bmatrix}[/mathjaxinline].
To estimate the direction in which the empirical variance is the largest, we need to calculate the covariance matrix. The covariance matrix, denoted as [mathjaxinline]\mathbf{S}[/mathjaxinline], is a square matrix that contains the variances and covariances of the data points.
To calculate the covariance matrix, we first need to calculate the mean of each dimension. In this case, since we have 2-dimensional data, we need to calculate the mean of the x-values and the mean of the y-values.
The mean of the x-values is given by:
[mathjaxinline]\displaystyle \mu _{x}\ =\ \frac{1}{4}( 0\ +\ 0\ +\ 1\ +\ ( -1)) \ =\ 0[/mathjaxinline]
The mean of the y-values is given by:
[mathjaxinline]\mu _{y}\ =\ \frac{1}{4}( 2\ +\ ( -2)\ +\ 1\ +\ ( -1)) \ =\ 0[/mathjaxinline]
Next, we calculate the covariance between the x and y dimensions. The covariance between two dimensions is given by:
[mathjaxinline]\displaystyle \text{cov}( x,y) \ =\ \frac{1}{n-1}\sum _{i=1}^{n}( x_{i}\ -\ \mu _{x})( y_{i}\ -\ \mu _{y})[/mathjaxinline]
Let's calculate the covariance:
For [mathjaxinline]\displaystyle \mathbf{x}^{(1)}( 0,2)[/mathjaxinline]:
[mathjaxinline]\displaystyle \text{cov}( x,y) \ =\ \frac{1}{4}( 0\ -\ 0)( 2\ -\ 0)\ =\ 0[/mathjaxinline]
For [mathjaxinline]\displaystyle \mathbf{x}^{(2)}( 0,-2)[/mathjaxinline]:
[mathjaxinline]\displaystyle \text{cov}( x,y) \ =\ \frac{1}{4}( 0\ -\ 0)( -2\ -\ 0)\ =\ 0[/mathjaxinline]
For [mathjaxinline]\displaystyle \mathbf{x}^{(3)}( 1,1)[/mathjaxinline]:
[mathjaxinline]\displaystyle \text{cov}( x,y) \ =\ \frac{1}{4}( 1\ -\ 0)( 1\ -\ 0)\ =\ \frac{1}{4}[/mathjaxinline]
For [mathjaxinline]\displaystyle \mathbf{x}^{(4)}( -1,-1)[/mathjaxinline]:
[mathjaxinline]\displaystyle \text{cov}( x,y) \ =\ \frac{1}{4}( -1\ -\ 0)( -1\ -\ 0)\ =\ \frac{1}{4}[/mathjaxinline]
Now, we can construct the covariance matrix [mathjaxinline]\displaystyle \mathbf{S}[/mathjaxinline] using the covariances we calculated:
[mathjaxinline]\displaystyle \mathbf{S}\ =\begin{pmatrix} \frac{1}{4} & 0\\ 0 & \frac{1}{4} \end{pmatrix}[/mathjaxinline]
To find the eigenvalues and eigenvectors of the covariance matrix, we need to solve the following equation:
[mathjaxinline]\displaystyle \mathbf{Sv} \ =\ \lambda \mathbf{v}[/mathjaxinline]
First, let's calculate the determinant of [mathjaxinline]\displaystyle \mathbf{S}\ -\ \lambda \mathbf{I}[/mathjaxinline], where [mathjaxinline]\displaystyle \mathbf{I}[/mathjaxinline] is the identity matrix:
[mathjaxinline]\displaystyle \mathbf{S}\ -\ \lambda \mathbf{I}\ =\begin{pmatrix} \frac{1}{4} & 0\\ 0 & \frac{1}{4} \end{pmatrix} -\begin{pmatrix} \lambda & 0\\ 0 & \lambda \end{pmatrix}[/mathjaxinline]
[mathjaxinline]\displaystyle \mathbf{S}\ -\ \lambda \mathbf{I}\ =\begin{pmatrix} \frac{1}{4} -\lambda & 0\\ 0 & \frac{1}{4} -\lambda \end{pmatrix}[/mathjaxinline]
To find the eigenvalues, set the determinant of [mathjaxinline]\displaystyle \mathbf{S}\ -\ \lambda \mathbf{I}[/mathjaxinline] equal to zero:
[mathjaxinline]\displaystyle \det(\mathbf{S}\ -\ \lambda \mathbf{I})\ =\ \left| \frac{1}{4} -\lambda \right| \left| \frac{1}{4} -\lambda \right|\ =\ 0[/mathjaxinline]
This gives us two eigenvalues:
[mathjaxinline]\displaystyle \lambda ^{(1)}\ =\frac{1}{4}\ \text{and}\ \lambda ^{(2)}\ =0[/mathjaxinline]
To find the eigenvectors, substitute each eigenvalue into [mathjaxinline]\displaystyle \mathbf{Sv} \ =\ \lambda \mathbf{v}[/mathjaxinline] and solve for [mathjaxinline]\displaystyle \mathbf{v}[/mathjaxinline].
For [mathjaxinline]\displaystyle \lambda ^{(1)}\ =\frac{1}{4}[/mathjaxinline]:
[mathjaxinline]\displaystyle \left( \frac{1}{4} -\lambda ^{(1)}\right) v_{1}\ =0[/mathjaxinline]
[mathjaxinline]\displaystyle \left( \frac{1}{4} -\frac{1}{4}\right) v_{1}\ =0[/mathjaxinline]
[mathjaxinline]\displaystyle 0v_{1}\ =0[/mathjaxinline]
Since the equation is satisfied for any value of [mathjaxinline]\displaystyle v_{1}[/mathjaxinline], we can choose a value of [mathjaxinline]\displaystyle v_{1}[/mathjaxinline] (except 0). Let's choose [mathjaxinline]\displaystyle v_{1}\ =\ 1[/mathjaxinline].
For [mathjaxinline]\displaystyle \lambda ^{(2)}\ =\ 0[/mathjaxinline]:
[mathjaxinline]\displaystyle \left( \frac{1}{4} -\lambda ^{(2)}\right) v_{2}\ =0[/mathjaxinline]
[mathjaxinline]\displaystyle \left( \frac{1}{4} -0\right) v_{2}\ =0[/mathjaxinline]
[mathjaxinline]\displaystyle \frac{1}{4} v_{2}\ =0[/mathjaxinline]
Since [mathjaxinline]\displaystyle \frac{1}{4} v_{2}\ =\ 0[/mathjaxinline], [mathjaxinline]\displaystyle v_{2}\ =\ 0[/mathjaxinline].
Therefore, the eigenvectors are:
[mathjaxinline]\displaystyle \mathbf{v}_{\lambda ^{(1)}}\ =\begin{pmatrix} 1\\ 0 \end{pmatrix}\ \text{and}\ \mathbf{v}_{\lambda ^{(2)}}\ =\begin{pmatrix} 0\\ 0 \end{pmatrix}[/mathjaxinline]
Note that any scalar multiple of an eigenvector is also an eigenvector. In this case, we can also choose [mathjaxinline]\displaystyle \mathbf{v}_{\lambda ^{(1)}}\ =\begin{pmatrix} 2\\ 0 \end{pmatrix}[/mathjaxinline] or any other non-zero multiple of [mathjaxinline]\displaystyle \mathbf{v}_{\lambda ^{(1)}}[/mathjaxinline].