NotesAtHKU

Matrices are represented by capitial letters: $A\in R^{r\times c}$ . $r \times c$ are the dimensions of a given matrix

Basic operations

Addition

Matrices can perform per-element addition if their dimensions match.

$\begin{bmatrix} 1 & 2 & 3 \\ 4 & 5 & 6 \end{bmatrix} + \begin{bmatrix} 6 & 5 & 4 \\ 3 & 2 & 1 \end{bmatrix} = \begin{bmatrix} 7 & 7 & 7 \\7 & 7 & 7 \end{bmatrix}$

Scalar multiplication

Matrices can perform per-element multiplication with a scalar.

$\begin{bmatrix} 1 & 2 & 3 \\3 & 2 & 1 \end{bmatrix} \times 2 = \begin{bmatrix} 2 & 4 & 6 \\6 & 4 & 2 \end{bmatrix}$

Matrix-matrix multiplication

Matrices can only multiply if their inner dimensions match, otherwise there will be no solution.

If a matrix is multiplied by another matrix, each element in the result matrix is the sum of it's corresponding row in the original matrix performing per-element multiplication on it's corresponding col in the applying matrix

$\begin{bmatrix} 1 & 2 \\3 & 4 \end{bmatrix} \times \begin{bmatrix} 1 & 2 \\3 & 4 \end{bmatrix}=\begin{bmatrix} 1\cdot 1 + 2 \cdot 3 & 1\cdot 2 + 2\cdot 4 \\3\cdot 1 + 4\cdot 3&3\cdot 2 + 4\cdot 4 \end{bmatrix} = \begin{bmatrix} 7 & 10 \\15 & 22 \end{bmatrix}$

Transposition

The function $A^T$ transposes the matrix $A$ . It swaps the rows and columns of a matrix.

$\begin{bmatrix} 1 & 2 \\3&4 \end{bmatrix}^T \equiv \begin{bmatrix} 1 & 3 \\2&4 \end{bmatrix}\quad \begin{bmatrix} 1 & 2 & 3 \end{bmatrix}^T \equiv \begin{bmatrix} 1 \\2\\3 \end{bmatrix}$

Solving linear systems

Identity matrix

The identity matrix of $I\in \mathbb{R}^{n\times n}$ with square dimensions is the matrix with diagonal elements of 1 and others 0.

$I\in \mathbb{R}^{3\times 3}=\begin{bmatrix} 1 & 0 & 0 \\0&1&0\\0&0&1 \end{bmatrix}$

Row-echelon form

A matrix with a bottom-left or top-right triangle of 0's (excluding the diagonal) is considered to be in row-echelon form.

$\begin{bmatrix} 1 & 2 & 3 \\0&1&2\\0&0&1 \end{bmatrix}\text{ and }\begin{bmatrix} 1 & 0 & 0 \\1&2&0\\0&0&1 \end{bmatrix}$ are both considered in row-echelon form

Augmented Matrix and Gaussian Elimination

Augmented matrices are an important tool to solve linear equations. The number of rows is equal to the number of variables in the equation.

GE are operations by rows, performed on augmented matrices. The possible moves are:

Addition between rows ( $R_1 + R_2$ adds row 2 to row 1, leaving row 2 unchanged)
Multiplication of a row by a scalar ( $\frac{1}{2}R_1$ )
Swapping rows ( $R_1\leftrightarrow R_2$ )

Strict variables

Consider the following linear system. We can take the coefficients and write it into AM form as followed:

\begin{matrix} x_1 + 2x_2 = 3 \\ 2x_1 + 5x_2 = 10 \end{matrix}\xRightarrow{\hspace{0.5cm}} \left[\begin{array}{cc|c} 1 & 2 & 3 \\ \underbrace{2}_{x_1} & \underbrace{5}_{x_2} & 10 \end{array}\right]

Then, we perform GE on the created augmented matrix to make the left side row-echelon.

\left[\begin{array}{cc|c} 1 & 2 & 3 \\ 2 & 5 & 10 \end{array}\right] \xrightarrow{R_2-2R_1} \left[\begin{array}{cc|c} 1 & 2 & 3 \\ 0 & 1 & 4 \end{array}\right]

Deriving from the AM, we know:

\begin{matrix} x_1 + 2x_2 = 3 \\ x_2 = 4 \end{matrix}

So the solution is:

\begin{bmatrix} 3-2\cdot 4 \\ 4 \end{bmatrix}\equiv \begin{bmatrix} -5 \\ 4 \end{bmatrix}

Inconsistent systems

We say a system is inconsistent if a full row of 0 on the left is equal to a non-zero number. We can also conclude that there are no solutions to the system. The following is an example of a system with no solutions:

\begin{matrix} x_1 + 2x_2 = 3 \\ 2x_1 + 4x_2 = 10 \end{matrix}\xRightarrow{\hspace{0.5cm}} \left[\begin{array}{cc|c} 1 & 2 & 3 \\ 2 & 4 & 10 \end{array}\right]\xrightarrow{R_2-2R_1} \left[\begin{array}{cc|c} 1 & 2 & 3 \\ \htmlClass{text-red-500}{0} & \htmlClass{text-red-500}{0} & \htmlClass{text-red-500}{4} \end{array}\right]

Free variable systems

Free variables are variables with their corresponding columns having no leading 1s before 0s in the augmented matrix. If a system has free variables, the solution exists in a space, defined by the general solution.

Consider the following system with free variables, we can derive:

\left[\begin{array}{cccc|c} 1 & 0 & \htmlClass{text-blue-500}{-5} & \htmlClass{text-blue-500}{0} & 8 & 3 \\ 0 & 1 & \htmlClass{text-blue-500}{-4} & \htmlClass{text-blue-500}{-1} & 0 & 6 \\ 0 & 0 & \htmlClass{text-blue-500}{\underbrace{0}_{free}} & \htmlClass{text-blue-500}{\underbrace{0}_{free}} & 1 & 0 \end{array}\right]\xRightarrow{\hspace{0.5cm}}\begin{matrix} x_1 -5x_3+8x_5 = 3 \\ x_2-4x_3-x_4=6 \\ x_5=0 \end{matrix}

We can ignore the free variables, and substitute $x_5$ into both equations, deriving the general solution to be:

x=\begin{bmatrix} 3+5x_3 \\ 6+x_4+4x_3 \\ x_3 \\ x_4 \\ 0 \end{bmatrix}

Matrix determinant

Determinant is a property of a matrix, represented by $\|A\|$ or $det(A)$ . Only square matrices have a determinant.

Finding the value of determinant

The determinant of a 2 times 2 matrix

$for\quad A=\begin{bmatrix} a & b \\c&d \end{bmatrix},\quad |A| = ad - bc$

The determinant of a n times n matrix

Choose any row or column, then for each element, ignore their corresponding row and column to give the co-factor of the element.

$det(A) = \sum(a\times det(\text{co-factor})\times (-1)^{r_a+c_a})$

Where $a$ is an element from the picked row or column, and $r_a c_a$ are the row and column number of a

$|\begin{smallmatrix}1&2&3\\1&1&1\\3&2&1\end{smallmatrix}|=1|\begin{smallmatrix} 1&1\\2&1 \end{smallmatrix}|-2|\begin{smallmatrix} 1&1\\3&1 \end{smallmatrix}|+3|\begin{smallmatrix} 1&1\\3&2 \end{smallmatrix}|=0$

Determinant of row-echelon form matrices

Product of diagonal elements. $|\begin{smallmatrix} 1&2&3\\0&2&3\\0&0&3 \end{smallmatrix}|=1\cdot 2\cdot 3=6$

Vector dependency
Matrix inverse
Matrix eigenpairs

Vectors

Vectors are matrices with a single column, represented by letters with an arrow above them. $\overrightarrow{v} = \begin{bmatrix} 1 \\2\\3 \end{bmatrix}$

Dot product

The dot product of two vectors $\vec{a}$ and $\vec{b}$ is essentially $\vec{a}^T\cdot\vec{b}$

Vector length

$\|\|\vec{v}\|\|$ gives the length of a vector. $||\begin{bmatrix} a & b & \dots \end{bmatrix}^T||=\sqrt[]{a^2 + b^2 + \dots}$

Unit vectors

They are vectors with the length of one. $\frac{\vec{v}}{\|\|\vec{v}\|\|}$ transforms the vector to a unit vector.

Vector dependency

If $|\begin{smallmatrix} v_1 & v_2 & \dots \end{smallmatrix}|=0$ , the vectors are dependent of each other

Orthogonal projection

The orthogonal projection of $a$ onto $b$ is $proj_ba = (\frac{a^Tb}{b^Tb})b$ .

The component of $a$ perpendicular to $b$ is given by $a-proj_ba$

Orthonormal matrices

Orthonormal matrices have vector columns length 1, and which any product of two vectors is 0. For an orthonormal matrix $A$ , $AA^T = I$

Expressing a vector as a linear combination

We can express a vector $\vec{v}$ as the linear combination of other vectors $a, b, \dots$ by:

$v = proj_av + proj_bv + \dots$

Vector spans

Vectors have the same span if they are linearly dependent

The dimension of $span(A)$ is given by the number of non-zero rows after GE

Matrix inverse

Inverse is a property and function of a matrix, represented by $A^{-1}$

Matrix inversibility

The matrix has an inverse if $\|A\|\ne 0$ , and that it is a square matrix.

Finding the inverse of a 2 times 2 matrix

$A^{-1} = \frac{1}{|A|}\begin{bmatrix} d & -b \\-c &a \end{bmatrix}$

Finding the inverse of a n times n matrix

$\left[\begin{array}{c|c} A & I \end{array}\right]\xrightarrow{GE}\left[\begin{array}{c|c} I & A^{-1} \end{array}\right]$

Matrix eigenpairs

Matrices have properties eigenvalues $\lambda_n$ and eigenvectors $v_n$ . They must satisfy the condition $Av=\lambda v$ , and has the following properties:

$\lambda_n$ and $v_n$ come in pairs (each value correspond to a vector)
Matrices can only have $\lambda_1, \lambda_2 \dots \lambda_n$ where $n$ is the smallest dimension of the matrix
An $v_n$ includes all vectors of its multiple, and is non-zero By convention, $\lambda_1 > \lambda_2 > \dots > \lambda_n$

Solving for eigenvalues

Note that we can rearrange the condition to $(A-I\lambda)v=0$ , and into $\|A-I\lambda\| = 0$ . Rearrange into polynomial equation and solve for $\lambda$ by first trial and error (if degree $> 2$ ) and then factorization.

The eigenvalues of $\begin{bmatrix} 1 & 1 \\4&1 \end{bmatrix}$ can be found by $|\begin{smallmatrix} 1-\lambda&1\\4&1-\lambda \end{smallmatrix}|=0:\quad(1-\lambda)^2 - 4 = 0\quad\rightarrow\quad\lambda_1=3,\ \lambda_2=-1$

Algebratic and Geometric Multiplicity

AM is the number of times the value of $\lambda$ occur (e.g. $(1-\lambda)^2=0, \lambda_1=1$ has $AM=2$ )

GM is the size of nullspace (line of 0s) when plugging $\lambda$ into $(A-I\lambda)v=0$

Solving for eigenvectors

Plug each $\lambda$ into $(A-I\lambda)v=0$ , perform GE, then let the deterministic variable(s) as 1.

For $\lambda_1=3$ and $A=\begin{bmatrix} 1 & 1 & 1 \\1&1&1\\1&1&1 \end{bmatrix}$ :

$(A-3\lambda)v=0 \xRightarrow{\hspace{0.5cm}} \left[\begin{array}{ccc|c} -2 & 1 & 1 & 0 \\1&-2&1&0\\1&1&-2&0 \end{array}\right]\xrightarrow{GE}\left[\begin{array}{ccc|c} -2 & 1 & 1 & 0 \\0&-1&1&0\\\htmlClass{text-blue-500}{0}&\htmlClass{text-blue-500}{0}&\htmlClass{text-blue-500}{0}&\htmlClass{text-blue-500}{0} \end{array}\right]\begin{matrix} \ \\\ \\\htmlClass{text-blue-500}{GM=1} \end{matrix}$

From the augmented matrix: $-2x_1+x_2+x_3=0,\quad -x_2+x_3=0$ .

As $x_3$ is in both equations, let $x_3=1$ : $x_2=1, x_1=1\rightarrow v_1=\begin{bmatrix}1 \\1\\1\end{bmatrix}$

Solving for eigenvectors with GM 2

Find the other eigenvector with $v = z_2 - proj_{z_1}z_2$

For $v=\begin{bmatrix} -x_2-2x_3 \\x_2\\x_3 \end{bmatrix}$ , find the two solution forms as $z_1=\begin{bmatrix} -1 \\1\\0 \end{bmatrix},\ z_2\begin{bmatrix} -2 \\0\\1 \end{bmatrix}$

Then let $z_1=v_1$ , and solve for $v_2$ with $v_2=z_2-proj_{z_1}z_2=\begin{bmatrix}-2 \\0\\1\end{bmatrix} - \frac{2+0+0}{1+1+0}\begin{bmatrix}-1 \\1\\0\end{bmatrix}$

Related properties:

Diag
Quad
Diff

Matrix diagonalization

Condition

A matrix can only be diagonalized if all of it's eigenpairs has their $AM = GM$ .

Eigendecomposition

$A=VDV^{-1},\quad\text{where }V=\begin{bmatrix}v_1 \\\dots\\v_n\end{bmatrix},\ D=\begin{bmatrix}\lambda_1 & 0 & 0 \\0&\ddots&0\\0&0&\lambda_n\end{bmatrix}$

We can also derive $A^k=VD^kV^{-1}$

Differential equations

Given $x(0)$ , solve for $x(t)$ with the following steps:

Solve for eigenpairs
$xk = cve^{\lambda t} + \dots + cve^{\lambda t}$
Solve c by augmented matrix $\left[\begin{array}{ccc|c}v_1 & \dots & v_n & x(0)\end{array}\right]$

Quadratic form

A quadratic equation can be written into the form $Q(x)=xAx^T$ , where $A$ is a symmetric matrix.

Symmetric matrices

A matrix is symmetric if $A=A^T$

Linear to matrix quadratic

\begin{aligned} Q(x) & =ax_1^2+bx_2^2+cx_3^2+dx_1x_2+ex_2x_3+fx_1x_3 \\ & =\begin{bmatrix} x_1 & x_2 & x_3 \end{bmatrix}\begin{bmatrix} a & \frac{d}{2} & \frac{f}{2} \\\frac{d}{2}&b&\frac{e}{2}\\\frac{f}{2}&\frac{e}{2}&c \end{bmatrix}\begin{bmatrix} x_1 \\x_2\\x_3 \end{bmatrix} \end{aligned}

Definitiveness of a quadratic

For $Q(x)=xAx^T$ , considering the eigenvalues of $A$ :

$Q(x)$ is positive definitive if $\forall\lambda > 0$
$Q(x)$ is negative definitive if $\forall\lambda < 0$
$Q(x)$ is indefinite if otherwise
$Q(x)$ is p/n semi-definitive if the conditions are inclusive (i.e. $\lambda \geq or \leq 0$ )

Minimum and maximum values

$\lambda_{max} = max\{xAx^T:\|\|x\|\|=1\}$

$\lambda_{min} = min\{xAx^T:\|\|x\|\|=1\}$

Apply the corresponding $v\rightarrow x$ to find the min/max of $Q(x)$ . Note that $\|\|x\|\|=1$ is a required restriction on $Q(x)$ , otherwise its size would be infinite.

Matrices