Comprehensive Linear Algebra Notes
Table of Contents
1. Introduction to Linear Algebra
Linear Algebra is a branch of mathematics that deals with vector spaces, linear transformations, systems of linear equations, and matrices. It provides a framework for solving problems across various fields including physics, engineering, computer science, economics, and data science.
Key Concepts in Linear Algebra
- Vectors: Quantities with magnitude and direction
- Matrices: Rectangular arrays of numbers
- Linear Equations: Equations of the form a₁x₁ + a₂x₂ + ... + aₙxₙ = b
- Vector Spaces: Sets closed under vector addition and scalar multiplication
- Linear Transformations: Functions between vector spaces that preserve vector addition and scalar multiplication
- Eigenvalues and Eigenvectors: Special values and vectors related to linear transformations
Why is Linear Algebra Important?
Linear algebra provides tools for understanding multi-dimensional space, solving systems of equations, and analyzing data. It's fundamental to modern mathematics, computer science, and engineering. Applications include computer graphics, machine learning, quantum mechanics, economics, and statistics.
2. Vectors and Vector Operations
A vector is a mathematical object that has both magnitude (size) and direction. In an n-dimensional space, a vector has n components.
In 2D space, vectors can be represented as: v = (v₁, v₂)
In 3D space, vectors can be represented as: v = (v₁, v₂, v₃)
Vector Representations
Vectors can be represented in different ways:
- Geometric Representation: Arrows in space, with length indicating magnitude and orientation indicating direction
- Component Representation: Ordered lists of numbers (v₁, v₂, ..., vₙ)
- Algebraic Representation: Linear combinations of basis vectors v = v₁e₁ + v₂e₂ + ... + vₙeₙ
Basic Vector Operations
Vector Addition
The addition of two vectors u = (u₁, u₂, ..., uₙ) and v = (v₁, v₂, ..., vₙ) is:
u + v = (u₁ + v₁, u₂ + v₂, ..., uₙ + vₙ)
Example: Add the vectors u = (3, 1, 4) and v = (2, 5, -1)
u + v = (3+2, 1+5, 4+(-1)) = (5, 6, 3)
Scalar Multiplication
The multiplication of a vector v = (v₁, v₂, ..., vₙ) by a scalar c is:
cv = (cv₁, cv₂, ..., cvₙ)
Example: Find 3v where v = (2, -1, 5)
3v = 3(2, -1, 5) = (3×2, 3×(-1), 3×5) = (6, -3, 15)
Dot Product
The dot product of two vectors u = (u₁, u₂, ..., uₙ) and v = (v₁, v₂, ..., vₙ) is:
u · v = u₁v₁ + u₂v₂ + ... + uₙvₙ
Example: Find the dot product of u = (1, 3, -2) and v = (4, 0, 5)
u · v = (1×4) + (3×0) + (-2×5) = 4 + 0 + (-10) = -6
Cross Product (3D Vectors Only)
The cross product of two 3D vectors u = (u₁, u₂, u₃) and v = (v₁, v₂, v₃) is:
u × v = (u₂v₃ - u₃v₂, u₃v₁ - u₁v₃, u₁v₂ - u₂v₁)
Example: Calculate u × v where u = (2, 1, 3) and v = (1, 0, 4)
u × v = ((1×4) - (3×0), (3×1) - (2×4), (2×0) - (1×1))
= (4 - 0, 3 - 8, 0 - 1) = (4, -5, -1)
Vector Properties
Vector Magnitude (Length)
The magnitude of a vector v = (v₁, v₂, ..., vₙ) is:
|v| = √(v₁² + v₂² + ... + vₙ²)
Example: Find the magnitude of v = (3, 4)
|v| = √(3² + 4²) = √(9 + 16) = √25 = 5
Unit Vectors
A unit vector is a vector with a magnitude of 1. To find the unit vector in the direction of v, divide v by its magnitude:
û = v / |v|
Example: Find the unit vector in the direction of v = (3, 4)
|v| = 5 (from previous example)
û = v / |v| = (3, 4) / 5 = (3/5, 4/5) = (0.6, 0.8)
Vector Projections
The scalar projection of u onto v is:
projvu = (u · v) / |v|
The vector projection of u onto v is:
projvu = ((u · v) / |v|²) v
Example: Find the scalar and vector projections of u = (2, 3) onto v = (4, 0)
u · v = (2×4) + (3×0) = 8
|v| = √(4² + 0²) = 4
Scalar projection: projvu = 8 / 4 = 2
Vector projection: projvu = (8 / 16) × (4, 0) = (2, 0)
3. Matrices and Matrix Operations
A matrix is a rectangular array of numbers, symbols, or expressions arranged in rows and columns. A matrix with m rows and n columns is called an m × n matrix.
Matrix Representation
A general m × n matrix A is represented as:
Special Types of Matrices
- Square Matrix: A matrix with the same number of rows and columns (m = n)
- Identity Matrix (I): A square matrix with 1s on the main diagonal and 0s elsewhere
- Zero Matrix (0): A matrix where all elements are 0
- Diagonal Matrix: A square matrix where all non-diagonal elements are 0
- Upper Triangular Matrix: A square matrix where all elements below the main diagonal are 0
- Lower Triangular Matrix: A square matrix where all elements above the main diagonal are 0
- Symmetric Matrix: A square matrix equal to its transpose (A = Aᵀ)
Basic Matrix Operations
Matrix Addition
If A and B are m × n matrices, their sum A + B is the m × n matrix where each element is the sum of the corresponding elements of A and B:
(A + B)ᵢⱼ = Aᵢⱼ + Bᵢⱼ
Example: Add the matrices A = [1 2/3 4] and B = [5 6/7 8]
A + B = [1+5 2+6/3+7 4+8] = [6 8/10 12]
Scalar Multiplication
If A is an m × n matrix and c is a scalar, then cA is the m × n matrix where each element is c times the corresponding element of A:
(cA)ᵢⱼ = c × Aᵢⱼ
Example: Calculate 3A where A = [1 2/3 4]
3A = 3 × [1 2/3 4] = [3×1 3×2/3×3 3×4] = [3 6/9 12]
Matrix Multiplication
If A is an m × n matrix and B is an n × p matrix, their product AB is the m × p matrix where:
(AB)ᵢⱼ = Σₖ₌₁ⁿ Aᵢₖ × Bₖⱼ
In other words, the (i,j) entry of AB is the dot product of the ith row of A and the jth column of B.
Example: Calculate AB where A = [1 2/3 4] and B = [5 6/7 8]
(AB)₁₁ = (1×5) + (2×7) = 5 + 14 = 19
(AB)₁₂ = (1×6) + (2×8) = 6 + 16 = 22
(AB)₂₁ = (3×5) + (4×7) = 15 + 28 = 43
(AB)₂₂ = (3×6) + (4×8) = 18 + 32 = 50
AB = [19 22/43 50]
Matrix Transpose
The transpose of an m × n matrix A is the n × m matrix Aᵀ where:
(Aᵀ)ᵢⱼ = Aⱼᵢ
In other words, the rows of A become the columns of Aᵀ and vice versa.
Example: Find the transpose of A = [1 2 3/4 5 6]
Aᵀ = [1 4/2 5/3 6]
Matrix Properties
Determinant (for Square Matrices)
The determinant of a square matrix A, denoted det(A) or |A|, is a scalar value that provides information about the matrix. For a 2×2 matrix:
|A| = |a b/c d| = ad - bc
Example: Calculate the determinant of A = [3 5/2 4]
|A| = (3×4) - (5×2) = 12 - 10 = 2
Matrix Inverse (for Square Matrices)
The inverse of a square matrix A, denoted A⁻¹, is a matrix such that AA⁻¹ = A⁻¹A = I (the identity matrix).
For a 2×2 matrix A = [a b/c d], if |A| ≠ 0, then:
A⁻¹ = (1/|A|) × [d -b/-c a]
Example: Find the inverse of A = [3 5/2 4]
|A| = 2 (from previous example)
A⁻¹ = (1/2) × [4 -5/-2 3] = [2 -2.5/-1 1.5]
Matrix Rank
The rank of a matrix is the dimension of the vector space generated by its rows or columns. It equals the maximum number of linearly independent rows or columns.
Example: Find the rank of A = [1 2 3/2 4 6/3 5 7]
Row 2 = 2 × Row 1, so Row 2 is linearly dependent on Row 1.
Row 3 is not a multiple of Row 1, so it's linearly independent.
The matrix has 2 linearly independent rows, so rank(A) = 2.
4. Systems of Linear Equations
A system of linear equations is a collection of one or more linear equations involving the same variables. In matrix form, it can be written as Ax = b, where A is the coefficient matrix, x is the variable vector, and b is the constant vector.
Matrix Form of Linear Systems
Consider the system:
a₁₁x₁ + a₁₂x₂ + ... + a₁ₙxₙ = b₁
a₂₁x₁ + a₂₂x₂ + ... + a₂ₙxₙ = b₂
...
aₘ₁x₁ + aₘ₂x₂ + ... + aₘₙxₙ = bₘ
This can be written in matrix form as:
Solution Methods
Gaussian Elimination
Gaussian elimination is a method for solving systems of linear equations by transforming the augmented matrix [A|b] into row echelon form using elementary row operations.
Example: Solve the system using Gaussian elimination:
x + 2y + 3z = 14
2x + y + z = 8
3x + y + 2z = 14
Step 1: Write the augmented matrix
Step 2: Perform row operations to get an upper triangular matrix
R₂ = R₂ - 2R₁: [1 2 3 | 14/0 -3 -5 | -20/3 1 2 | 14]
R₃ = R₃ - 3R₁: [1 2 3 | 14/0 -3 -5 | -20/0 -5 -7 | -28]
R₃ = R₃ - (5/3)R₂: [1 2 3 | 14/0 -3 -5 | -20/0 0 1/3 | 5/3]
Step 3: Back-substitution
From row 3: z = 5
From row 2: -3y - 5z = -20 → -3y - 5(5) = -20 → -3y = 5 → y = -5/3
From row 1: x + 2y + 3z = 14 → x + 2(-5/3) + 3(5) = 14 → x - 10/3 + 15 = 14 → x = 14 - 15 + 10/3 = -1 + 10/3 = 1/3
Solution: x = 1/3, y = -5/3, z = 5
Cramer's Rule
For a system Ax = b where A is an n × n matrix with |A| ≠ 0, the solution is:
xᵢ = |Aᵢ| / |A|
where Aᵢ is the matrix formed by replacing the ith column of A with the column vector b.
Example: Use Cramer's rule to solve:
2x + y = 5
x + 3y = 10
A = [2 1/1 3], b = [5/10]
|A| = (2×3) - (1×1) = 6 - 1 = 5
A₁ = [5 1/10 3], |A₁| = (5×3) - (1×10) = 15 - 10 = 5
A₂ = [2 5/1 10], |A₂| = (2×10) - (5×1) = 20 - 5 = 15
x = |A₁|/|A| = 5/5 = 1
y = |A₂|/|A| = 15/5 = 3
Matrix Inverse Method
For a system Ax = b where A is an n × n invertible matrix, the solution is:
x = A⁻¹b
Example: Solve the system using the matrix inverse method:
2x + y = 5
x + 3y = 10
A = [2 1/1 3], b = [5/10]
|A| = 5 (from previous example)
A⁻¹ = (1/5) × [3 -1/-1 2] = [3/5 -1/5/-1/5 2/5]
x = A⁻¹b = [3/5 -1/5/-1/5 2/5] × [5/10]
= [(3/5×5) + (-1/5×10)/(-1/5×5) + (2/5×10)]
= [3 - 2/-1 + 4] = [1/3]
Types of Solutions for Linear Systems
- Unique Solution: The system has exactly one solution (when rank(A) = rank([A|b]) = n for an n-variable system)
- Infinitely Many Solutions: The system has infinitely many solutions (when rank(A) = rank([A|b]) < n)
- No Solution: The system is inconsistent (when rank(A) < rank([A|b]))
Homogeneous Systems
A homogeneous system is of the form Ax = 0. It always has at least the trivial solution x = 0. It has non-trivial solutions if and only if |A| = 0 (i.e., A is singular).
5. Linear Transformations
A linear transformation T from a vector space V to a vector space W is a function T: V → W that preserves vector addition and scalar multiplication:
T(u + v) = T(u) + T(v) for all u, v ∈ V
T(cu) = cT(u) for all u ∈ V and scalar c
Matrix Representation of Linear Transformations
Every linear transformation T: ℝⁿ → ℝᵐ can be represented by an m × n matrix A such that T(x) = Ax for all x ∈ ℝⁿ.
Example: Find the matrix A of the linear transformation T: ℝ² → ℝ² that rotates vectors counterclockwise by 90°.
For the standard basis vectors:
T(e₁) = T(1,0) = (0,1)
T(e₂) = T(0,1) = (-1,0)
Therefore, A = [0 -1/1 0]
Common Linear Transformations in ℝ²
Rotation
Rotation by angle θ (counterclockwise):
A = [cos(θ) -sin(θ)/sin(θ) cos(θ)]
Example: Find the result of rotating the vector (3,2) by 45° counterclockwise.
A = [cos(45°) -sin(45°)/sin(45°) cos(45°)] = [1/√2 -1/√2/1/√2 1/√2] ≈ [0.707 -0.707/0.707 0.707]
T(3,2) = A[3/2] = [0.707×3 + (-0.707)×2/0.707×3 + 0.707×2] ≈ [2.121 - 1.414/2.121 + 1.414] ≈ [0.707/3.535]
Scaling
Scaling by factors a and b in the x and y directions:
A = [a 0/0 b]
Example: Scale the vector (3,2) by a factor of 2 in the x-direction and 3 in the y-direction.
A = [2 0/0 3]
T(3,2) = A[3/2] = [2×3 + 0×2/0×3 + 3×2] = [6/6]
Reflection
Reflection across the x-axis: A = [1 0/0 -1]
Reflection across the y-axis: A = [-1 0/0 1]
Reflection across the line y = x: A = [0 1/1 0]
Example: Reflect the vector (3,2) across the x-axis.
A = [1 0/0 -1]
T(3,2) = A[3/2] = [1×3 + 0×2/0×3 + (-1)×2] = [3/-2]
Shear
Horizontal shear with factor k: A = [1 k/0 1]
Vertical shear with factor k: A = [1 0/k 1]
Example: Apply a horizontal shear with factor 2 to the vector (3,2).
A = [1 2/0 1]
T(3,2) = A[3/2] = [1×3 + 2×2/0×3 + 1×2] = [3 + 4/2] = [7/2]
Properties of Linear Transformations
- Kernel (Null Space): ker(T) = {v ∈ V | T(v) = 0}, the set of vectors that T maps to zero
- Range (Image): range(T) = {T(v) | v ∈ V}, the set of all possible outputs of T
- Rank: dim(range(T)), the dimension of the range of T
- Nullity: dim(ker(T)), the dimension of the kernel of T
Rank-Nullity Theorem
If T: V → W is a linear transformation and V is finite-dimensional, then:
dim(V) = rank(T) + nullity(T)
6. Eigenvalues and Eigenvectors
An eigenvector of a square matrix A is a non-zero vector v such that Av = λv for some scalar λ. The scalar λ is called an eigenvalue of A.
Finding Eigenvalues and Eigenvectors
To find the eigenvalues and eigenvectors of a matrix A:
- Find the eigenvalues by solving the characteristic equation: |A - λI| = 0
- For each eigenvalue λ, find the eigenvectors by solving the homogeneous system: (A - λI)v = 0
Example: Find the eigenvalues and eigenvectors of A = [3 1/1 3]
Step 1: Find the eigenvalues
|A - λI| = |[3-λ 1/1 3-λ]| = (3-λ)² - 1 = (3-λ)² - 1 = 9 - 6λ + λ² - 1 = λ² - 6λ + 8
(λ - 4)(λ - 2) = 0
λ = 4 or λ = 2
Step 2: Find the eigenvectors
For λ = 4:
(A - 4I)v = [3-4 1/1 3-4]v = [-1 1/1 -1]v = 0
This gives us the equation: -v₁ + v₂ = 0 → v₁ = v₂
So, v = t(1, 1) for any t ≠ 0. A normalized eigenvector is v = (1/√2, 1/√2)
For λ = 2:
(A - 2I)v = [3-2 1/1 3-2]v = [1 1/1 1]v = 0
This gives us the equation: v₁ + v₂ = 0 → v₁ = -v₂
So, v = t(1, -1) for any t ≠ 0. A normalized eigenvector is v = (1/√2, -1/√2)
Diagonalization
A square matrix A is diagonalizable if there exists an invertible matrix P and a diagonal matrix D such that A = PDP⁻¹. The columns of P are the eigenvectors of A, and the diagonal entries of D are the corresponding eigenvalues.
Example: Diagonalize the matrix A = [3 1/1 3] from the previous example.
From the previous example, we found:
λ₁ = 4 with eigenvector v₁ = (1, 1)
λ₂ = 2 with eigenvector v₂ = (1, -1)
P = [1 1/1 -1]
D = [4 0/0 2]
We can verify that A = PDP⁻¹:
P⁻¹ = (1/2) × [1 1/1 -1] = [1/2 1/2/1/2 -1/2]
PD = [1 1/1 -1] × [4 0/0 2] = [4 2/4 -2]
PDP⁻¹ = [4 2/4 -2] × [1/2 1/2/1/2 -1/2] = [3 1/1 3] = A
Applications of Eigenvalues and Eigenvectors
- Differential Equations: Eigenvalues help determine the stability of solutions
- Principal Component Analysis (PCA): Eigenvectors of the covariance matrix represent principal components
- Quantum Mechanics: Eigenvalues represent possible outcomes of measurements
- Vibration Analysis: Eigenvalues represent natural frequencies of vibration
- PageRank Algorithm: The webpage ranking is based on the principal eigenvector of the web graph
Properties of Eigenvalues and Eigenvectors
- The trace of a matrix equals the sum of its eigenvalues
- The determinant of a matrix equals the product of its eigenvalues
- If v is an eigenvector of A with eigenvalue λ, then v is also an eigenvector of A^n with eigenvalue λ^n
- Similar matrices have the same eigenvalues
7. Applications of Linear Algebra
Computer Graphics
Linear algebra is fundamental to computer graphics for operations such as:
- Transformations: Rotation, scaling, translation, and projection of objects
- Camera Positioning: Defining viewpoints and perspectives
- Lighting and Shading: Computing light reflections and shadows
- Ray Tracing: Calculating ray intersections with objects
Example: In 3D graphics, a point (x, y, z) can be rotated around the z-axis by angle θ using the matrix:
R_z(θ) = [cos(θ) -sin(θ) 0/sin(θ) cos(θ) 0/0 0 1]
Machine Learning and Data Science
Linear algebra provides the mathematical foundation for many machine learning algorithms:
- Linear Regression: Finding the best-fit line through data points
- Principal Component Analysis (PCA): Reducing data dimensionality
- Singular Value Decomposition (SVD): Used in recommendation systems
- Neural Networks: Matrix operations for forward and backward propagation
Example: In linear regression, we solve Xβ = y for β, where X is the data matrix, y is the target vector, and β contains the regression coefficients.
The solution is β = (X^T X)^(-1) X^T y
Quantum Mechanics
Linear algebra is essential in quantum mechanics:
- State Vectors: Quantum states are represented as vectors in Hilbert space
- Operators: Physical observables are represented as linear operators
- Eigenvalue Problems: Measurement outcomes are eigenvalues of operators
- Transformations: Quantum gates are represented as unitary matrices
Example: The Pauli matrices are important in quantum mechanics:
σ_x = [0 1/1 0], σ_y = [0 -i/i 0], σ_z = [1 0/0 -1]
Cryptography
Linear algebra plays a role in various cryptographic systems:
- Hill Cipher: Uses matrix multiplication for encryption and decryption
- Error-Correcting Codes: Based on linear transformations over finite fields
- Lattice-Based Cryptography: Uses high-dimensional lattices
Example: In the Hill cipher, a message is encrypted using a key matrix K:
Ciphertext = K × Plaintext (mod 26)
Decryption: Plaintext = K^(-1) × Ciphertext (mod 26)
Engineering and Physics
Linear algebra is widely used in engineering and physics:
- Structural Analysis: Solving systems of equations for forces and displacements
- Circuit Analysis: Applying Kirchhoff's laws leads to systems of linear equations
- Control Theory: State-space representations of dynamic systems
- Mechanics: Rigid body motion, stress and strain analysis
Example: In control theory, a linear time-invariant system can be represented as:
ẋ = Ax + Bu
y = Cx + Du
where x is the state vector, u is the input vector, y is the output vector, and A, B, C, D are matrices.
Economics and Finance
Linear algebra is applied in various economic and financial models:
- Input-Output Models: Analyzing relationships between economic sectors
- Portfolio Optimization: Minimizing risk for a given expected return
- Econometrics: Estimating parameters in economic models
Example: In Leontief's input-output model, the equation (I - A)x = d relates total production x to final demand d, where A is the matrix of technical coefficients.
