用四个子空间给「求逆」分类

设 $A$ 是 $m\times n$ 矩阵，秩为 $r$。整个讨论围绕那张熟悉的图：

• 行空间 $\subseteq\mathbb R^n$，维数 $r$；
• 零空间 $\subseteq\mathbb R^n$，维数 $n-r$；
• 列空间 $\subseteq\mathbb R^m$，维数 $r$；
• 左零空间（$A^{\mathsf T}$ 的零空间）$\subseteq\mathbb R^m$，维数 $m-r$。

逆之所以有时不存在，是因为零空间里的向量被 $A$ 送到零，没有任何映射能把它们「复活」。所以能否求逆，取决于零空间有多大。

时间轴

四种情形

情形一：满秩方阵

$r=m=n$，$A$ 可逆，左右两侧都得到单位矩阵：

$$A^{-1}A=AA^{-1}=I.$$

情形二：满列秩，左逆

列线性无关，$r=n$，$A$ 的零空间只有零向量。此时 $A^{\mathsf T}A$ 是 $n\times n$ 对称矩阵，并且可逆。

定义

$$A_{\text{left}}=(A^{\mathsf T}A)^{-1}A^{\mathsf T}.$$

它是 $n\times m$ 矩阵，并且

$$A_{\text{left}}A=(A^{\mathsf T}A)^{-1}A^{\mathsf T}A=I_n.$$

这就是最小二乘里反复出现的组合。列无关时，正规方程的系数矩阵可逆，最小二乘有唯一解。

注意维度：$A_{\text{left}}A$ 是 $n\times n$ 单位阵；但反过来的 $AA_{\text{left}}$ 是 $m\times m$，不可能是单位阵（因为 $n<m$）。

情形三：满行秩，右逆

行线性无关，$r=m$，$A^{\mathsf T}$ 的零空间只有零向量。此时 $AA^{\mathsf T}$ 可逆。

定义

$$A_{\text{right}}=A^{\mathsf T}(AA^{\mathsf T})^{-1}.$$

它是 $n\times m$ 矩阵，并且

$$AA_{\text{right}}=I_m.$$

因为 $r=m<n$，零空间维数为 $n-m>0$，所以只要 $Ax=b$ 有解，就有无穷多个解。

另一侧的乘积是投影

左逆放在右边，得不到单位阵，但得到的是一个熟悉的矩阵：

$$AA_{\text{left}}=A(A^{\mathsf T}A)^{-1}A^{\mathsf T}=P_{\text{col}},$$

这是到列空间的投影矩阵。它尽量充当单位阵：在列空间上等于恒等，在左零空间上等于零。

同理，

$$A_{\text{right}}A=A^{\mathsf T}(AA^{\mathsf T})^{-1}A=P_{\text{row}},$$

这是到行空间的投影矩阵。

这解释了一个常见疑问：为什么左逆只能「左乘」才得到单位阵？因为另一侧的空间维度不够，投影是能做到的最好结果。

一般情形：伪逆

当 $r<\min(m,n)$ 时，两侧都有零空间，真正的逆不存在。但 $A$ 在行空间与列空间之间是一个完美的双射。

为什么是双射

设 $x,y$ 都在行空间，且 $Ax=Ay$。则

$$A(x-y)=0,$$

所以 $x-y$ 在零空间中。可是 $x-y$ 又是两个行空间向量的差，仍在行空间中。零空间与行空间正交，交集只有零向量，因此

$$x-y=0.$$

这就证明了：行空间中的不同向量，被 $A$ 送到列空间中的不同向量。

同时，行空间与列空间维数都是 $r$，所以这是一个可逆映射。它的逆就是伪逆 $A^+$。

用 SVD 构造

设

$$A=U\Sigma V^{\mathsf T}.$$

$\Sigma$ 是 $m\times n$ 对角矩阵，前 $r$ 个对角元 $\sigma_1,\dots,\sigma_r>0$，其余为零。

把 $\Sigma$ 的每个非零对角元取倒数，并转置成 $n\times m$：

$$\Sigma^+=\operatorname{diag}\left(\frac{1}{\sigma_1},\dots,\frac{1}{\sigma_r},0,\dots,0\right).$$

则

$$A^+=V\Sigma^+U^{\mathsf T}.$$

两个乘积都是投影

$$AA^+=U\Sigma\Sigma^+U^{\mathsf T}=P_{\text{col}},$$ $$A^+A=V\Sigma^+\Sigma V^{\mathsf T}=P_{\text{row}}.$$

$\Sigma\Sigma^+$ 是 $m\times m$，前 $r$ 个对角元为 $1$，其余为 $0$；$\Sigma^+\Sigma$ 是 $n\times n$，结构相同。它们分别把向量投影到列空间与行空间。

为什么这是「最好的逆」

伪逆在所有可能的选择中最小：零空间方向上没有信息，任何取值都无法被检验，取零是最自然的。多填任何数都不会改变 $AA^+$ 与 $A^+A$ 的结果，只会让伪逆变大。

补充：伪逆满足四个 Penrose 条件：$AA^+A=A$、$A^+AA^+=A^+$、$(AA^+)^{\mathsf T}=AA^+$、$(A^+A)^{\mathsf T}=A^+A$。前两条说明它在可行范围内尽量充当逆，后两条说明两个乘积都是正交投影。

统计学为什么需要伪逆

最小二乘的核心是解正规方程

$$A^{\mathsf T}A\hat x=A^{\mathsf T}b.$$

只要 $A$ 满列秩，$A^{\mathsf T}A$ 可逆，一切顺利。但现实中经常出现重复测量或线性相关的特征，此时 $A^{\mathsf T}A$ 奇异，正规方程有无穷多解。

伪逆给出的

$$\hat x=A^+b$$

是其中范数最小的解。这正是统计软件在共线性情形下采用的默认处理方式。

常见误区

1. 以为长方矩阵也有双侧逆。维度不匹配，$AA_{\text{left}}$ 是 $m\times m$，不可能等于 $I_n$。
2. 把左逆与右逆的公式记反。左逆用 $A^{\mathsf T}A$，右逆用 $AA^{\mathsf T}$。
3. 以为伪逆等于「先转置再求逆」。只有当 $A$ 的奇异值全为 $1$ 时才碰巧如此。
4. 忽略零空间自由度。伪逆在零空间方向上不做任何「恢复」，这是信息论上无法做到的。

本讲结论

• 四种情形由 $m,n,r$ 的关系决定。
• 满列秩：左逆 $(A^{\mathsf T}A)^{-1}A^{\mathsf T}$，另一侧乘积是列空间投影。
• 满行秩：右逆 $A^{\mathsf T}(AA^{\mathsf T})^{-1}$，另一侧乘积是行空间投影。
• 一般情形：$A^+=V\Sigma^+U^{\mathsf T}$，在行空间与列空间之间是真正的逆。
• $AA^+$ 与 $A^+A$ 分别是到列空间、行空间的正交投影。
• 伪逆给出的最小二乘解是范数最小的解，适用于 $A^{\mathsf T}A$ 奇异的情形。

理解检验

1. 为什么长方矩阵不可能有双侧逆？

参考答案：设 $A$ 是 $m\times n$。若存在左逆 $B$（$n\times m$）使 $BA=I_n$，则 $\operatorname{rank}A\ge\operatorname{rank}(BA)=n$，结合 $\operatorname{rank}A\le n$ 得 $\operatorname{rank}A=n$。若同时存在右逆 $C$ 使 $AC=I_m$，则 $\operatorname{rank}A\ge m$，得 $\operatorname{rank}A=m$。两者同时成立只能有 $m=n$。更直接的代数证明是：由 $BA=I_n$、$AC=I_m$，

$$B=B(AC)=(BA)C=C,$$

于是 $B=C$，同一矩阵既是左逆又是右逆，$A$ 为方阵且可逆。所以当 $m\ne n$ 时，最多只有单侧逆。

2. 左逆与右逆的公式中，为什么一个用 $A^{\mathsf T}A$，另一个用 $AA^{\mathsf T}$？

参考答案：左逆要在左边得到 $I_n$，需要 $n\times n$ 的可逆矩阵，而 $A^{\mathsf T}A$ 正是 $n\times n$；它在列无关（$r=n$）时可逆，因为 $A^{\mathsf T}Ax=0\Rightarrow x^{\mathsf T}A^{\mathsf T}Ax=\lVert Ax\rVert^2=0\Rightarrow Ax=0\Rightarrow x=0$。于是 $(A^{\mathsf T}A)^{-1}A^{\mathsf T}$ 左乘 $A$ 得 $I_n$。右逆要在右边得到 $I_m$，需要 $m\times m$ 矩阵，$AA^{\mathsf T}$ 正是 $m\times m$，在行无关（$r=m$）时可逆，同理 $A^{\mathsf T}(AA^{\mathsf T})^{-1}$ 右乘 $A$ 得 $I_m$。维度决定了各自该用哪个乘积。

3. 证明 $A$ 在行空间上是单射，并指出证明中用到了哪两个子空间的性质。

参考答案：设 $x,y$ 都在行空间，且 $Ax=Ay$。则 $A(x-y)=0$，所以 $x-y\in N(A)$。另一方面，行空间对减法封闭，$x-y$ 仍在行空间中，即 $x-y\in C(A^{\mathsf T})$。用到的第一个性质是零空间与行空间正交（$N(A)\perp C(A^{\mathsf T})$），第二个是行空间是子空间（对线性组合封闭）。正交子空间的交集只有零向量，所以 $x-y=0$，即 $x=y$。行空间与列空间维数同为 $r$，单射加上维数相等就得到双射，其逆正是伪逆。

4. 为什么 $AA^+$ 是投影而不是单位阵？它的零空间是什么？

参考答案：$AA^+=U\Sigma\Sigma^+U^{\mathsf T}$，其中 $\Sigma\Sigma^+$ 是 $m\times m$ 对角阵，前 $r$ 个对角元为 $1$、其余 $m-r$ 个为 $0$。它在列空间（由 $u_1,\dots,u_r$ 张成）上是恒等映射，在左零空间（由 $u_{r+1},\dots,u_m$ 张成）上是零映射。只有当 $r=m$ 时它才等于单位阵；一般情形 $m-r>0$，左零空间方向被抹掉，所以只能是投影。它的零空间正是左零空间 $N(A^{\mathsf T})$，维数 $m-r$。

5. 当 $A^{\mathsf T}A$ 奇异时，最小二乘为什么需要伪逆？伪逆选出的解有什么特殊性？

参考答案：$A^{\mathsf T}A$ 奇异意味着 $A$ 的列相关（$r<n$），正规方程 $A^{\mathsf T}A\hat x=A^{\mathsf T}b$ 有无穷多个解——它们相差零空间中的任意向量，而最小二乘只约束「残差最小」，无法区分这些解。伪逆给出

$$\hat x=A^+b=V\Sigma^+U^{\mathsf T}b,$$

它把 $b$ 在左零空间上的分量投影掉，再把列空间分量按 $1/\sigma_i$ 映回行空间，因此在零空间方向上取零。于是 $\hat x$ 是所有最小二乘解中范数最小的那一个（最小范数最小二乘解），这也是统计软件在共线性下采用的默认解。

与前后课程的关系

前几讲分别处理了特征值、正定、相似与 SVD；这一讲把 SVD 用于非方阵求逆，把四个子空间、秩与最小二乘串成一个完整分类。下一讲是全课程总结，用旧考题回顾这些主线。

Classifying inverses with the four subspaces

Let $A$ be $m\times n$ of rank $r$. Everything turns on the familiar picture:

• row space $\subseteq\mathbb R^n$, dimension $r$;
• null space $\subseteq\mathbb R^n$, dimension $n-r$;
• column space $\subseteq\mathbb R^m$, dimension $r$;
• left null space (null space of $A^{\mathsf T}$) $\subseteq\mathbb R^m$, dimension $m-r$.

Inverses fail exactly because vectors in the null space are sent to zero, and no map can bring them back. How much can be inverted depends on how large the null spaces are.

Timeline

Four cases

Case one: full-rank square

$r=m=n$, so $A$ is invertible and both sides give the identity:

$$A^{-1}A=AA^{-1}=I.$$

Case two: full column rank, left inverse

The columns are independent, $r=n$, and the null space of $A$ is trivial. Then $A^{\mathsf T}A$ is $n\times n$, symmetric, and invertible.

Define

$$A_{\text{left}}=(A^{\mathsf T}A)^{-1}A^{\mathsf T}.$$

It is $n\times m$, and

$$A_{\text{left}}A=(A^{\mathsf T}A)^{-1}A^{\mathsf T}A=I_n.$$

This is the combination that appears throughout least squares. With independent columns, the normal equations have an invertible coefficient matrix and a unique least-squares solution.

Watch the dimensions: $A_{\text{left}}A$ is $n\times n$. The other order, $AA_{\text{left}}$, is $m\times m$ and cannot be the identity because $n<m$.

Case three: full row rank, right inverse

The rows are independent, $r=m$, and the null space of $A^{\mathsf T}$ is trivial. Then $AA^{\mathsf T}$ is invertible.

Define

$$A_{\text{right}}=A^{\mathsf T}(AA^{\mathsf T})^{-1}.$$

It is $n\times m$, and

$$AA_{\text{right}}=I_m.$$

Since $r=m<n$, the null space has dimension $n-m>0$, so whenever $Ax=b$ is solvable it has infinitely many solutions.

The other product is a projection

Put the left inverse on the right and you do not get the identity, but something familiar:

$$AA_{\text{left}}=A(A^{\mathsf T}A)^{-1}A^{\mathsf T}=P_{\text{col}},$$

the projection onto the column space. It does the best a matrix can do: identity on the column space, zero on the left null space.

Similarly,

$$A_{\text{right}}A=A^{\mathsf T}(AA^{\mathsf T})^{-1}A=P_{\text{row}},$$

the projection onto the row space.

This explains why a left inverse only works from the left: the other side does not have the right dimension, and a projection is the best available result.

The general case: the pseudoinverse

When $r<\min(m,n)$, both null spaces are nonzero and no true inverse exists. But $A$ is a perfect bijection between the row space and the column space.

Why it is a bijection

Let $x,y$ lie in the row space and suppose $Ax=Ay$. Then

$$A(x-y)=0,$$

so $x-y$ lies in the null space. But $x-y$ is also a difference of two row-space vectors, hence lies in the row space. The null space and the row space are orthogonal complements, so their intersection is only the zero vector. Therefore

$$x-y=0.$$

So distinct row-space vectors map to distinct column-space vectors. Both spaces have dimension $r$, so the map is invertible. Its inverse is the pseudoinverse $A^+$.

Constructing it from SVD

Let

$$A=U\Sigma V^{\mathsf T}.$$

$\Sigma$ is $m\times n$ with $\sigma_1,\dots,\sigma_r>0$ on the diagonal and zeros elsewhere.

Reciprocate each nonzero diagonal entry and transpose to shape $n\times m$:

$$\Sigma^+=\operatorname{diag}\left(\frac{1}{\sigma_1},\dots,\frac{1}{\sigma_r},0,\dots,0\right).$$

Then

$$A^+=V\Sigma^+U^{\mathsf T}.$$

Both products are projections

$$AA^+=U\Sigma\Sigma^+U^{\mathsf T}=P_{\text{col}},$$ $$A^+A=V\Sigma^+\Sigma V^{\mathsf T}=P_{\text{row}}.$$

$\Sigma\Sigma^+$ is $m\times m$ with $r$ ones and zeros; $\Sigma^+\Sigma$ is $n\times n$ with the same structure. They project onto the column and row spaces respectively.

Why this is the best inverse

The pseudoinverse is the smallest possible choice. Along the null spaces there is no information, so no value can be tested; setting it to zero is natural. Adding anything there would not change $AA^+$ or $A^+A$, only make the pseudoinverse larger.

Background: the pseudoinverse satisfies the four Penrose conditions: $AA^+A=A$, $A^+AA^+=A^+$, $(AA^+)^{\mathsf T}=AA^+$, and $(A^+A)^{\mathsf T}=A^+A$. The first two make it an inverse where possible; the last two say both products are orthogonal projections.

Why statistics needs it

Least squares centers on the normal equations

$$A^{\mathsf T}A\hat x=A^{\mathsf T}b.$$

When $A$ has full column rank, $A^{\mathsf T}A$ is invertible and everything is fine. But repeated measurements or linearly dependent features make $A^{\mathsf T}A$ singular, and the normal equations have infinitely many solutions.

The pseudoinverse gives

$$\hat x=A^+b,$$

the solution of smallest norm. That is exactly what statistical software does by default under collinearity.

Common mistakes

1. Expecting a rectangular matrix to have a two-sided inverse. The dimensions forbid it: $AA_{\text{left}}$ is $m\times m$, not $I_n$.
2. Swapping the left and right inverse formulas. Left uses $A^{\mathsf T}A$; right uses $AA^{\mathsf T}$.
3. Thinking the pseudoinverse is just "transpose then invert". That happens only when all singular values equal one.
4. Ignoring null-space freedom. The pseudoinverse cannot recover anything along the null space; no inverse could.

Takeaways

• The four cases are determined by the relation among $m$, $n$, and $r$.
• Full column rank: left inverse $(A^{\mathsf T}A)^{-1}A^{\mathsf T}$; the other product is the column-space projection.
• Full row rank: right inverse $A^{\mathsf T}(AA^{\mathsf T})^{-1}$; the other product is the row-space projection.
• General case: $A^+=V\Sigma^+U^{\mathsf T}$ is a genuine inverse between row and column spaces.
• $AA^+$ and $A^+A$ are orthogonal projections onto the column and row spaces.
• The pseudoinverse gives the minimum-norm least-squares solution when $A^{\mathsf T}A$ is singular.

Check your understanding

1. Why can a rectangular matrix never have a two-sided inverse?

Reference answer: Let $A$ be $m\times n$. If a left inverse $B$ ($n\times m$) satisfies $BA=I_n$, then $\operatorname{rank}A\ge\operatorname{rank}(BA)=n$, and since $\operatorname{rank}A\le n$ we get $\operatorname{rank}A=n$. If at the same time a right inverse $C$ satisfies $AC=I_m$, then $\operatorname{rank}A\ge m$, so $\operatorname{rank}A=m$. Both can hold only if $m=n$. A direct algebraic proof: from $BA=I_n$ and $AC=I_m$,

$$B=B(AC)=(BA)C=C,$$

so $B=C$ is simultaneously a left and a right inverse, making $A$ square and invertible. Hence for $m\ne n$ at most one side can exist.

2. Why does the left inverse use $A^{\mathsf T}A$ while the right inverse uses $AA^{\mathsf T}$?

Reference answer: A left inverse must produce $I_n$ on the left, which requires an invertible $n\times n$ matrix — and $A^{\mathsf T}A$ is exactly $n\times n$. It is invertible when the columns are independent ($r=n$), because $A^{\mathsf T}Ax=0\Rightarrow x^{\mathsf T}A^{\mathsf T}Ax=\lVert Ax\rVert^2=0\Rightarrow Ax=0\Rightarrow x=0$. Hence $(A^{\mathsf T}A)^{-1}A^{\mathsf T}$ times $A$ on the left gives $I_n$. A right inverse must produce $I_m$, requiring an $m\times m$ matrix; $AA^{\mathsf T}$ is exactly that and is invertible when the rows are independent ($r=m$), so $A^{\mathsf T}(AA^{\mathsf T})^{-1}$ gives $I_m$ on the right. The dimensions dictate which product to use.

3. Prove that $A$ is injective on the row space, and identify which two subspace facts the proof uses.

Reference answer: Take $x,y$ in the row space with $Ax=Ay$. Then $A(x-y)=0$, so $x-y\in N(A)$. On the other hand the row space is closed under subtraction, so $x-y\in C(A^{\mathsf T})$. The first fact used is $N(A)\perp C(A^{\mathsf T})$: the null space is orthogonal to the row space. The second is that the row space is a subspace, hence closed under linear combinations. Two orthogonal subspaces meet only at zero, so $x-y=0$, i.e. $x=y$. Since the row and column spaces both have dimension $r$, injectivity plus equal dimensions gives a bijection, and its inverse is the pseudoinverse.

4. Why is $AA^+$ a projection rather than the identity? What is its null space?

Reference answer: $AA^+=U\Sigma\Sigma^+U^{\mathsf T}$, where $\Sigma\Sigma^+$ is the $m\times m$ diagonal matrix with the first $r$ diagonal entries $1$ and the remaining $m-r$ entries $0$. It acts as the identity on the column space (spanned by $u_1,\dots,u_r$) and as zero on the left null space (spanned by $u_{r+1},\dots,u_m$). It equals the identity only when $r=m$; in general $m-r>0$ and the left-null-space directions are killed, so it can only be a projection. Its null space is precisely the left null space $N(A^{\mathsf T})$, of dimension $m-r$.

5. When $A^{\mathsf T}A$ is singular, why does least squares need the pseudoinverse, and what is special about the solution it selects?

Reference answer: $A^{\mathsf T}A$ singular means the columns of $A$ are dependent ($r<n$), so the normal equations $A^{\mathsf T}A\hat x=A^{\mathsf T}b$ have infinitely many solutions — they differ by arbitrary null-space vectors, and "least residual" alone cannot distinguish them. The pseudoinverse gives

$$\hat x=A^+b=V\Sigma^+U^{\mathsf T}b,$$

which projects away the component of $b$ in the left null space and maps the column-space component back through $1/\sigma_i$; it assigns zero on the null space. Among all least-squares solutions, $\hat x$ is therefore the one of minimum norm (the minimum-norm least-squares solution), which is the default that statistical software uses under collinearity.

How this connects to the course

The preceding lectures covered eigenvalues, positive definiteness, similarity, and SVD. This one applies SVD to inverting non-square matrices and ties the four subspaces, rank, and least squares into one classification. The next lecture is the final review, revisiting these threads through old exam problems.