Singular Vector

Let \(u\) and \(v\) be a pair of singular vectors of \(A\) . i.e.

\begin{equation*} Av = \sigma u \end{equation*}

\(u^T A = \sigma v^T\)

Then,

• Scaling doesn't matter. i.e. \(ku\) and \(kv\) are also singular vectors
• Both vectors have same magnitude \(||u|| = ||v||\)

• \(v\) is (right) singular vector of \(A\) iff \(v\) is eigenvector of \(A^TA\)

  and \(u\) is (left) singular vector of \(A\) iff \(u\) is eigenvector of \(AA^T\)

• Singular vectors are orthogonal

\begin{align*} u_i \cdot u_j = 0 \textrm{ for } i \neq j \\ v_i \cdot v_j = 0 \textrm{ for } i \neq j \end{align*}

Caveat: Singular vectors corresponding to distinct singular values are orthogonal. But, if the singular values are non distinct (equal) then the singular vectors need not be othogonal. However, given a set of nonorthonal singular vectors, we can construct an orthogonal basis for that set. So, even for a set of singular values where not all values are distinct, we can create a corresponding set of orthogonal singular vectors.

Proof: Since \(v\) is eigenvector of \(A^TA\) which is a real symmetric matrix, its eigenvectors are orthogonal. Proof for othogonality of eigenvector for symmetric matrices follows from definition of self adjoint operator.

• Span of top-\(k\) singular vectors is the best fit \(k\) -dimensional subspace for the rows of \(A\)
• The partial decomposition obtained by using only the top \(k\) singular vectors is the best rank-\(k\) approximation to \(A\)

\begin{equation*} v_1 = argmax_{||v|| = 1} ||Av|| \end{equation*} \begin{equation*} v_2 = argmax_{||v|| = 1, v \cdot v1 = 0} ||Av|| \end{equation*}