Singular Value Decomposition as a Noise Reduction Scheme

We will see that in the presence of noise one has to expect that all zero eigenvalues $\sigma_i$ (of the noise-free case) will be changed into non-zero numbers, small but non-zero! So the simple method of the above section breaks down and the result of the computation will be in almost all cases $n'=n$, no matter how large $n$ is. To deal with this problem we introduce two new matrices

$$\quad C = \left(c_1,c_2,\ldots,c_n\right) \ \ ,$$ (36)

$$\Sigma = \mbox{diag}\left(\sigma_1,\sigma_2,\ldots,\sigma_n\right) \quad.$$ (37)

The elements $\sigma_i$ of $\Sigma$ are called the singular values of $X$. Using these definitions, the definition of $\Xi$ and eq. (34) we get the important relation

$$\quad \left(XC\right)^T\left(XC\right) = \Sigma^2 \quad.$$ (38)

Recall that $\{c_i\}_{i=1}^n$ is by definition an orthonormal basis of ${\bf R}^n$. So the $(N,n)$-matrix $\tilde X\equiv XC$ can be interpreted as the projection of the trajectory matrix onto this new basis. Then $\tilde X$ is the trajectory matrix expressed in the basis $c_i$, and $\tilde X^T\tilde X$ in the above equation is the covariance matrix in the same new basis. If one considers the general form of a covariance matrix, $\Xi = \frac{1}{N}\sum_{i=1}^{N}x_ix_i^T$, then it is clear that $\Xi$ measures the correlation of all the vectors $x_i$, averaged over the entire trajectory. Thus the fact that the product $\tilde X^T\tilde X$ gives a diagonal matrix (eq. (40)) shows that in the basis $\{c_i\}$ the vectors of the trajectory ($x_i^TC$) are uncorrelated ¹³. What is more, we can see from eq. (40) that $\sigma_i^2$ is proportional to the extent to which the trajectory varies in the $c_i$-direction. One can think of the trajectory as exploring on the average an ellipsoid in ${\bf R}^n$, the directions and lengths of the axes of which are given by the $c_i$ and the $\sigma_i^2$, respectively. This picture is very valuable for the discussion of the effect of noise on the trajectory: If there is, for example, a floor of white noise (i.e. we have $v_i=\overline {v}_i+\xi$, where $\overline {v}_i$ is the deterministic contribution to the observable and $\xi$ stands for the noise) then we get for the covariance matrix an expression like

$$\quad \Xi = \overline \Xi + \left<\xi^2\right>\cdot I_n \quad,$$ (39)

where $\left<.\right>$ denotes the time average, $I_n$ is the $(n,n)$-identity matrix and $\overline \Xi$ is the deterministic contribution we have discussed until now. Diagonalization of $\Xi$ for this ``white noise case'' yields the eigenvalues

$$\quad \sigma_i^2 = \overline {\sigma}_i^2+\left<\xi^2\right>, \ \ i=1,2,\ldots,n \quad.$$ (40)

Notice that in the presence of noise all eigenvalues are non-zero (since $\left<\xi^2\right>>0$), even for those directions $c_i$ which are not explored by the deterministic movement of the system. Thus the influence of noise on our analysis is to make us think that the (deterministic) trajectory explores all directions of ${\bf R}^n$, instead of only $n'$! The solution of this problem is found by considering the singular value decomposition of $X$:

$$\quad X = S\Sigma C^T \quad.$$ (41)

($S$ is the $(N,n)$-matrix formed of the eigenvectors of $\Theta$.) We will see below that this decomposition is useful, because $\Sigma$ appears as one of the factors constituting $X$. We want to find those entries of $\Sigma$ which are obviously non-zero only due to the effects of noise. One possibility to do this is to measure, in addition to the true time series, a time series which consists only of the noise and then to compute the corresponding mean square projections onto the $c_i$-directions (which we found for the true time series). These quantities are to be compared with the respective $\sigma_i$ (of the true time series) and if both are found to be of the same order of magnitude then we know that this particular $\sigma_i$ is only non-zero due to the noise; in other words, the corresponding direction $c_i$ is noise-dominated. A straightforward strategy (which uses the special form of eq. (43)) to get rid of this most significant influence of noise is to set those selected entries equal to zero¹⁴. Thus we arrive at the following corrected equation for the approximate deterministic part $\overline{X}'$ of the trajectory matrix:

$$\overline{X}' = S \overline{\Sigma}' C^T$$ (42)

$$\overline{\Sigma}' = \mbox{diag} \left(\overline{\sigma}_1',\overline{\sigma}_2', \ldots,\overline{\sigma}_n'\right)$$ (43)

$$\overline{\sigma}_i' = \left\{ \begin{array}{c@{\quad {\rm if}\quad}c} \sigma_i & \sigma_i>{\rm noise} \\ 0 & \sigma_i\approx{\rm noise} \end{array} \right.$$ (44)

(Notice that the $\overline{\sigma}_i'$ are not the deterministic component $\overline{\sigma_i}$ of the $\sigma_i$; but, by construction, $\overline{\sigma_i}'$ and $\overline{\sigma_i}$ should be approximately the same.), which can be simplified to get a representation of the corrected trajectory matrix $\overline{X}'$ which is most easy to work with:

$$\quad \overline{X}' = \sum_{ { \{i\mid i\in\{1,\ldots,n\}... ...{\rm noise \ floor} \} } } \left( Xc_i \right) c_i^T \quad.$$ (45)

This equation is nice to work with since the $c_i$ and $\sigma_i$ are easy to compute by diagonalization of the covariance matrix. The sum in eq. (47) will run over $d$ summands, and of course we expect $d\approx n'$. We relabel the $\{c_i\}_{i=1}^{n}$ such that the first $d$ of them correspond to those eigenvalues $\sigma_i$ which are not noise-dominated. Then we know, after having eliminated the effect of noise as far as possible, that the trajectory is confined to a $d$-dimensional subspace of ${\bf R}^n$ which is spanned by $\{c_i\}_{i=1}^{d}$. So we can take ${\bf R}^d$ as the embedding space instead of ${\bf R}^n$. Finally we get the following vectors on the trajectory in ${\bf R}^d$:

$$\quad \left(x_i^T c_1,x_i^T c_2, \ldots, x_i^T c_d \right)^T, \ \ i=0,1,2,\ldots,N-1 \quad,$$ (46)

and it is these vectors which we can now plot in $d$ dimensions (taking e.g. two- or three-dimensional cross-sections) to get the geometric picture of the attractor which we have been aiming at.

Footnotes

... uncorrelated¹³

This result justifies the choice $J=1$ in section 3.1: since in a well-chosen basis ($\{c_i\}$) the vectors which form the trajectory are uncorrelated, the lag-time does not influence our results and we can choose it at our convenience.

... zero¹⁴

One should be aware of the fact that this strategy does not remove all effects of the noise on the trajectory: The noise contribution to those $\overline{\sigma}_i$ which are non-zero according to eq. (46) remains unchanged. It is not totally clear to me why Broomhead and King in [7] do not propose to subtract the noise floor from all eigenvalues $\sigma_i$ by replacing $\Sigma$ in eq. (43) with $\left(\Sigma-diag \left( \left<\xi_1^2\right>,\ldots,\left<\xi_n^2\right> \right) \right)$ (or with $\left(\Sigma-\left<\xi^2\right>\cdot I_n\right)$ in the case of white noise); this would (on the average!) remove the noise floor from all eigenvalues. Maybe the reason for not doing this are:
$\bullet$ we would not get eq. (47) in this nice form which is easy to handle numerically;
$\bullet$ the effect of noise on the first $n'$ components is assumed to be not very large;
$\bullet$ for the calculation of $n'$ the numerical values of the $\sigma_i$ do not matter anyhow, apart from being above or in the noise floor.