Time-frequency
Fourier transform allows one to known the frequency content of a signal, but we then “loose” the time information. For example, the spectrum of two piano notes played simutanously is given on Fig. while the spectrum of the same two notes played successively is given on Fig. If we can recognize what are the frequency of the two played notes, we cannot distinguished from the spectrum when the notes are played. The question is then to construct a representation given the evolution of the frequency content with the time.
The simple idea of time-frequency analysis si to perform a local spectral analysis of the signal thanks to a sliding window. We will then compute the correlation of the signal with time-frequency atoms.
Let $g$ be a smooth window “well localized” around $t=0$. This window is called the analysis window
Short Time Fourier Transform Let $x\in L^1(\mathbb{R})\cap L^2(\mathbb{R})$. The short time Fourier transform (STFT) of $x$ with the analysis window $g\in L^1(\mathbb{R})\cap L^2(\mathbb{R})$ is the function $X$ of two variables given by $$ \begin{aligned} X\ :\ \mathbb{R}\times\mathbb{R} & \rightarrow \mathbb{C} \\ (b,\nu) & \mapsto X(b,\nu) = \left\langle x; g(t-b)e^{2i\pi\nu t}\right\rangle = \int_{\mathbb{R}} x(t)\boldsymbol{g}(t-b)e^{-2i\pi\nu t}\mathrm{d}t \end{aligned} $$
In the following, we will denote by $g_{b,\nu}(t)= g(t-b)e^{2i\pi\nu t}$ the time-freuqency atoms of the time-frequency decomposition. For a fixed $(b,\nu)$ the decomposition is localized around the time-frequency point $(b,\nu)$ thanks to the time localization of $g$ and the frequency localization of $e^{2i\pi\nu t}$. We can now define the spectrogram of a signal
Spectrogram Let $x\in L^1(\mathbb{R})\cap L^2(\mathbb{R})$ and $X(b,\nu)$ is STFT with the analysis window $g$. The spectrogram is given by the positive function of two variabels $(b,\nu)\mapsto \left| X(b,\nu)\right|^2$.
Because of the Heisenberg uncertainty (see this exercise), a time-frequency atoms cannot be well localized both in time and frequency.
The STFT preserved the energy and can be inverted
Energy preservation Let $x\in L^1(\mathbb{R})\cap L^2(\mathbb{R})$ a signak and $X(b,\nu)$ its STFT with the analysis window $g$. Then $$ \int_{\mathbb{R}} x^2(t)\mathrm{d}t =\frac{1}{ \Vert g\Vert ^2}\int_{\mathbb{R}}\int_{\mathbb{R}}|X(b,\nu)|^2 db\mathrm{d}\nu. $$
Inversion Let $x\in L^1(\mathbb{R})\cap L^2(\mathbb{R})$ a signak and $X(b,\nu)$ its STFT with the analysis window $g$. Then $$ \forall t\in \mathbb{R}\quad x(t)=\frac{1}{|g|^2}\int_{\mathbb{R}}\int_{\mathbb{R}}X(b,\nu)g(t-b)e^{2i\pi\nu t} db\mathrm{d} \nu $$
The short time Fourier transform is characterized by the time-frequency atoms $g_{b,f}$, which are themself defined by the window $g$. In practice, we have to well chose this window in order to define the STFT. Several choices are possible. The most simple window is the rectangular function:
$$ g(t)= \begin{cases} 1 & \text{ if }t\in [-\frac{T_0}{2};\frac{T_0}{2}]\\ 0& \text{ otherwise }. \end{cases} $$ This function can be well localized in time, depending on the choice $T_0$. However, its Fourier transform $$ \widehat{g}(f)=\text{sinc}(T_0\nu) $$ is badly localized in frequency.
The “optimal” choice for the window depends of its time-frequency spread. On the Figure, one can see that whatever the considered atoms, its Heisenberg box is the same, given by the resolution of $g$. A nice STFT should be well localized in time and in frequency. Ideally, we should choose the window with the smaller Heisenberg box, that is, the gaussian window. However, in practice, other consideration can be taken into account when the discretization come into play.
The STFT gives us a time-frequency analysis, but is limited by the Heisenberg uncertainty principle. For some signals, the most informative part can be in the singularity and some irregular structures, as for images the eyes being most sensitive to edges or some medical signal such as electro-encephalogram in order to detect epileptics zone. For such application, wavelets transform has been introduced in the 80’s.