Home Good examples Faded examples Noisy examples

Audio context encoder

This website accompanies the work published on IEEE TASLP.
The code used can be found here.
We studied the ability of deep neural networks (DNNs) to restore missing audio content based on its context. We focused on gaps in the range of tens of milliseconds, a condition which has not received much attention yet. The proposed DNN structure was trained on audio signals containing music and musical instruments, separately, with 64-ms long gaps. The input to the DNN was the context, i.e., the signal surrounding the gap, transformed into time-frequency (TF) coefficients. Two networks were analyzed, a DNN with complex-valued TF coefficient output and another one producing magnitude TF coefficient output, both based on the same network architecture. We found significant differences in the inpainting results between the two DNNs. In particular, we discuss the observation that the complex-valued DNN fails to produce reliable results outside the low frequency range. We demonstrated a generally good usability of the proposed DNN structure for generating complex audio signals like music.
Encoder architecture:

Good spectrogram

Decoder architecture:

Good spectrogram

Now we introduce sound examples generated with the network. They are divided into three classes according to our perception of them. These are: good, noisy and faded. The faded class presents samples where we hear as if the algorithm did a fade-in and fade-out on the gap.

Good examples

Trumpet

On this example, we find a string signal with a few harmonics. Both networks achieve reconstructions which are difficult to detect.
  • Left: Ground truth
  • Center: Magnitude (28.7 dB SNRms)
  • Right: Complex (20.9 dB SNRms)

Trumpet

This example features a synthesized string signal, with several harmonics and a constant vibrato. Both networks achieve reconstructions that are difficult to detect.
  • Left: Ground truth
  • Center: Magnitude (29.2 dB SNRms)
  • Right: Complex (27.3 dB SNRms)

Trumpet

On this example, we find a trumpet signal with a lot of information on high frequencies. Neither of the networks achieve very high SNRms values, nevertheless, it is quite hard to hear any artifacts.
  • Left: Ground truth
  • Center: Magnitude (11.4 dB SNRms)
  • Right: Complex (4.1 dB SNRms)

Trumpet

On this example, we find a synthetic signal with a lot of modulations. Both networks represent this modulations in some way. For the magnitude network, even on very high frequencies the modulations are still inpainted. For the complex network, above 5Khz there is little information.
  • Left: Ground truth
  • Center: Magnitude (7.8 dB SNRms)
  • Right: Complex (5.1 dB SNRms)

Faded examples

Trumpet

On this example, we find a pulsated string signal. Both networks achieve very high SNRms values, but they both present a faded artifact. This is quite easy to hear for the complex network and it is not as present for the magnitude network.
  • Left: Ground truth
  • Center: Magnitude (35.2 dB SNRms)
  • Right: Complex (29.6 dB SNRms)

Trumpet

On this example, we find a string signal. Again we find a faded artifact that is clearer on the complex network than the magnitude network. In this case, the SNRms are quite low.
  • Left: Ground truth
  • Center: Magnitude (8.7 dB SNRms)
  • Right: Complex (2.9 dB SNRms)

Noisy examples

Trumpet

Here we find a very low frequency synthesized signal. A clear noise burst can be heard on the gap.
  • Left: Ground truth
  • Center: Magnitude (13.7 dB SNRms)
  • Right: Complex (10.3 dB SNRms)

Trumpet

Here we find a low frequency signal. Interestingly, the magnitude network produced a harmonic that is not present on the original signal, and which can be clearly heard. The complex network's reconstruction also has a noisy characteristic, but it is not as obvious.
  • Left: Ground truth
  • Center: Magnitude (23.4 dB SNRms)
  • Right: Complex (24.1 dB SNRms)

Quote of the day: phase life