Two frequency and phase controls Drag either red point horizontally to change frequency and vertically to change phase. The horizontal axis is logarithmic.

The impossibility of perfect audio stretching

Inspired by Julius O. Smith III’s book Spectral Audio Signal Processing

Suppose we want to stretch a song to twice its original length. How can we do this?

We will investigate this question using an audio clip of two sine waves. Drag the red points to change their frequencies (which control how quickly the waves oscillate) and phases (which control the starting position of the waves), then press play to hear the result.

The waveform can be described by the following equation:

$$y(t) = \sin(2 \pi f_1 t + \phi_1) + \sin(2 \pi f_2 t + \phi_2)$$

where $t$ is time, $f_1$ and $f_2$ are frequencies, and $\phi_1$ and $\phi_2$ are phases.

Where the beats come from

We can use trigonometric identities to rewrite the waveform as:

$$y(t) = 2\cos\left(\pi(f_1 - f_2)t + \frac{\phi_1 - \phi_2}{2}\right)\sin\left(\pi(f_1 + f_2)t + \frac{\phi_1 + \phi_2}{2}\right)$$

This is the product of two waves: one wave with frequency $|f_1 + f_2|/2$ and the second with frequency $|f_1 - f_2|/2$. The second wave modulates the amplitude of the first wave, creating beats. The beat frequency is the frequency difference $|f_1 - f_2|$: 440 and 445 Hz (cycles per second) produce five beats per second, or ten beats in this two-second clip.

Why is the beat frequency not half the difference in frequencies as the equation seems to suggest? A single cycle of the modulating wave has both a positive and negative component. Perceived loudness only depends on the magnitude of the modulating wave, which peaks twice in a single cycle, so we hear two beats per cycle. Also, note that the phases $\phi_1$ and $\phi_2$ of the original waves don't affect the beat frequency. Instead, they affect only the phase of the beats.

In the context of this signal, we define "perfect" stretching as increasing the signal's duration while preserving the component frequencies and stretching the amplitude envelope (so the number of beats is preserved). Next, we will try different ways to achieve this.

Simple stretching

If we stretch the waveform to twice its original length, every frequency is halved. The result is slower, but also an octave lower (half the original frequency).

This is generally not what we mean by stretching audio: we want more time without changing the original pitch.

Preserving frequencies

What if we keep the original frequencies and just play the signal for twice as long? This works well for a single sine wave, but not when we have multiple sine waves.

The original two-second clip contains ten beats. Over four seconds, the same frequencies produce twenty beats instead of ten.

Ideally, we'd like the stretched signal to contain the same number of beats as the original so that the timing of the beats is "stretched" by a factor of two.

Slowing the beat frequency

To preserve the number of beats, we can slow the beat frequency. Because beat frequency is the difference between the component frequencies, we can make those frequencies more similar.

Changing 445 Hz to 442.5 Hz halves the beat frequency from 5 Hz to 2.5 Hz. The four-second result now contains the original ten beats.

Larger frequency gaps

Now, consider a wider frequency separation: 440 Hz and 660 Hz. The beat frequency is now 220 Hz, which is too high for humans to perceive distinct beats.

Instead, the pair of frequencies is heard as a musical interval, which you may recognize as a perfect fifth.

Maintaining beat count

We can again preserve the beat count by halving the frequency difference. But the new upper frequency, 550 Hz, is noticeably different from 660 Hz. The beat pattern stretches correctly, but the pitch is distorted.

Preserving frequencies again

If we instead preserve 440 Hz and 660 Hz while extending the duration, the stretched audio sounds like the original. The beat count doubles, but those beats are too fast to perceive individually.

This suggests a general strategy: when the two frequencies are nearby, preserve the beat count, and when they are far apart, preserve the frequencies.

Moderately close frequencies

But what happens between the two extremes of nearby and far-apart frequencies? In this example, the frequencies are close enough for their beats to be perceivable (you may perceive them as a "rough" texture), yet far enough apart for their pitch difference to be audible.

At 440 Hz and 460 Hz, this two-second clip contains forty beats.

Note also that there is no sharp perceptual boundary between hearing beats and hearing two distinct tones. Instead, as the two frequencies get closer, the beats become slower and more noticeable, while the pitch difference becomes harder to distinguish.

Preserving frequencies fails

Suppose we keep both frequencies fixed and extend the signal to four seconds. The pitch survives, but the number of beats doubles from forty to eighty. This is not what we want...

Preserving beat count fails

Instead, suppose we halve the frequency difference to preserve forty beats over four seconds. The upper tone then moves from 460 Hz to 450 Hz.

The beat count now matches that of the original clip, but the pitch has changed noticeably from the original signal.

Towards modern stretching algorithms

We cannot stretch our two-tone signal while keeping both its component frequencies and the beat count. For this signal, any stretching method that changes the phases and frequencies of the two sine waves must distort either the frequencies or the beats. The choice of which to preserve depends on which aspects of the signal are more important.

For more complex signals, like music or voice, a more general principle often applies: any stretching procedure will produce some forms of distortion, so the optimal procedure to use depends on which properties of the signal (such as pitch, rhythm, or timbre) are most important to preserve.

Because signals often sound simpler over short-time windows, common TSM algorithms work by breaking up the signal into temporal chunks, applying a stretching transformation that preserves the desired properties within each chunk, then reassembling the chunks into the full signal. The example below illustrates this idea for a simple melody: here each 0.05 second chunk in the signal is repeated.

Conclusion

While this simple chunking method roughly works, it introduces audible artifacts at chunk boundaries. This is because the waves may not align perfectly at the edges of chunks, creating discontinuities that we perceive as clicking sounds. These discontinuities can be avoided when every frequency in the signal completes an integer number of cycles within the chunk length so that the phases of each sine wave are the same at the chunk boundaries.

Modern TSM algorithms aim to soften these artifacts by more carefully using a number of methods, including aligning the phases of neighboring chunks, overlapping chunks together, or using different chunk sizes for different frequency components of the signal. By exploiting the structure of real audio, modern TSM algorithms can do this so well that distortions are nearly imperceptible.

Nevertheless, stretching an audio signal almost always introduces some distortions: there is no single way to "perfectly" stretch a signal. Instead, the choice of stretching method depends on which properties of the signal are most important to preserve.