Bug 423337 - Waveform display for longer audio track goes out of sync with played or rendered content
Summary: Waveform display for longer audio track goes out of sync with played or rende...
Status: RESOLVED FIXED
Alias: None
Product: kdenlive
Classification: Applications
Component: Video Display & Export (show other bugs)
Version: unspecified
Platform: Other Linux
: NOR grave
Target Milestone: ---
Assignee: Jean-Baptiste Mardelle
URL:
Keywords:
: 410798 422139 (view as bug list)
Depends on:
Blocks:
 
Reported: 2020-06-21 23:53 UTC by info
Modified: 2021-03-31 12:26 UTC (History)
5 users (show)

See Also:
Latest Commit:
Version Fixed In:


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description info 2020-06-21 23:53:24 UTC
SUMMARY

STEPS TO REPRODUCE
1. Import an AV clip from Camera (here e.g.: *.MTS; AVHCD; Full-HD 50 fps + "original" Audio 48kHz stereo)
2. Import a separately recorded "new" audio file (here: e.g. 44.1kHz or 48kHz stereo AIF)
3. Try to manually sync-align the "new" audio file with the "original" audio track by looking at the displayed waveforms in the timeline.

(Maybe cut out some unneeded sceenes from both sources, and/or maybe try to compensate for slightly different sample clocks or for an initial problem in the waveform display generator that may require manual alignment of the two tracks further down.)

(If you want me to, I can also try it again using an aiff copy of the original camera soundtrack as "new" audio file. My error report does *not* refer to problems resulting from different sample frequencies or unsynchronized sample clocks on different recorders.)

OBSERVED RESULT
After a few minutes, the waveform displayed in the timeline preview for the "new" audio file will be out of sync with its actual audio content.
When I go e.g. 5 minutes into the material, and try to align the "new" audio track to the "original" audio by looking at both waveforms in the timeline at that position, and then play them both together at that position - I'll get a distinct echo or delay effect - even though the two displayed waveforms may be nicely aligned. The offset between displayed waveform and actually played audio content may exceed a few tenths of a second, i.e. a total desynchronization between piano playing video and audio.
-
Thus, it is rather impossible to use the waveform display to align separately recorded audio tracks to camera-recorded audio/video.
-
This is true, no matter whether the two audio tracks use the same or a different sample rate.

EXPECTED RESULT
The waveform display in the timeline should accurately represent the audio in this track at any position down into the audio material.
Thus, when aligning two audio tracks by looking at their waveform previews at some given position in the timeline, the preview-played and the rendered audio of these two tracks at that position in the timeline should also be aligned.

SOFTWARE/OS VERSIONS
Windows: 
macOS: 
Linux/KDE Plasma: kdenlive 18.x; 19.x; 20.07 
(available in About System)
KDE Plasma Version:
KDE Frameworks Version:
Qt Version:

ADDITIONAL INFORMATION
I suspect that the waveform display uses a waveform preview generator that goes out of sync with the actual waveform content - OR some built in synchronization that ties camera recorded audio to the video frames is unavailable for separate audio tracks, so that audio positions from the timeline do not refer to the expected position in the audio file. Possibly because of some integer/rounding problem or the like that adds up after processing a large number of samples.
-
I know that kdenlive has some master/slave/audio-auto-align functionality, but I have not succeeded in trying that out. I actually doubt that this will work when I only want to retain multiple individual scenes from the original recording, resulting in multiple audio-and-video snippets to be aligned and then crossfaded.
-
Additional problem, less severe: kdenlive apparently allows positioning of an audio clip only in full frame increments.
-
Additional problem: When GPU acceleration is *dis*abled, the preview playback of Full HD 50 fps video is so slow (on an older i7 cpu) that it is impossible to judge whether Audio/Video are in sync. Additionally, simple audio fade-in/fade-out effects are lost. - When GPU acceleration is *en*abled, preview play is sufficiently fast, and fade-in fade-out are there - but: *rendering* a clip will never finish. And transition between both settings is ... difficult.
-
Therefore, it's almost impossible (for me...) to replace camera recorded audio with externally recorded audio tracks in kdenlive. Simply, because the waveform displays are unreliable, and even quickly changing between a working video preview and working test renderings is impractical.
-
I've tested that with various versions of kdenlive on various machines, and even converted audio recordings externally to the same sample rate etc. It's a very sad restriction, because otherwise, kdenlive looks rather nice.
-
Simply searching for "waveform" in the kdenlive bug tracker returns a few errors that *might* be related. But their status is mostly "repo" (??) and/or they are old, or might describe different problems - so I don't know whether this problem is being handled. I can provide any additional testing if that is required.
-
Kind regards and thank you for looking into this - and for the otherwise very interesting work with kdenlive - js
Comment 1 info 2020-06-22 01:08:49 UTC
Follow up:
(1) I did try the "Set Audio Reference" / "Align Audio to Reference" functionality. It actually worked. Thank you for this feature, especially given the limited accuracy waveform display, this really saves the day. :-)
(2) The results of this feature are quite close to what I found by manual adjustment from listening to both tracks at the same time, or comparing audio/video in the play-preview.
(3) The size of the error is as follows: Maybe 2..5 minutes after the start of the recording, the displayed waveform in the timeline for the external audio (provided in 48 kHz already) is early by ca. 2 frames; and about 20..30 mins later, it is early by about 18 frames. This means that in the timeline, the displayed waveform for the "new" audio track needs to be positioned 18 frames before the displayed waveform of the "original" camera recorded audio, to make  audio from both tracks appear in sync (in the playback preview window that is - I still need to see what comes out in the rendering).
Comment 2 Henrique Sant'Anna 2020-09-21 14:40:34 UTC
I can confirm that audio waveform thumbnails on timeline will get out of sync with real sound by several frames, representing around 500 ms of delay from the real sound to the corresponding displayed waveform, and will increase over the timeline progress.

Tested with 20.08.1 linux distro package (arch): Confirmed.
Tested with kdenlive-20.11.70-8a08988-x86_64.appimage: Confirmed.

Tried to clean project cache to regenerate audio thumbnails, closing kdenlive and than reopening the project, but it will be regenerated out of sync again. Tested with and without ffmpeg audio thumbnails, but with the same results.

kdenlive-20.11.70-8a08988-x86_64.appimage seems to regenerate audio thumbnails much faster than 20.08.1 linux distro package.
Comment 3 emohr 2021-03-07 17:09:04 UTC
*** Bug 410798 has been marked as a duplicate of this bug. ***
Comment 4 emohr 2021-03-07 17:13:09 UTC
*** Bug 422139 has been marked as a duplicate of this bug. ***
Comment 5 Jean-Baptiste Mardelle 2021-03-12 17:27:53 UTC
Git commit e0c0a3df27969261ec4bb211e662e61070291d4e by Jean-Baptiste Mardelle.
Committed on 12/03/2021 at 17:23.
Pushed by mardelle into branch 'master'.

Improve audio thumbnail offset on clip cut or longer clips.
Related to #973

M  +1    -0    src/monitor/view/kdenliveclipmonitor.qml
M  +2    -1    src/timeline2/view/qml/ClipAudioThumbs.qml
M  +25   -22   src/timeline2/view/qml/timeline.qml
M  +6    -3    src/timeline2/view/qml/timelineitems.cpp

https://invent.kde.org/multimedia/kdenlive/commit/e0c0a3df27969261ec4bb211e662e61070291d4e
Comment 6 farid 2021-03-14 01:06:17 UTC
Hey folks, can you try the latest daily build [1], or compile from master, and let us know if it is fixed for you after latest commit.

Thanks for you report

[1] https://binary-factory.kde.org/job/Kdenlive_Nightly_Appimage_Build/lastSuccessfulBuild/artifact/
Comment 7 farid 2021-03-16 12:39:25 UTC
This has been confirmed as fixed, so will close it here as well. Please reopen if you encounter issues. Thanks for your report.
Comment 8 info 2021-03-22 17:36:32 UTC
Thank you very much to all of you for fixing this.

I've also tried it out (shortly) now, and it looks good. This is great!

Kind regards, js
Comment 9 farid 2021-03-22 17:57:46 UTC
(In reply to info from comment #8)
> Thank you very much to all of you for fixing this.
> 
> I've also tried it out (shortly) now, and it looks good. This is great!
> 
> Kind regards, js

Thanks for the feedback.
Comment 10 Colin Brash 2021-03-31 12:26:05 UTC
(In reply to farid from comment #6)
> Hey folks, can you try the latest daily build [1], or compile from master,
> and let us know if it is fixed for you after latest commit.
> 
> Thanks for you report
> 
> [1]
> https://binary-factory.kde.org/job/Kdenlive_Nightly_Appimage_Build/
> lastSuccessfulBuild/artifact/
Have been using 20.12.2 AppImage and it's working like a charm!! Thanks to all who have worked on this issue it's very much appreciated.