SUMMARY Project file corruption due to random changes in clip lengths. STEPS TO REPRODUCE 1. Create project. 2. Edit videos/clips, then save project. 3. Open project to continue editing. OBSERVED RESULT 'Invalid Clip' errors are shown and clips are removed from project. Track times have changed in project files that stop working even when no edits were performed based on last 'good' saved project file. EXPECTED RESULT No invalid clip errors. Using latest release of .appimage file... lsb_release -a No LSB modules are available. Distributor ID: Ubuntu Description: Ubuntu 18.04.2 LTS Release: 18.04 Codename: bionic plasmashell --version plasmashell 5.12.7 kf5-config --version Qt: 5.9.5 KDE Frameworks: 5.44.0 kf5-config: 1.0 Additional information about my system can be found at: https://github.com/mltframework/mlt/issues/445 ADDITIONAL INFORMATION Hi, my goal was simply to edit a quick video with kdenlive, but I've found myself chasing the source of a project corruption bug for the past week. I was using the latest appimage build and I did some tests with mlt (see below). This bug is extremely frustrating because it has happened several times after I've put a few hours into editing a real project. The version of kdenlive I used was: md5sum kdenlive-19.04.1b-x86_64.appimage f0ab2ecf3f606e1f1fc36e2a3929f9f7 kdenlive-19.04.1b-x86_64.appimage I have include a .tar archive that contains all iterations of my project files from scratch, kdenlive caches, and executable stdout and sterr outputs. I also have screenshots of the problem, and the source video files I was editing for download here at: http://www.robertelder.ca/kbugreport.tar (1.4GiB) Pay special attention to the following project files: second-kdenlive-bug/good16.kdenlive second-kdenlive-bug/bad17.kdenlive second-kdenlive-bug/bad18.kdenlive These 3 files are supposed to represent the *EXACT SAME* project (just opening in kdenlive, then immeditely 'save as' from previous), but if you diff them you'll note the track times and lengths are different. I think is causing the 'corruption' possibly due to an impossible constraint? For some reason, this change seems to happen more often if I open the project from a desktop shortcut by double clicking on the project file. The worst part is that the open will be *successful* and everything will look normal, but then saving the project will save the corruption to disk. On the next open, saved work will be lost and not importable. At one time I thought I could avoid the bug by only launching from command line then doing project open, but I lost another project file to that as well. The changes and 'corruptedness' seems to have a random component since this morning I was not able to open 'bad17.kdenlive' without errors (note that bad17 has changed times from good16.kdenlive). However, I not seem to be able to open it, but I then just re-saved it as bad18.kdenlive immediately after opening. Again, bad18.kdenlive has changed track times and is visibly corrupted with 'Invalid' tracks when opening it. I also see a difference in relative/absolute paths which I don't think causes a problem, but I'm not sure why that changes. The folder 'kdenlive-bug' is a more minimal example of the bug, but the saved files show what happens when you save the file after the 'Invalid' clips have been removed. This one is I'm less concerned about because it doesn't seem to result in a silently corrupted saved project file that you walk away from and assume it's fine till you open it. Note that the phrase 'SUSPICIOUS: we weren't expecting a producer when parsing the timeline' is found the the executable output multiple times. Related? Other things to note in save session output: qml: keyframe model changed............ qml: loaded clip: 26245 , ID: 12 , index: 3 , TYPE: AV qrc:/qml/Clip.qml:164: TypeError: Cannot call method 'reload' of null I ran kdenlive in the following different ways: # When starting a new project ./kdenlive-19.04.1b-x86_64.appimage # This is how I recored the session output for debugging purposes. ./kdenlive-19.04.1b-x86_64.appimage good1.kdenlive > session3_output.txt 2>&1 # I also used the following /usr/share/applications/kdenlive.desktop entry so I could double click on project files: [Desktop Entry] Exec=/home/robert/kdenlive-bug/kdenlive_hook.sh %F Name=kdenlive NoDisplay=false Terminal=true Type=Application # The hook script above is included in this submission. It it just a hook to log the kdenlive output. Note that both projects were located at the location '/home/robert/kdenlive-bug' when doing this experimenting. The 'second-kdenlive-bug' should be renamed to 'kdenlive-bug' if you want to reproduce my steps. I also have included: - Saved kdenlive caches before and after I encountered the error with this project. - Saved kdenlive executable output for every editing session with kdenlive. - The source video and project file history from the afflicted projects. I followed the troubleshooting guide which notes that I should check my ffplay output (which works) and test out playing a file with 'melt'. I do experience problems doing simple things like 'melt color:red' on the command line (I don't see any red), and I filed a bug in the mlt repo: https://github.com/mltframework/mlt/issues/445 I received a response from Dan Dennedy suggesting that the problem wasn't in mlt, but that I should instead look into a source of corruption in the kdenlive project saved files, which I believe I may have found. Other notes: - A couple times I saw 'No profile found for clip' when adding one of the clips. This only happens *sometimes* although most of the time it doesn't appear at all when adding to project. Related? Screenshot included. Please let me know if you'd like me to compile and test a special branch, or add some debug printf statements and return the output or something. I have been compilling different versions/combinations of mlt/kdenlive from source for the past week so I'm starting to get familiar with the process. There doesn't seem to be any good alternative video editors for Linux, so at the moment fixing this bug seems to be the only way that I'll be able to get my video editing done.
Disclaimer: I haven't looked deep into that yet. I just want to point out that if showing a red producer with melt doesn't work for you, there is a stream of things that may break, since your melt installation seems broken. We need to make sure the simple things work before attempting more complex ones. Maybe you can try to build melt from source, and install it with sudo make install (make sure to uninstall any previously installed version / package), and report what happens when you try to output the red consumer.
(In reply to alcinos from comment #1) > Disclaimer: I haven't looked deep into that yet. > I just want to point out that if showing a red producer with melt doesn't > work for you, there is a stream of things that may break, since your melt > installation seems broken. We need to make sure the simple things work > before attempting more complex ones. > > Maybe you can try to build melt from source, and install it with sudo make > install (make sure to uninstall any previously installed version / package), > and report what happens when you try to output the red consumer. Hi alcinos, thanks for your response. I did (on at least one occasion), build melt from source and actually sudo make install it. I think I did that with v6.16? Same result when playing video/color:red. Having said, that, there are a lot of dependencies that the 'melt' executable relies on and I'm not 100% sure I uninstalled all its dependencies. Therefore, I did what I figured would be a stronger test which is build melt in a directory in /tmp/, and then do '. setenv' to let the mlt package do whatever it does to set paths, and then test the local melt executable that way without install. Same result. Here is the bug I filed for melt: https://github.com/mltframework/mlt/issues/445 Dan Dennedy suggested that perhaps there is an issue with my SDL2 (a dependency of mlt?). Even if that's the case, I would argue that kdenlive should be smart enough to detect this subtle corruption condition and at least issue warnings to the user for this situation where, clearly, an unmodified project file will have its track lengths changed without any modifications to the project. In such a case, regular warnings that say something like 'Warning: track length changed' might arouse suspicion and make it a bit faster to track down bugs. As of this time, I am not aware of any action item I can take to further assist in fixing this issue. The mlt bug I filed has been closed, and I don't know enough about SDL2 or how it interacts with mlt to know what to investigate there. Please suggest further courses of action.
Ok, let's try to get to the bottom of this. First note: your Qt version (Qt: 5.9.5) is old, and it has known bug. This means that you have to stick to the AppImage. Now, using this appimage try this: 1. Create a new project 2. Go to Project -> add color clip, chose whatever color you want 3. It will add a color clip to the bin. Check whether you can visualize it 4. Add it on the timeline, resize it and move it a bit, and check if you can preview the timeline 5. Save, close the program, then open again the project and see if you can continue editing.
Thanks. I have followed these instructions and created a project with 2 separate saved points. I can load both of them without issue. I can also render an output file that looks ok and renders properly on my computer. See all artifacts at: wget http://www.robertelder.ca/colortest1.tar Also, you noted that my desktop QT version is old. As I understand the .appimage file gets around the outdated dependency problem by bundling all dependences. My question is, what important dependencies does it *not* bundle? I extracted it and I can see that it contains a copy of melt, but if this problem is specific to me, then it must be in a very low-level dependency.
The appimage should contain everything you need, I wound not worry about that at this point. So, if this simple test with a color producer works, we need to look further. Can you try to reproduce the same test, replacing the color producer with one video clip that you know for a fact it was getting flagged as "invalid clip" by kdenlive?
Created attachment 120207 [details] 6.kdenlive
Created attachment 120208 [details] 7.kdenlive
Hi, I just did a test to delete one of the solid colour clips with one of the clips that I have trouble with. I then saved it, closed kdenlive, then re-opened the file with no problems. Then, I made 0 modifications and 'saved as' the file again with another name. I then did a diff of the two 'idential' project files and found some of the xml tags moved around, but the content was the same. Then, I re-opened the latest project file and cut the offending clip into 3 major sections, deleted off a couple slices, and re-arranged the last clip so it was in the middle two on the end. Then I saved this file as '6.kdenlive'. Then, I found something interesting: If I open 6.kdenlive, then make no modifications, save as to 7.kdenlive, I can see what I think are functional differences in the 'children' section. See attached two files as an example (do a vim -d on them). Note that I can still load the project and it looks fine at this point as I can still edit/watch everything, but it seems fishy that these number are changing. I just did a re-open of 6.kdenlive and saved it as '7-again.kdenlive' which is idential to 7.kdenlive so at least it might be a deterministic change? Actually, after staring at the diff for a bit, I can see that the change affect the 'kdenlive:docproperties.groups' JSON array by re-arranging the order of the objects. I don't know enough about kdenlive internals to know if that is really a functional difference or not, but it certainly doesn't make reproducibility easier.
The difference you see is in the encoding of the groups, it doesn't matter for the issue at hand. Where is the media located? Is it on an external hard-drive for example? Is it possible that you have a faulty disk that causes the file to be read with errors?
Hi, Thanks for the response. For the ordering of the group encodings, I would like to cast a vote that the ordering be canonicalized somehow, or at least made predictable so that doing 'save as' with a project with no changes produces a new .kdenlive project with idential hash checksums (just for debugability). I'm fairly sure it's not a bad disk problem, I've experienced this in testing from my SSD and larger spinning disk. Here's the one I'm using now: After doing sudo smartctl -a /dev/sdb SMART overall-health self-assessment test result: PASSED ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_FAILED RAW_VALUE 1 Raw_Read_Error_Rate 0x000b 100 100 062 Pre-fail Always - 0 2 Throughput_Performance 0x0005 100 100 040 Pre-fail Offline - 0 3 Spin_Up_Time 0x0007 122 122 033 Pre-fail Always - 2 4 Start_Stop_Count 0x0012 098 098 000 Old_age Always - 3200 5 Reallocated_Sector_Ct 0x0033 100 100 005 Pre-fail Always - 0 7 Seek_Error_Rate 0x000b 100 100 067 Pre-fail Always - 0 8 Seek_Time_Performance 0x0005 100 100 040 Pre-fail Offline - 0 9 Power_On_Hours 0x0012 068 068 000 Old_age Always - 14199 10 Spin_Retry_Count 0x0013 100 100 060 Pre-fail Always - 0 12 Power_Cycle_Count 0x0032 099 099 000 Old_age Always - 1990 191 G-Sense_Error_Rate 0x000a 100 100 000 Old_age Always - 0 192 Power-Off_Retract_Count 0x0032 100 100 000 Old_age Always - 28 193 Load_Cycle_Count 0x0012 097 097 000 Old_age Always - 32587 194 Temperature_Celsius 0x0002 115 115 000 Old_age Always - 52 (Min/Max 12/64) 196 Reallocated_Event_Count 0x0032 100 100 000 Old_age Always - 0 197 Current_Pending_Sector 0x0022 100 100 000 Old_age Always - 0 198 Offline_Uncorrectable 0x0008 100 100 000 Old_age Offline - 0 199 UDMA_CRC_Error_Count 0x000a 200 200 000 Old_age Always - 0 223 Load_Retry_Count 0x000a 100 100 000 Old_age Always - 0 SMART Error Log Version: 1 No Errors Logged So, I spent the entire day messing around with this to try and figure out why it was getting corrupted, and I'm pretty sure I've found *a* bug. Not sure if it's the one we're looking for. Long story short is that you should be looking for reproducibility in the ability to open a project, make *absolutely no changes*, then save the file. What I see happening is I can create a new project, add a single clip. Don't add it to timeline, add to project. Save project as. Re-open new project, change nothing, save as. Repeat this like 6 times, then diff the project files and you'll see a trend like this: a.kdenlive: <producer id="producer0" in="00:00:00.000" out="00:21:54.121"> b.kdenlive: <producer id="producer0" in="00:00:00.000" out="00:21:54.087"> c.kdenlive: <producer id="producer0" in="00:00:00.000" out="00:21:54.054"> d.kdenlive: <producer id="producer0" in="00:00:00.000" out="00:21:54.021"> e.kdenlive: <producer id="producer0" in="00:00:00.000" out="00:21:53.987"> f.kdenlive: <producer id="producer0" in="00:00:00.000" out="00:21:53.954"> g.kdenlive: <producer id="producer0" in="00:00:00.000" out="00:21:53.921"> Note that the time length keeps going down. If you add the clip to the timeline, you can see it move around and if you add multiple clips, empty spots slowly creep in. I did some tests to see if the change is deterministic, and doing 6 tries of save as from a single point seem to deterministicly change the track length. I also noticed that sometimes when I add a clip there is a message about creating a 'profile' for the clip. Sometimes it is a popup dalog you have answer, and sometimes it's a small sub-window in the clips area (why the difference?). I found that if you click cancel or ignore the message you get this time 'drifting' effect after save, but it never asks for a profile again. If I click 'create' the profile, it seems to stop drifting on multiple saves, so perhaps I really *have* to create a profile? If so, that message should be much more forceful rather than allowing subtle drifting and blank parts in the track. Another important point: Sometimes, I don't get that 'create profile' message at all. For example, if I add the Screencast 2019-05-21 14:55:40.mp4 video, then save exit. Re-open project, then add VID_20190520_093206251.mp4, I get *absolultely no* message about creating profiles. Whereas, if I add VID_20190520_093206251.mp4 first I get a mandatory prompt for it. Here are the source videos to test with: wget "http://www.robertelder.ca/Screencast 2019-05-21 14:55:40.mp4" wget "http://www.robertelder.ca/VID_20190520_093206251.mp4" No project file is necessary, create a new project from scratch. Now, that's one definite issue, but unfortunately, I was not able to reproduce the corruption issue all day today when launching kdenlive appimage from the command-line. I can often re-produce it when launching kdenlive by double-clicking on a .kdenlive project though. I have install .desktop file associated as document above in the first post. I don't know what the difference is between opening via command-line, and through the 'open with' association. Sometimes, I can open a project no problems from the 'open with' file association, and edit, encode whatever no problems. I even logged the environment variables from using the open with method, then set them manually in shell and tried to reproduce that way. Could this be a race condition? My working theory is that upon opening the project, kdenlive is somehow re-computing the 'lengths' of videos in some non-determinstic way. Most of the time, I see the clip length trend down after re-saving multiple times, but I have logged at least one instance of it going up. If one clip ends up with an impossible overlap, impossible high length, can that cause the 'Invalid clip' error?
Thank you very much with the in-depth exploration! When you don't accept to create the profile, or switch to an existing profile, it means that potentially the fps of the project and the fps of your video samples don't match. My theory, based on what you show, is that the fps conversion math (used for example to compute the length of the clips in frames), will create small but real rounding errors, and when you repeat that over and over the errors will compound. I'm a bit short on time right now but will definitely take a look soon, if confirmed it would be a serious bug. To be sure I reproduce exactly what you did, would you mind giving me the default fps of your projects in kdenlive? It should be displayed in the title of the app when you open a fresh kdenlive.
Created attachment 120236 [details] screenshot for fps.
Here are exact repo steps: mkdir /tmp/test cd /tmp/test wget https://files.kde.org/kdenlive/release/kdenlive-19.04.1b-x86_64.appimage $ md5sum kdenlive-19.04.1b-x86_64.appimage # f0ab2ecf3f606e1f1fc36e2a3929f9f7 kdenlive-19.04.1b-x86_64.appimage chmod u+x kdenlive-19.04.1b-x86_64.appimage wget "http://www.robertelder.ca/Screencast 2019-05-21 14:55:40.mp4" ./kdenlive-19.04.1b-x86_64.appimage # Add clip 'Screencast 2019-05-21 14:55:40.mp4'. Don't add to timeline. If prompted to create profile for clip, click 'Cancel'. # After clip has been added, do save as 1.kdenlive in current directory. ./kdenlive-19.04.1b-x86_64.appimage 1.kdenlive # Then change *nothing* and 'save as' 2.kdenlive ./kdenlive-19.04.1b-x86_64.appimage 2.kdenlive # Then change *nothing* and 'save as' 3.kdenlive ./kdenlive-19.04.1b-x86_64.appimage 3.kdenlive # Then change *nothing* and 'save as' 4.kdenlive ./kdenlive-19.04.1b-x86_64.appimage 4.kdenlive # Then change *nothing* and 'save as' 5.kdenlive ./kdenlive-19.04.1b-x86_64.appimage 5.kdenlive # Then change *nothing* and 'save as' 6.kdenlive # Then observe that the project files include different times: robert@robert-msi-ge62:/tmp/test$ grep -Hn 'id="producer0"' *.kdenlive 1.kdenlive:4: <producer id="producer0" in="00:00:00.000" out="00:21:54.121"> 2.kdenlive:4: <producer id="producer0" in="00:00:00.000" out="00:21:54.087"> 3.kdenlive:4: <producer id="producer0" in="00:00:00.000" out="00:21:54.054"> 4.kdenlive:4: <producer id="producer0" in="00:00:00.000" out="00:21:54.021"> 5.kdenlive:4: <producer id="producer0" in="00:00:00.000" out="00:21:53.987"> 6.kdenlive:4: <producer id="producer0" in="00:00:00.000" out="00:21:53.954">
Thanks for your investigation. I can confirm the behavior described in your last comment. It only happend on project profiles with non integer fps, and the clip duration seems to vary slightly between saves. I still have to investigate if the problem is in MLT or Kdenlive, and if this has real consequences, will get back to you soon on that. Not sure this is the cause of your "invalid clip" errors though.
Git commit bc32c54a408d283de1a0568301dcd3c626967632 by Jean-Baptiste Mardelle. Committed on 23/05/2019 at 06:25. Pushed by mardelle into branch 'Applications/19.04'. Do not use MLT producer's get_length_time methd as it changes the way the length property is stored, causing inconsistencies (clock vs smpte_df) M +1 -1 src/mltcontroller/clipcontroller.cpp https://invent.kde.org/kde/kdenlive/commit/bc32c54a408d283de1a0568301dcd3c626967632
Ok, we seem to have a regression that prevents clips with a colon ':' to load correctly. This is the reason for the invalid clip error message. working on it
Awesome, glad to hear. I look forward to trying out the next release :)
Git commit efbf6bd1002174b893befd1f216f594ba0db1839 by Jean-Baptiste Mardelle. Committed on 24/05/2019 at 09:14. Pushed by mardelle into branch 'Applications/19.04'. Fix invalid clip on project opening M +5 -24 src/bin/projectitemmodel.cpp M +10 -6 src/mltcontroller/clipcontroller.cpp https://invent.kde.org/kde/kdenlive/commit/efbf6bd1002174b893befd1f216f594ba0db1839
Ok, so I think the issue should now be solved in the latest AppImage: https://files.kde.org/kdenlive/release/kdenlive-19.04.1c-x86_64.appimage.mirrorlist Please test if possible and let us know.
Fantastic, I need to do some video editing this weekend, so this comes at just the right time. Thank you for the quick turn-around.
Hi, a brief update: - I've tried out the new kdenlive-19.04.1c-x86_64.appimage and the time drifting after saves issue appears to be fixed for me. - I've edited a fairly large project based on the same files, and I haven't gotten the 'Invalid clip' error again so that appears to be fixed. I have also been testing through opening with a desktop app shortcut which seemed to trigger the issue before. - I do still think there is still one bug which happens on rendering. I haven't dug into the source, but I have a hunch that there is another instance of the same colon related bug. If I render videos that have the ':' character in them, the output rendered video just shows blank white frames. I was able to work around this by renaming all videos in my project to not include the ':' character. Also, for whatever reason, I don't get this error if I run the kdenlive app image directly from the command-line. Not sure why, most of the environment variables are the same. If you need a set of repo steps for rendering bug, let me know. I anticipate that the previous repo steps should be easily adapted.
Hi, just curious if the bug I mentioned in my last comment is still on your radar: "If I render videos that have the ':' character in them, the output rendered video just shows blank white frames. I was able to work around this by renaming all videos in my project to not include the ':' character.". It's a bit different than the bug I reported, but I can file a separate bug for it if you want. If you need repo steps, let me know. For now I'll just keep re-naming all my files to ones that kdenlive can work with, but it would be cool if it worked out of the box. It makes doing backups easier without all the renaming. Also, is there anywhere I can do 'donations' to the kdenlive project (not kde overall) to help keep the bug fixers well fed? You guys should consider setting up a patreon.
I would personally consider this bug thread resolved and I think the remaining issue I had is very different than the one described by this thread. I have filed a different bug at : https://bugs.kde.org/show_bug.cgi?id=408460 If you want me to update it to 'resolved', when I'm satisfied, let me know.
Dear Bug Submitter, This bug has been in NEEDSINFO status with no change for at least 15 days. Please provide the requested information as soon as possible and set the bug status as REPORTED. Due to regular bug tracker maintenance, if the bug is still in NEEDSINFO status with no change in 30 days the bug will be closed as RESOLVED > WORKSFORME due to lack of needed information. For more information about our bug triaging procedures please read the wiki located here: https://community.kde.org/Guidelines_and_HOWTOs/Bug_triaging If you have already provided the requested information, please mark the bug as REPORTED so that the KDE team knows that the bug is ready to be confirmed. Thank you for helping us make KDE software even better for everyone!
This bug has been in NEEDSINFO status with no change for at least 30 days. The bug is now closed as RESOLVED > WORKSFORME due to lack of needed information. For more information about our bug triaging procedures please read the wiki located here: https://community.kde.org/Guidelines_and_HOWTOs/Bug_triaging Thank you for helping us make KDE software even better for everyone!