Bug 411260 - "f=mkv" or "f=matroska" produces MP4 file output
Summary: "f=mkv" or "f=matroska" produces MP4 file output
Status: RESOLVED UPSTREAM
Alias: None
Product: kdenlive
Classification: Applications
Component: Video Display & Export (show other bugs)
Version: 19.08.0
Platform: Microsoft Windows Microsoft Windows
: NOR normal
Target Milestone: ---
Assignee: Jean-Baptiste Mardelle
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2019-08-25 05:24 UTC by Jonathan Gilbert
Modified: 2019-12-02 14:56 UTC (History)
3 users (show)

See Also:
Latest Commit:
Version Fixed In:
fritzibaby: Brainstorm+


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Jonathan Gilbert 2019-08-25 05:24:00 UTC
SUMMARY
Previous kdenlive versions such as 18.12.2-beta would produce an MKV file if you specified "f=mkv" in the render job properties. "mkv" and "matroska" are both listed as valid formats by "melt -query formats". But when "f=mkv" is supplied to 19.08.0, the render succeeds but the output file is in MP4 format.

STEPS TO REPRODUCE
1. Create a custom render format definition with parameters such as:

properties=x264-medium f=mkv vcodec=libx264 acodec=flac g=120 crf=9 global_quality=500 me_method=umh

In my testing, any valid set of parameters that specify "f=mkv" trigger the bug.

2. Render all or part of the timeline.

OBSERVED RESULT
The output file has MP4 format according to tools like "ffprobe".

EXPECTED RESULT
The output file has Matroska (MKV) format.

SOFTWARE/OS VERSIONS
Windows: Windows 10 1903
KDE Frameworks Version: 5.61.0
Qt Version: 5.13.0
MLT Version: 6.16.0
Comment 1 Jonathan Gilbert 2019-09-20 07:25:36 UTC
I have been reviewing the background of this report and tried to reproduce a previous Kdenlive version producing an MKV file as I remembered. From fresh installations of 17.12 and 18.12.2, I saw the same behaviour as 19.08, ignoring the f="mkv" in the format specification and writing an MP4 container into the output file despite happily giving it the .mkv extension.

I poked a bit deeper and found the avformat presets files in ./share/mlt off the installation folder. The x264 presets in this directory (and, I'm sure, others as well) include a directive "f=mp4" at the top of the file, and I found that if I edit this to "f=mkv" in the preset file, I get the desired behaviour I want. I conjecture that my experience in the distant past was with an installation that had been modified in this way, which is how I have memories of Kdenlive writing MKV files.

So, this changes this bug report, but it does not obviate it. It is not a regression; rather, Kdenlive never did allow the user to specify the output format, even though it does invite you to set up your own render format definitions, in which "f=mkv" is accepted syntax and "mkv" can be specified for the file extension.
Comment 2 Jonathan Gilbert 2019-09-20 07:31:31 UTC
I have just done a follow-up test. It appears that if "f=mp4" is entirely removed from the presets file (no 'f' option value specified), then a value specified in the render format definition is in fact respected. I can change my output file format by switching between "f=mp4" and "f=mkv" in the render format definition in the Kdenlive UI. This isn't possible with the distribution version of e.g. the "x264-medium" preset file.

I also tested what would happen if the render format definition also omitted the 'f' option, and the result appears to be that the MLT consumer automatically detects the format based on the extension in the filename provided. If "x264-medium" is entirely missing "f=mp4" and my render format has neither "f=mp4" nor "f=mkv", then setting the output filename to "Output.mp4" creates an MP4 file and setting it to "Output.mkv" creates an MKV file with no other changes.

I would propose, then, that at least some of the MLT libavformat presets files distributed with Kdenlive have their 'f' option removed, allowing the user to specify the format via the Kdenlive UI either explicitly or implicitly via the filename.
Comment 3 Jonathan Gilbert 2019-09-20 07:51:05 UTC
With further experimentation, I have determined that the preset files in question are sourced directly from the MLT release and aren't part of the Kdenlive codebase. As such, I believe I need to take this issue upstream.
Comment 4 Jonathan Gilbert 2019-09-20 08:29:23 UTC
Alright, I have feedback from the MLT side. This appears to be a bug on the Kdenlive side after all. When it generates the <consumer> tag in the MLT script, the parameters get reordered apparently randomly (maybe they're passing through a hash table on the way?). Dan Dennedy of the MLT project pointed out that the order of the XML attributes is important. They are processed left-to-right, and so when it encounters the `properties` option to load a preset, it is at that point that the preset is loaded and all of its parameters are applied. If `f="mkv"` appears to the left of `properties`, then the preset file can overwrite it. According to Dan, Kdenlive should always be putting `properties="preset"` as the very first XML attribute.
Comment 5 Jonathan Gilbert 2019-09-20 08:41:11 UTC
I did a bit more research. Kdenlive is using the Qt `QDomDocument` type to represent an XML DOM in-memory while building the MLT script. It turns out that between Qt 4 and Qt 5, this DOM representation got reworked, and the Qt 5 version uses a hash table to store attributes. In addition, for security reasons, the hash provider is salted with a unique seed on each run. So, with `QDomDocument`, it is flat-out impossible to have any control over the order of attributes. Simply loading an XML document and re-saving it will scramble all the attributes in the file. Since MLT is very sensitive to the order of the attributes, I believe Kdenlive's render widget will have to be reworked to use a different XML library.
Comment 6 emohr 2019-09-21 08:59:35 UTC
Thank you very much for such indeep research. I opened issue: https://invent.kde.org/kde/kdenlive/issues/359 so the dev can work on it.
Comment 7 Vincent PINON 2019-09-21 11:12:59 UTC
Please drop the "properties=x264-medium" in your MKV encoding profile, it sources /usr/share/mlt/presets/consumer/avformat/x264-medium which starts with f=mp4.
just copy the settings you are interested in from that file...

I think "f=mkv vcodec=libx264 crf=15 acodec=flac" should be enough
Comment 8 Jonathan Gilbert 2019-09-21 15:36:33 UTC
I actually have a further update. It seems this is more properly a bug in MLT, because the XML standard says that attribute order should not be a semantic difference -- that's why Qt 5 feels it is okay to start using a hash for them -- and MLT is violating that by processing files differently for different attribute orders. Kdenlive may not need to make a change because MLT is going to make a change to eliminate the dependency on attribute order simply by first explicitly searching for and processing "properties" and then processing everything else. I don't know when that'll happen or what the release cadence is, but it may be sufficient simply to wait for the next MLT version and build the next Kdenlive release against that. If MLT might not get updated quickly enough then switching to an XML approach that is deterministic in its attribute order may still make sense, but there is a chance it might not be necessary.
Comment 9 Jonathan Gilbert 2019-09-29 03:45:29 UTC
MLT master now has a fix for this issue, so as long as it has another release before the next Kdenlive release, and Kdenlive updates to take that release, this bug should be resolved. :-)
Comment 10 farid 2019-12-02 14:56:58 UTC
This seems to be fixed upstream.

https://github.com/mltframework/mlt/issues/477

Please test and reopen if necessary.