Bug 461975 - Broken EXR and TIFF when exporting from some color spaces
Summary: Broken EXR and TIFF when exporting from some color spaces
Status: REOPENED
Alias: None
Product: krita
Classification: Applications
Component: File formats (show other bugs)
Version: 5.1.3
Platform: Flatpak Linux
: NOR normal
Target Milestone: ---
Assignee: amyspark
URL:
Keywords:
Depends on: 462799 462925
Blocks:
  Show dependency treegraph
 
Reported: 2022-11-18 04:46 UTC by Geekley
Modified: 2023-01-17 12:35 UTC (History)
1 user (show)

See Also:
Latest Commit:
Version Fixed In:


Attachments
Test files - source images, broken files and exportation script (3.38 MB, application/zstd)
2022-11-18 04:46 UTC, Geekley
Details
Dolphin previews of original sent files from "broken" folder (75.94 KB, image/png)
2022-12-09 07:50 UTC, Geekley
Details
Dolphin previews of "broken", retrying script now to ensure flatten image (82.53 KB, image/png)
2022-12-09 07:53 UTC, Geekley
Details
Exported files after retrying script now to ensure flatten image (1.02 MB, application/zstd)
2022-12-09 08:10 UTC, Geekley
Details
Example file that Krita reads incorrectly (fully white issue) (29.07 KB, image/x-exr)
2022-12-20 22:11 UTC, Geekley
Details

Note You need to log in before you can comment on or make changes to this bug.
Description Geekley 2022-11-18 04:46:39 UTC
Created attachment 153852 [details]
Test files - source images, broken files and exportation script

SUMMARY
I've made some files with the objective of testing whether Krita (especially via command-line) is exporting files correctly from all color spaces. When the KRA file is in certain color spaces, exporting to EXR and TIFF formats creates broken files. I'm attaching a compressed TAR with all files used for testing, and with a bash shell script to facilitate exporting all files at once.

STEPS TO REPRODUCE
1. Extract the attached file and enter the Krita-ColorTest folder.
2. On the subfolders, check the bad files generated from conversion; they have similar names to the corresponding source KRA file in the top-level folder. E.g. *.exr.kra means a KRA file is bad when exporting to EXR. *.exr+tif.kra is bad when exporting to both EXR and TIFF.
3. In Krita, make sure the last used export settings for EXR and TIFF don't use any weird/advanced option that could cause incompatibility. E.g. make sure exportation will flatten the image. Since there's no command-line flags for such options, you might have to manually export some arbitrary file just to make sure the correct settings will be remembered when exporting from command-line.
4. Make sure Krita GUI is closed before running the command-line, so it doesn't freeze (bug #460631).
5. Then, you can run the shell script to re-export those files via command line. They'll be generated next to the source, not on the subfolders.
6. Sometimes exporting from Krita GUI gives a different result from the command-line (bug!), but this result is still broken for one reason or another.

OBSERVED RESULT
The source KRA files use paint and vector layers.
- Sometimes vector layers aren't exported.
However, even if you convert all layers to paint layers, they'll still be broken because of other problems:
- No files export alpha channel correctly (result often has opaque background, sometimes image is totally empty).
- Most files in Gray/Alpha color space export Gray channel incorrectly (often file is green).
- Some files appear broken in other viewers (in addition to the problems above when opening in Krita), e.g. some TIFF previews are all messed up in Dolphin.
- Files in "encodedBadly" folder have incorrect colors because the result expects linear colors but they didn't convert from sRGB, so they're lighter than the original. This can be seen by inspecting the "0.5" gray in the middle. Triangle was sRGB gray (should be close to #808080) and circle was linear gray (should be 0.5f in linear space, equivalent to #bcbcbc); tertiary colors also use the same logic.

EXPECTED RESULT
Files must look like the original, at least when opening them in Krita (but preferably also in other apps like Gwenview, GIMP). Colors must correspond too. If the format doesn't support something, it'd be better to have a lossy or heavier conversion than a broken or badly encoded file.

SOFTWARE/OS VERSIONS
Operating System: Kubuntu 22.10
KDE Plasma Version: 5.25.5
KDE Frameworks Version: 5.98.0
Qt Version: 5.15.6
Kernel Version: 5.19.0-23-generic (64-bit)
Graphics Platform: X11
Processors: 4 × Intel® Core™ i5-5300U CPU @ 2.30GHz
Memory: 11.1 GiB of RAM
Graphics Processor: Mesa Intel® HD Graphics 5500

ADDITIONAL INFORMATION
I don't expect broken images in the color spaces not included in the attachment. I had made a similar test for 5.1.1 for (pretty much) all color models, exporting to PNG, EXR and TIFF. However, files are CC0, so feel free use them and this script method to make conversion tests in Krita development if desired. Such automated scripts can help testing many files at once, so I suggest using a similar method to test:
- Whether conversions are breaking files.
- Whether command-line conversions are different from GUI exporting.
Comment 1 amyspark 2022-12-06 01:27:36 UTC
I've started triaging this bug. There are some interesting insights on the characteristics of the broken TIFF files, I'll write back when I have a fuller picture.
Comment 2 amyspark 2022-12-07 22:42:43 UTC
Ok, I've tested this and have a few observations.

> OBSERVED RESULT
> The source KRA files use paint and vector layers.
> - Sometimes vector layers aren't exported.

[BUG] I was able to consistently reproduce the latter error with your files, but only when not flattening the files. Something's failing inside LibTiff and not in our code. I'll try and debug it on my end.

> However, even if you convert all layers to paint layers, they'll still be broken because of other problems:
> - No files export alpha channel correctly (result often has opaque background, sometimes image is totally empty).

This is correct and consistent with `Store Alpha Channel (transparency)`  being unset by default on your end.  The latter is actually consequence of the error above (as we never checked the return value of LibTiff when writing each layer).

> - Most files in Gray/Alpha color space export Gray channel incorrectly (often file is green).

[BUG] Can confirm this with the Tev viewer. All GrayA images store as green, will check the code in light of https://lists.aswf.io/g/openexr-dev/topic/82133584?p=Created,,,20,2,0,0::,,,0,0,0,82133584 .

> - Some files appear broken in other viewers (in addition to the problems above when opening in Krita), e.g. some TIFF previews are all messed up in Dolphin.

I don't have Dolphin, but all files (when flattened) show correctly in macOS Preview and the Windows 10 Photo Viewer (except for the CMYK 32f whose depth seems to be unsupported). It'd be great if you could send the affected files here plus previews of each, to understand what the graphical glitches are.

> - Files in "encodedBadly" folder have incorrect colors because the result expects linear colors but they didn't convert from sRGB, so they're lighter than the original. This can be seen by inspecting the "0.5" gray in the middle. Triangle was sRGB gray (should be close to #808080) and circle was linear gray (should be 0.5f in linear space, equivalent to #bcbcbc); tertiary colors also use the same logic.

[BUG] Can confirm with the Tev viewer. Will dig deeper into this.
Comment 3 Geekley 2022-12-09 07:48:26 UTC
> > - Sometimes vector layers aren't exported.
> [BUG] I was able to consistently reproduce the latter error with your files, but only when not flattening the files.
Did you test both GUI and command line? I ran a quick test here:
`flatpak run org.kde.krita --export CMYK-8-cp.tif.kra --export-filename CMYK-8-cp.tif`
and I got the inconsistency I said before:
> 6. Sometimes exporting from Krita GUI gives a different result from the command-line (bug!), but this result is still broken for one reason or another.

> This is correct and consistent with `Store Alpha Channel (transparency)`  being unset by default on your end.
I'm a bit confused because I thought I always enable alpha. So I just tested TIFF exportation and I found something really weird in the GUI. "Flatten the image" was checked and "Store alpha" was unchecked and disabled/locked. When I uncheck flatten, alpha becomes checked and still locked. When I check flatten again, alpha becomes checked and UNLOCKED! This has to be a bug and might have caused either the issue itself or some confusion in my testing. I don't know if it's always like this or if it happens on some specific case.

> It'd be great if you could send the affected files here plus previews of each, to understand what the graphical glitches are.
I'll add just the screenshots for now (little time). The "broken" folder is from the original files I sent. The "broken-retry" is what I tried again now, double-checking that it should use flatten image (in case it didn't do that the first time); both were generated from that command line script.
Comment 4 Geekley 2022-12-09 07:50:55 UTC
Created attachment 154449 [details]
Dolphin previews of original sent files from "broken" folder
Comment 5 Geekley 2022-12-09 07:53:13 UTC
Created attachment 154450 [details]
Dolphin previews of "broken", retrying script now to ensure flatten image
Comment 6 Geekley 2022-12-09 08:10:02 UTC
Created attachment 154452 [details]
Exported files after retrying script now to ensure flatten image

I kept them in the same folder as the originals. I hope I did it right.
Comment 7 amyspark 2022-12-09 10:43:24 UTC
> This has to be a bug and might have caused either the issue itself or some confusion in my testing. I don't know if it's always like this or if it happens on some specific case.

Yes, that's a bug I found first hand when trying your files here. I'll try and commit the first set of patches (this issue and the EXR grayscale import/export) today so you can try with the nightlies of tonight.
Comment 8 amyspark 2022-12-10 20:54:37 UTC
(In reply to amyspark from comment #2)
> [BUG] I was able to consistently reproduce the latter error with your files,
> but only when not flattening the files. Something's failing inside LibTiff
> and not in our code. I'll try and debug it on my end.

We recently changed the code so that all file accesses went through Qt, in order to support Android's content URLs. We had not noticed that LibTiff actually requires simultaneously read and write mode to save multi-page TIFFs, so we set it as write-only.

I'll clean up the code while at it, as it'll help be more maintainable long-term.
Comment 9 Bug Janitor Service 2022-12-11 18:53:24 UTC
A possibly relevant merge request was started @ https://invent.kde.org/graphics/krita/-/merge_requests/1671
Comment 10 amyspark 2022-12-12 15:25:18 UTC
Git commit 179a85f477627256aa2db22ac6ef8f7b6cb54b63 by L. E. Segovia.
Committed on 12/12/2022 at 15:23.
Pushed by lsegovia into branch 'master'.

TIFF: fix LibTiff requiring read/write access to write multi-page files

M  +4    -3    plugins/impex/tiff/kis_tiff_export.cc

https://invent.kde.org/graphics/krita/commit/179a85f477627256aa2db22ac6ef8f7b6cb54b63
Comment 11 amyspark 2022-12-12 15:27:41 UTC
Git commit 229cca731c977cbbc2914da45f3d3a4eb4edbccc by L. E. Segovia.
Committed on 12/12/2022 at 15:27.
Pushed by lsegovia into branch 'krita/5.1'.

TIFF: fix LibTiff requiring read/write access to write multi-page files
(cherry picked from commit 179a85f477627256aa2db22ac6ef8f7b6cb54b63)

M  +4    -3    plugins/impex/tiff/kis_tiff_export.cc

https://invent.kde.org/graphics/krita/commit/229cca731c977cbbc2914da45f3d3a4eb4edbccc
Comment 12 Geekley 2022-12-20 22:07:51 UTC
> I'll try and commit the first set of patches (this issue and the EXR grayscale import/export) today so you can try with the nightlies of tonight.
Cool! I've tried 5.1.4 and I can see the checkbox issue was fixed and the green issue too. Files are still broken (mostly because of the vector layers), but there has been improvements.

Some EXR files (e.g. GrayA-16-linear.exr) that were opening as fully transparent in Krita before (when preview was green), now at least show something, but colors that should be grayscale currently appear as fully white in Krita (but are grayscale in Gwenview, GIMP, Dolphin previews). So, it's not just writing, there must be some additional issue in reading EXR files too.
Comment 13 Geekley 2022-12-20 22:11:31 UTC
Created attachment 154718 [details]
Example file that Krita reads incorrectly (fully white issue)
Comment 14 amyspark 2022-12-27 02:52:30 UTC
Git commit ee625a5f0dc63eb23f5a9e913837d01e76921d0e by L. E. Segovia.
Committed on 27/12/2022 at 02:52.
Pushed by lsegovia into branch 'master'.

EXR: fix opening files with only luma channel

M  +23   -5    plugins/impex/exr/exr_converter.cc

https://invent.kde.org/graphics/krita/commit/ee625a5f0dc63eb23f5a9e913837d01e76921d0e
Comment 15 amyspark 2022-12-27 02:53:16 UTC
Git commit 132211ddd13a34a56ab46c5c89282b63a955f4fc by L. E. Segovia.
Committed on 27/12/2022 at 02:53.
Pushed by lsegovia into branch 'krita/5.1'.

EXR: fix opening files with only luma channel
(cherry picked from commit ee625a5f0dc63eb23f5a9e913837d01e76921d0e)

M  +23   -5    plugins/impex/exr/exr_converter.cc

https://invent.kde.org/graphics/krita/commit/132211ddd13a34a56ab46c5c89282b63a955f4fc
Comment 16 amyspark 2022-12-27 02:55:20 UTC
I'm working on supporting vector layers, at least by converting them to bitmaps. The code is, however, so old that it'll take me a while before I can show some reliable results.

I'll CC this bug once I have news; but as this bit is specifically a wishlist one, I've marked this bug as resolved.

Thanks for your testing and the heads up!
Comment 17 Geekley 2023-01-16 22:13:54 UTC
I've tested with 5.1.5 and, just a reminder, some issues here (other than the vector layer thing) still persist.

TIFF exportation is weirdly behaving differently when the file exists - it's not deterministic. Each time the file is overwritten, file size keeps increasing. This issue happens on both GUI and command-line exportation (I tested on this file RGBA-8-sRGB). I didn't test this before, but it seems plausible to assume this was introduced with the read-write mode thing (it doesn't happen on EXR). Though minor, I'd say it's still a bug.

It's still having inconsistent results between command-line and GUI exportation of the same file with (presumably*) same settings.
I tested RGBA-8-sRGB.tif.kra and when exporting from GUI, the file is perfect, even vector layers are OK.
When I had tested command-line though, at first on batch exportation (all-kra-convert.sh), I got broken files.
Then I tested again, exporting only this file:
    flatpak run org.kde.krita --export RGBA-8-sRGB.tif.kra --export-filename RGBA-8-sRGB-sh.tif
And the file wasn't broken this time, though it wasn't perfect like from GUI (some text** isn't shown). So, unless it has something to do with settings, there's inconsistency between GUI and command-line exports. I tried batch again but they weren't broken this time, so who knows what happened...
* (I say presumably because unfortunately there's no way to enforce/ensure specific exportation settings, so I expect it to use last used settings; a feature to set those via command-line/json would be great and also significantly help testing).
** May be related to this warning - though it's strange that the issue is only via command-line:
    krita.file: WARNING: 'font-size-adjust' SVG attribute is not supported!
Comment 18 Geekley 2023-01-16 22:35:11 UTC
First issue: Maybe it's easy to solve by just deleting the file before re-creating it on read-write mode or something like that.

Second issue: Actually, it might not be related to that warning after all; on closer inspection, this font-size-adjust attribute is on the text that's being shown correctly (imported/pasted from Inkscape svg), not on the one that is not rendered (Legend vector layer made in Krita). In any case, if this is more related to the vector layer support feature and out-of-scope, then ignore this part I guess.
Comment 19 amyspark 2023-01-17 12:35:43 UTC
> First issue: Maybe it's easy to solve by just deleting the file before re-creating it on read-write mode or something like that.

This is because libtiff's one of the few libraries that need to go through Qt's QIODevice to work in Android. IIRC, libtiff does not reuse IFDs on rewrite, so we'll have to open the file with truncation mode to keep it from ballooning.