Bug 463788

Summary: Paint Layer with Transform Mask can't be flattened
Product: [Applications] krita Reporter: Halcyoen <halcyoen>
Component: Tools/TransformAssignee: Dmitry Kazakov <dimula73>
Status: RESOLVED FIXED    
Severity: normal CC: dimula73, halla
Priority: NOR Keywords: regression
Version First Reported In: 5.1.4   
Target Milestone: ---   
Platform: Microsoft Windows   
OS: Microsoft Windows   
Latest Commit: https://invent.kde.org/graphics/krita/commit/59d7555716f0f276c1aae5a1a9605650a7643890 Version Fixed/Implemented In:
Sentry Crash Report:
Attachments: file with layer and transform mask

Description Halcyoen 2023-01-03 18:09:14 UTC 
SUMMARY
when trying to flatten the layer that has a transform mask in the attached .kra file the operation never seems to complete. krita doesnt lock up per say but either it gets stuck in some kind of loop or something about transform masks makes this operation take an *unreasonably* long time to finish. 

i understand that non-destructible editing comes at a certain performance cost but not to this degree, even at larger dimensions.

STEPS TO REPRODUCE
1. open the attached file
2. right click the base layer and select "flatten layer"
Comment 1 Halcyoen 2023-01-03 18:12:51 UTC 
Created attachment 154995 [details]
file with layer and transform mask
Comment 2 Halcyoen 2023-01-03 18:14:32 UTC 
additionally, ive tested this on a number of older versions of krita and am able to reproduce the issue on all of them.
Comment 3 Halla Rempt 2023-01-04 09:29:15 UTC 
Thank you for the report. I can confirm the issue. It does look like there's an infinite loop or deadlock somewhere. Curiously, I can actually cancel the image, after which the operation restarts and gets to 97%, and then hangs again. The gui remains somewhat responsive.
Comment 4 Halla Rempt 2023-01-04 09:46:48 UTC 
I tested with 4.3.0, and there it flattened correctly, so this is a regression. With 4.4.8, the operation was much slower than with 4.3.0, but still worked correctly. The regression starts with 5.0.
Comment 5 Dmitry Kazakov 2023-01-19 12:52:01 UTC 
Hi, Halcyoen!

Could you please publish the example file again? Google says "the owner moved the file into trash".
Comment 6 Dmitry Kazakov 2023-01-20 09:43:28 UTC 
Git commit 59d7555716f0f276c1aae5a1a9605650a7643890 by Dmitry Kazakov.
Committed on 20/01/2023 at 09:39.
Pushed by dkazakov into branch 'master'.

Fix double extension of the update rect when flattening transform masks

M  +1    -1    libs/image/kis_layer_utils.cpp

https://invent.kde.org/graphics/krita/commit/59d7555716f0f276c1aae5a1a9605650a7643890
Comment 7 Halcyoen 2023-01-20 20:12:07 UTC 
my bad, i forgot that i had had to upload it as a drive link. the file has been restored and the link should work again.