Bug 387061

Also, pressing R on keyboard, to compose a reply does show the text of the message, eventually. It takes a very long time (minutes) to open the compositor window with the quoted original text (plain text) in it and until the window opens one of the CPU cores is constantly at 100 %

The test message is precisely located at
https://lists.opensuse.org/opensuse-factory/2017-11/msg00487.html

It's a 1.06MB text (or see the attachement here)

Confirmed here on the same configuration.

14:32:41 - kmail2(30224) - org.kde.pim.messagelist: MessageList::Core::View::mousePressEvent: Left hit with selectedIndexes().count() ==  3
14:32:41 - kmail2(30224) - org.kde.pim.messagelist: MessageList::Core::View::slotSelectionChanged: View message selected [ "[opensuse-factory] New Tumbleweed snapshot 20171117 released!" ]
14:32:41 - kmail2(30224) - org.kde.pim.kmail: KMReaderWin::setMessage: void KMReaderWin::setMessage(const Akonadi::Item &, MimeTreeParser::UpdateMode) QSplitter(0x164f310, name="splitter2")
14:32:41 - kmail2(30224) -  : Empty filename passed to function
14:32:41 - kmail2(30224) -  : Empty filename passed to function
14:32:41 - kmail2(30224) -  : Empty filename passed to function
14:32:41 - kmail2(30224) -  : Empty filename passed to function
14:32:41 - kmail2(30224) - org.kde.pim.mimetreeparser: MimeTreeParser::NodeHelper::setNodeUnprocessed: Node UNprocessed:  0x7fa7500d5bf0
14:32:41 - kmail2(30224) - org.kde.pim.mimetreeparser: MimeTreeParser::NodeHelper::setNodeUnprocessed: Node UNprocessed:  0x7fa7500d5bf0
14:32:41 - kmail2(30224) - org.kde.pim.mimetreeparser: MimeTreeParser::NodeHelper::setNodeDisplayedEmbedded: SET NODE:  0x7fa7500d5bf0 true
14:32:41 - kmail2(30224) - org.kde.pim.mimetreeparser: MimeTreeParser::NodeHelper::setNodeDisplayedEmbedded: SET NODE:  0x7fa7500d5bf0 true
14:32:41 - kmail2(30224) - org.kde.pim.mimetreeparser: MimeTreeParser::NodeHelper::setNodeDisplayedEmbedded: SET NODE:  0x7fa7500d5bf0 true
14:32:41 - kmail2(30224) - org.kde.pim.mimetreeparser: MimeTreeParser::NodeHelper::setNodeProcessed: Node processed:  "" "Content-Type: text/plain; charset=\"us-ascii\""
14:32:41 - kmail2(30224) - org.kde.pim.messageviewer.semantic: SemanticRenderer::render: ========================================= Semantic Rendering
14:32:41 - kmail2(30224) -  : Empty filename passed to function
14:32:41 - kmail2(30224) -  : Empty filename passed to function
14:32:41 - kmail2(30224) -  : Empty filename passed to function
[cut]
14:32:44 - kmail2(30224) -  : Empty filename passed to function
14:32:44 - kmail2(30224) -  : Empty filename passed to function
14:32:44 - kmail2(30224) - org.kde.pim.mimetreeparser: MimeTreeParser::NodeHelper::overallEncryptionState: 

  KMMsgEncryptionState: 78
14:32:44 - kmail2(30224) - org.kde.pim.mimetreeparser: MimeTreeParser::NodeHelper::overallSignatureState: 

  KMMsgSignatureState: 78
14:32:44 - kmail2(30224) - org.kde.pim.webengineviewer: WebEngineViewer::WebEngineView::relativePosition: Relative Position -1
14:32:44 - kmail2(30224) - org.kde.pim.gravatar: Gravatar::GravatarCache::loadGravatarPixmap:  hashStr "e479a3497689cc53440f813115e566c3b8ae2aa06f49fd2939920d760a381b15"

(and the content isn't displayed)

As I learned now from vkrause, there is a 2MB limit inside QtWEbEngine (http://doc.qt.io/qt-5/qwebengineview.html#setHtml). So we will have to rework the part that pushed the content to not use setHtml and use instead a file.

Confirmed for openSuse Leap 42.3

*** Bug 388451 has been marked as a duplicate of this bug. ***

> As I learned now from vkrause, there is a 2MB limit inside QtWEbEngine
> (http://doc.qt.io/qt-5/qwebengineview.html#setHtml). So we will have to
> rework the part that pushed the content to not use setHtml and use instead a
> file.

Or go back to using QtWebKit (ReBooted), which is better for just about anything but full-blown browsers anyway? ;)

*** Bug 375445 has been marked as a duplicate of this bug. ***

Still a problem in version 5.10.3, Qt5.12.2, KDE Frameworks 5.57.0. Any target version for the fix?

(In reply to Rodney Baker from comment #10)
> Still a problem in version 5.10.3, Qt5.12.2, KDE Frameworks 5.57.0. Any
> target version for the fix?

No there is no target version for the fix. This is a limitation by the way we load the mail into QtWebEngine.Someone needs to rewrite the code in messagelib, that we don't use QWebEngineView::setHtml anymore and use QWebEngineView::load instead. If anyone wants to dive into that. Feel free to ask.

I have no idea where or how to do that. I tried cloning the kmail source but I can find neither messagelib nor any instance (using grep -r) of a call to QWebEngineView::setHtml within kmail. My programming experience is limited to Visual Basic and Ruby, so this might be a step too far at the moment.

(In reply to Rodney Baker from comment #12)
> I have no idea where or how to do that. I tried cloning the kmail source but
> I can find neither messagelib nor any instance (using grep -r) of a call to
> QWebEngineView::setHtml within kmail. My programming experience is limited
> to Visual Basic and Ruby, so this might be a step too far at the moment.

kdepim is about 50 repos. The repo you search is https://cgit.kde.org/messagelib.git. 
And the file:
messageviewer/src/htmlwriter/webengineparthtmlwriter.cpp:l66
To build up a development environment look at: https://community.kde.org/KDE_PIM/Docker

On Monday, 13 May 2019 7:55:39 ACST Sandro Knauß wrote:
> https://bugs.kde.org/show_bug.cgi?id=387061
> 
> --- Comment #13 from Sandro Knauß <sknauss@kde.org> ---
> (In reply to Rodney Baker from comment #12)
> 
> > I have no idea where or how to do that. I tried cloning the kmail source
> > but I can find neither messagelib nor any instance (using grep -r) of a
> > call to QWebEngineView::setHtml within kmail. My programming experience
> > is limited to Visual Basic and Ruby, so this might be a step too far at
> > the moment.
> kdepim is about 50 repos. The repo you search is
> https://cgit.kde.org/messagelib.git.
> And the file:
> messageviewer/src/htmlwriter/webengineparthtmlwriter.cpp:l66
> To build up a development environment look at:
> https://community.kde.org/KDE_PIM/Docker

Thanks for the pointer. I'm a complete newbie when it comes to c++, so this 
may be beyond me, but I'll see how I go. So far what I see doesn't make a lot 
of sense, but I guess I'll have to compare the docs for the two methods and 
see what needs to change, first. Hopefully this doesn't turn out to be like 
peeling an onion (layer upon layer upon layer...).

2+ years is a long time for a regression of this magnitude to go unfixed. Is there any way to get a kdepim developer to look into this soon?

*** Bug 415051 has been marked as a duplicate of this bug. ***

*** Bug 378928 has been marked as a duplicate of this bug. ***

(In reply to Sandro Knauß from comment #5)
> As I learned now from vkrause, there is a 2MB limit inside QtWEbEngine
> (http://doc.qt.io/qt-5/qwebengineview.html#setHtml). So we will have to
> rework the part that pushed the content to not use setHtml and use instead a
> file.

Hi, if there is a limit of 2MB than I am wondering why is it not possible to show ASCII content of 1.3 MB. I think 1.3 is less than 2, isn't it?

See my bug report: https://bugs.kde.org/show_bug.cgi?id=415051

Something should be done because it is a bad user experience when the user clicks on a new email. There is no warning, no nothing. It is a peace of cake for other email clients to show large messages.

Qt's text APIs don't use bytes, but work on UTF-16 code points. The limit is actually 1 million of those and each uses 16 bit (2 byte).

(In reply to Christoph Feck from comment #20)
> Qt's text APIs don't use bytes, but work on UTF-16 code points. The limit is
> actually 1 million of those and each uses 16 bit (2 byte).

Thanks for this information.
You have mentioned "Qt's text API". Kwrite can show text-files larger then 50MB. Sorry I am a user. I am looking into solutions. By the way I have upgraded from Fedora 21 to 31. Something significant has changed in between because it wasn't a big deal for Kmail to show ASCII emails larger then 50MB.

KWrite/Kate doesn't use QtWebEngine. The limitation is there, probably to prevent minute-long hangs when trying to layout a 100 MB HTML file.

Git commit 991eb9c20286bdce2458d7dbc17765dd7b0d7b38 by Laurent Montel.
Committed on 13/01/2020 at 06:35.
Pushed by mlaurent into branch 'master'.

Fix Bug 387061 - Large messages don't display in the viewer pane (eg. New Tumbleweed snapshot 20171117 released!)

FIXED-IN: 5.14.0

M  +17   -1    messageviewer/src/htmlwriter/webengineparthtmlwriter.cpp
M  +2    -1    messageviewer/src/htmlwriter/webengineparthtmlwriter.h

https://commits.kde.org/messagelib/991eb9c20286bdce2458d7dbc17765dd7b0d7b38

Tested locally and now works as expected. Many thanks, Laurent!

*** Bug 419298 has been marked as a duplicate of this bug. ***

REGRESSION - this was fixed but the issue has recurred in version 5.14.1 (20.04.0).

(In reply to Rodney Baker from comment #26)
Confirmed. I see this regression as well.

Yep confirmed.
It created too many encoding problem so  I decided to reverse this patch.

So, is it possible to make it conditional i.e. below a certain size threshold, use the preferred method, but if a message is above that threshold, use the alternative (working but reverted) method from this patch?

it can be a good idea.
I will look at if it's possible.
it can break encoding for some big email but it's less critical as not be able to load it I think

A possibly relevant merge request was started @ https://invent.kde.org/pim/messagelib/-/merge_requests/3

Git commit 706aad16c5927cee5ff6a6bf6d608f8842c4e9db by Wolfgang Bauer.
Committed on 30/07/2020 at 08:31.
Pushed by mlaurent into branch 'release/20.08'.

Fix size threshold for "big email" codepath

The maximum size supported by QWebEngineView::setContent() is 2 MB, not 2 GB.
FIXED-IN: 5.15.0

M  +1    -1    messageviewer/src/htmlwriter/webengineparthtmlwriter.cpp

https://invent.kde.org/pim/messagelib/commit/706aad16c5927cee5ff6a6bf6d608f8842c4e9db

Sorry, it's taken a while for me to be able to verify on my system due to other issues, but I can confirm it is now working as expected. Thanks for your work - much appreciated!

Created attachment 134744 [details]
message not displayed

Looks like the issue is back:
Operating System: openSUSE Tumbleweed 20210108
KDE Plasma Version: 5.20.5
KDE Frameworks Version: 5.77.0
Qt Version: 5.15.2

(In reply to Axel Braun from comment #35)
> Looks like the issue is back:
The issue us not really "back". Rather, it was actually not *fully* fixed, for every possible email.

As proposed in comment#29, messagelib does use a temporary file now for larger emails, because setting the content with QWebEngineView::setContent() or QWebEngineView::setHTML() has a 2MB limit (see comment#5).

But the problem is that AFAICT it is impossible to know in advance how big the message exactly is *after* QWebEngine's internal processing.

To quote from the docs:
> Note: Content larger than 2 MB cannot be displayed, because setHtml() converts 
> the provided HTML to percent-encoding and places data: in front of it to
> create the URL that it navigates to. Thereby, the provided code becomes a URL
> that exceeds the 2 MB limit set by Chromium.

So apparently your mail is an "edge case" that is smaller than 2000000 bytes (the current threshold in messagelib), but that still becomes too large for QtWebEngine/Chromium after the percent-encoding done internally.

Larger emails, like the one mentioned in the title, do (still) display fine now though, because they are correctly detected as being too large.

It should be easy to fix for that particular mail by lowering the size limit, but the lower limit may still be too high for certain other emails. OTOH, setting it too low is not good either, as that way of displaying mails seems to cause problems with encoding (see comment#28).

But I just noticed that the docs also say:
> If the content is too large, the loadFinished() signal is triggered with success=false.
Maybe this could be exploited as trigger for using a temporary file instead of a size check?
Just an idea, I don't know if it's really possible to do that (it would probably be quite tricky to implement at least, I fear).

The bug has not been fixed as far as I can see. KMail 5.15.3 is available for Fedora 33. It doesn't work when I try to open ASCII-emails larger then 2 MB. I got ASCII emails from 30 MB up to 55 MB. The system starts to swap by opening such an email (8 GB RAM is installed and a CPU with 4 cores). All I can do is to delete this email as soon as KMail responds not to overload the system. This isn't a good user experience.
Is there anything I can do? Let me know if you still need more informtaion.

Additional information:

I have observed the system activity today by opening an ASCII-email by the size of 35 MB. The application "System Activity" shows me a CPU usage of 13% (QtWebEngineProcess) and RAM consumption of 4.317.684 K. I think this is massive, isn't it? Scrolling through the email is almost impossible.

A pure text email of 35Mb is massive too, but eating around 4Gb of RAM for that isn't just massive. That's what you meant, right, not 4Mb for displaying just the part that fits in your window (which would seem reasonable)? 4Gb, that's about 120 times the size of the email.

I knew KMail would become a powerkids' tool with the design decision to use a full-fledged-browser-in-a-widget (QtWebEngine) for rendering everything including pure text emails but I wasn't expecting this. Then again, I don't usually get this kind of long letters by email so I have no idea how any other MUA would handle them.

If the account you got this email on has a web/browser interface it would be interesting to estimate the resources an actual browser (GChrome or Chromium) would use to display it, for comparison.

It is working for me (plain text emails around 41MB - log files from an automated backup script), but it is very slow to load the email. The CPU fan does ramp up slightly when doing so, but at least it loads, and once it loads, scrolling is usable. 

By comparison, "View Message Source" uses almost no CPU or RAM, and displays the message pretty much instantly. 

Thunderbird displays the message absolutely fine, too, with no noticeable hit on system resources or CPU load.

(In reply to RJVB from comment #39)
> A pure text email of 35Mb is massive too, but eating around 4Gb of RAM for
> that isn't just massive. That's what you meant, right, not 4Mb for
> displaying just the part that fits in your window (which would seem
> reasonable)? 4Gb, that's about 120 times the size of the email.

Yes I mean 4.317.684 K (QtWebEngineProcess).

> I knew KMail would become a powerkids' tool with the design decision to use
> a full-fledged-browser-in-a-widget (QtWebEngine) for rendering everything
> including pure text emails but I wasn't expecting this. Then again, I don't
> usually get this kind of long letters by email so I have no idea how any
> other MUA would handle them.

I get this kind of long emails every day. This is the output from backup scripts executed by Cron.

> If the account you got this email on has a web/browser interface it would be
> interesting to estimate the resources an actual browser (GChrome or
> Chromium) would use to display it, for comparison.

Unfortunately this account doesn't have a Web-Interface, but I did a test on another account with Web-Interface.
The plain text has a size of 20 MiB.

In Comparison:
QtWebEngineProcess (KMail) is eating 1.609.232 K.
Firefox (Web-Interface) is eating 156.204 K.

Firefox can display this e-mail within 2 seconds and I can scroll very smooth through  the content by dragging the scrollbar.

(In reply to Attila from comment #41)
> Firefox can display this e-mail within 2 seconds and I can scroll very
> smooth through  the content by dragging the scrollbar.
Have you tried in Chromium though? (which is what QtWebEngine is based upon)

TBH, I think even QtWebKit would struggle with these large mails, according to my experiences with OBS build logs... (KHTML served better in this regard, but that's completely dead now)

Anyway, this is kind of a different problem than comment#35 (or comment#0).
I don't see how it could be fixed from the kmail side, except for not trying to display mails larger than a certain size at all. (or maybe just display it as plain text instead then, but that probably means a large rewrite of the code, and I'm not sure it's possible at all)

(In reply to Wolfgang Bauer from comment #42)

> Have you tried in Chromium though? (which is what QtWebEngine is based upon)

Current versions of Firefox tend to consume more memory than Chromium - in fact,I understand that the Firefox Quantum engine uses bits and pieces from all 3 open source web engines and picks the fastest one.

> TBH, I think even QtWebKit would struggle with these large mails

That would be simple to test in konqueror with the webkit backend, or Otter-Browser, and the rebooted QtWebKit. Or in epiphany.
1 caveat emptor: we don't actually know what clever tricks the web email interface pulls.

> Anyway, this is kind of a different problem than comment#35 (or comment#0).
> I don't see how it could be fixed from the kmail side, except for not trying
> to display mails larger than a certain size at all. (or maybe just display
> it as plain text instead then, but that probably means a large rewrite of
> the code, and I'm not sure it's possible at all)

- Don't dump the entire message into the rendering widget but use a paged approach
- wrapper code can be written that connects to different backends. If you don't want to support QtWebkit, fine, but there's also QTextBrowser which should be sufficient even for simple HTML emails (it's what Qt's assistant uses by default since whatever Qt version that obsoleted QtWebkit). I've done a compile-time version of such wrapper code for KDevelop's help browser (which supports both QWE and QWK). I think it should be possible at least to chose between backends as a startup option.

Out of curiosity: does that humongous QWE process exit when you close the message view, or do you have to quit KMail to recuperate all that RAM?

(In reply to RJVB from comment #43)
> > TBH, I think even QtWebKit would struggle with these large mails
> 
> That would be simple to test in konqueror with the webkit backend, or
> Otter-Browser, and the rebooted QtWebKit. Or in epiphany.
> 1 caveat emptor: we don't actually know what clever tricks the web email
> interface pulls.
I do use QtWebKit with konqueror, and opening large OBS logs (which are just text files) do take a while.

> > Anyway, this is kind of a different problem than comment#35 (or comment#0).
> > I don't see how it could be fixed from the kmail side, except for not trying
> > to display mails larger than a certain size at all. (or maybe just display
> > it as plain text instead then, but that probably means a large rewrite of
> > the code, and I'm not sure it's possible at all)
> 
> - Don't dump the entire message into the rendering widget but use a paged
> approach
> - wrapper code can be written that connects to different backends. If you
> don't want to support QtWebkit, fine, but there's also QTextBrowser which
> should be sufficient even for simple HTML emails (it's what Qt's assistant
> uses by default since whatever Qt version that obsoleted QtWebkit). I've
> done a compile-time version of such wrapper code for KDevelop's help browser
> (which supports both QWE and QWK). I think it should be possible at least to
> chose between backends as a startup option.
That's what I meant with a "large rewrite" amongst other things.
Please note that I'm not a kmail developer though.

> Out of curiosity: does that humongous QWE process exit when you close the
> message view, or do you have to quit KMail to recuperate all that RAM?
I think that the QWE process is reused.
I'm not sure though, nor if it frees the resources when you switch to a different mail (I think it does).

(In reply to Wolfgang Bauer from comment #44)

> I do use QtWebKit with konqueror, and opening large OBS logs (which are just
> text files) do take a while.

TBH, Konqueror is such a multipurpose tool that I have no idea what backend/engine/kpart it uses for rendering pure text.
Actually, when I open /var/log/syslog I get a view that suggest very strongly that the Kate text editor kpart is used. That is, if I open the file from within Konqueror. If I try to start konqueror with a file it will start but open the file in the system handler (Kate for text files), very funny.

> That's what I meant with a "large rewrite" amongst other things.
> Please note that I'm not a kmail developer though.

Neither am I, and you're right in assuming it could be more work than you'd expect.

> I'm not sure though, nor if it frees the resources when you switch to a
> different mail (I think it does).

In my experience that doesn't always mean the resources are available immediately, for RAM at least. But I digress...

From a user perspective, it's more than reasonable to expect KMail to display large plain text messages as well as other common MUAs do. Rather than using QtWebEngine to display plain text, shouldn't an efficient method such as the one employed by "View Source" be used for all text/plain MIME parts?

(In reply to RJVB from comment #45)
> (In reply to Wolfgang Bauer from comment #44)
> 
> > I do use QtWebKit with konqueror, and opening large OBS logs (which are just
> > text files) do take a while.
> 
> TBH, Konqueror is such a multipurpose tool that I have no idea what
> backend/engine/kpart it uses for rendering pure text.
I'm talking about opening a text file in the HTML renderer, whatever is the default.

Konqueror can of course use other kparts to display files (katepart e.g.), but that's not the topic here.

(In reply to Greg Rivers from comment #46)
> From a user perspective, it's more than reasonable to expect KMail to
> display large plain text messages as well as other common MUAs do. Rather
> than using QtWebEngine to display plain text, shouldn't an efficient method
> such as the one employed by "View Source" be used for all text/plain MIME
> parts?
I don't think so, no. Without a large rewrite anyway, as mentioned.

KMail does add HTML stuff to an EMail even if it's just plain text, such as the header, links to attachments, and so on.

(In reply to Wolfgang Bauer from comment #42)
> (In reply to Attila from comment #41)
> > Firefox can display this e-mail within 2 seconds and I can scroll very
> > smooth through  the content by dragging the scrollbar.
> Have you tried in Chromium though? (which is what QtWebEngine is based upon)

I have tried Google Chrome today. It is comparable to Firefox. Chrome displays the same e-mail in 2 seconds. There is just one difference. Chrome takes 15 seconds more to scroll down to the bottom of the e-mail. This would be acceptable as well. Chrome eats 347.676 K. This is not bad, is it?