Bug 113794 - new history viewer is slow
Summary: new history viewer is slow
Status: RESOLVED FIXED
Alias: None
Product: kopete
Classification: Applications
Component: History Plugin (show other bugs)
Version: unspecified
Platform: unspecified Linux
: NOR normal
Target Milestone: ---
Assignee: Kopete Developers
URL:
Keywords:
: 122141 122748 (view as bug list)
Depends on:
Blocks:
 
Reported: 2005-10-03 20:27 UTC by Mathieu Jobin
Modified: 2009-12-20 18:49 UTC (History)
4 users (show)

See Also:
Latest Commit:
Version Fixed In:


Attachments
Message (12.59 KB, text/plain)
2005-12-08 18:27 UTC, steffenp
Details
Fixed patch for regexp handling (765 bytes, patch)
2005-12-09 20:17 UTC, Tommi Rantala
Details
Performance improvement (3.73 KB, patch)
2006-03-26 09:28 UTC, Tommi Rantala
Details

Note You need to log in before you can comment on or make changes to this bug.
Description Mathieu Jobin 2005-10-03 20:27:24 UTC
Version:           0.10.92 (0.11 Beta 1) (using KDE 3.4.91 (beta1, >= 20050910), Gentoo)
Compiler:          gcc version 3.3.6 (Gentoo 3.3.6, ssp-3.3.6-1.0, pie-8.7.8)
OS:                Linux (i686) release 2.6.12-gentoo-r6

I would not say I dislike the new ViewHistory interface... its probably a good improovement... although, I liked the old one too and never complain about it. 

but searching in the new one IS MUCH SLOWER... old one was snappy, not kopete freeze for too many seconds on every search. I call it a regression.
Comment 1 Matthias Granberry 2005-10-03 23:15:37 UTC
It looks like the search also hangs forever(with 0% CPU used) when doing searches.  I have to re-click 'Search' in order to complete the search.  Is this what you're seeing?
Comment 2 Mathieu Jobin 2005-10-04 18:49:27 UTC
actually, i don't know which interface is better, but the old one allowed me to search for a specific quote in the conversation. the new one only give me on which day this quote has been said... and god knows how much you can talk with someone on one day. that you gotta read through it (ctrl-f is not even available)

the old interface could have just been modified to give a configurable context of 3, 5 or 10 lines. anyway... maybe hightlighting would help the new one ? 
Comment 3 Mathieu Jobin 2005-10-13 06:02:12 UTC
I think it is related to the History slowness because I have that option that get the last 7 message from the history when I open a new chat window.

it takes between 15 and 30 seconds for me to open a chat window, depending on who I'm clicking on. which seems to directly depends with how much discussion I had with this person. Even though I click on the same person three times in a row after closing the window, reopening it always take as long.

thanks

Comment 4 Marc Cramdal 2005-10-26 12:18:36 UTC
Matthias> ok, I will try to fix this little bug.

Mathieu> highlighting is already used to show the word you find. About slowness, do you speak of the slowness of the search ? or the slowness of the chatwindow opening. Even though history was modified, the way it loads the last history items in your chatwindow wasn't changed actually.
Comment 5 Matthias Granberry 2005-10-26 16:50:31 UTC
Actually, I already took a crack at in subversion but forgot to mention it here.  The signal used to start the search is the same as the one that cancels the search was being sent twice and the progress bar wasn't getting set to 'done' when a search was cancelled.  That buglet's fixed now.  I don't see exactly what Mathieu is complaining about.  Nothing's slow here but I only have a few months of light IM history on my primary computer.
Comment 6 Mathieu Jobin 2005-10-27 18:07:23 UTC
opening history window, doing a search and opening chat window. I'm very surprised loading method for last history message for chat windows has not changed. It's much slower.

would also be possible that my kopete is slower because I have a kolab addressbook ?


Comment 7 Tommi Rantala 2005-10-27 20:01:02 UTC
Just browsing the history becomes quite slow when the number (and length) of messages increase. Some IRC logs give me the following results:

77 messages => >0.27s to display
205 messages => >2.4s to display

kopete: BEGIN: void HistoryDialog::dateSelected(QListViewItem*)
kopete:   BEGIN: QValueList<Kopete::Message> HistoryLogger::readMessages(QDate)
kopete:   END__: QValueList<Kopete::Message> HistoryLogger::readMessages(QDate) - Took 0.07s
kopete:   BEGIN: void HistoryDialog::setMessages(QValueList<Kopete::Message>)
kopete:     msgs size=77
kopete:   END__: void HistoryDialog::setMessages(QValueList<Kopete::Message>) - Took 0.2s
kopete: END__: void HistoryDialog::dateSelected(QListViewItem*) - Took 0.27s


kopete: BEGIN: void HistoryDialog::dateSelected(QListViewItem*)
kopete:   BEGIN: QValueList<Kopete::Message> HistoryLogger::readMessages(QDate)
kopete:   END__: QValueList<Kopete::Message> HistoryLogger::readMessages(QDate) - Took 1.7s
kopete:   BEGIN: void HistoryDialog::setMessages(QValueList<Kopete::Message>)
kopete:     msgs size=205
kopete:   END__: void HistoryDialog::setMessages(QValueList<Kopete::Message>) - Took 0.67s
kopete: END__: void HistoryDialog::dateSelected(QListViewItem*) - Took 2.4s
Comment 8 steffenp 2005-12-08 18:17:50 UTC
I have also observed slowness when browsing the history (opening the window or clicking the previous button).

A did a little debugging and found out that kopetemessage.cpp spents lots of time in Message::setBody() on this statement:

theBody.replace( QRegExp( QString::fromLatin1(".*<body.*>\\s+(.*)\\s+</body>.*") ), QString::fromLatin1("\\1") );

If have a few messages (decompiled source code) in my history that stall kopete for _minutes_. Here ist the gdb bt:

#0  0xb6917b05 in QGArray::at () from /usr/lib/libqt-mt.so.3
#1  0xb69d72d8 in QMemArray<int>::operator[] () from /usr/lib/libqt-mt.so.3
#2  0xb6d09b6b in QRegExpEngine::matchHere () from /usr/lib/libqt-mt.so.3
#3  0xb6d0aee1 in QRegExpEngine::badCharMatch () from /usr/lib/libqt-mt.so.3
#4  0xb6d0b2e7 in QRegExpEngine::match () from /usr/lib/libqt-mt.so.3
#5  0xb6d0f04a in QRegExp::search () from /usr/lib/libqt-mt.so.3
#6  0xb6d18a4c in QString::replace () from /usr/lib/libqt-mt.so.3
#7  0xb7e4a960 in Kopete::Message::setBody (this=0xbfb248a8, body=@0xbfb24944,
    f=Kopete::Message::RichText) at kopetemessage.cpp:214
#8  0xb5b9e5f1 in HistoryLogger::readMessages () from /usr/lib/kde3/kopete_history.so

My system is Debian unstable, libqt3-mt 3.3.5-3, kopete 3.4.3-3.

I wrote a short test program that invokes the regexp on the same string but I was not able reproduce the behaviour, it took only seconds.
Comment 9 steffenp 2005-12-08 18:27:11 UTC
Created attachment 13826 [details]
Message

Here is a sample 349 line message. Send this to someone, close the chat window
and try to view the history. On my sytem it takes minutes until the history
window is displayed.
Comment 10 Matthias Granberry 2005-12-08 18:33:45 UTC
Thanks.  I've been doing performance tuning for the Real Job for the last few days, so I'll profile kopete too and see if I can get this taken care of.
Comment 11 Tommi Rantala 2005-12-08 20:28:50 UTC
Hi, try this:

Index: kopetemessage.cpp
===================================================================
--- kopetemessage.cpp   (revision 485489)
+++ kopetemessage.cpp   (working copy)
@@ -216,7 +216,7 @@
        {
                //This is coming from the RichTextEditor component.
                //Strip off the containing HTML document
-               theBody.replace( QRegExp( QString::fromLatin1(".*<body.*>\\s+(.*)\\s+</body>.*") ), QString::fromLatin1("\\1") );
+               theBody.replace( QRegExp( QString::fromLatin1("<body.*>\\s+(.*)\\s+</body>") ), QString::fromLatin1("\\1") );

                //Strip <p> tags
                theBody.replace( QString::fromLatin1("<p>"), QString::null );


The .* are really not needed, right? Cuts down processing time of #9 from 51s to 0.39s.
Comment 12 Matthias Granberry 2005-12-08 21:04:52 UTC
I don't think this is equivalent.  The first strips everything outside the body as well, while the second just strips the <body> </body> tags, so if the RichTextEditor component feeds us anything outside of the <body> tags, it won't get nuked.  If it doesn't put anything there then the new regexp is fine.

<html><body>text</body></html> would become <html>text</html>
Comment 13 steffenp 2005-12-08 22:29:54 UTC
Usability wise even 0.39 s per entry seems too long. I usually have my history size set to 200 items. Maybe replacing it by a short code snippet that uses QString::find("<body") would be more feasible?
Comment 14 Tommi Rantala 2005-12-09 20:17:04 UTC
Created attachment 13840 [details]
Fixed patch for regexp handling

#12: You're right, that change was not correct. This is what I intended. I've
just added some unit testing (libkopete/tests/) and it seems to work.
Comment 15 Brian Smith 2006-02-26 19:01:13 UTC
*** Bug 122141 has been marked as a duplicate of this bug. ***
Comment 16 Brian Smith 2006-02-26 19:02:28 UTC
*** Bug 122748 has been marked as a duplicate of this bug. ***
Comment 17 Marc Cramdal 2006-03-25 23:49:25 UTC
Tommi > How much does this patch improve speed ?
Comment 18 Tommi Rantala 2006-03-26 09:28:29 UTC
Created attachment 15311 [details]
Performance improvement

Here's an improved patch. Any realistic logs take about 15% processing time
compared to unpatched Kopete.

For example one log that took 3 seconds to load now takes about 0.45 seconds.
Comment 19 Chani 2006-03-28 23:56:47 UTC
> Here's an improved patch. Any realistic logs take about 15% processing time
> compared to unpatched Kopete.
>
> For example one log that took 3 seconds to load now takes about 0.45 seconds.


didn't make much difference to me. kopete normally takes about 26
seconds to load all history; with the patch it took 25.
Comment 20 Michael Frister 2006-05-23 15:46:22 UTC
*** This bug has been confirmed by popular vote. ***
Comment 21 Tommi Rantala 2006-07-27 16:06:00 UTC
SVN commit 566905 by rantala:

Commit the patch partially.

Reduces time taken by HistoryLogger::readMessages(QDate) from 25 seconds
to 0.5 seconds with a 200kb log file.

CCBUG: 113794


 M  +10 -2     kopetemessage.cpp  


--- trunk/KDE/kdenetwork/kopete/libkopete/kopetemessage.cpp #566904:566905
@@ -83,7 +83,11 @@
 	{
 		//This is coming from the RichTextEditor component.
 		//Strip off the containing HTML document
-		this->body.replace( QRegExp( QLatin1String(".*<body.*>\\s+(.*)\\s+</body>.*") ), QLatin1String("\\1") );
+		if (this->body.contains(QLatin1String("<body"))) {
+			QRegExp rx( QLatin1String("<body[^>]*>\\s+(.*)\\s+</body>") );
+			if (rx.indexIn(this->body) != -1)
+				this->body = rx.cap(1);
+		}
 
 		//Strip <p> tags
 		this->body.replace( QLatin1String("<p>"), QString::null );
@@ -192,7 +196,11 @@
 	{
 		//This is coming from the RichTextEditor component.
 		//Strip off the containing HTML document
-		theBody.replace( QRegExp( QLatin1String(".*<body.*>\\s+(.*)\\s+</body>.*") ), QLatin1String("\\1") );
+		if (theBody.contains(QLatin1String("<body"))) {
+			QRegExp rx( QLatin1String("<body[^>]*>\\s+(.*)\\s+</body>") );
+			if (rx.indexIn(theBody) != -1)
+				theBody = rx.cap(1);
+		}
 
 		//Strip <p> tags
 		theBody.replace( QLatin1String("<p>"), QString::null );
Comment 22 Mathieu Jobin 2006-07-29 11:53:33 UTC
sugoi, thank you very much
Comment 23 Iuri Fiedoruk 2006-11-15 23:56:44 UTC
I'm using Kopete 0.12.3 on Ubuntu Edgy and I'm still having problems with slow appearing chat windows because of history plugin. Any chance we got a release to fix this soon?
Comment 24 George Kiagiadakis 2008-07-23 13:14:45 UTC
I cannot reproduce the initial bug with KDE 4.0.99. Search is snappy enough, however clicking on a large conversation in the history viewer can take up to 3 seconds to show up, using a phenom 3-core processor @ 2.1 GHz. Not sure if this can be improved somehow...
Comment 25 Cyrill Helg 2008-08-24 20:46:13 UTC
I agree that the inital problem is solved, using kopete trunk version.