Bug 55872

Summary:	kspread slow to open large csv spreadsheets
Product:	[Applications] calligrasheets	Reporter:	celer
Component:	general	Assignee:	Laurent Montel <montel>
Status:	RESOLVED DUPLICATE
Severity:	normal	CC:	nicolasg
Priority:	NOR
Version:	1.2
Target Milestone:	---
Platform:	Mandrake RPMs
OS:	Linux
Latest Commit:		Version Fixed In:
Sentry Crash Report:

Description celer 2003-03-12 16:43:48 UTC

Version:           1.2 (using KDE 3.0.3)
Installed from:    Mandrake Linux Cooker i586 - Cooker
Compiler:          gcc version 3.2 (Mandrake Linux 9.0 3.2-1mdk)
OS:          Linux (i686) release 2.4.19-19mdkcustom

When opening a 1.8MB csv file in kspread it takes far too long. 

I have a 1.2Ghz 512MB laptop and it takes about 5-10 minutes to open a 1.8MB csv file.  When opening the same document in excell 97 on a 400Mhz celeron with 96MB ram it takes approximately 45 seconds. Once the file has been opened kspread is completely non-responsive when having to do anything. It appears that just redrawing takes about 5 minutes. Finally kspread consummed 1.3GB of memory in ram and swap.

The spread sheet was  50 columns by 17500 rows, and looked like so
TEXT, TEXT, 0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0...

Comment 1 Norbert Andres 2003-04-04 01:12:02 UTC

Subject: Re: kspread slow to open large csv spreadsheets

Hi, 

this bug is partly fixed in CVS Head: it still takes a long time for loading 
the data into KSpreads internal structure (it's faster than before though), 
but KSpread is now not "non-responsive" anymore.
The problem that loading such big files is really slow, is nothing that can be 
solved easily. We plan to do this rewrite of the internal structure of 
KSpread for 1.4. 

Regards
Norbert

Comment 2 Nicolas Goutte 2004-09-05 15:00:37 UTC

See also bug #59510

Comment 3 Nicolas Goutte 2004-09-05 16:16:01 UTC

CVS commit by goutte: 

Improve file reading:
- use directly QIODevice::readAll
- release memory when the input file itself is not needed anymore
  (should help with bug #55872 by giving a little more memory.)
CCMAIL:55872@bugs.kde.org


  M +3 -2      csvimport.cc   1.28


--- koffice/filters/kspread/csv/csvimport.cc  #1.27:1.28
@@ -1,4 +1,5 @@
 /* This file is part of the KDE project
    Copyright (C) 1999 David Faure <faure@kde.org>
+   Copyright (C) 2004 Nicolas GOUTTE <goutte@kde.org>
 
    This library is free software; you can redistribute it and/or
@@ -96,6 +97,5 @@ KoFilter::ConversionStatus CSVFilter::co
     //    csv_delimiter = config[0];
 
-    QByteArray inputFile(in.size());
-    in.readBlock(inputFile.data(), in.size());
+    QByteArray inputFile( in.readAll() );
     in.close();
 
@@ -103,4 +103,5 @@ KoFilter::ConversionStatus CSVFilter::co
     if (!dialog->exec())
         return KoFilter::UserCancelled;
+    inputFile.resize( 0 ); // Release memory (input file content)
 
     ElapsedTime t( "Filling data into document" );

Comment 4 Nicolas Goutte 2004-09-05 16:19:46 UTC

As the bug seems only to complain about the time after that the dialog is shown, this means that it is very similar to bug #59510 now.

Have a nice day!

*** This bug has been marked as a duplicate of 59510 ***