Bug 59510 - Handles large spreadsheets "extremely" poorly
Summary: Handles large spreadsheets "extremely" poorly
Status: RESOLVED FIXED
Alias: None
Product: calligrasheets
Classification: Applications
Component: general (show other bugs)
Version: 1.2.90
Platform: OpenSUSE Linux
: NOR normal with 58 votes (vote)
Target Milestone: ---
Assignee: Laurent Montel
URL:
Keywords:
: 55872 113530 142572 (view as bug list)
Depends on:
Blocks:
 
Reported: 2003-06-08 13:33 UTC by Sean Clarke
Modified: 2007-07-28 09:39 UTC (History)
5 users (show)

See Also:
Latest Commit:
Version Fixed In:


Attachments
CSV testcase (390.62 KB, text/plain)
2006-02-13 07:09 UTC, Philippe Rigault
Details
Mortgage amortization (27.28 KB, application/x-kspread)
2006-12-16 06:43 UTC, Krishna Sethuraman
Details

Note You need to log in before you can comment on or make changes to this bug.
Description Sean Clarke 2003-06-08 13:33:05 UTC
Version:           1.2.90 (using KDE KDE 3.1.2)
Installed from:    SuSE RPMs
OS:          Linux

When loading a large spreadsheet (native format) into Kspread, Kspread uses a massive amount of memory (310MB) and takes a huge amount of time to open (7 minutes).

StarOffice (Linux and Windows) and Excel opens file in seconds.

May be related to bug 55872, but this is using a native format file, file is only just a 1MB in size.
Comment 1 Sean Clarke 2003-06-08 13:41:01 UTC
I can't attach the Kspread document as it is 1.3MB in size (there is an upload limit of 
1MB). 
 
If you require the file, send me an email or an ftp location (I can put it on a server 
and you can download it via HTTP if you wish?). 
 
I also have the file in CSV, StarOffice and Excel formats if that is any use. 
Comment 2 Ariya Hidayat 2003-06-10 00:56:59 UTC
Halo Sean,

KSpread is known to have problem with large files. We are working on that, but
not for the current release.

About the test file, yes I can download the file given the URL. The other
solution would be to cut part of the file (until it fits in 1 MB or less) and
attach it here. I guess 1 MB or 1.3 MB would not make any big difference.
Comment 3 Sean Clarke 2003-06-10 01:44:13 UTC
The test file can be found at : 
http://homepage.ntlworld.com/sean.e.clarke/Master_Database_9256.ksp 
 
Just bear in mind that when you open it Kspread "may" want up to 310MB of RAM. 
 
Comment 4 Sean Clarke 2003-08-29 00:43:10 UTC
Just tested this again with KOffice 1.3 b3 (1.2.92), problem still present.  
Comment 5 Sean Clarke 2004-02-01 00:39:13 UTC
Problem still present in KOffice 1.3 release - now uses 336MB.
Don't quite understand how a ~1.3MB file can cause an application to use >330MB of RAM. Application doesn't crash, once loaded it works as normal albeit slower.
Comment 6 Nicolas Goutte 2004-09-05 16:19:47 UTC
*** Bug 55872 has been marked as a duplicate of this bug. ***
Comment 7 Sean Clarke 2004-12-13 00:22:03 UTC
Still present in 1.3.5
Comment 8 Inge Wallin 2005-05-18 18:55:34 UTC
Cannot find the file when I wanted to test it.  Shall I close the bug now?  :-)
Comment 9 Tomas Mecir 2006-02-06 13:09:43 UTC
*** Bug 113530 has been marked as a duplicate of this bug. ***
Comment 10 Philippe Rigault 2006-02-13 07:09:09 UTC
Created attachment 14666 [details]
CSV testcase

Simple testcase: CSV file with 10,000 rows x 10 columns, all cells set to '1'

Loads instantly (<1 sec) in OpenOffice.org
Hogs CPU for ages (23 min !) in kspread. It does not eat up much RAM though, so
this is clearly not system-limited in this case.

Timings one on a Dual Opteron 2.0 GHz with 4GB RAM.

This is not a 'Normal' bug. It is a grave one that makes kspread unusable for
most academic and scientific users, who routinely exchange thousands of lines
of data in text/CSV.
Comment 11 Philippe Rigault 2006-02-13 07:17:38 UTC
Note following comment#10: Once the spreadsheet is loaded, exporting to CSV is instantaneous. 
Comment 12 Jo Øiongen 2006-04-17 14:11:52 UTC
Is this bug related to bug 85372?
Comment 13 Ernest ter Kuile 2006-11-26 16:15:56 UTC
kspread 1.6.0 (KDE 3.5.5, Debian Package 4:3.5.5a.dfsg.1-3 (4.0))
Linux (x86_64) release 2.6.18.3

Simple testcase described above does not load instantly, but does within 3 seconds. 
That makes it acceptable, no?

Timing is on AMD64 3000 with 1GB 
Comment 14 celer 2006-11-26 17:11:14 UTC
Hmm, probably so. I wonder if the bug got fixed or if the hardware got 
faster.

Thanks,

celer

+--[celer]--+

On Sun, 26 Nov 2006, Ernest ter Kuile wrote:

[bugs.kde.org quoted mail]
Comment 15 Philippe Rigault 2006-11-27 22:36:28 UTC
Indeed, there is an improvement with regard to the problem described in comment #10 (tested with kspread 1.6.1).

Here are my measurements for loading the CSV testcase on a Sun Ultra20 (DualCore AMD Opteron 2.4GHz, 2GB RAM):
 2 sec: Time from opening the file to the "Import" menu
 8 sec: Time from starting import to fully loaded document

Note that this is still much (5x) slower than OpenOffice.org-calc (respective timings are 0 sec and 2 sec).

There is still a _big_ problem with regard to memory usage in kspread.
Look at pmap measurements (http://web.hexapodia.org/~adi/pmap.c)

Kspread, when started with an empty sheet, uses the following memory:
  mapped:   269708 KB writable/private: 9412 KB shared: 1928 KB
and when the CSV testcase is loaded shows:
  mapped:   621012 KB writable/private: 350912 KB shared: 1928 KB
which is an increase of 340 MB in private memory.

In comparison:
OpenOffice empty sheet (look at this hog, 90MB for an empty sheet)
  mapped:   534084 KB writable/private: 90768 KB shared: 12644 KB
OpenOffice loaded with CSV testcase
  mapped:   659616 KB writable/private: 109000 KB shared: 12940 KB
which shows an increase of only 19 MB.

If I load a 30,000 row x 30 columns CSV document, the numbers are:


Kspread loads in 36 sec (8 sec + 28 sec) memory increases by 970 MB
  mapped:  1244076 KB writable/private: 973976 KB shared: 1928 KB

OpenOffice loads in 5 sec (0 sec + 5 sec) memory increases by 50 MB
  mapped:   703300 KB writable/private: 141048 KB shared: 12940 KB
Comment 16 Krishna Sethuraman 2006-12-16 06:43:36 UTC
Created attachment 18950 [details]
Mortgage amortization

Mortgage amortization
Comment 17 Krishna Sethuraman 2006-12-16 06:47:58 UTC
I can beat that.  See 'Mortgage amortization', open it in kspread, and just use the arrow keys to move around.  It's got about 200 rows, and moving to the bottom and then back around caused kspread to hang and eat 500M of memory on my system before I killed it.

This .ksp was generated using kspread 1.2 or 1.3, but it loads and displays data ok under my version of kspread.  But moving around it or changing the numbers at the top (initial amount, interest rate, monthly payment) seems to cause it to reproducibly hang.  The .ksp is all of 27kb, about 2Mb uncompressed.

Kubuntu 6.06 running kde 3.5.5 and koffice 1.6 as backports:
Qt: 3.3.6
KDE: 3.5.5
KSpread: 1.6.0
Comment 18 Philippe Rigault 2006-12-16 16:11:49 UTC
You seem to describe a different bug, since  'Mortgage amortization' loads quite quickly. It looks like this one has trouble with rendering its (many) formulas. 
Comment 19 Stefan Nikolaus 2007-04-16 16:32:02 UTC
*** Bug 142572 has been marked as a duplicate of this bug. ***
Comment 20 Stefan Nikolaus 2007-07-28 09:39:22 UTC
Closing.

The state for the current development version:
The file in #10 loads fast enough and the memory consumption got reduced.
The file in #17 loads okay and memory consumption is okay.

Can't say anything about the reporter's file, because it's not accessible anymore.