Bug 286275 - QIF import consumes all memory and is killed by OS
Summary: QIF import consumes all memory and is killed by OS
Status: RESOLVED FIXED
Alias: None
Product: kmymoney
Classification: Applications
Component: general (show other bugs)
Version: 4.6.1
Platform: Gentoo Packages Linux
: NOR normal
Target Milestone: ---
Assignee: KMyMoney Devel Mailing List
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2011-11-10 16:00 UTC by Mac Michaels
Modified: 2011-11-13 09:37 UTC (History)
0 users

See Also:
Latest Commit:
Version Fixed In:
Sentry Crash Report:


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Mac Michaels 2011-11-10 16:00:37 UTC
Version:           4.6.1 (using KDE 4.6.5) 
OS:                Linux

Exported QIF file with account and Stock symbol information from Quicken.  
Imported this file to KMyMoney using the default built in database.  
KMyMoney showed a  progress bar that went to 100% then reset the bar and started on the next part of the import.  It ran for a while, the system slowed way down, then KMyMoney was killed by the OS.  

I changed KMyMoney to use MYSQL as the database.  
It does _not_ crash.
It took about 12 hours to complete the import of 50 categories, 30 accounts, only 2 transactions as the transaction data was not exported, and a very large number of stock prices for different dates on over 100 stocks.
This completed OK.

There appear to be 2 bugs.
1) KMyMoney does not manage memory properly when accessing the internal database.
2) QIF import uses a very poor algorithm somewhere as it runs far too slowly.  

Reproducible: Always

Steps to Reproduce:
1) Export a QIF from quicken with account information and stock information for data that has accumulated for about 15 years.  
2) Import QIF into a freshly installed KMyMoney.  


Actual Results:  
KMyMoney is killed by the OS after it uses all available memory.

Expected Results:  
Import of QIF should complete correctly.

NOTE: expected result was obtained when using a MYSQL database.


OS: Linux (x86_64) release 3.0.6-gentoo
Compiler: x86_64-pc-linux-gnu-gcc
Comment 1 Thomas Baumgart 2011-11-12 17:41:02 UTC
The cause of the problem is the vast amount of price entries in the QIF file: we're talking 81.500 (eighty one thousand and five hundred) of them. I believe it's the rollback mechanism of MyMoneyMap that causes the huge memory consumption.
Comment 2 Thomas Baumgart 2011-11-13 09:37:02 UTC
Git commit 6f28b1c9fa3c0684d184d3bfc1d638de933a1f91 by Thomas Baumgart.
Committed on 13/11/2011 at 10:31.
Pushed by tbaumgart into branch 'master'.

BUG: 286275

Don't create multiple rollback records in case the same object
gets modified more than once during the same transaction. Keeping
the state before the transaction started once is enough.

M  +38   -2    kmymoney/mymoney/storage/mymoneymap.h

http://commits.kde.org/kmymoney/6f28b1c9fa3c0684d184d3bfc1d638de933a1f91