Bug 286275

Summary: QIF import consumes all memory and is killed by OS
Product: [Applications] kmymoney Reporter: Mac Michaels <thebitpit>
Component: generalAssignee: KMyMoney Devel Mailing List <kmymoney-devel>
Status: RESOLVED FIXED    
Severity: normal    
Priority: NOR    
Version: 4.6.1   
Target Milestone: ---   
Platform: Gentoo Packages   
OS: Linux   
Latest Commit: Version Fixed In:
Sentry Crash Report:

Description Mac Michaels 2011-11-10 16:00:37 UTC 
Version:           4.6.1 (using KDE 4.6.5) 
OS:                Linux

Exported QIF file with account and Stock symbol information from Quicken.  
Imported this file to KMyMoney using the default built in database.  
KMyMoney showed a  progress bar that went to 100% then reset the bar and started on the next part of the import.  It ran for a while, the system slowed way down, then KMyMoney was killed by the OS.  

I changed KMyMoney to use MYSQL as the database.  
It does _not_ crash.
It took about 12 hours to complete the import of 50 categories, 30 accounts, only 2 transactions as the transaction data was not exported, and a very large number of stock prices for different dates on over 100 stocks.
This completed OK.

There appear to be 2 bugs.
1) KMyMoney does not manage memory properly when accessing the internal database.
2) QIF import uses a very poor algorithm somewhere as it runs far too slowly.  

Reproducible: Always

Steps to Reproduce:
1) Export a QIF from quicken with account information and stock information for data that has accumulated for about 15 years.  
2) Import QIF into a freshly installed KMyMoney.  


Actual Results:  
KMyMoney is killed by the OS after it uses all available memory.

Expected Results:  
Import of QIF should complete correctly.

NOTE: expected result was obtained when using a MYSQL database.


OS: Linux (x86_64) release 3.0.6-gentoo
Compiler: x86_64-pc-linux-gnu-gcc
Comment 1 Thomas Baumgart 2011-11-12 17:41:02 UTC 
The cause of the problem is the vast amount of price entries in the QIF file: we're talking 81.500 (eighty one thousand and five hundred) of them. I believe it's the rollback mechanism of MyMoneyMap that causes the huge memory consumption.
Comment 2 Thomas Baumgart 2011-11-13 09:37:02 UTC 
Git commit 6f28b1c9fa3c0684d184d3bfc1d638de933a1f91 by Thomas Baumgart.
Committed on 13/11/2011 at 10:31.
Pushed by tbaumgart into branch 'master'.

BUG: 286275

Don't create multiple rollback records in case the same object
gets modified more than once during the same transaction. Keeping
the state before the transaction started once is enough.

M  +38   -2    kmymoney/mymoney/storage/mymoneymap.h

http://commits.kde.org/kmymoney/6f28b1c9fa3c0684d184d3bfc1d638de933a1f91