Bug 308247 - KMymoney crashes on importing QIF
Summary: KMymoney crashes on importing QIF
Status: RESOLVED FIXED
Alias: None
Product: kmymoney
Classification: Applications
Component: general (show other bugs)
Version: 4.6.1
Platform: Debian stable Linux
: NOR crash
Target Milestone: ---
Assignee: KMyMoney Devel Mailing List
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2012-10-11 15:14 UTC by Denis
Modified: 2012-10-14 21:23 UTC (History)
1 user (show)

See Also:
Latest Commit:
Version Fixed In:


Attachments
attachment-2567-0.html (1.65 KB, text/html)
2012-10-11 22:08 UTC, Denis
Details
attachment-2567-1.dat (1 bytes, multipart/alternative)
2012-10-11 22:08 UTC, Denis
Details
powerni.qif (669 bytes, application/octet-stream)
2012-10-11 22:08 UTC, Denis
Details
attachment-19562-0.html (1.98 KB, text/html)
2012-10-12 17:55 UTC, Denis
Details
attachment-19562-1.dat (1 bytes, multipart/alternative)
2012-10-12 17:55 UTC, Denis
Details
uscurrent.qif (12.85 KB, application/octet-stream)
2012-10-12 17:55 UTC, Denis
Details
attachment-19987-0.html (2.91 KB, text/html)
2012-10-14 17:45 UTC, Denis
Details
attachment-19987-1.dat (1 bytes, multipart/alternative)
2012-10-14 17:45 UTC, Denis
Details
uscurrent6.qif (13.00 KB, application/octet-stream)
2012-10-14 17:45 UTC, Denis
Details

Note You need to log in before you can comment on or make changes to this bug.
Description Denis 2012-10-11 15:14:41 UTC
Application: kmymoney (4.6.1)
KDE Platform Version: 4.8.5 (4.8.5)
Qt Version: 4.8.1
Operating System: Linux 3.2.0-23-generic i686
Distribution: Linux Mint 13 Maya

-- Information about the crash:
I clicked Import on a QIF file produced by MoneyManagerEX application. Linux Mint 13.

The crash can be reproduced every time.

-- Backtrace:
Application: KMyMoney (kmymoney), signal: Segmentation fault
Using host libthread_db library "/lib/i386-linux-gnu/libthread_db.so.1".
[KCrash Handler]
#7  0x00335d42 in AlkValue::AlkValue(QString const&, QChar const&) () from /usr/lib/libalkimia.so.4
#8  0x0012456d in MyMoneyMoney::MyMoneyMoney (this=0xbffeb740, pszAmount=...) at /build/buildd/kmymoney-4.6.1/kmymoney/mymoney/mymoneymoney.cpp:114
#9  0x083ce5a5 in MyMoneyQifProfile::value (this=0xb327ccc, def=..., valuein=...) at /build/buildd/kmymoney-4.6.1/kmymoney/converter/mymoneyqifprofile.cpp:712
#10 0x083e9871 in MyMoneyQifReader::processTransactionEntry (this=0xb327ca8) at /build/buildd/kmymoney-4.6.1/kmymoney/converter/mymoneyqifreader.cpp:1187
#11 0x083ebae8 in MyMoneyQifReader::processQifEntry (this=0xb327ca8) at /build/buildd/kmymoney-4.6.1/kmymoney/converter/mymoneyqifreader.cpp:659
#12 0x083ec443 in MyMoneyQifReader::slotProcessData (this=0xb327ca8) at /build/buildd/kmymoney-4.6.1/kmymoney/converter/mymoneyqifreader.cpp:386
#13 0x083ec68d in MyMoneyQifReader::qt_static_metacall (_o=<optimized out>, _id=<optimized out>, _a=<optimized out>, _c=<optimized out>) at /build/buildd/kmymoney-4.6.1/obj-i686-linux-gnu/kmymoney/converter/mymoneyqifreader.moc:64
#14 0x04580c01 in QMetaCallEvent::placeMetaCall (this=0x92d0998, object=0xb327ca8) at kernel/qobject.cpp:525
#15 0x04589c7b in QObject::event (this=0xb327ca8, e=0x92d0998) at kernel/qobject.cpp:1195
#16 0x01551ed4 in notify_helper (e=0x92d0998, receiver=0xb327ca8, this=0x8efa238) at kernel/qapplication.cpp:4559
#17 QApplicationPrivate::notify_helper (this=0x8efa238, receiver=0xb327ca8, e=0x92d0998) at kernel/qapplication.cpp:4531
#18 0x0155730d in QApplication::notify (this=0x92d0998, receiver=0xb327ca8, e=0x92d0998) at kernel/qapplication.cpp:4288
#19 0x00644e01 in KApplication::notify(QObject*, QEvent*) () from /usr/lib/libkdeui.so.5
#20 0x0456e97e in QCoreApplication::notifyInternal (this=0x8ef22e8, receiver=0xb327ca8, event=0x92d0998) at kernel/qcoreapplication.cpp:876
#21 0x04572ad8 in sendEvent (event=<optimized out>, receiver=<optimized out>) at ../../include/QtCore/../../src/corelib/kernel/qcoreapplication.h:231
#22 QCoreApplicationPrivate::sendPostedEvents (receiver=0x0, event_type=0, data=0x8ed2130) at kernel/qcoreapplication.cpp:1500
#23 0x04572e0c in QCoreApplication::sendPostedEvents (receiver=0x0, event_type=0) at kernel/qcoreapplication.cpp:1393
#24 0x045a1494 in sendPostedEvents () at ../../include/QtCore/../../src/corelib/kernel/qcoreapplication.h:236
#25 postEventSourceDispatch (s=0x8ef8b80) at kernel/qeventdispatcher_glib.cpp:279
#26 0x062c0d86 in g_main_context_dispatch () from /lib/i386-linux-gnu/libglib-2.0.so.0
#27 0x062c1125 in ?? () from /lib/i386-linux-gnu/libglib-2.0.so.0
#28 0x062c1201 in g_main_context_iteration () from /lib/i386-linux-gnu/libglib-2.0.so.0
#29 0x045a1887 in QEventDispatcherGlib::processEvents (this=0x8ed2ea0, flags=...) at kernel/qeventdispatcher_glib.cpp:424
#30 0x0160aaaa in QGuiEventDispatcherGlib::processEvents (this=0x8ed2ea0, flags=...) at kernel/qguieventdispatcher_glib.cpp:204
#31 0x0456d50d in QEventLoop::processEvents (this=0xbffec414, flags=...) at kernel/qeventloop.cpp:149
#32 0x0456d7a9 in QEventLoop::exec (this=0xbffec414, flags=...) at kernel/qeventloop.cpp:204
#33 0x04572eba in QCoreApplication::exec () at kernel/qcoreapplication.cpp:1148
#34 0x0154fa74 in QApplication::exec () at kernel/qapplication.cpp:3820
#35 0x0808a493 in runKMyMoney (splash=0xbffec568, a=<optimized out>) at /build/buildd/kmymoney-4.6.1/kmymoney/main.cpp:282
#36 0x080889db in main (argc=0, argv=0x0) at /build/buildd/kmymoney-4.6.1/kmymoney/main.cpp:181

Reported using DrKonqi
Comment 1 allan 2012-10-11 16:13:03 UTC
Created attachment 74485 [details]
attachment-2567-0.html

Importing of QIF files is very dependant on the actual structure of the file, because the QIF format is very loose.  Is MoneyManagerEX able to import the file?

What size is the file?
Is there the possibility that you could provide the file, or a simplified version that still shows the problem, with no personal info included?  Or, could you send the file to me, privately.
Comment 2 Denis 2012-10-11 22:08:14 UTC
Thank you,
The file size is 669 bytes.
MoneyManagerEX is able to import the file OK.
Attached is the file.


On Thu, Oct 11, 2012 at 9:13 AM, allan <agander93@gmail.com> wrote:

> https://bugs.kde.org/show_bug.cgi?id=308247
>
> allan <agander93@gmail.com> changed:
>
>            What    |Removed                     |Added
>
> ----------------------------------------------------------------------------
>                  CC|                            |agander93@gmail.com
>
> --- Comment #1 from allan <agander93@gmail.com> ---
> Importing of QIF files is very dependant on the actual structure of the
> file,
> because the QIF format is very loose.  Is MoneyManagerEX able to import the
> file?
>
> What size is the file?
> Is there the possibility that you could provide the file, or a simplified
> version that still shows the problem, with no personal info included?  Or,
> could you send the file to me, privately.
>
> --
> You are receiving this mail because:
> You reported the bug.
>
Comment 3 Denis 2012-10-11 22:08:15 UTC
Created attachment 74486 [details]
attachment-2567-1.dat
Comment 4 Denis 2012-10-11 22:08:15 UTC
Created attachment 74487 [details]
powerni.qif
Comment 5 allan 2012-10-12 09:53:12 UTC
(In reply to comment #4)
> Created attachment 74487 [details]
> powerni.qif

The problem relates to the '$' items in the file.  The '$' signifies the value of a split in the transaction, but these transactions have no split category or memo or percentage, so there seems no need for the '$' symbols.  Removing them all allows the file to be imported.  Most of them are in transactions with an 'M' item, which sometimes is empty, but there is one transaction with  '$' and no 'M'.

So, I don't see he rationale here, and, if it's important (ie., you have lots more similar files, you may need to contact MoneyManagerEX.

That said, KMM should not crash when handling them, and I'll look into that next.
Comment 6 allan 2012-10-12 10:55:31 UTC
@ Thomas
I need your guidance here, I'm afraid.

What constitutes a valid split?  The problem transactions here have a split amount but no 'S' split category, although they do have a 'L' category item.

I have eliminated the cause of the crash, in extractSplits(listqSplits), where the absence of the 'S' bypassed the creation of a listqSplits entry.  However, if I create a istqSplits entry on the presence of a '$' split amount, there is no split category so the imported transaction is flagged as having no category, which was present, but with an 'L' entry.

If the apparently useless '$' is edited out, these transactions import with the 'L' category assigned, so the user has no need of editing the transaction category.

So, to me, the best way would be to ignore (or flag) any split amount entry which isn't accompanied by a 'S' split category.

Any other ideas, please?
Comment 7 Thomas Baumgart 2012-10-12 12:31:36 UTC
@Allen
I would do the following (line numbers refer to current git master):

a) move the definition of QList<qSplit> listqSplits; right before the if statement on line 1135.
b) change the signature of extractSplits() to return a boolean
c) replace the if statement on line 1135 with the following

   if( ! extractSplits(listqSplits)) {

d) change the logic in extractSplits in such a way that it returns true only if KMyMoney finds all the information needed to process them. If one is missing, return false, which will in case process the L entry found. In case no L is present, the importer will create an unbalanced transaction.
Comment 8 allan 2012-10-12 12:57:31 UTC
Created attachment 74504 [details]
attachment-19562-0.html

(In reply to comment #7)
> @Allen
> I would do the following (line numbers refer to current git master):
> 
> a) move the definition of QList<qSplit> listqSplits; right before the if
> statement on line 1135.
> b) change the signature of extractSplits() to return a boolean
> c) replace the if statement on line 1135 with the following
> 
>    if( ! extractSplits(listqSplits)) {
> 
> d) change the logic in extractSplits in such a way that it returns true only
> if KMyMoney finds all the information needed to process them. If one is
> missing, return false, which will in case process the L entry found. In case
> no L is present, the importer will create an unbalanced transaction.

I wasn't expecting all that, but thanks.  It seems to work nicely, but I'll do some more testing, to make sure nothing else complains.
I've excluded the split memo from the test for needed items.
Comment 9 Denis 2012-10-12 17:55:40 UTC
I am not familiar with the structure of .qif files, but I will copy your
comments to MoneyManager (which incidentally cannot import split
transactions properly, even from its own qif files).

I am attaching a larger qif file which contains many more varied
transactions than the previous one. It is a US account but with UK style
date dd-mm-yyyy.


On Fri, Oct 12, 2012 at 2:53 AM, allan <agander93@gmail.com> wrote:

> https://bugs.kde.org/show_bug.cgi?id=308247
>
> --- Comment #5 from allan <agander93@gmail.com> ---
> (In reply to comment #4)
> > Created attachment 74487 [details]
> > powerni.qif
>
> The problem relates to the '$' items in the file.  The '$' signifies the
> value
> of a split in the transaction, but these transactions have no split
> category or
> memo or percentage, so there seems no need for the '$' symbols.  Removing
> them
> all allows the file to be imported.  Most of them are in transactions with
> an
> 'M' item, which sometimes is empty, but there is one transaction with  '$'
> and
> no 'M'.
>
> So, I don't see he rationale here, and, if it's important (ie., you have
> lots
> more similar files, you may need to contact MoneyManagerEX.
>
> That said, KMM should not crash when handling them, and I'll look into that
> next.
>
> --
> You are receiving this mail because:
> You reported the bug.
>
Comment 10 Denis 2012-10-12 17:55:41 UTC
Created attachment 74505 [details]
attachment-19562-1.dat
Comment 11 Denis 2012-10-12 17:55:41 UTC
Created attachment 74506 [details]
uscurrent.qif
Comment 12 allan 2012-10-13 16:25:01 UTC
(In reply to comment #9)
> I am not familiar with the structure of .qif files, but I will copy your
> comments to MoneyManager (which incidentally cannot import split
> transactions properly, even from its own qif files).
> 
> I am attaching a larger qif file which contains many more varied
> transactions than the previous one. It is a US account but with UK style
> date dd-mm-yyyy.

Another crash, again because of a split problem. --
D13/08/2012
T-23.97
PPaypal USA
N
LTransfer
MOBDII analyser, Fridge thermometers
$-23.97
SHome:Miscellaneous
$9.00
EHome:Miscellaneous 9.00
SCar:Maintenance
$14.97
ECar:Maintenance 14.97
^
I believe the $-23.97 split is the problem - it shouldn't be there.  The following six lines compose two good transactions, but the presence of the additional $-23.97 split line causes them to get mis-matched, resulting in the crash.  Removing that line allows the rest if the file to import without problem, the remaining splits being OK.

So far as the crash is concerned, again it shouldn't happen, and I think I've fixed this too.

I've needed to do some more tuning of extractSplits() to allow the unedited file to import, with just that one transaction flagged as unbalanced, and in need of user attention.

I still have a residual problem, with one of my test files, containing three splits, where the second split, which has no memo, picks up the memo from the preceeding split, but that should be simple, I think/hope, then more testing.
Comment 13 allan 2012-10-14 11:17:55 UTC
Created attachment 74541 [details]
attachment-19987-0.html

Git commit 81d6396a66534f2e95eac89aab19282575554f97 by Allan Anderson.
Committed on 14/10/2012 at 12:35.
Pushed by allananderson into branch 'master'.

M  +31   -17   kmymoney/converter/mymoneyqifreader.cpp
M  +1    -2    kmymoney/converter/mymoneyqifreader.h

http://commits.kde.org/kmymoney/81d6396a66534f2e95eac89aab19282575554f97
Comment 14 Denis 2012-10-14 17:45:34 UTC
You are right the  $-23.97 split should not be there, because a transfer
should not have a split. My mistake in entering it. I corrected all such
incorrect entries but it still crashes. Attached is the corrected file.
I wonder if there are some residual files causing this. I removed KMyMoney
and associated apps, and as many residual files in the KMyMoney folder
(there are some I am unable to delete). Reinstalled and still crashes.
I will try importing qif file with no transfers and no splits.

On Sat, Oct 13, 2012 at 9:25 AM, allan <agander93@gmail.com> wrote:

> https://bugs.kde.org/show_bug.cgi?id=308247
>
> --- Comment #12 from allan <agander93@gmail.com> ---
> (In reply to comment #9)
> > I am not familiar with the structure of .qif files, but I will copy your
> > comments to MoneyManager (which incidentally cannot import split
> > transactions properly, even from its own qif files).
> >
> > I am attaching a larger qif file which contains many more varied
> > transactions than the previous one. It is a US account but with UK style
> > date dd-mm-yyyy.
>
> Another crash, again because of a split problem. --
> D13/08/2012
> T-23.97
> PPaypal USA
> N
> LTransfer
> MOBDII analyser, Fridge thermometers
> $-23.97
> SHome:Miscellaneous
> $9.00
> EHome:Miscellaneous 9.00
> SCar:Maintenance
> $14.97
> ECar:Maintenance 14.97
> ^
> I believe the $-23.97 split is the problem - it shouldn't be there.  The
> following six lines compose two good transactions, but the presence of the
> additional $-23.97 split line causes them to get mis-matched, resulting in
> the
> crash.  Removing that line allows the rest if the file to import without
> problem, the remaining splits being OK.
>
> So far as the crash is concerned, again it shouldn't happen, and I think
> I've
> fixed this too.
>
> I've needed to do some more tuning of extractSplits() to allow the unedited
> file to import, with just that one transaction flagged as unbalanced, and
> in
> need of user attention.
>
> I still have a residual problem, with one of my test files, containing
> three
> splits, where the second split, which has no memo, picks up the memo from
> the
> preceeding split, but that should be simple, I think/hope, then more
> testing.
>
> --
> You are receiving this mail because:
> You reported the bug.
>
Comment 15 Denis 2012-10-14 17:45:35 UTC
Created attachment 74542 [details]
attachment-19987-1.dat
Comment 16 Denis 2012-10-14 17:45:35 UTC
Created attachment 74543 [details]
uscurrent6.qif
Comment 17 allan 2012-10-14 19:36:18 UTC
(In reply to comment #14)
> You are right the  $-23.97 split should not be there, because a transfer
> should not have a split. My mistake in entering it. I corrected all such
> incorrect entries but it still crashes. Attached is the corrected file.
> I wonder if there are some residual files causing this. I removed KMyMoney
> and associated apps, and as many residual files in the KMyMoney folder
> (there are some I am unable to delete). Reinstalled and still crashes.
> I will try importing qif file with no transfers and no splits.
> 

This file seems to have the same problem as your original - 
!Account
NUS Key Current 1610
TChecking
^
!Type:Cash
D25/01/2012
T0.15
PKey
N
LIncome:Interest
M
^
D27/01/2012
T-1,942.76
PPaypal USA
N
LTransfer
M
$-1,942.76
^
etc
 The second transaction, and many others, still has an unbalanced split - $-1,942.76, with no split category.  As a quick test, I started replacing all '$'  with 'M' and the import went much further.
Comment 18 allan 2012-10-14 21:23:58 UTC
Just a clarification.
When I said "...and the import went much further.", that was using the unpatched KMM, and it crashed beyond the section I'd edited.  Using the patched KMM, the file imported without crashing.