Bug 317655

Are you able to provide a simple QIF file that demonstrates these points, obviously with personal details replaced.

(In reply to comment #1)
> Are you able to provide a simple QIF file that demonstrates these points,
> obviously with personal details replaced.

I don't know how to anonymize the QIF file. If you could help me do that, I will provide the file.

(In reply to comment #2)
> (In reply to comment #1)
> > Are you able to provide a simple QIF file that demonstrates these points,
> > obviously with personal details replaced.
> 
> I don't know how to anonymize the QIF file. If you could help me do that, I
> will provide the file.

I don't know if you are thinking of the anonymize feature within KMM, but that works from the internal structures and accounts within a KMM loaded file to produce an anonymized file.  That is not the case here, as the problem you see occurs during the actual file load.  Taking an anonymized file in this case would show only the import result, rather than the original QIF file.

 A QIF file is just a simple text file in a very simple structure.  An anonymized version would have to be the result of manual editing.  I suspect your file is not of a trivial size, and it may be that it would involve quite a bit of work to manually edit everything.

So, if you are able to identify the individual transaction, you could perhaps disguise that and send it, to me direct if you prefer, and I'll see if I am able to produce a file showing the problem.  Every transaction will be enclosed  in "^" characters.  Include these too, so I know I have the whole transaction.

Really, in this case, it sounds as though Quicken have stepped outside their published file format, but that may not be the case.  So, we'll see what can be done.

> Secondly, KMyMony made duplicate empty categories for several subcategories.

Do you see any pattern to these?  Can you produce examples?

Created attachment 78547 [details]
QIF with mixed transfer and deposit in same transaction from Quicken

this file is a selected output from Quicken showing that it could list the deposit as including transfer and non-transfer. I.e. some of the deposit came from other accounts, e.g. cash deposited to bank, but other parts of the same transaction involved depositing checks which came from certain outside sources. KMyMoney threw away the non-transfer parts of such transactions.

Created attachment 78548 [details]
Just Categories output from Quicken

This is just the categories. In my case, some of the subcategories got duplicated when importing to KMyMoney.

(In reply to comment #3)
> (In reply to comment #2)
> > (In reply to comment #1)
> > > Are you able to provide a simple QIF file that demonstrates these points,
> > > obviously with personal details replaced.
> > 
> > I don't know how to anonymize the QIF file. If you could help me do that, I
> > will provide the file.
> 
> I don't know if you are thinking of the anonymize feature within KMM, but
> that works from the internal structures and accounts within a KMM loaded
> file to produce an anonymized file.  That is not the case here, as the
> problem you see occurs during the actual file load.  Taking an anonymized
> file in this case would show only the import result, rather than the
> original QIF file.
> 
>  A QIF file is just a simple text file in a very simple structure.  An
> anonymized version would have to be the result of manual editing.  I suspect
> your file is not of a trivial size, and it may be that it would involve
> quite a bit of work to manually edit everything.
> 
> So, if you are able to identify the individual transaction, you could
> perhaps disguise that and send it, to me direct if you prefer, and I'll see
> if I am able to produce a file showing the problem.  Every transaction will
> be enclosed  in "^" characters.  Include these too, so I know I have the
> whole transaction.
> 
> Really, in this case, it sounds as though Quicken have stepped outside their
> published file format, but that may not be the case.  So, we'll see what can
> be done.
> 
> > Secondly, KMyMony made duplicate empty categories for several subcategories.
> 
> Do you see any pattern to these?  Can you produce examples?

Files attached!

I've looked at every line in your qif file, but cannot see a problem.  Every split seems to go to the required account or category.  The only query really is with line 6 - "B756.75".  So far, I haven't found a description of the B prefix, or what the amount represents, assuming it is an amount.  I thought it might have been a balance, but, following  an !Account identifier,  the Statement balance amount prefix should be $.

Can you pinpoint a particular line in a particular transaction and say what you see as the problem?

(In reply to comment #7)
> I've looked at every line in your qif file, but cannot see a problem.  Every
> split seems to go to the required account or category.  The only query
> really is with line 6 - "B756.75".  So far, I haven't found a description of
> the B prefix, or what the amount represents, assuming it is an amount.  I
> thought it might have been a balance, but, following  an !Account
> identifier,  the Statement balance amount prefix should be $.
> 
> Can you pinpoint a particular line in a particular transaction and say what
> you see as the problem?

The specific issue is: when a particular transaction includes both a transfer and other category deposits in the same transaction, then importing into KMyMoney drops the non-transfer sections. In other words, KMyMoney doesn't bring an error message, but when I did the import and then looked at the transactions, the ones that had both a transfer and other categories were missing information.

Yes, I understand what your issue is, but, as I said, every line in the file appears to be handled correctly.
As I said before, 
> Can you pinpoint a particular line in a particular transaction and say what
> you see as the problem?

I've now also imported your category list and I see just one duplication, NUtilities:Telephone:Cell.  I don't see "several".  I'm using the development version, but have just tried with the stable 4.6.3, and get the same result, that same single duplication.

(In reply to comment #9)
> Yes, I understand what your issue is, but, as I said, every line in the file
> appears to be handled correctly.
> As I said before, 
> > Can you pinpoint a particular line in a particular transaction and say what
> > you see as the problem?
> 
> I've now also imported your category list and I see just one duplication,
> NUtilities:Telephone:Cell.  I don't see "several".  I'm using the
> development version, but have just tried with the stable 4.6.3, and get the
> same result, that same single duplication.

Well, I didn't see a clear pattern, but I got duplications of about half of the subcategories

> Well, I didn't see a clear pattern, but I got duplications of about half of
> the subcategories

I was working with a new kmy file.  Have you tried that, or can you?

(In reply to comment #11)
> > Well, I didn't see a clear pattern, but I got duplications of about half of
> > the subcategories
> 
> I was working with a new kmy file.  Have you tried that, or can you?

Yes I can, but I did. I'm new to KMyMoney so this import is all I ever did. I always started fresh and have never had any pre-existing KMyMoney files or data.

(In reply to comment #12)
> (In reply to comment #11)
> > > Well, I didn't see a clear pattern, but I got duplications of about half of
> > > the subcategories
> > 
> > I was working with a new kmy file.  Have you tried that, or can you?
> 
> Yes I can, but I did. I'm new to KMyMoney so this import is all I ever did.
> I always started fresh and have never had any pre-existing KMyMoney files or
> data.

OK.  I'd started to think, from this, and have just found that re-importing the same categories list file duplicates all sub-categories, but not the top level categories.  The duplicate cell phone one became a triple.

I'll take a look at that anyway.

Puzzled!!!

I've fixed the problem of duplicating sub-categories on reimporting.

However, there still is the problem of one particular sub-category being duplicated with just a single import.  Initially, both sub-categories have the same name, "Cell", and both appear to have the same id.  If I edit the name of the first one, to "CellA", that one keeps its name, but the second one now shows as "CellA".  If I delete the second one, it disappears, but if I now try to edit or delete the first one,  an error is flagged "Unable to modify account 'Cell'. Cause: Unknown account id 'A000092".  Similarly, after deleting the first one, the second one can't be edited/deleted.
Re-parenting one creates a new item.
I haven't as yet found where all this happens.

The problem of one particular sub-category being duplicated on import is clarified a bit.  That particular sub-category is the only third-level one.  I've now reproduced the issue on a another, imported, third-level one.  Manually created third-level ones appear OK.

I'm stuck, I'm afraid.
After loading the QIF file, and looking at AccountsModel::slotObjectAdded, the problem item appears only once.  However, in Categories view, it appears twice, also twice seen in AccountsViewFilterProxyModel, with a value of 0.00.  If I edit a transaction to use that category, only one of the two entries shows that value.  I can delete either entry, but not both.  "Unable to delete category wedB. Cause: Unknown account id 'A000010"

If I attempt to delete the one that links to the edited transaction, I'm told the transaction needs to be assigned a different category, but the drop down doesn't show the second version of the duplicate category.  Also, when editing a transaction to use the category, that drop down also shows only one of the duplicates.

So, it appears the category view is wrong, but elsewhere, the category is OK.

I've spent some time investigating, but am unable to see a cause.  Can anyone assist, please?  Thanks.

I think I have sussed this, but really need a second opinion as this is my first brush with models.
I'm importing this simple QIF file, taken from the OP's Quicken file.

!Type:Cat
NUtilities:Telephone:Cell
DCell Phone
E
^

The structure is imported correctly, except that the lowest level sub-category gets duplicated, or so it appears in Categories view.  Elsewhere in KMM, all looks OK.  Adding a fourth level results in two third level items, the first having two fourth level subs, and the second having one.

What I think is happening is that, when the 'N' line is processed, the whole structure is created, including all sub-categories.  Then, when the second level item is created, a further copy of its sub-category gets created.

I've traced this to accountsmodel.cpp, line74-76.

if (acc.accountList().count() > 0) {qDebug()<<"74 AccountsModel:loadSubAccounts"<<acc.accountList();
        loadSubAccounts(model, item, favoriteAccountsItem, acc.accountList());
}

I thought at first that manual entry was correct, but that was because I was creating each level separately.  Creating all three levels in one action produced the same effect.

There was a further, separate, problem, in that, if the OP's complete QIF file was re-imported, there was wholesale duplication of categories.  This, I fixed with -

@@ -906,13 +908,19 @@ void MyMoneyQifReader::processCategoryEntry(void)
   }
 
   // check if we can find the account already in the file
-  MyMoneyAccount acc = kmymoney->findAccount(account, MyMoneyAccount());
+  MyMoneyAccount acc = kmymoney->findAccount(account, parentAccount);

Hopefully, I'm not overlooking something.

> Hopefully, I'm not overlooking something.

Sadly, I was.  More investigation needed.

(In reply to comment #18)
> > Hopefully, I'm not overlooking something.
> 
> Sadly, I was.  More investigation needed.

The original proposed fix did fix the initial duplication of sub-accounts on their creation.  However, it caused another problem, with existing third and subsequent level categories being dropped in Categories view.

So, the original fix was removed, allowing loading of sub-categories again.  Instead, 
void loadSubAccounts(QStandardItemModel *model, QStandardItem *accountsItem, QStandardItem *favoriteAccountsItem, const QStringList& list) was changed, to detect attempts to load again a sub-category which had already been loaded.

Overall, the change is as follows -
 kmymoney/converter/mymoneyqifreader.cpp |    2 +-
 kmymoney/models/accountsmodel.cpp       |   21 ++++++++++++++++++---
 kmymoney/models/accountsmodel.h         |    1 +
 3 files changed, 20 insertions(+), 4 deletions(-)

So, it's not a huge change, but this is my first involvement with models, and it may be that there is a better way.

I'll publish the patch here, but let me know if Reviewboard would be better.

Created attachment 78824 [details]
Patch to fix Categories view sub-category duplication issues.

In deed, Reviewboard is better suited for this. You can even attach the reviewboard entry to this bug when creating it. A first question though: any reason why you made addedCategoriesList a static?

See https://git.reviewboard.kde.org/r/110022/ .

Initially, during this recursive process, the categories have not yet been added to the KMM file, so I add them to a new addedCategoriesList, which is scanned on each pass, to avoid this duplication.  Because the process is recursive, I made the list static, to ensure the higher levels are retained between passes when descending the tree.

Git commit 593973cd5c6237db698efd27fcc2290309f48da4 by Allan Anderson.
Committed on 11/04/2013 at 23:01.
Pushed by allananderson into branch 'master'.
categories. Also, when manually creating sub-categories, some may appear two
or more times in Categories view.
Remove redundant duplicateCategory flag, as per ReviewBoard.
Change addedCategoriesList to AccountsModel::Private and reference it through
the d-pointer.  Fixed minor style issue.

M  +1    -1    kmymoney/converter/mymoneyqifreader.cpp
M  +19   -0    kmymoney/models/accountsmodel.cpp

http://commits.kde.org/kmymoney/593973cd5c6237db698efd27fcc2290309f48da4

The bug has been reopended because the related patch introduces a new bug, see blocked bug.

Git commit 6c0582fefcb1f0e9ae134fa3d880b2258c8d8991 by Ralf Habacker.
Committed on 25/06/2017 at 20:32.
Pushed by habacker into branch '4.8'.

Fix QIF debug import.

MyMoneyQifReader::startImport() needs to trigger data processing and
return true to not abort the import.

M  +3    -3    kmymoney/converter/mymoneyqifreader.cpp

https://commits.kde.org/kmymoney/6c0582fefcb1f0e9ae134fa3d880b2258c8d8991

Git commit 86ad993c5e9c4db198618f382dd7f28f67be9654 by Ralf Habacker.
Committed on 26/06/2017 at 09:10.
Pushed by habacker into branch '4.8'.

Fix implementation of KMyMoneyApp::findAccount()

If no parent account is provided KMyMoneyApp::findAccount()
did a search only from the first parent in the local generated
list of parent accounts.

Instead it should search all top level accounts which is
required to be in sync with createAccount() used by qif importer.
FIXED-IN:4.8.1

M  +6    -1    kmymoney/kmymoney.cpp
M  +14   -0    kmymoney/kmymoney.h

https://commits.kde.org/kmymoney/86ad993c5e9c4db198618f382dd7f28f67be9654