Bug 313793

A zipped anonymised KMyMoney file demonstrating the bug is located here:-

http://www.filefactory.com/file/6x4nlut2l7r3/n/Documents_anon_xml_zip

I have seen this before and I thought I had raised a bug report but as I can't find it I'll just add my comments here.
This may be caused by creating accounts and entering data retrospectively. Taking investment A000469 as an example. The date of 2008-10-31 reported by the checker is actually the opening date of the account as held on the <ACCOUNT> record in the field opened="2008-10-31". The transactions take place prior to this date as can be seen by looking at the entrydate field on the <TRANSACTION> records. The opening date for an investment (shares or mutual funds) seems to get set to today's date by KMyMoney when the investment is first added.
I have no idea why the checker insists on a price for the account opening date. Surely a price is only needed when a transaction takes place. I seem to remember I fixed this on my file by editing the XML and changing the opening date to be the date of the first transaction.

Perhaps the consistency checker should also note when there is a transaction with a date prior to the opening date of the account?  I don't know if it should automatically fix it (by changing the opening date?)  or just put out a warning.

Yes, I agree. Or maybe just change the opening date to the date of the first transaction, and issue an info message with the change.
That is because sometime during 0.9 to 1.0 we added a check to prevent from entering transactions prior to the opening date, but this is a very old file, where transaction had already been entered.
I'm surprised it doesn't have problems with reports, which is why we added the constraint.

"I have no idea why the checker insists on a price for the account opening date. "
I'm not sure it's right to say it 'insists' on a price.  The output message starts off "* Potential problem with investments/currencies", so, it's issuing a warning to the user about an inconsistency.  I think it's reasonable, when checking an investment account, to expect there to be prices.
In the case of the first four instances, there are no prices in the prices table, possibly the user deleted them when closing the accounts.
As these accounts have been closed, is it reasonable to perform such a check?  I think it probably is, as the  security and the transactions are still in the file, which needs to be consistent.
Then there is the question of the presence of dates prior to the opening date.  There is no check for a new price being entered prior to the opening date.  Should this be prohibited, or should the user be given the means to correct the date?

I'll have to check the algorithm, but perhaps the warning can be avoided if there is a check for any price, even if it is before the opening date.

Thinking again about "Should this be prohibited, or should the user be given the means to correct the date?", if entering an earlier date were stopped, even then the user would need the means to correct the date.

A transaction earlier than the opening date is now generally caught because the transaction can't be edited (or entered in the first place?)  As I understand it, this bug happened with a file from a very early version of KMM, before that restriction was added.  I think it is reasonable to change the opening date to the earliest transaction date in the account, with a notice to the user.  I suspect that if an account opening date is explicitly set by the user, it WILL be before any transaction, and the problem is allowing the default opening date, and then trying to back-fill an old account, in which case the change of opening date shouldn't be a problem for the user.

If I understand correctly the code in this area, it is involved in checking the prices list, not the actual transactions.  It certainly used to be that prices in transactions were not added to the prices list.  I cannot remember if this was from QIF import, which I'm sure I did fix, and/or from manual input. (I've just checked, and manually entered transactions do now have their price added.)
A price *can* be entered manually with a date before the opening date.  Oh, this is interesting, a transaction can be entered with a pre-opening date.  I didn't expect that.

It was the case that a transaction could be entered with a date prior to the opening date. You can't do that anymore, but we are dealing with old files here.

So, we have two possible fixes here: check for transactions prior to the opening date and verify if prices are entered for the date of those transactions, or fix the opening date to the date of the first transactions. We could even add both fixes.

I accidentally clicked Save then.  Apologies.

It is also possible to edit that transaction and then enter it, still with the pre-opening date.

[The following is off-topic]
HOWEVER, things go a bit strange.  After entering the edited transaction, it appears to be accepted, but the transaction does not close.  Switching then to another view seems to work, but it is not possible to switch back to Ledger view.  Then I noticed that the ledger account selection box is still visible.  If I then select a different account, Ledger view is now working.  I can return to the original account and the edited transaction appears normal. This is unrelated and I'll need to open a new bug.

Now, it is working correctly after an edit.  Oh, I see.  If I edit the quantity of shares and then click within the body of the transaction, but not in an editable field, the transaction does not close correctly.  If instead I click OK, then all is well.

(In reply to comment #10)
> It was the case that a transaction could be entered with a date prior to the
> opening date. You can't do that anymore, but we are dealing with old files
> here.
>
It seems you can.  See my comment 11.

> So, we have two possible fixes here: check for transactions prior to the
> opening date and verify if prices are entered for the date of those
> transactions, or fix the opening date to the date of the first transactions.
> We could even add both fixes.

It needs to be born in mind, too, that the opening transaction might not have a price,
for instance opening with an add shares transaction, which is not unusual.

(In reply to comment #8)
> I think it is reasonable to change
> the opening date to the earliest transaction date in the account, with a
> notice to the user.  I suspect that if an account opening date is explicitly
> set by the user, it WILL be before any transaction, and the problem is
> allowing the default opening date, and then trying to back-fill an old
> account, in which case the change of opening date shouldn't be a problem for
> the user.

But as I understand it, the user has no control over this opening date. It is set to today by KMyMoney when the stock is added. The opening date we are discussing is not the opening date of the investment account (which can be set by the user) It is an opening date which is set by KMyMoney for the stock or fund within th investment account.

Yes, that's my understanding too.

(In reply to comment #1)
> A zipped anonymised KMyMoney file demonstrating the bug is located here:-
> 
> http://www.filefactory.com/file/6x4nlut2l7r3/n/Documents_anon_xml_zip

Mike, just a small point here.  Lower down this page is a link which allows
adding attachments.  Easier to access.

"Add an attachment"
https://bugs.kde.org/attachment.cgi?bugid=313793&action=enter

"but this is a very old file, where transaction had already been entered".

It is indeed!  Much of the data in this file was imported from Quicken quite a few years ago - into KmyMoney 0.00 or 1.0 as I remember.  I haven't looked in too much detail, but I noticed that the date claimed to be the "opening date" in the first three entries in the consistency checker error is actually one of the dates on which much of the data was imported from Quicken - just in case that might be relevant.

Allan - thanks for the pointer towards the file attachment feature.  I did try it but got bounced because the file was too large.

Typo in last comment....  It should have read "into KMyMoney 0.9 or 1.0 as I remember".

Created attachment 76715 [details]
Anonymised KMyMoney data file demonstrating the problem

Off topic....

Apologies to Allan.  It appears that my original attempt at uploading the sample file was bounced because I'd attempted to upload the un-zipped version.  I've now uploaded the zipped file as a local attachment.  As you say.... it's a heck of a lot more convenient than having it hosted somewhere else.

(In reply to comment #10)
> It was the case that a transaction could be entered with a date prior to the
> opening date. You can't do that anymore, but we are dealing with old files
> here.
> 
> So, we have two possible fixes here: check for transactions prior to the
> opening date and verify if prices are entered for the date of those
> transactions, or fix the opening date to the date of the first transactions.
> We could even add both fixes.

Just wanted to check if you are, or intend, looking into this further?

I'm looking into it. Waiting for some feedback from Thomas on which would be the best way to fix this.

I've posted a review request with a fix https://git.reviewboard.kde.org/r/122746/

Git commit dbd547a503e14e146d7f9d2b1b0781f05ad3e0b6 by Cristian Oneț.
Committed on 08/04/2015 at 05:57.
Pushed by conet into branch 'master'.

Add a stage to the consistency check to fix transaction post dates.

If the transaction is moved to another post date the price information
is moved along with it for investments transactions. This fixes the
reported issue, saving the file once will fix all price issues.

Also add the possibility to save the consistency check log to a file
or copy it into the clipboard.

Make sure that the user enters a valid transaction date by checking
that the transaction date is after the opening date of each and every
account involved in the transaction. Until now we only checked that
the transaction date is after the opening date of the account in which
is being entered.

Remove the check for "Sell" action since there is no such action.
REVIEW: 122746

M  +16   -2    kmymoney/dialogs/transactioneditor.cpp
M  +73   -9    kmymoney/kmymoney.cpp
M  +10   -0    kmymoney/kmymoney.h
M  +31   -2    kmymoney/mymoney/mymoneyfile.cpp

http://commits.kde.org/kmymoney/dbd547a503e14e146d7f9d2b1b0781f05ad3e0b6

Git commit 0ce50018f87b2e19386001ac2f141aafb7119a44 by Cristian Oneț.
Committed on 08/04/2015 at 06:01.
Pushed by conet into branch '4.7'.

Add a stage to the consistency check to fix transaction post dates.

If the transaction is moved to another post date the price information
is moved along with it for investments transactions. This fixes the
reported issue, saving the file once will fix all price issues.

Also add the possibility to save the consistency check log to a file
or copy it into the clipboard.

Make sure that the user enters a valid transaction date by checking
that the transaction date is after the opening date of each and every
account involved in the transaction. Until now we only checked that
the transaction date is after the opening date of the account in which
is being entered.

Remove the check for "Sell" action since there is no such action.
REVIEW: 122746
(cherry picked from commit dbd547a503e14e146d7f9d2b1b0781f05ad3e0b6)

M  +16   -2    kmymoney/dialogs/transactioneditor.cpp
M  +73   -9    kmymoney/kmymoney.cpp
M  +10   -0    kmymoney/kmymoney.h
M  +31   -2    kmymoney/mymoney/mymoneyfile.cpp

http://commits.kde.org/kmymoney/0ce50018f87b2e19386001ac2f141aafb7119a44

Just discovery this feature the hard way: it corrupted all my data.
It occurred when doing an import today (attached the log). 
It thinks that all  my categories are accounts and comparing their open date with the transaction. How can I disable it?
Shouldn't it ask me before to proceed?

Created attachment 92415 [details]
kmymoney file corrupted by consistency check option when doin import

(In reply to Gilberto Caetano de Andrade from comment #25)
> Just discovery this feature the hard way: it corrupted all my data.
> It occurred when doing an import today (attached the log). 
> It thinks that all  my categories are accounts and comparing their open date
> with the transaction. How can I disable it?
> Shouldn't it ask me before to proceed?

For me "consistency check" means just check/report/identify problems and not doing anything. It is up to me analyze and take the decision or not to fix the problems.

(In reply to Gilberto Caetano de Andrade from comment #25)
> Just discovery this feature the hard way: it corrupted all my data.
> It occurred when doing an import today (attached the log). 
> It thinks that all  my categories are accounts and comparing their open date
> with the transaction. How can I disable it?
> Shouldn't it ask me before to proceed?

Other thing, every time I open the file it shows the save button activated and I have to save or cancel even though not doing any modification.

(In reply to Gilberto Caetano de Andrade from comment #26)
> Created attachment 92415 [details]
> kmymoney file corrupted by consistency check option when doin import

What do you mean by "corrupted", from what I see you imported some transactions in some income expense accounts prior to the opening date of those accounts, which, after the consistency check runs, get properly adjusted.

This only happens when historic data is imported.

(In reply to Cristian Oneț from comment #29)
> (In reply to Gilberto Caetano de Andrade from comment #26)
> > Created attachment 92415 [details]
> > kmymoney file corrupted by consistency check option when doin import
> 
> What do you mean by "corrupted", from what I see you imported some
> transactions in some income expense accounts prior to the opening date of

They are not accounts! They are Categories. Before this feature, I could analyze old transactions and create new Categories for them. This feature should not touch category's date creation or anything else.
As I said, it should touch nothing, just check/report/identify problems.

> those accounts, which, after the consistency check runs, get properly
> adjusted.
> 
> This only happens when historic data is imported.

That's the point, there is no historic data imported. They are new data coming from my bank account(via OFX). Kmymoney is touching old data, making changes it shouldn't.

(In reply to Gilberto Caetano de Andrade from comment #30)
> (In reply to Cristian Oneț from comment #29)
> > (In reply to Gilberto Caetano de Andrade from comment #26)
> > > Created attachment 92415 [details]
> > > kmymoney file corrupted by consistency check option when doin import
> > 
> > What do you mean by "corrupted", from what I see you imported some
> > transactions in some income expense accounts prior to the opening date of
> 
> They are not accounts! They are Categories. Before this feature, I could
> analyze old transactions and create new Categories for them. This feature
> should not touch category's date creation or anything else.
> As I said, it should touch nothing, just check/report/identify problems.

I don't agree that it should only identify problems. Since the opening date of accounts (categories included) can only be edited one by one it would have beed a pain for all users having historic data (transactions older than catogiry creation date) to update their catrgories data. AFAIK the opening date of regular accounts is not adjusted. 

> 
> > those accounts, which, after the consistency check runs, get properly
> > adjusted.
> > 
> > This only happens when historic data is imported.
> 
> That's the point, there is no historic data imported. They are new data
> coming from my bank account(via OFX). Kmymoney is touching old data, making
> changes it shouldn't.

Sorry, but transactions from 12-2013 seem historic to me. Having a transaction in an account (even category) before it's opening date could cause incorrect reports so this needs to be fixed.

(In reply to Cristian Oneț from comment #31)
> (In reply to Gilberto Caetano de Andrade from comment #30)
> > (In reply to Cristian Oneț from comment #29)
> > > (In reply to Gilberto Caetano de Andrade from comment #26)
> > > > Created attachment 92415 [details]
> > > > kmymoney file corrupted by consistency check option when doin import
> > > 
> > > What do you mean by "corrupted", from what I see you imported some
> > > transactions in some income expense accounts prior to the opening date of
> > 
> > They are not accounts! They are Categories. Before this feature, I could
> > analyze old transactions and create new Categories for them. This feature
> > should not touch category's date creation or anything else.
> > As I said, it should touch nothing, just check/report/identify problems.
> 
> I don't agree that it should only identify problems. Since the opening date
> of accounts (categories included) can only be edited one by one it would
> have beed a pain for all users having historic data (transactions older than
> catogiry creation date) to update their catrgories data. AFAIK the opening
> date of regular accounts is not adjusted. 
> 
> > 
> > > those accounts, which, after the consistency check runs, get properly
> > > adjusted.
> > > 
> > > This only happens when historic data is imported.
> > 
> > That's the point, there is no historic data imported. They are new data
> > coming from my bank account(via OFX). Kmymoney is touching old data, making
> > changes it shouldn't.
> 
> Sorry, but transactions from 12-2013 seem historic to me. Having a
> transaction in an account (even category) before it's opening date could
> cause incorrect reports so this needs to be fixed.

I agree with you(about an account date) but not the category. It (the category/tag date) doesn’t interfere in any report - I think it shouldn't!
The kmymoney message is fine:
"The consistency check has found some issues in your data. Details are presented below. Those issues that could not be corrected automatically need to be solved by the user."

So, is my responsibility to change my  data. And If kmymoney can do it automatically, I should enable or disable it whenever I want.

(After restore my data) It just happened right now when trying to fix the account date, it altered all category's date.

PS1: sorry for delay, English is not my native language
PS2: unable to re-size the log window(attached)

Created attachment 92432 [details]
data updated automatically and unable to re-size log window

Within KMM, a category IS an account, even if the user does not think of it like that.  Since it is how the internal bookkeeping is done, a category needs to follow all the rules for an account.

If the only thing that was changed is the date the category was opened (or created) why do you consider that corruption?  If there is a transaction with a date prior to the opening date, that is incorrect.  

Can you run a consistency check on your data before you do any import?  I don't know if it matters, but it is not clear to me if the problem is already in your .kmy file, or is only created by the import.

Perhaps we need to change the wording of the message to be more clear about what issues KMM will fix itself and what issues the user must resolve.

(In reply to Jack from comment #34)
> Within KMM, a category IS an account, even if the user does not think of it
> like that.  Since it is how the internal bookkeeping is done, a category
> needs to follow all the rules for an account.
I'm sorry about that. :(

> 
> If the only thing that was changed is the date the category was opened (or
> created) why do you consider that corruption?  
Because I was not considering Category/Tag an account. And in bookkeeping is not account either.
But I can't do anything about it since Kmymoney was done that way.

> If there is a transaction
> with a date prior to the opening date, that is incorrect.  
Again, I agree with you: TRANSACTION.

> Can you run a consistency check on your data before you do any import?  I
> don't know if it matters, but it is not clear to me if the problem is
> already in your .kmy file, or is only created by the import.
It doesn't matter.

> Perhaps we need to change the wording of the message to be more clear about
> what issues KMM will fix itself and what issues the user must resolve.
That's a good idea.

After thinking a while about the issue, I came up with the idea to eliminate the usage of the opening date for categories towards the user all together by setting it fixed to 1900-01-01 internally and not showing it in the account edit dialog and other UI locations. As Jack has mentioned, we do need the internally as category is basically a synonym for account. This is not only the case inside KMyMoney but the mechanics of double entry bookkeeping. The above mentioned change should solve the import issues we see with categories after the recent changes of the consistency check logic. Comments welcome.

Thomas I think your proposal is a good idea.

It also sounds good to me, certainly for new files and new categories.  I'm not sure what is best for existing files where the problem is found by the consistency check.  In general, I would agree to change the opening date to this new earlier value.  Do you propose to change just the category with the issue, or all categories? 

As I remember, until now, the consistency check either simply informed the user of the issue, or, in certain cases, it just made the necessary correction.  In this case would it make sense to offer the user a choice: 1) do nothing and let the user fix the file manually, 2) move the date of the transaction  (that is the current option, right?) 3) change the opening date of this category, or 4) change the opening date of ALL categories.

As part of the consistency check I would set all categories to that date. Changing only those, that do show a problem is not sufficient for the import case which requires the date to be updated before a transaction with an offending date is imported. 

User choice: the current logic only has a mechanism to inform the user but cannot request feedback. In general, I don't want to change that (right now).

A transaction date is never changed, at least after http://quickgit.kde.org/?p=kmymoney.git&a=commit&h=af7939608b3aa305a86b0ad6196beacb52c63108 has been added to the repository. Changing transaction dates in the back of the user is not a good idea as it immediately creates inaccuracy of your financial data.

Changing the opening date of the categories does not create any harm as it is not used elsewhere (except in the check for transaction storage).

Sounds good to me.

Git commit 7a4901c668871882bba75068bf1256f2bf00dba0 by Thomas Baumgart.
Committed on 16/05/2015 at 06:20.
Pushed by tbaumgart into branch 'master'.

Eliminate usage of opening date for categories

Set the opening date for newly created categories to 1900-01-01 and fix
existing ones to have that date during the consistency check. Since the
opening date of an income/expense account (aka category) is not used
throughout the application, it should not interfer with the check that
the opening dates of all accounts referenced in a transaction are prior
to the transaction's post date. Setting the opening date of a category
to the above mentioned value makes sure that a category is always
'opened' before the transaction took place.

The UI elements for the opening date have been removed from the
account/category edit dialog as they are not used anymore.
GUI:

M  +8    -2    kmymoney/dialogs/knewaccountdlg.cpp
M  +1    -0    kmymoney/dialogs/knewaccountdlg.h
M  +16   -16   kmymoney/dialogs/knewaccountdlgdecl.ui
M  +2    -0    kmymoney/kmymoney.cpp
M  +13   -0    kmymoney/mymoney/mymoneyfile.cpp
M  +71   -0    kmymoney/mymoney/mymoneyfiletest.cpp
M  +1    -0    kmymoney/mymoney/mymoneyfiletest.h

http://commits.kde.org/kmymoney/7a4901c668871882bba75068bf1256f2bf00dba0

I've spent a happy morning clearing all my consistency check errors.

In case it helps anyone, many of mine were the result of the opening price not being saved in the Prices table, even though the price was available in the transaction.

I was puzzled by this at first, as I had submitted a fix for this many moons ago.  However, what I realised was that the missing prices were for old transactions, which pre-dated the fix.

All that was necessary was to open each transaction concerned for editing then closing it again without making any change.  Then on to the next one.

*** Bug 359695 has been marked as a duplicate of this bug. ***