Bug 431373 - Incorrect Balance Values in Reconcile View of Ledger
Summary: Incorrect Balance Values in Reconcile View of Ledger
Status: RESOLVED WORKSFORME
Alias: None
Product: kmymoney
Classification: Applications
Component: general (show other bugs)
Version: 5.1.1
Platform: Appimage Linux
: NOR normal
Target Milestone: ---
Assignee: Matthew Grimes
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2021-01-09 23:59 UTC by alrtzgkse7o9
Modified: 2023-05-14 03:45 UTC (History)
1 user (show)

See Also:
Latest Commit:
Version Fixed In:


Attachments
collection of screenshots and data. (126.39 KB, application/pdf)
2021-01-09 23:59 UTC, alrtzgkse7o9
Details
screen shot of Credit card reconcile. Total of cleared transaction not same as brown bar total (15.68 KB, image/png)
2023-02-26 16:25 UTC, Matthew Grimes
Details

Note You need to log in before you can comment on or make changes to this bug.
Description alrtzgkse7o9 2021-01-09 23:59:33 UTC
Created attachment 134695 [details]
collection of screenshots and data.

SUMMARY

When viewing the ledger for reconciliation of credit cards and investment, the balance values reach a state where they are incorrect no matter removing/adding transactions. This issue appears randomly for an account but is persistent once the account is affected; reverting to a backup is the only method that I've found to resolve the incorrect balance values. 

The issue was initially experienced with the SQLite database for the credit card account. I reverted to a backup with the XML "database" which corrected the credit card balance values. 

The second instance of incorrect balance values was experienced within an investment account ledger. The storage format being XML.

This issue was also present in 5.1.0. I'm new to using reconciliation, so I cannot say whether this is issue existed in earlier versions.


STEPS TO REPRODUCE

1. <Unknown>
2. Start reconcile
3. Observe that the balance values are incorrect.


OBSERVED RESULT

[Ledger View, not reconcile]
1. Buy 32.74 shares -> Balance: 32.74
--investment account reconciled--
2. Buy 32.904 shares -> Balance: 35.644

[Reconcile]
1. Buy 32.904 shares -> Balance: 37.730


EXPECTED RESULT

[Ledger View, not reconcile view]
1. Buy 32.74 shares -> Balance: 32.74
--investment account reconciled--
2. Buy 32.904 shares -> Balance: 35.644

[Reconcile]
1. Buy 32.904 shares -> Balance: 35.644


SOFTWARE/OS VERSIONS

Linux OS: Fedora 32 
KMyMoney Version 5.1.0-599a575 AppImage
KMyMoney Version 5.1.1-4ebcb55 AppImage
KMyMoney Version 5.1.1-a1a0fcb AppImage
- KDE Frameworks 5.51.0
- Qt 5.11.3 (built against 5.11.3)
- The xcb windowing system


ADDITIONAL INFORMATION

Screenshots and data attached.
Comment 1 Jack 2021-01-10 01:05:13 UTC
First, reconciling it not really supported for Investment accounts.  It works (in that it does something) but you are really on your own there.  Reconciliation deals with cash balance and there is no cash in an Investment account.  I do reconcile all my Investment accounts, but the balances are not affected, that I have ever seen.  Also note (I don't know if it affects your case) the balance in an investment ledger is the currently owned number of shares of the security reference in that transaction, not any sort of account grand total or value.

With that said, the reconciliation process should not affect any balance, since it doesn't add or alter any transaction.  In accounts other than Investment, it changes Cleared transactions to Reconciled, but it does not do that for Investment accounts.

I assume your line "--investment account reconciled--" indicates the divider bar in the ledger from a previous reconciliation.  If not - please explain.  Also, your numbers do not make any sense.  If the first buy is 32.74 shares, then yes, the balance should be 32.74.  If the second buy is 32.904 shares, then the balance should be 65.644 shares, but if the second by is 2.904, the balance is correct.  Also, after reconciliation, you should still have the two buy transactions.  Are there any other transactions displayed?

Separately, if you can provide a sample .kmy  or .xml file which demonstrates the problem with a credit card account, that would help.
Comment 2 alrtzgkse7o9 2021-01-10 02:04:54 UTC
Thank you for the quick response.

>First, reconciling it not really supported for Investment accounts.
Right. I was told the same thing when I asked in chat. I am primarily reconciling to move the Last Reconciled bar with no concern for the cash value; I also manually verify the security balances affected during the period, which is where I noticed that they were not adding up correctly.


>Also note (I don't know if it affects your case) the balance in an investment ledger is the currently owned number of shares of the security reference in that transaction, not any sort of account grand total or value.
I am aware of the distinction that the balances (or number of shares) are specific to each security. The number of total shares for the affected security are inaccurate when reconciling, yet accurate in the non-reconcile ledger view.


>I assume your line "--investment account reconciled--" indicates the divider bar in the ledger from a previous reconciliation.
Correct. 


>Also, your numbers do not make any sense.
Whoops. Corrected (and expanded) values here and in my original attachment.

OBSERVED RESULT

[Ledger View, not reconcile]
1. Buy 32.74 shares -> Balance: 32.74
--investment account reconciled--
2. Buy 2.904 shares -> Balance: 35.644
3. Buy 0.203 shares -> Balance: 35.847
4. Buy 2.622 shares -> Balance: 38.469
5. Buy 2.590 shares -> Balance: 41.059


[Reconcile (ledger)]
--Last reconciliation--
1. Buy 2.904 shares -> Balance: 37.730
2. Buy 0.203 shares -> Balance: 37.933
3. Buy 2.622 shares -> Balance: 40.555
4. Buy 2.590 shares -> Balance: 43.145

EXPECTED RESULT

[Ledger View, not reconcile view]
1. Buy 32.74 shares -> Balance: 32.74
--investment account reconciled--
2. Buy 2.904 shares -> Balance: 35.644
3. Buy 0.203 shares -> Balance: 35.847
4. Buy 2.622 shares -> Balance: 38.469
5. Buy 2.590 shares -> Balance: 41.059

[Reconcile (ledger)]
--Last reconciliation--
1. Buy 2.904 shares -> Balance: 35.644
2. Buy 0.203 shares -> Balance: 35.847
3. Buy 2.622 shares -> Balance: 38.469
4. Buy 2.590 shares -> Balance: 41.059


>Also, after reconciliation, you should still have the two buy transactions.  Are there any other transactions displayed?
Yes, I have dozens of transactions afterwards for the affected security throughout the year, among some other securities. I did not have an easy way to redact the remaining transactions from my original screenshot. I have added further entries to the OBSERVED RESULT section.

In the reconciliation ledger view, the balance for the first transaction is incorrect. The subsequent math is valid for transactions and their change in the balance (number of total shares for the security), but the balance is incorrect due to the first transaction share count being incorrect.

>Separately, if you can provide a sample .kmy  or .xml file which demonstrates the problem with a credit card account, that would help.
If it happens, again, I'll find a way to share the data. The affected data was in a SQLite3 database that I wrote off as a one-off quirk of SQLite3 and deleted. After restoring from a .kmy (XML) backup and re-entering the credit card transactions, the values were correct. 


FURTHER THOUGHTS

(this is a stretch and I don't think it should be related, but adding just in case) 
I've also recently been using the CSV importer. I can't say for certain at this point (it was several weeks ago), but I may have imported the credit card transactions in the original situation. For certain, I can say that I have imported the last few months of transactions related to the affected investment security. However, all of the transactions and splits for the affected account retrieved by getsplitpart.pl appear valid and add up correctly. 

Is there somewhere else in the XML file that is touched by the CSV importer that could be referenced by the reconcile wizard when it is calculating its version of the ledger?
Comment 3 alrtzgkse7o9 2021-01-10 02:21:21 UTC
Ah ha! I figured out the cause.

One of the later transactions (2020-10-22) for the affected security was marked as reconciled. When I clear the "reconciled" state for that transaction, that removed the extra shares that were showing up.

This is both expected and unexpected behaviour. Expected because the reconciled balance totals include all reconciled transactions. Unexpected because it does not seem obvious that transactions beyond (later/after) the period being reconciled would be included.

So, my thoughts are that it might be good to filter out reconciled transactions with postdates after the period being reconciled or maybe have a warning if any exist. But, it is definitely an edge case.

STEPS TO REPRODUCE
1. Mark a transaction outside/later the affected time period as "reconciled"
2. Start Reconcilation process for the earlier time period
3. The earlier time period will include the shares/value from _all_ reconciled transactions for the affected security.
Comment 4 Bug Janitor Service 2021-01-25 04:33:14 UTC
Dear Bug Submitter,

This bug has been in NEEDSINFO status with no change for at least
15 days. Please provide the requested information as soon as
possible and set the bug status as REPORTED. Due to regular bug
tracker maintenance, if the bug is still in NEEDSINFO status with
no change in 30 days the bug will be closed as RESOLVED > WORKSFORME
due to lack of needed information.

For more information about our bug triaging procedures please read the
wiki located here:
https://community.kde.org/Guidelines_and_HOWTOs/Bug_triaging

If you have already provided the requested information, please
mark the bug as REPORTED so that the KDE team knows that the bug is
ready to be confirmed.

Thank you for helping us make KDE software even better for everyone!
Comment 5 Thomas Baumgart 2021-01-25 15:37:24 UTC
Updating status
Comment 6 Matthew Grimes 2023-01-19 16:37:41 UTC
Problem is with checking account also Jan 2023.
Comment 7 Thomas Baumgart 2023-01-29 13:42:58 UTC
Git commit d193f80579bed84a0cce4ca17ba336b815ea507e by Thomas Baumgart.
Committed on 29/01/2023 at 13:40.
Pushed by tbaumgart into branch 'master'.

Improve handling of reconciliation

The previous implementation caused all transactions marked as reconciled
to be hidden no matter if they are within the period of of the statement
or not. Also, transactions not marked reconciled but have a post date
prior to the start of the statement were not correctly displayed during
the reconciliation process. This caused the balances to become
"incorrect".

This has been changed to show all transactions from the date of the
first non-reconciled one even if that is prior to the last
reconciliation. All reconciled transactions that have a post date that
is within the period of the statement are also not hidden anymore and
taken into account when calculating the balances. Transactions younger
than the statement date are not taken into account for balance
calculation as before.

I am not sure if this fixes the reported problem 431373 but it is
certainly an improvement.
Related: bug 438936
FIXED-IN: master

M  +21   -11   kmymoney/models/ledgerfilter.cpp
M  +16   -0    kmymoney/models/ledgerfilter.h
M  +2    -0    kmymoney/views/reconciliationledgerviewpage.cpp
M  +5    -0    kmymoney/wizards/endingbalancedlg/checkingstatementinfowizardpage.cpp
M  +7    -0    kmymoney/wizards/endingbalancedlg/checkingstatementinfowizardpage.h
M  +92   -37   kmymoney/wizards/endingbalancedlg/kendingbalancedlg.cpp
M  +9    -0    kmymoney/wizards/endingbalancedlg/kendingbalancedlg.h

https://invent.kde.org/office/kmymoney/commit/d193f80579bed84a0cce4ca17ba336b815ea507e
Comment 8 Thomas Baumgart 2023-02-01 09:54:05 UTC
Is the problem still present after the recent changes to the code?
Comment 9 Bug Janitor Service 2023-02-16 03:48:26 UTC
Dear Bug Submitter,

This bug has been in NEEDSINFO status with no change for at least
15 days. Please provide the requested information as soon as
possible and set the bug status as REPORTED. Due to regular bug
tracker maintenance, if the bug is still in NEEDSINFO status with
no change in 30 days the bug will be closed as RESOLVED > WORKSFORME
due to lack of needed information.

For more information about our bug triaging procedures please read the
wiki located here:
https://community.kde.org/Guidelines_and_HOWTOs/Bug_triaging

If you have already provided the requested information, please
mark the bug as REPORTED so that the KDE team knows that the bug is
ready to be confirmed.

Thank you for helping us make KDE software even better for everyone!
Comment 10 Matthew Grimes 2023-02-26 16:25:35 UTC
Created attachment 156761 [details]
screen shot of Credit card reconcile. Total of cleared transaction not same as brown bar total

This is a screen shot of Credit card reconcile attempt in my Kmoney. Software indicates proper starting balance amount, I entered the ending balance amount and marked my cleared transactions in the register. The puck brown summary bar indicates 1 charge and 20 payments. Should be 20 charges and one payment. The totals shown in the summary bar are correct, ( $961.62 cleared, and $236 payment)  but the program shows a cleared amount of $11,053.60 which is not correct, and does not agree with the brown bar amount. The difference value under the register balance indicates $1,451.24 instead of zero.  I get a warning (image #2) that says there is a difference between statement and cleared amounts and asks for confirmation to proceed. Reconcile of my checking account does not have this problem.
Comment 11 Jack 2023-04-14 22:36:04 UTC
Your second attachment was of the error message, not of the reconcile ledger.  I occasionally also receive that error pop-up, but it has not seemed to ever actually cause any problems for me, and that includes checking, credit card, and investment accounts.  Can you create a test file which shows the problem that you can attach?
Comment 12 Matthew Grimes 2023-04-14 22:45:45 UTC
When you say that you need the "reconcile ledger", are your referring to the actual transaction data file for the credit card transactions or the whole file of all my accounts, or something else?  Is there a way to only send the credit card transactions? How do I do that?




Sent with Proton Mail secure email.

------- Original Message -------
On Friday, April 14th, 2023 at 6:36 PM, Jack <bugzilla_noreply@kde.org> wrote:


> https://bugs.kde.org/show_bug.cgi?id=431373
> 
> Jack ostroffjh@users.sourceforge.net changed:
> 
> 
> What |Removed |Added
> ----------------------------------------------------------------------------
> Status|REPORTED |NEEDSINFO
> Resolution|--- |WAITINGFORINFO
> 
> --- Comment #11 from Jack ostroffjh@users.sourceforge.net ---
> 
> Your second attachment was of the error message, not of the reconcile ledger.
> I occasionally also receive that error pop-up, but it has not seemed to ever
> actually cause any problems for me, and that includes checking, credit card,
> and investment accounts. Can you create a test file which shows the problem
> that you can attach?
> 
> --
> You are receiving this mail because:
> You are the assignee for the bug.
> You are on the CC list for the bug.
Comment 13 Bug Janitor Service 2023-04-29 03:46:13 UTC
Dear Bug Submitter,

This bug has been in NEEDSINFO status with no change for at least
15 days. Please provide the requested information as soon as
possible and set the bug status as REPORTED. Due to regular bug
tracker maintenance, if the bug is still in NEEDSINFO status with
no change in 30 days the bug will be closed as RESOLVED > WORKSFORME
due to lack of needed information.

For more information about our bug triaging procedures please read the
wiki located here:
https://community.kde.org/Guidelines_and_HOWTOs/Bug_triaging

If you have already provided the requested information, please
mark the bug as REPORTED so that the KDE team knows that the bug is
ready to be confirmed.

Thank you for helping us make KDE software even better for everyone!
Comment 14 Bug Janitor Service 2023-05-14 03:45:59 UTC
This bug has been in NEEDSINFO status with no change for at least
30 days. The bug is now closed as RESOLVED > WORKSFORME
due to lack of needed information.

For more information about our bug triaging procedures please read the
wiki located here:
https://community.kde.org/Guidelines_and_HOWTOs/Bug_triaging

Thank you for helping us make KDE software even better for everyone!