Bug 420761 - After upgrade from Fedora 31 to 32, one of my checking accounts shows a huge negative "Cleared" balance
Summary: After upgrade from Fedora 31 to 32, one of my checking accounts shows a huge ...
Status: RESOLVED FIXED
Alias: None
Product: kmymoney
Classification: Unclassified
Component: general (show other bugs)
Version: 5.0.8
Platform: Other Linux
: NOR major (vote)
Target Milestone: ---
Assignee: KMyMoney Devel Mailing List
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2020-04-29 16:41 UTC by schoenes-rad
Modified: 2020-06-05 04:06 UTC (History)
4 users (show)

See Also:
Latest Commit:
Version Fixed In: 5.1.0


Attachments
Compiled on F31 (120.80 KB, image/png)
2020-05-09 22:28 UTC, Brendan
Details
Compiled on F32 (120.70 KB, image/png)
2020-05-09 22:29 UTC, Brendan
Details
F32 Test File (1.99 KB, application/x-kmymoney)
2020-05-09 22:29 UTC, Brendan
Details
attachment-14636-0.html (1.48 KB, text/html)
2020-05-18 18:39 UTC, Dawid Wróbel
Details

Note You need to log in before you can comment on or make changes to this bug.
Description schoenes-rad 2020-04-29 16:41:59 UTC
SUMMARY


STEPS TO REPRODUCE
1. Start KMyMoney
2. Check the account in question
3. Shows huge negative "Cleared" balance amount

OBSERVED RESULT
"Cleared" differs highly from "Balance"

EXPECTED RESULT
Both amounts should be identical once reconciliation has been performed.

SOFTWARE/OS VERSIONS

Linux/KDE Plasma: Fedora 32 Workstation, 64bit
(available in About System)
KDE Plasma Version: 
KDE Frameworks Version: 5.68.0
Qt Version: Qt 5.13.2 (built against 5.13.2)
GNOME 3.36.1
Windowing System X11 (xcb)

ADDITIONAL INFORMATION

This behaviour has started immediately after upgrading Fedora from release 31 to 32 last night.

Here's how to reproduce the problem:

1. With KMyMoney open maintaining the bank account in question, select Reconcile...
2. Enter correct bank balance and date
3. Click Continue on Interest/Charges dialogue
4. Balances show as perfect (Statement = Cleared), Difference zero
5. Once you click Finish to end Reconciliation, the Reconciliation Report dialogue pops up.  All is still fine
6. Once the Reconciliation report dialogu has been acknowledged by clicking OK, the incorrect figures are back again.


BTW - I took a copy (!) of my .kmy file to a different computer running under LXLE where an earlier version of KMyMoney (4.xx.xx) is installed.  All balances are correct there.  None of the described behaviour can be reproduced on that machine.

- nothing follows -
Comment 1 Brendan 2020-04-29 17:30:15 UTC
I can confirm this bug and add my observations.

Immediately before upgrading from F31 to F32 on one of my systems, I compiled and installed KMM from the master branch.

After upgrading my system to F32, I again compiled and installed KMM from the master branch. Then I saw this bug report so I opened KMM and found the cleared amount did not match the balance as expected in at least one checking account and one savings account.

I then opened the exact same KMM file on another system that was still running F31 and a recently compiled version of KMM. The cleared amount matched the balance as expected.

I went back to the system running F32 and removed the version of KMM compiled in F32 and reinstalled the version of KMM that was compiled in F31 immediately before the upgrade and the cleared amounts went back to normal.

It appears that this bug is being caused by the compile process on F32.
Comment 2 Rex Dieter 2020-04-29 19:50:23 UTC
Fedora32 ships with gcc-10, possible that's relevant.

To confirm, you're saying both the fedora packaged and local-build behaves the same way on f32?
Comment 3 Brendan 2020-04-29 23:05:15 UTC
I can only comment on the local build. I have not tried the package provided by Fedora.
Comment 4 Rex Dieter 2020-04-30 02:54:04 UTC
Unfortunately that didn't answer my question.  Does the local build on f32 work ok or not?
Comment 5 Brendan 2020-04-30 12:47:32 UTC
See my first comment and let me know what question you still need answered.
Comment 6 Rex Dieter 2020-04-30 15:06:39 UTC
It wasn't clear to me, that's why I asked for clarification.  My guess is that your local build on f32 was bad too, which supports my theory that this is related to building with gcc 10 somehow.  Either a compiler bug, or a code issue exposed by gcc10
Comment 7 Thomas Baumgart 2020-04-30 15:21:11 UTC
Do the unit tests show different results or do they pass on both F31 and F32?
Comment 8 Brendan 2020-04-30 15:27:24 UTC
I guess I was not clear. I will try again.

I did not test the package provided with F32.

1) F31 & KMM compiled on F31: NO BUG

2) F32 & KMM Compiled on F32: BUG

3) F32 & KMM Compiled on F31: NO BUG

I assume KMM from Fedora for F32 is compiled with F32 so it probably has the same bug as option 2 above. I will download the latest release version and test it on F32.
Comment 9 Rex Dieter 2020-04-30 20:57:41 UTC
running tests on f31:

The following tests FAILED:
          1 - appstreamtest (Failed)
         38 - qsqlcipher-test (Failed)

1 is a false positive
      Start  1: appstreamtest
 1/42 Test  #1: appstreamtest .........................***Failed    0.06 sec
CMake Error at /usr/share/ECM/kde-modules/appstreamtest.cmake:32 (message):
  File '/usr/share/metainfo/org.kde.kmymoney.appdata.xml' does not exist.

Assumes kmymoney appstream file is already installed, instead of testing the one in the build directory

38/42 Test #38: qsqlcipher-test .......................***Failed    0.02 sec
********* Start testing of qsqlciphertest *********
Config: Using QtTest library 5.13.2, Qt 5.13.2 (x86_64-little_endian-lp64 shared (dynamic) release build; by GCC 9.3.1 20200317 (Red Hat 9.3.1-1))
FAIL!  : qsqlciphertest::initTestCase() 'drivers.contains(sqlDriverName)' returned FALSE. ()
   Loc: [/var/tmp/kdecache-rdieter/BUILDROOT/kmymoney-5.0.8/kmymoney/plugins/sqlcipher/tests/qsqlcipher-test.cpp(42)]
PASS   : qsqlciphertest::cleanupTestCase()
Totals: 1 passed, 1 failed, 0 skipped, 0 blacklisted, 1ms
********* Finished testing of qsqlciphertest *********

Similar assumption, assuming sql driver in builddir is usable?


On f32,

The following tests FAILED:
          1 - appstreamtest (Failed)
          9 - mymoneyfile-test (Failed)
         23 - mymoneytransactionfilter-test (Failed)
         31 - reports-chart-test (Failed)
         33 - reports-pivottable-test (Failed)
         38 - qsqlcipher-test (Failed)

 9/42 Test  #9: mymoneyfile-test ......................***Failed    0.12 sec
********* Start testing of MyMoneyFileTest *********
Config: Using QtTest library 5.14.2, Qt 5.14.2 (x86_64-little_endian-lp64 shared (dynamic) release build; by GCC 10.0.1 20200328 (Red Hat 10.0.1-0.11))
...
FAIL!  : MyMoneyFileTest::testCountTransactionsWithSpecificReconciliationState_transactionWithUnwantedReconcileState() Compared values are not the same
   Actual   (m->countTransactionsWithSpecificReconciliationState(accountId, eMyMoney::TransactionFilter::State::NotReconciled)): 1
   Expected (0)                                                                                                                : 0
   Loc: [/builddir/build/BUILD/kmymoney-5.0.8/kmymoney/mymoney/tests/mymoneyfile-test.cpp(2345)]
...
FAIL!  : MyMoneyFileTest::testClearedBalance() Compared values are not the same
   Loc: [/builddir/build/BUILD/kmymoney-5.0.8/kmymoney/mymoney/tests/mymoneyfile-test.cpp(2543)]

23/42 Test #23: mymoneytransactionfilter-test .........***Failed    0.10 sec
********* Start testing of MyMoneyTransactionFilterTest *********
Config: Using QtTest library 5.14.2, Qt 5.14.2 (x86_64-little_endian-lp64 shared (dynamic) release build; by GCC 10.0.1 20200328 (Red Hat 10.0.1-0.11))
...
FAIL!  : MyMoneyTransactionFilterTest::testMatchTransactionState() Compared values are not the same
   Actual   (filter.matchingSplits(transaction).size()): 4
   Expected (1)                                        : 1
   Loc: [/builddir/build/BUILD/kmymoney-5.0.8/kmymoney/mymoney/tests/mymoneytransactionfilter-test.cpp(454)]
...
FAIL!  : MyMoneyTransactionFilterTest::testMatchTransactionTypeAllTypes() '!filter.match(transaction)' returned FALSE. ()
   Loc: [/builddir/build/BUILD/kmymoney-5.0.8/kmymoney/mymoney/tests/mymoneytransactionfilter-test.cpp(575)]
FAIL!  : MyMoneyTransactionFilterTest::testMatchTransactionTypeDeposits() Compared values are not the same
   Actual   (filter.matchingSplits(transaction2).size()): 1
   Expected (0)                                         : 0
   Loc: [/builddir/build/BUILD/kmymoney-5.0.8/kmymoney/mymoney/tests/mymoneytransactionfilter-test.cpp(623)]
FAIL!  : MyMoneyTransactionFilterTest::testMatchTransactionTypePayments() Compared values are not the same
   Actual   (filter.matchingSplits(transaction).size()): 2
   Expected (0)                                        : 0
   Loc: [/builddir/build/BUILD/kmymoney-5.0.8/kmymoney/mymoney/tests/mymoneytransactionfilter-test.cpp(663)]
FAIL!  : MyMoneyTransactionFilterTest::testMatchTransactionTypeTransfers() Compared values are not the same
   Actual   (filter.matchingSplits(transaction).size()): 3
   Expected (0)                                        : 0
   Loc: [/builddir/build/BUILD/kmymoney-5.0.8/kmymoney/mymoney/tests/mymoneytransactionfilter-test.cpp(724)]

31/42 Test #31: reports-chart-test ....................***Failed   10.09 sec
********* Start testing of ChartTest *********
Config: Using QtTest library 5.14.2, Qt 5.14.2 (x86_64-little_endian-lp64 shared (dynamic) release build; by GCC 10.0.1 20200328 (Red Hat 10.0.1-0.11))
PASS   : ChartTest::initTestCase()
libEGL warning: MESA-LOADER: failed to open swrast (search paths /usr/lib64/dri)

(probably harmless, buildsystem cannot use gui)

33/42 Test #33: reports-pivottable-test ...............***Failed    0.15 sec
********* Start testing of reports::PivotTableTest *********
Config: Using QtTest library 5.14.2, Qt 5.14.2 (x86_64-little_endian-lp64 shared (dynamic) release build; by GCC 10.0.1 20200328 (Red Hat 10.0.1-0.11))
...
FAIL!  : reports::PivotTableTest::testAdvancedFilter() 'networth_f4.m_grid["Asset"].m_total[eActual][11] == moCheckingOpen + moChild' returned FALSE. ()
   Loc: [/builddir/build/BUILD/kmymoney-5.0.8/kmymoney/plugins/views/reports/core/tests/pivottable-test.cpp(671)]
Comment 10 Rex Dieter 2020-04-30 20:59:00 UTC
(Temporary) link to full f32 build.log,
https://kojipkgs.fedoraproject.org//work/tasks/7321/43947321/build.log
Comment 11 Thomas Baumgart 2020-05-01 05:30:08 UTC
Had a quick glimpse at the log. Looks like the failing MyMoneyFile test is related to/dependent on the failing MyMoneyTransactionFilter tests.

@Rex: as a side note, bcooksley added some magic to run the graphical chart tests on the KDE CI infra. I have no idea what he did in detail, though.
Comment 12 toki07 2020-05-09 19:02:37 UTC
I have the same issue. It looks like the first line in the register is not taken in account. When adding a line with a value of zero just before, and marking it as cleared, the result is good.
Comment 13 Brendan 2020-05-09 19:39:10 UTC
Nice catch, hopefully that will help find the cause.

I'm a little concerned that this might be affecting other calculation that are not as obvious.
Comment 14 Jack 2020-05-09 20:38:35 UTC
Can someone provide a simply kmy file showing the problem?  Is the reconciling process necessary to show the problem, or is it just seen opening the file?

I'm on Gentoo, but have gcc-10 installed, and KMM seems to work fine for me compiled with gcc-10 from git head 5.0 branch.  Also, for me, only the appimage test fails.  However, I'm using ninja instead of make and that may well make a difference.  I've just kicked off a build using make, and will report on the results.

I also suspect it might matter whether the various tools used were themselves compiled with gcc-9 or gcc-10.
Comment 15 Jack 2020-05-09 21:09:29 UTC
Compiling with make (4.3) and gcc (10.1.0) gives no test errors at all.
Comment 16 Brendan 2020-05-09 22:27:56 UTC
The suggested fix did not make fix to my accounts. Adding a zero transaction before the first transaction and marking it cleared or reconciled did not change the Cleared value at the bottom.

In my savings account, the Cleared value at the bottom of the page is equal to the account balance as long as everything is marked cleared or reconciled in the F31 version of KMM. In the F32 version, it's the cleared value minus the opening balance amount in the account which was in 2003. This results in a cleared amount that is a very large number. Again, adding a zero amount transaction before the opening balance transaction did not change this, which is surprising since subtracting zero from the account balance should work. I have not saved the file which may be necessary for this to reset.

My checking account is not that straight forward, I have not be able to figure out which numbers are being used to get the cleared amount and adding a zero amount transaction at he beginning changed the cleared amount but not by an amount I can make sense of. it makes no difference if the new zero amount first transaction is marked cleared, reconciled or unmarked.

Changing the cleared/reconciled status of the first couple of transactions in my checking account is causing all kinds of strange behavior. At one point the cleared amount at the bottom match the account balance as it should but ti's not consistent and I can't get back to that point. At one point I uncleared a $500 transaction that used to be the first transaction and the cleared amount changed by about $40,000. It should be around $1,400.

For what it's worth, the savings account cleared amount is a huge negative number and the checking is a huge positive number.

When I switch back to the last version of KMM that I compiled on F31 the problem goes away. To be clear, this is all happening on the same F32 machine that I upgraded last week. I am able to remove the F32 compiled version and reinstall the F31 version with my KMM build scripts.

I did make a test file and took 2 screen captures from it to show the difference between the F31 cleared amount and the F32 cleared amount. of course they don't match the theory that it's the balance minus the first transaction.
Comment 17 Brendan 2020-05-09 22:28:53 UTC
Created attachment 128315 [details]
Compiled on F31
Comment 18 Brendan 2020-05-09 22:29:18 UTC
Created attachment 128316 [details]
Compiled on F32
Comment 19 Brendan 2020-05-09 22:29:50 UTC
Created attachment 128317 [details]
F32 Test File
Comment 20 Jack 2020-05-09 22:45:14 UTC
Brendan - I don't see any opening balance amounts in either account in the attached kmy file.  I don't even see an opening balance account, so I don't know how you think the opening balance plays any role here at all.  If the version of KMM I just compiled with gcc-10, I see what you show in the F31 picture.
Comment 21 Brendan 2020-05-18 18:06:40 UTC
I didn't mean to say opening balance, I meant to say the first transaction. In my case it was called "Opening Balance Adjustment" causing me to write the wrong thing. Sorry for the confusion.

I just did a test with F31 and F32. I changed the status of a $2.36 deposit from Reconciled to not marked. In KMM compiled on F31 the cleared amount changed by $2.36. In the F32 compiled version the cleared amount changed by $5,497.04. This happens with payments or deposits from my savings account.

When I do the same thing for a $2.50 deposit, the cleared amount changes by 14,616.78. 

When I do the same thing on an older $2.50 deposit, the cleared amount changes by 138,203.33.

Very odd.

The cleared amount is a positive number in KMM/F31. When I unclear a deposit the cleared amount drops as expected. In KMM/F32 the cleared amount is about 10 times greater in absolute value but it's negative. When I unclear a deposit, it gets smaller (bigger negative number) so at least it's going in the right direction.

I added a couple of transactions to my test file. There are 5 transactions. Changing the status of the first 3 has no affect on the cleared amount. The most recent 2 transactions work as expected. The more things I try the stranger this gets.

Note that F32 is up to date so if this is a bug in the compiler it hasn't been fixed yet.
Comment 22 Jack 2020-05-18 18:33:05 UTC
I'm pretty much at a loss here.  I don't see any evidence of this, but I'll ask anyway for you to confirm there are no currency conversions involved anywhere here, right?  Everything in US Dollars?

Also, just to avoid confusion of changing the displayed order of transactions when changing state, try to make each transaction on a different date.  It shouldn't make any difference, but we're already taking about stuff that doesn't make any sense.
Comment 23 Dawid Wróbel 2020-05-18 18:39:41 UTC
Created attachment 128584 [details]
attachment-14636-0.html

On top of what Jack suggested, may I also ask that you perform the same
tests using Skrooge? If you confirm it has the same problem it would point
to LibAlkimia, which is a library that both share.

On Mon, May 18, 2020 at 2:33 PM Jack <bugzilla_noreply@kde.org> wrote:

> https://bugs.kde.org/show_bug.cgi?id=420761
>
> --- Comment #22 from Jack <ostroffjh@users.sourceforge.net> ---
> I'm pretty much at a loss here.  I don't see any evidence of this, but
> I'll ask
> anyway for you to confirm there are no currency conversions involved
> anywhere
> here, right?  Everything in US Dollars?
>
> Also, just to avoid confusion of changing the displayed order of
> transactions
> when changing state, try to make each transaction on a different date.  It
> shouldn't make any difference, but we're already taking about stuff that
> doesn't make any sense.
>
> --
> You are receiving this mail because:
> You are the assignee for the bug.
Comment 24 Dawid Wróbel 2020-05-18 19:22:27 UTC
If you decide to test Skrooge as well, please make sure it is compiled with the same versions of LibAlkimia and libMPIR/libGMP as KMyMoney.
Comment 25 Brendan 2020-05-19 15:48:21 UTC
In my test file, I made all the dates different. The first 3 were the same date and the last 2 were today. Once I mde them all different, only the first transaction ignored the cleared status.

Testing on Skrooge will have to wait. I'll have to assume the Fedora package of Skrooge uses the same versions of the libraries that I'm using when I compile KMM.

I know this is all pretty confusing so let me repeat what I said earlier.

KMM does not have a problem when I run the last version of KMM that I compiled on F31 immediately prior to upgrading to F32, even when I'm running the previously compiled version of KMM on F32. My KMM compile scripts have 2 steps, one that downloads and builds from fresh source every time and another that removes the existing version and installs any version that I still have in my build area. This makes it trivial for me to switch between any compiled version that has not be deleted from my system.

I'm not sure what this means but I thought it might be important.
Comment 26 Thomas Baumgart 2020-05-21 11:31:20 UTC
Git commit a8386dc173f51fd56c5c9e42fef5fa5f2df477b7 by Thomas Baumgart.
Committed on 21/05/2020 at 11:31.
Pushed by tbaumgart into branch 'master'.

Fix transaction filter due to compiler issue with g++ 10

g++ version 10 seems to have a problem with our ancient union and struct
based bitmap handling.

Changing it to be based on QFlags solves the problem.

M  +61   -65   kmymoney/mymoney/mymoneytransactionfilter.cpp
M  +15   -16   kmymoney/mymoney/mymoneytransactionfilter.h
M  +3    -3    kmymoney/plugins/sql/mymoneystoragesql.cpp

https://invent.kde.org/office/kmymoney/commit/a8386dc173f51fd56c5c9e42fef5fa5f2df477b7
Comment 27 Thomas Baumgart 2020-05-21 11:32:36 UTC
Git commit 9e1e074d0a2232d6528b64567e7424956da70aee by Thomas Baumgart.
Committed on 21/05/2020 at 11:32.
Pushed by tbaumgart into branch '5.0'.

Fix transaction filter due to compiler issue with g++ 10

g++ version 10 seems to have a problem with our ancient union and struct
based bitmap handling.

Changing it to be based on QFlags solves the problem.
FIXED-IN: 5.0.9

(cherry picked from commit a8386dc173f51fd56c5c9e42fef5fa5f2df477b7)

M  +61   -65   kmymoney/mymoney/mymoneytransactionfilter.cpp
M  +15   -16   kmymoney/mymoney/mymoneytransactionfilter.h
M  +3    -3    kmymoney/plugins/sql/mymoneystoragesql.cpp

https://invent.kde.org/office/kmymoney/commit/9e1e074d0a2232d6528b64567e7424956da70aee
Comment 28 Brendan 2020-05-21 14:18:53 UTC
Works for me. Thanks Thomas.
Comment 29 Rex Dieter 2020-05-21 17:11:35 UTC
Thanks, can also confirm relevant autotests now pass.

Patched update submitted for Fedora 32:
https://bodhi.fedoraproject.org/updates/FEDORA-2020-a3cb50e3dd
Comment 30 schoenes-rad 2020-05-22 12:42:22 UTC
I wish I could confirm the solution for my bank account.  Three out of four accounts do perfectly reconcile but still not the one I reported in the beginning.

Any idea?
Comment 31 schoenes-rad 2020-05-22 12:48:19 UTC
Just checked - my version is still 5.0.8 .  Will 5.0.9 be released to the Fedora project repository any time soon?
Comment 32 schoenes-rad 2020-05-22 13:07:32 UTC
Downloaded latest AppImage KMyMoney-5.0.8-9e1e074-x86_64.AppImage and tested my ref. bank account - reconciles and tallies OK.

Will wait for next repository update (last one was yesterday 20200521).  Hope to get it soon.

Thanks for the fix!
Comment 33 Rex Dieter 2020-05-22 13:24:54 UTC
Re: comment 31
"Will 5.0.9 be released to the Fedora project repository any time soon?"

When kmymoney 5.0.9 is released, yes.  At this moment, it is not (yet).
Comment 34 Brendan 2020-05-22 14:51:40 UTC
If you have any interest in compiling from source I can send you a link to my build scripts. Once you have the dependencies installed it's very easy to install from git or the released tars.
Comment 35 schoenes-rad 2020-05-23 11:05:42 UTC
Thanks Brendan - I'll consider that.  For the moment I am fine with the app image I downloaded.