Bug 345655

Created attachment 91803 [details]
Exemple described in the bug report

*** Bug 346000 has been marked as a duplicate of this bug. ***

Confirming as I also have the problem on git master as of 29th March. I also had the consistency check errors.
I think this is a regression as I am sure I didn't have the same issue in February on an older version of the git master.

Backing out the patch from Bug 303026 fixes the rounding problem for me.

I confirm that patch from bug 303026 is to blame !

Flagging as a regression

Will patch from bug 303026 be reverted ? Difference between kmymoney's reconciled value and my bank's value is growing slowly...

Hello
is anybody working on that issue? I sued to work with Kmymoney for quite a while and I noticed this bug some time ago. Actually I expected the community to fix such a obvious issue more rapidely.
This bug is really annoying and actually makes KMM useless with investment accounts :-(

BR Sven

As I understand it, all current developer time is going to the conversion to KDE Frameworks.  This issue is unlikely to be addressed until after that is complete.  

In addition, while this may certainly be annoying, if does not make KMM useless.  I track several investment accounts in KMM, with multiple brokers, and it works just fine for me.  Remember, nobody is being paid to work on KMM.  (You may also want to be careful about your choice of words, as "sued" has a particular legal implication in English, definitely in the US.)

(In reply to Jack from comment #9)
> As I understand it, all current developer time is going to the conversion to
> KDE Frameworks.  This issue is unlikely to be addressed until after that is
> complete.  
> 
> In addition, while this may certainly be annoying, if does not make KMM
> useless.  I track several investment accounts in KMM, with multiple brokers,
> and it works just fine for me.  Remember, nobody is being paid to work on
> KMM. 

>  (You may also want to be careful about your choice of words, as "sued"
> has a particular legal implication in English, definitely in the US.)

I took that to be a typo for 'used'.

Hi Jack,
first sorry for the typo and for being that impatient. It supposed to be "used" and of course not "sued"!
I can understand that reworking towards the new framework takes resources.

I found two issues pointing to rounding problems in the data base. this seems to match better with my experience. However I wonder how one can work with investments not having problems with rounding? Actually I would not even have figured out if I wasn't syncing my investment with my current account. There I download a transaction let's say 100.00 €. unfortunately I cannot directly download the transactions from my depot(no clue how to connect it to internet banking is it was possible). Therefore I add a stock transaction manually using my receipt from the bank e.g.
1stock for 99.9956€
The resulting payment shown in the depot is 100.00€ (correctly rounded)
When I try to match now the manual transaction with the downloaded I get an error that they do not match because 100.00€ != 100.00€
You may say OK, delete the downloaded transaction and all is fine. But internally it seems not to round the umbers since sooner or later the account does not match anymore since the hidden digits sum up to >1ct.
For me this really makes using kmm a nightmare at the moment. 
Can u confirm that this is the same bug? Or do you have an idea how to work around?
I'd really appreciate any help here.

First, sorry I didn't recognize that as a typo. :-)
Second, I'm going to have to review the details of this bug in more detail to be sure I really understand what is happening before I make any recommendations.  I do agree that there are issues with rounding, especially with investment accounts and securities that are tracked at many decimal places.  

For now, there are a few things you might do.  First, can you create a test kmy file that shows the issue?  Just the investment and the checking accounts, for example.  Will the mismatch happen even if you manually enter the bank transaction at exactly 100?  If so you could send the file, so someone can look to see what is happening.  You can also do that yourself.  A .kmy file is just a gzipped xml file.  (Always make a backup!)  If you can find the various transactions in the file, you can see how KMM is exactly recording the amounts.  The other option is that you could see if you can create an anonymized file, and post that, if it still shows the problem.

Separately, even if you cannot directly map  your KMM account to your online investment account, see if your broker lets you download the transactions - hopefully as OFX or possible QIF.  KMM can import either of those.  You can work separately to see if you actually can do a direct connect to your broker.

One other thing to check is the default precision set in KMM.  Select the Settings/Configure KMyMoney menu item, and on the General view, the Global tab, what is the Price precision set to?  Mine is set to 4 digits.  I'm not certain, but I think that may affect the precision to which things are rounded (but not necessarily displayed).  I certainly agree that if it is saying that 100 != 100 (I use dollars not pounds, but I don't think that should matter) then something is amiss.

Created attachment 99202 [details]
test case rounding

Thanks for your support. 
please find a test case in the file.
3 scenarios:
stock A: visible mismatch between broker and current account
Stock B: can match two transactions
Stock C: invisible mismatch 

Hope that helps for further investigation.

Created attachment 99235 [details]
Analysis of erroneous transaction

Thanks for the file that demonstrates the problem very well. Due to *not* rounding the value of the third split, the transaction becomes unbalanced (sum of all splits must be zero for that). The attached spreadsheet shows the transaction in its XML format as well the broken down fields. The one with the yellow background is the source and the one with the red background is causing the problem.

Now I wonder how you entered this transaction into KMyMoney. Can you describe this in detailed steps so that we can reproduce the problem? I have a gut feeling that the rounding was removed due to some other problem. I want to make sure to not introduce a regression, but git will certainly help us to find out, once we have located the spot in the source.

Hi Thomas,
I'm not sure in which order I did enter the date for the example. Tehrefore I try to reproduce the scenarios:
the numbers stay same:
Name "Stock B"
amount = 0,649
price/share = 157,689
refund commission -2,44 (used wrong German term provision)
Sum automatically filled: 99,90
(However actual value (w/o rouding) 99,900161)
-----------
Scenario A:
1.) I entered the data in the Invest account:
2.) entering transaction in check account:
(investment transaction already shown here)
category investment
amount 99,90
--------------
3.) I match the two transactions 
"KMM has matched the transaction (result above)"
4.) seems(!) to have matched correctly now
5.) modified Note in check account Scenario A
--> exclamation mark has missing assignment 0,00
##########
Scenario B:
1.) enter check account
2.) enter invest account
3.) save:
  * Anzahl Anteile auf Wert der Split-Buchung „T000000000000000019“ gesetzt.
* Mögliches Problem mit Investitionen/Währungen
  * Für die Investition 'Test invest' ist am Eröffnungsdatum '2015-03-01' kein Wert festgesetzt.
    Bitte geben die den Wert der Investition am Tag der Eröffnung an.

Beendet: 1 Problem behoben. 1 Problem noch vorhanden.
4.) back to check acount and matching: 
-->failure
value of split transactions for check in conflict: (-99,90, -99,90)
############
Scenario C:
1.) enter invest account
2.) save 
--> error missing assignment of 0,00
3.) enter check account
4.) match
--> transaction matched, but exclamation mark persists (missing assignment of 0,00)
############

As you could see, it is not so easy to see a straight forward behaviour which is easily repeadable.
However I tried my best. please find the matching file below.
Thanks Sven

Created attachment 99237 [details]
test Scenario as described

I just created https://git.reviewboard.kde.org/r/128061/ . It would be nice, if you someone can test it against git master.

Referring back to comments #3, #4,  #5, and https://bugs.kde.org/show_bug.cgi?id=303026, does that issue now need revisiting?

Allan, good point. The modification is almost identical, though it differs a bit. The rounding problem caught in https://bugs.kde.org/show_bug.cgi?id=303026 could still happen. Can you please comment on reviewboard so that we can think of how to deal with the situation? It would be nice if you can provide me with a testcase (probably just different values) that fails.

Git commit 88040d6c3f43ee6781911087dd12801e890fd8b1 by Thomas Baumgart.
Committed on 13/06/2016 at 12:07.
Pushed by tbaumgart into branch 'master'.

Reduce the precision of shares and values

The problem described in bug 345655 is caused by a fractional part of the
splits value that is larger than the fractional part of the accounts currency.
This leads to the effect, that amounts are displayed with correctly
but the underlying amount used for calculations is not equal to the
amount displayed. This will end up in KMyMoney presenting the user
strange messages like 'Unassigned value of 0.00' which do not make
sense. The difference is in reality between 0 and 0.01, e.g. 0.008.
FIXED-IN: 4.8.0
REVIEW: 128061

M  +32   -7    kmymoney/mymoney/mymoneyfile.cpp
M  +5    -0    kmymoney/mymoney/mymoneyfile.h
M  +127  -0    kmymoney/mymoney/mymoneyfiletest.cpp
M  +1    -0    kmymoney/mymoney/mymoneyfiletest.h

http://commits.kde.org/kmymoney/88040d6c3f43ee6781911087dd12801e890fd8b1

*** Bug 359471 has been marked as a duplicate of this bug. ***

Git commit dd5bfdd80e64ec30ac10b13c6a0dd7aa818bd1d2 by Cristian Oneț, on behalf of Thomas Baumgart.
Committed on 13/06/2016 at 13:22.
Pushed by conet into branch 'frameworks'.

Reduce the precision of shares and values

The problem described in bug 345655 is caused by a fractional part of the
splits value that is larger than the fractional part of the accounts currency.
This leads to the effect, that amounts are displayed with correctly
but the underlying amount used for calculations is not equal to the
amount displayed. This will end up in KMyMoney presenting the user
strange messages like 'Unassigned value of 0.00' which do not make
sense. The difference is in reality between 0 and 0.01, e.g. 0.008.
FIXED-IN: 4.8.0
REVIEW: 128061

M  +32   -7    kmymoney/mymoney/mymoneyfile.cpp
M  +5    -0    kmymoney/mymoney/mymoneyfile.h
M  +127  -0    kmymoney/mymoney/mymoneyfiletest.cpp
M  +1    -0    kmymoney/mymoney/mymoneyfiletest.h

http://commits.kde.org/kmymoney/dd5bfdd80e64ec30ac10b13c6a0dd7aa818bd1d2

It looks to me that https://bugs.kde.org/show_bug.cgi?id=303026 still shows its problem.  See comment #18.  A new manually created copy of its sell transaction still objects if a fee of 12.95 is entered.

I know you are all up to your eyebrows, and doing fantastic work

I've just reopened this to ensure it is not overlooked, as there is still, I think, an outstanding issue.  https://bugs.kde.org/show_bug.cgi?id=345655#c23.

Hi Folks,
eventually I got the RPM package provided by Opensuse. 
I can confirm, for my "productive" file it is not working as expected.
Great work! Thanks a lot for the effort!

However one question remains (but perhaps not related to this bug)
I double checked and corrected all my investment transactions and the (visible) amount of the account also matches the amount downloaded from the bank.
However it is marked in red as if it would not match?
Is there a way to check where a potential mismatch might be?

Cheers

Sven

I meant it IS WORKING! :-)
Sorry for the typo!
unfortunately I cannot edit my comment anymore :-(

(In reply to sven.keller from comment #25)
> Hi Folks,
> eventually I got the RPM package provided by Opensuse. 
> I can confirm, for my "productive" file it is not working as expected.
> Great work! Thanks a lot for the effort!
> 
> However one question remains (but perhaps not related to this bug)
> I double checked and corrected all my investment transactions and the
> (visible) amount of the account also matches the amount downloaded from the
> bank.
> However it is marked in red as if it would not match?
> Is there a way to check where a potential mismatch might be?
> 
> Cheers
> 
> Sven

Can you be a bit more precise?  Is it just the date field?  If you hover the pointer, does that produce a clue?  Might it be that the opening date for the account is after the transaction date?  If so, edit the account's opening date to match the oldest likely transaction.

Actually, if the answer to the previous comment does not resolve the issue, it would be as well to open a new bug, as your problem is not to do with rounding.

@Allan I agree it may not be related to this rouding issue. 
I just like to clarify how I could give more details wether is is a different bug or part of this one.

I don't really get the point of your question?
Where should I hover the pointer above? 
The opening date and the date of the first transaction match.
Also the balance on the right column of the ledgers is same as "balance" underneath, "cleared" and "online statement balance" whereas the latter is marked with red background instead yellowish as it supposed to be. So I assume the system detected a mismatch.
However during save action I do not get an error or warning related to this. 
How can I help you given more helpful details?

I made an assumption that it could be the date field that was red. Obviously now that's not the issue.  So, I'm missing something.

Are you able to attach a screen-shot to the bug?

Allan 

In Settings/Configure KMM/Colors, a negative value may be set to show as red.  Might that explain?

Created attachment 100268 [details]
screenshot

there it is..
I reconciled the account but when I want to finish it tells me:
"you are about to finish the reconciliation of this account with a difference between your bank statement and the transactions marked as cleared. are you sure..."

Ah, we're on reconciliation.  It is indicating a discrepancy with cleared transactions.  Uncleared transactions will be excluded from reconciliation.

So, in that account, click Not marked in the status (top right).  Make sure all those transactions are cleared.

It might help in any further difficulty to just reconcile part of the account, into smaller chunks , if that helps to show where the problem lies.

Allan

(In reply to sven.keller from comment #31)
I do not know exactly what that particular error message is in tended to say.  I get it fairly often, with no apparent issued with the account.  As Alan said, review everything in the account since the previous reconciliation to assure the correct status of all transactions.  However, given the same account balance, cleared balance, and online balance, it is possible there really are not any problems, other than the spurious error message.

Make lots of backups!

As you recommended I filtered "not  marked " elements and got an empty list. This however comes with no surprise since I manually reconciled all entries while searching for the "odd one" causing the rounding problem. 
So I finished reconciliation and got the report without issues (apart from the red bar at the botom...
perhaps you could tell me how I can find those 4 entries (balance, online statement, balance below the table and cleared) in the xml to see if there is a mismatch?

Here's some diagnostic instructions that you should run on your data file (I assume it is a standard KMyMoney file).

We need to figure out the account id of the account in question. Here's how to do that. First we uncompress the data and store that in its own file which we use for further analysis.
% zcat xxx.kmy > xxx.xml
% grep '<ACCOUNT ' xxx.xml | grep 'Giro'

Replace xxx with your filename. Single quotes and spaces are important here. As argument to grep use part of the name of your account. Here I get something like the following line (you may see more entries - pick the one that is your account):

  <ACCOUNT currency="EUR" description="" parentaccount="AStd::Asset" opened="2016-01-01" number="111406" lastmodified="2016-07-24" type="1" id="A000001" lastreconciled="2016-05-25" institution="I000001" name="1200 Giro RaiBa">

The id in which we are interested of my example account is A000001. That is what you need from your file.

Next we try to extract some data from your file. Depending on the order of the attributes stored in the file, some arguments might be different for you. Here's a line from my file so that you can check if they are identical:

% grep 'SPLIT ' xxx.xml | head -n 1
    <SPLIT payee="" reconcileflag="2" shares="64133/50" reconciledate="2016-01-04" action="" bankid="" account="A000001" number="" value="64133/50" memo="" id="S0001"/>

If the attributes for you are in the same order then things should work as explained here. Otherwise you will have to play with the -f argument of the first cut command. 

Replace the # signs after the A with the id determiined above. The argument for the -d of the second cut command  single quote, double quote, single quote.  

% grep 'SPLIT ' xxx.xml | grep A###### | cut -d= -f4 | cut -d'"' -f2 | cut -d/ -f2 | sort | uniq

This should give a short list (mine as sample here):

1
10
100
2
20
25
4
5
50

These are the denominators of the shares which I am interested in. Next we need the same for the online balances. That is a lot easier.

% grep lastStatementBalance xxx.xml | cut -d= -f 3 | cut -d/ -f2 | sort | uniq

Here is my output (don't worry about the trailing double quote here)

100"
2"
20"
25"
50"


Can you post your lists here? Once done, you can get rid of xxx.xml again.

kraxel@kerbos:~/> grep 'SPLIT' MyMoney-nopgp_delete-me.kmy.uncompressed |grep A000217  | cut -d= -f4 | cut -d'"' -f2 | cut -d/ -f2 | sort | uniq
1
10
100
100000
1000000
12500
125000
2
20
2000
2000000
25
25000
250000
31250
4
40000
5
50
5000
50000
500000
62500
"2008-09-16" memo
"2008-09-30" memo
"2008-12-30" memo
"2009-02-02" memo
"2009-05-04" memo
"2009-07-01" memo
"2009-08-14" memo
"2009-08-31" memo
"2009-09-01" memo
"2009-09-15" memo
"2009-11-02" memo
"2009-12-01" memo
"2010-01-22" memo
"2010-02-16" memo
"2010-02-22" memo
"2010-03-31" memo



kraxel@kerbos:~/> grep lastStatementBalance  MyMoney-nopgp_delete-me.kmy.uncompressed | cut  -d= -f 3 | cut -d/ -f2 | sort | uniq
1"
10"
100"
20"
25"
4"
50"

Ah, what I somewhat expected and what probably leads to the problems you encounter. Those denominators > 100 represent a value with a higher precision than 2 decimal digits (I expect the account being a checking account and with EUR as currency this would be 2 decimal digits) Example: 1/100 and 11/1000 are both shown as 0,01 but in fact the latter is 0,011. So KMyMoney tells you that the values are different which they are but you will not see it on screen.

Using a bit more grep magic on the xml file one can get the transaction ID of such a transaction.  If you run

% grep -B 20 '62500' xxx.xml | grep -B 20 'SPLIT '

it should show you 20 lines before the split with a denominator of 62500. We are interested in the starting TRANSACTION tag for this split and its attribute id, which is a T followed by an 18 digit number. If you don't see it, increase the value of 20.

You can use that id directly in KMyMoney's 'Edit/Find transaction' dialog as the text to search. What happens to the splits of this transaction if you start editing it and enter it w/o changes or with slight changes e.g. in the memo field? Does the precision get adjusted to two digits?

fur the purpose of "mocking away the rounding error" I used more digits. I reversed it for this transaction to the actual values and the rounding seems to be OK.
The precision of this transaction also changed in the xml to xxx/100 
But it does not change the red bar in the ledger.

ps: there are likely more transactions like this for the same reason, but I could not find another "62500".

OK, so editing transactions solves the problem on a per-transaction basis (which was implemented by the fix). In order to resolve the problem with the red background for the online balance you need to edit all transaction with a denominator > 100  (seems you have some of those according to the list you provided) and not only the one with 62500 which I picked randomly from your list..

This has actually been instructive for me - that reconciliation error I have seen has always been in investment accounts, and I certainly have increased precision in many of the equities.  Now I'll have to watch for it to happen again, and see if I actually do have similar rounding problems that have simply never been large enough to be obvious in any of the visible transaction amounts.

Git commit c6e31c96a03ad5a7ec5e3695bbfa8336181ab81c by Łukasz Wojniłowicz.
Committed on 01/04/2017 at 06:16.
Pushed by wojnilowicz into branch 'master'.

Fix rounding problem with investments

Patch introduces rounding rules per security for fixing bug #372163. It
seems that this broker always rounds amounts down while my broker rounds
amounts depending on the outlying digit, so it couldn't work for both of
us without rules.

Rounding is done in InvestTransactionEditor because it has all needed
informations at hand.

No rounding of shares is done in InvestTransactionEditor::setupPrice.
Transaction from bug #372163 looks as follows:
brokerage:
shares = 1,009 ; value = 1,009
investment:
shares = -1 ; value = 1,009

InvestTransactionEditor::setupPrice causes brokerage to look as follows:
shares = 1,01 ; value = 1,009
As we can see shares and value diverge, which is unacceptable here.

Patch makes assumption that transaction has only single split of
stock/mutual fund/bond.
Related: bug 357784, bug 365177, bug 372163
FIXED-IN:5.0

Differential Revision: https://phabricator.kde.org/D5187

Signed-off-by: Łukasz Wojniłowicz <lukasz.wojnilowicz@gmail.com>

M  +32   -15   kmymoney/dialogs/investtransactioneditor.cpp
M  +31   -0    kmymoney/kmymoneyutils.cpp
M  +22   -0    kmymoney/kmymoneyutils.h
M  +39   -0    kmymoney/mymoney/mymoneysecurity.cpp
M  +27   -8    kmymoney/mymoney/mymoneysecurity.h
M  +1    -0    kmymoney/mymoney/storage/mymoneydbdef.cpp
M  +3    -0    kmymoney/mymoney/storage/mymoneystoragesql.cpp
M  +4    -3    kmymoney/widgets/transaction.cpp
M  +11   -0    kmymoney/wizards/newinvestmentwizard/kinvestmentdetailswizardpage.cpp
M  +104  -90   kmymoney/wizards/newinvestmentwizard/kinvestmentdetailswizardpagedecl.ui
M  +2    -0    kmymoney/wizards/newinvestmentwizard/knewinvestmentwizard.cpp

https://commits.kde.org/kmymoney/c6e31c96a03ad5a7ec5e3695bbfa8336181ab81c